TWI626587B - Neural processing unit that selectively writes back to neural memory either activation function output or accumulator value - Google Patents

Abstract

A neural network unit includes a programmable index specifying first and second actions, a first memory, a second memory, a plurality of neural processing units and a plurality of activation units. Each neural processing unit includes an accumulator and an arithmetic unit. The arithmetic unit performs a series of multiplication operations on a plurality of pairs of the first and second operands received by the first and second memories, and performs a series of addition operations on a generated series of products to accumulate an accumulated value to be stored in the accumulation. Device. The startup unit executes a startup function on this accumulated value to produce a result. When the indicator specifies the first action, the neural network unit writes the result generated by the activation unit into the first memory. When the indicator specifies the second action, the neural network unit writes the accumulated value in the accumulator into the first memory.

Description

Selectively write start function output or accumulator value Neural processing unit

This application claims the following international priority for the United States provisional application. The full text of these priority cases is incorporated in this case for reference.

This application is related to the following U.S. provisional applications that file simultaneously. The full texts of these related applications are incorporated in this case for reference.

In recent years, artificial neural networks (ANN) have regained attention. These studies are often called similar terms such as deep learning, computer learning, and so on. The increase in the computing power of general-purpose processors has also boosted people's interest in artificial neural networks now, decades later. Recent applications of artificial neural networks include language and image recognition. There seems to be an increasing need for increasing the computational efficiency and efficiency of artificial neural networks.

In view of this, the present invention provides a device including an array composed of N processing units, a first memory and a second memory. Each processing unit in the array of N processing units includes an accumulator, an arithmetic unit, a weight input and a multiplexer register. The accumulator has an output. The arithmetic unit has first, second, and third inputs, and performs operations on them to generate a result stored in the accumulator. The aforementioned first input receives the output of the accumulator. The weight input is received from the aforementioned second input to the arithmetic unit. Multiplexing The device has first and second data inputs, an output and a control input. The output of the multiplexing register is received from the aforementioned third input to the arithmetic unit. The control input controls the selection of the first and second data inputs. The output of the multiplex register is received by the second data input of the multiplex register of an adjacent processing unit. When the second data input is selected as the control input, the multiplex register of the N processing units It works as a spinner for the same N characters. The first memory system loads N weight characters of W columns, and provides N weight characters of one of the W columns to the corresponding weight inputs of the N processing units of the processing unit array. The second memory system loads N data characters of D rows, and provides N data characters of one of D rows to the corresponding first data of the multiplex register of N processing units of the processing unit array. Enter.

The invention also provides a processor. The processor includes an instruction set, an array composed of N processing units, a first memory and a second memory. The instruction set has architectural instructions to instruct the processor to operate. Each processing unit in the array of N processing units includes an accumulator, an arithmetic unit, a weight input, and a multiplexer register. The accumulator has an output. The arithmetic unit has first, second, and third inputs, and performs operations on them to generate a result stored in the accumulator. The aforementioned first input receives the output of the accumulator. The weight input is received from the aforementioned second input to the arithmetic unit. The multiplex register has first and second data inputs, an output and a control input. The output of the multiplex register is received from the third input to the arithmetic unit. The control input controls the first and second data. Input choice. Among them, the output of the multiplexer register is output by the second data of the multiplexer register of an adjacent processing unit. When received, when the control input selects the second data input, the multiplex registers of the N processing units work together as a rotator of the same N characters. The first memory system loads N weight characters of W columns, and provides N weight characters of one of the W columns to the corresponding weight inputs of the N processing units of the processing unit array. The second memory system loads N data characters of D rows, and provides N data characters of one of D rows to the corresponding first data of the multiplex register of N processing units of the processing unit array. Enter.

The invention also provides a computer program product encoded in at least one non-transitory computer usable medium for use by a computer device. The computer program product includes a computer-usable code included in the medium to describe a device. The computer-usable code includes a first code, a second code, and a third code. The first code designates an array of N processing units. Each processing unit in the array includes an accumulator, an arithmetic unit, a weight input, and a multiplexer register. The accumulator has an output. The arithmetic unit has first, second, and third inputs, and performs operations on them to generate a result stored in the accumulator. The aforementioned first input receives the output of the accumulator. The weight input is received from the aforementioned second input to the arithmetic unit. The multiplex register has first and second data inputs, an output and a control input. The output of the multiplex register is received from the third input to the arithmetic unit. The control input controls the first and second data. Input choice. The output of the multiplex register is received by the second data input of the multiplex register of an adjacent processing unit. When the second data input is selected as the control input, the multiplex register of the N processing units Work together as a rotation of the same N characters Device. The second code designates a first memory to load the N weight characters of W rows and provide the N weight characters of one of the W rows to the corresponding of the N processing units of the processing unit array. Weight entry. The third code designates a second memory to load the N data characters of D rows and provide the N data characters of one of the D rows to the multiplexing of the N processing units of the processing unit array. The corresponding first data of the register is input.

The specific embodiments used in the present invention will be further described by the following embodiments and drawings.

100‧‧‧ processor

101‧‧‧ instruction fetch unit

102‧‧‧Instruction cache

103‧‧‧Architecture Instructions

104‧‧‧Instruction translator

105‧‧‧microinstructions

106‧‧‧ Rename Unit

108‧‧‧ Reserved Station

112‧‧‧Other execution units

114‧‧‧Memory Subsystem

116‧‧‧General purpose register

118‧‧‧Media Register

121‧‧‧ Neural Network Unit

122‧‧‧Data Random Access Memory

124‧‧‧ Weight Random Access Memory

126‧‧‧Neural Processing Unit

127‧‧‧Control and status register

128‧‧‧Sequencer

129‧‧‧program memory

123,125,131‧‧‧Memory address

133,215,133A, 133B, 215A, 215B‧‧‧Result

202‧‧‧ Accumulator

204‧‧‧ Arithmetic Logic Unit

203,209,217,203A, 203B, 209A, 209B, 217A, 217B‧‧‧Output

205,205A, 205B‧‧‧Register

206,207,211,1811,206A, 206B, 207A, 207B, 211A, 211B, 1811A, 1811B, 711‧‧‧Input

208,705,208A, 208B‧‧‧Multiplex Register

213,713,803‧‧‧Control input

212‧‧‧Start function unit

242‧‧‧Multiplier

244‧‧‧ Adder

246,246A, 246B‧‧‧product

802‧‧‧ Multiplexer

1104‧‧‧column buffer

1112‧‧‧Start function unit

1400‧‧‧MTNN instruction

1500‧‧‧MFNN instruction

1432, 1532‧‧‧ functions

1402,1502‧‧‧‧Code field

1404‧‧‧src1 field

1406‧‧‧src2

1408, 1508‧‧‧‧gPr field

1412, 1512 ‧‧‧ Immediate field

1422, 1522 ‧‧‧ Address

1424, 1426, 1524 ‧‧‧ data blocks

1428, 1528 ‧‧‧selected columns

1434‧‧‧Control Logic

1504‧‧‧dst field

1602‧‧‧Read Port

1604‧‧‧write port

1606‧‧‧Memory Array

1702‧‧‧port

1704‧‧‧Buffer

1898‧‧‧operator selection logic

1896A‧‧‧Wide Multiplexer

1896B‧‧‧Narrow Multiplexer

242A‧‧‧ Wide Multiplier

242B‧‧‧Narrow Multiplier

244A‧‧‧Wide Adder

244B‧‧‧Narrow Adder

204A‧‧‧wide arithmetic logic unit

204B‧‧‧ Narrow Arithmetic Logic Unit

202A‧‧‧Wide Accumulator

202B‧‧‧Narrow Accumulator

212A‧‧‧Wide Start Function Unit

212B‧‧‧Narrow Start Function Unit

2402‧‧‧ Convolution Kernel

2404‧‧‧Data Array

2406A, 2406B‧‧‧ Data Matrix

2602, 2604, 2606, 2608, 2902, 2912, 2914, 2922, 2924, 2926, 2932, 2934, 2942,2944,2952,2954,2956,2923,2962,2964‧‧‧

3003‧‧‧ Random bit source

3005‧‧‧random bits

3002‧‧‧ Positive converter and output decimal point aligner

3004‧‧‧rounder

3006‧‧‧Multiplexer

3008‧‧‧Standard size compressor and saturator

3012‧‧‧Bit Selection and Saturator

3018‧‧‧corrector

3014‧‧‧Countdown Multiplier

3016‧‧‧Right shifter

3028‧‧‧Standard size transfer value

3022‧‧‧Hyperbolic Tangent Module

3024‧‧‧S Type Module

3026‧‧‧Soft add module

3032‧‧‧Multiplexer

3034‧‧‧Symbol Restorer

3036‧‧‧size converter and saturator

3037‧‧‧Multiplexer

3038‧‧‧Output Register

3402‧‧‧Multiplexer

3401‧‧‧Neural Processing Unit pipeline stage

3404‧‧‧ Decoder

3412, 3414, 3418 ‧‧‧ micro-computing

3416‧‧‧Micro Instructions

3422‧‧‧Mode Indicator

3502‧‧‧ Time-frequency generation logic

3504‧‧‧ Time-frequency reduction logic

3514‧‧‧Interface logic

3522‧‧‧Data Random Access Memory Buffer

3524‧‧‧weighted random access memory buffer

3512 ‧ ‧ mitigation indicators

3802‧‧‧Program Counter

3804‧‧‧Loop counter

3806‧‧‧Iteration counter

3912, 3914, 3916 ‧‧‧ fields

4901‧‧‧ Neural Processing Unit Group

4903‧‧‧Mask

4905,4907,5599‧‧‧Enter

The first diagram is a block diagram of a processor including a neural network unit (NNU).

The second diagram is a block diagram of a neural processing unit (NPU), one of the first diagrams.

The third diagram is a block diagram showing the use of N multiplexing registers of the N neural processing units of the neural network unit of the first diagram, for a row of data text obtained from the data random access memory of the first diagram Performs the operation of a rotator or circular shifter of the same N characters.

The fourth diagram is a table showing a program stored in the program memory of the neural network unit of the first diagram and executed by the neural network unit.

The fifth diagram is a timing diagram showing the execution of the program of the fourth diagram by the neural network unit.

The sixth diagram A is a block diagram showing that the neural network unit of the first diagram executes the program of the fourth diagram.

The sixth diagram B is a flowchart showing the processor of the first diagram executing a framework program to use a neural network unit to perform a typical multiply accumulate activation function operation of a neuron associated with a hidden layer of an artificial neural network. It works as if executed by the program in Figure 4.

The seventh diagram is a block diagram showing another embodiment of the neural processing unit of the first diagram.

The eighth diagram is a block diagram showing another embodiment of the neural processing unit of the first diagram.

The ninth figure is a table showing a program stored in the program memory of the neural network unit of the first figure and executed by the neural network unit.

The tenth diagram is a timing chart showing the execution of the program of the ninth diagram by the neural network unit.

The eleventh figure is a block diagram showing an embodiment of the neural network unit of the first figure. In the embodiment of FIG. 11, a neuron system is divided into two parts, that is, an activation function unit part and an arithmetic logic unit part (this part also includes a shift register part), and each activation function unit part is composed of multiple Part of the arithmetic logic unit is shared.

The twelfth figure is a timing chart showing that the neural network unit of the eleventh figure executes the program of the fourth figure.

The thirteenth figure is a timing chart showing the neural network unit of the eleventh figure executing the program of the fourth figure.

The fourteenth figure is a block diagram showing a move to neural network (MTNN) architecture instruction and its corresponding neural network unit of the first figure Part of the operation.

The fifteenth figure is a block diagram showing a move to neural network (MTNN) architecture instruction and its operation corresponding to the neural network unit of the first figure.

The sixteenth figure is a block diagram showing an embodiment of the data random access memory of the first figure.

The seventeenth figure is a block diagram showing an embodiment of the weighted random access memory and a buffer of the first figure.

The eighteenth figure is a block diagram showing a dynamically configurable neural processing unit, one of the first figures.

The nineteenth figure is a block diagram showing the embodiment according to the eighteenth figure, using the 2N multiplexing registers of the N neural processing units of the neural network unit of the first figure. The random access memory obtains a row of data text and performs the same operation as a rotator.

Figure 20 is a table showing a program stored in the program memory of the neural network unit of the first figure and executed by the neural network unit. The neural network unit has an embodiment as shown in figure 18. Neural Processing Unit shown.

The twenty-first diagram is a timing diagram showing a neural network unit executing the program of the twentieth diagram. The neural network unit has the neural processing unit shown in the eighteenth diagram executed in a narrow configuration.

The twenty-second figure is a block diagram showing the neural network unit of the first figure. This neural network unit has the neural processing unit shown in the eighteenth figure to execute the program of the twentieth figure.

The twenty-third figure is a block diagram showing another embodiment of the dynamically configurable neural processing unit in the first figure.

The twenty-fourth figure is a block diagram showing an example of a data structure used by the neural network unit of the first figure to perform a convolution operation.

The twenty-fifth figure is a flowchart showing that the processor of the first figure executes a framework program to perform a convolution operation of a convolution kernel using a neural network unit according to the data array of the twenty-fourth figure.

Figure 26A is a program list of a neural network unit program. This neural network unit program uses the convolution kernel of Figure 24 to perform a convolution operation of a data matrix and writes it back to random weights. Access memory.

Figure 26B is a block diagram showing an embodiment of some fields of the control register of the neural network unit in the first figure.

The twenty-seventh figure is a block diagram showing an example of the weighted random access memory filled with input data in the first figure. This input data is performed by the neural network unit in the first figure. ).

The twenty-eighth figure is a program list of a neural network unit program. This neural network unit program performs the common source operation of the input data matrix of the twenty-seventh figure and writes it back to the weight random access memory.

Figure 29A is a block diagram showing an embodiment of the control register of the first figure.

Figure 29B is a block diagram showing another embodiment of the control register of the first figure.

Figure 29C is a block diagram showing one embodiment of storing the reciprocal of Figure 29A in two parts.

The thirtieth figure is a block diagram showing an embodiment of an activation function unit (AFU) of the second figure.

The thirty-first figure is an example showing the operation of the activation function unit of the thirty-first figure.

The thirty-second figure is the second example showing the operation of the activation function unit of the thirty-third figure.

The thirty-third figure is a third example showing the operation of the activation function unit of the thirty-third figure.

The thirty-fourth figure is a block diagram showing some details of the processor and the neural network unit of the first figure.

Figure 35 is a block diagram showing a processor with a variable rate neural network unit.

Figure 36A is a timing diagram showing an example of the operation of a processor with a neural network unit in one of the general modes. This general mode operates at the main time frequency.

Figure 36B is a timing diagram showing an example of the operation of a processor with a neural network unit in a relaxation mode. The frequency of operation of the relaxation mode is lower than the main frequency.

Figure 37 is a flowchart showing the operation of the processor in Figure 35.

The thirty-eighth figure is a block diagram showing the sequence of the neural network units in detail.

Figure 39 is a block diagram showing some fields of the control and status register of the neural network unit.

The fortieth diagram is a block diagram showing an example of an Elman time recurrent neural network (RNN).

The forty-first diagram is a block diagram showing the random access memory and weight random access of the neural network unit when the neural network unit performs the calculation of the Elman time recurrent neural network associated with the forty diagram. An example of data allocation in memory.

Figure 42 is a table showing one of the programs stored in the neural network unit. This program is executed by the neural network unit and uses data and weights according to the configuration of figure 41 to achieve Elman time recurrent neural network

The forty-third diagram is a block diagram showing an example of Jordan time recurrent neural network.

The forty-fourth figure is a block diagram showing that when the neural network unit performs the calculation of the Jordan time recursive neural network associated with the forty-third figure, the data of the neural network unit is randomly stored in memory and weights are stored randomly. Take an example of data configuration in memory.

The forty-fifth figure is a table showing one of the programs stored in the neural network unit. This program is executed by the neural network unit and uses data and weights according to the configuration of the forty-fourth figure to achieve Jordan Time Recurrent Neural Network.

The forty-sixth figure is a block diagram showing an embodiment of a long short term memory (LSTM) cell.

The forty-seventh diagram is a block diagram showing the random access memory and weight random access of the data of the neural network unit when the neural network unit performs calculations related to the long-term and short-term memory cell layer of the forty-sixth diagram An example of data allocation in memory.

Figure 48 is a table showing the program stored in the neural network unit A program of memory. This program is executed by the neural network unit and uses data and weights according to the configuration of Figure 47 to achieve the calculation of long-term and short-term memory cells.

The forty-ninth figure is a block diagram showing an embodiment of a neural network unit. The neural processing unit group in this embodiment has output buffer masking and feedback capabilities.

Figure 50 is a block diagram showing the random access memory of the data of the neural network unit of Figure 49 when the neural network unit performs calculations related to the long-term and short-term memory cell layer of Figure 46. , An example of data allocation in weighted random access memory and output buffer.

Figure 51 is a table showing a program stored in the neural network unit's program memory. This program is executed by the neural network unit of Figure 49 and uses the data according to the configuration of Figure 50. Weights to achieve calculations related to long- and short-term memory cells.

The fifty-second figure is a block diagram showing an embodiment of a neural network unit. The neural processing unit group in this embodiment has output buffer masking and feedback capabilities, and shares an activation function unit.

Figure 53 is a block diagram showing the random access memory of the data of the neural network unit of Figure 49 when the neural network unit performs calculations related to the long-term and short-term memory cell layer of Figure 46. Another embodiment of the data allocation in the memory, the weighted random access memory and the output buffer.

Figure 54 is a table showing a program stored in the neural network unit's program memory. This program is executed by the neural network unit in Figure 49 and uses the data according to the configuration of Figure 53. And weights to achieve calculations related to long- and short-term memory cells.

The fifty-fifth figure is a block diagram showing a part of a neural processing unit according to another embodiment of the present invention.

The fifty-sixth diagram is a block diagram showing the calculation of the neural network unit when the neural network unit performs calculations of the Jordan time recurrent neural network associated with the forty-third diagram and uses the embodiment of the fifty-fifth diagram An example of data allocation in data random access memory and weighted random access memory.

Figure 57 is a table showing one of the programs stored in the neural network unit. This program is executed by the neural network unit and uses data and weights according to the configuration of Figure 56 to achieve Jordan. Time recurrent neural network.

Processor with architecture neural network unit

The first diagram is a block diagram of a processor 100 including a neural network unit (NNU) 121. As shown in the figure, the processor 100 includes an instruction fetch unit 101, an instruction cache 102, an instruction translator 104, a rename unit 106, multiple reservation stations 108, multiple media registers 118, multiple The general purpose register 116 includes a plurality of execution units 112 and a memory subsystem 114 outside the aforementioned neural network unit 121.

The processor 100 is an electronic device and serves as a central processing unit of an integrated circuit. The processor 100 receives the input digital data, processes the data according to the instructions fetched from the memory, and generates the processing result of the operation instructed by the instructions as its output. The processor 100 may be used in a desktop computer, a mobile device, or a tablet computer, and used for calculation, text Processing, multimedia display and web browsing applications. The processor 100 may be provided in an embedded system to control various devices including devices, mobile phones, smart phones, vehicles, and industrial controllers. The central processing unit is an electronic circuit (ie, hardware) that executes instructions of a computer program (or computer application or application) by performing operations on data, including arithmetic, logic, and input / output. Integrated circuits are a group of electronic circuits made from a small semiconductor material, usually silicon. Integrated circuits are also commonly used to represent wafers, microchips, or dies.

The instruction fetching unit 101 controls the operation of fetching the architectural instructions 103 to the instruction cache 102 from the system memory (not shown). The instruction fetch unit 101 provides a fetch address to the instruction cache 102 to specify the memory address of the cache line of the architecture instruction byte of the processor 100 to fetch to the cache memory 102. The selection of the fetch address is based on the current value of the instruction index (not shown) of the processor 100 or the program counter. Generally, the program counter is incremented according to the instruction size until a control instruction such as a branch, call, or return appears in the instruction stream, or an exception condition such as an interrupt, a trap, an exception, or an error occurs. Update the program counter with non-sequential addresses such as branch target address, return address, or exception vector. In a word, the program counter is updated according to the execution instructions of the execution units 112/121. The program counter can also be updated when an exception condition is detected, for example, the instruction translator 104 encounters an instruction 103 that is not defined in the instruction set architecture of the processor 100.

The instruction cache 102 stores architecture instructions 103 obtained from a system memory coupled to the processor 100. These architectural instructions 103 include a move to neural network (MTNN) instruction and a move by neural network The (MFNN) instruction is described in detail later. In one embodiment, the architecture instruction 103 is an instruction of the x86 instruction set architecture, and MTNN instruction and MFNN instruction are added. In this disclosure, an x86 instruction set architecture processor is understood to be a processor that produces the same result as the Intel® 80386® processor at the instruction set architecture layer when executing the same mechanical language instructions. However, other instruction set architectures, such as Advanced Reduced Instruction Set Machine Architecture (ARM), Sun's Scalable Processor Architecture (SPARC), or Enhanced Reduced Instruction Set Performance Computing Performance Optimization Architecture (PowerPC), It can also be used in other embodiments of the present invention. The instruction cache 102 provides the architectural instruction 103 to the instruction translator 104 to translate the architectural instruction 103 into a micro instruction 105.

The microinstruction 105 is provided to the renaming unit 106 and is finally executed by the execution unit 112/121. These micro-instructions 105 implement architectural instructions. In a preferred embodiment, the instruction translator 104 includes a first part for translating frequently executed and / or relatively less complex architectural instructions 103 into microinstructions 105. The instruction translator 104 also includes a second part having a microcode unit (not shown). The microcode unit has a microcode memory loaded with microcode instructions to execute complex and / or rarely used instructions in the architecture instruction set. The microcode unit also includes a microsequencer to provide a non-architecture microprogram counter (micro-PC) to the microcode memory. In a preferred embodiment, these micro-instructions are translated into micro-instructions 105 via a micro-translator (not shown). The selector selects the microinstruction 105 from the first part or the second part and provides it to the renaming unit 106 according to whether the microcode unit currently has the control right.

The renaming unit 106 renames the architecture register specified by the architecture instruction 103 to the physical register of the processor 100. Just a better In an embodiment, the processor 100 includes a rearrangement buffer (not shown). The renaming unit 106 assigns the reordering buffer items to each microinstruction 105 in a program order. In this way, the processor 100 can completely remove the micro instruction 105 and its corresponding architecture instruction 103 according to the program sequence. In one embodiment, the media register 118 has a width of 256 bits, and the general purpose register 116 has a width of 64 bits. In one embodiment, the media register 118 is an x86 media register, such as an advanced vector extension (AVX) register.

In one embodiment, each item of the rearrangement buffer has a storage space to store the result of the microinstruction 105. In addition, the processor 100 includes an architecture register file. The architecture register file has a physical register corresponding to each architecture register, such as the media register 118, the general purpose register 116, and other architecture registers. Device. (For a preferred embodiment, for example, the media register 118 and the general-purpose register 116 are different in size, and separate register files can be used to correspond to these two registers.) For micro Each source operand of an architecture register is specified in the instruction 105. The rename unit will use the reordering buffer directory of the latest microinstruction in the old microinstruction 105 written in the architecture register and fill in the microinstruction 105. The source operand field. When the execution unit 112/121 completes the execution of the microinstruction 105, the execution unit 112/121 writes its result into the reorder buffer item of the microinstruction 105. When the microinstruction 105 is removed, the removal unit (not shown) will write the result of the rearranged buffer field of this microinstruction into the register of the physical register file, which is associated with As a result, the register for the architectural purpose specified by the microinstruction 105 is removed.

In another embodiment, the processor 100 includes a physical register file, which has more physical registers than architecture registers. However, the processor 100 does not include an architecture register file, and the result buffer space does not include the result storage space. (For a preferred embodiment, because the size of the media register 118 and the general-purpose register 116 are different, a separate register file can be used to correspond to these two registers.) This processor 100 does It includes an index table with corresponding indexes of each architecture register. For each operand in the microinstruction 105 where the architecture register is specified, the rename unit will use an indicator pointing to a free register in the physical register file to fill in the destination operand field in the microinstruction 105. If there is no free register in the physical register file, the renaming unit 106 will temporarily suspend the pipeline. For each source operand in the microinstruction 105 where the architecture register is specified, the rename unit will use a pointer to the physical register file and assign it to the latest microinstruction in the old microinstruction 105 written to the architecture register. The index of the register is filled in the source operand field in the microinstruction 105. When the execution unit 112/121 finishes executing the micro instruction 105, the execution unit 112/121 writes the result to the register of the destination operand field of the micro instruction 105 in the physical register file. When the microinstruction 105 is removed, the removal unit will copy the value of the destination operand field of the microinstruction 105 to the indicator associated with the index table of the architectural purpose register specified by the removal microinstruction 105.

The reservation station 108 loads the micro-instructions 105 until the micro-instructions are issued to the execution units 112/121 for preparation for execution. When all the source operands of a microinstruction 105 are available and the execution units 112/121 are also available for execution, the preparation for the release of this microinstruction 105 is completed. The execution unit 112/121 is the rearrangement buffer or the structure register file described in the first embodiment, or the second embodiment The physical register file receives the register source operand. In addition, the execution unit 112/121 can directly receive the register source operand through the result transmission bus (not shown). In addition, the execution unit 112/121 may receive the immediate operand specified by the micro instruction 105 from the reservation station 108. The MTNN and MFNN architecture instructions 103 include an immediate operand to specify the function to be performed by the neural network unit 121. This function is provided by one or more micro-instructions 105 generated by the translation of the MTNN and MFNN architecture instructions 103, as described later. .

The execution unit 112 includes one or more load / store units (not shown). The memory subsystem 114 loads data and stores the data in the memory subsystem 114. In a preferred embodiment, the memory subsystem 114 includes a memory management unit (not shown). The memory management unit may include, for example, a plurality of lookaside buffers, a table movement ( table walk) unit, a level one data cache (and instruction cache 102), a level two unified cache, and a bus interface unit as an interface between the processor 100 and the system memory. In one embodiment, the processor 100 in the first figure is represented by one of a plurality of processing cores of a multi-core processor, and the multi-core processor shares a last-level cache memory. The execution unit 112 may include multiple integer units, multiple media units, multiple floating point units, and a branch unit.

The neural network unit 121 includes a weighted random access memory (RAM) 124, a data random access memory 122, N neural processing units (NPU) 126, a program memory 129, a sequencer 128, and more Control and status register 127. These neural processing units 126 are conceptually functioning as neurons in a neural network. Weight random access memory 124, data random access memory 122, and program memory 129 Both can be written and read through MTNN and MFNN architecture instructions 103, respectively. The weight random access memory 124 is arranged in W rows, each row having N weight characters, and the data random access memory 122 is arranged in D rows, each row having N data characters. Each data text and each weight text are a plurality of bits. In a preferred embodiment, they can be 8 bits, 9 bits, 12 bits, or 16 bits. Each data text is used as the output value (sometimes expressed by the activation value) of a neuron in the previous layer in the network, and each weight text is used as the weight of a link in the network that is associated with a neuron entering the current layer of the network . Although in many applications of the neural network unit 121, the text or operand loaded in the weight random access memory 124 is actually the weight associated with the connection into a neuron, it should be noted that in the neural network In some applications of the unit 121, the words loaded in the weight random access memory 124 are not weighted, but because these words are stored in the weight random access memory 124, they are still expressed in terms of "weight words". For example, in some applications of the neural network unit 121, for example, examples of convolution operations of the twenty-fourth to twenty-sixth graphs A or examples of common source operations of the twenty-seventh to twenty-eighth graphs The weight random access memory 124 will load objects other than weights, such as elements of a data matrix (such as image pixel data). Similarly, although in many applications of the neural network unit 121, the text or operand loaded in the data random access memory 122 is essentially the output value or activation value of the neuron, but it should be noted that in the neural network In some applications of the circuit unit 121, the text loaded in the data random access memory 122 is not the same, but because these texts are stored in the data random access memory 122, they are still expressed in terms of "data text". For example, in the neural network list In some applications of element 121, such as the example of the convolution operation of Figures 24 to 26A, the data random access memory 122 will load non-neuronal outputs, such as elements of a convolution kernel.

In one embodiment, the neural processing unit 126 and the sequencer 128 include combination logic, sequence logic, state machine, or a combination thereof. An architecture instruction (such as MFNN instruction 1500) loads the content of the state register 127 into one of the general purpose registers 116 to confirm the state of the neural network unit 121. For example, the neural network unit 121 has been completed from the program memory 129 A command or a program operates, or the neural network unit 121 is free to receive a new command or start a new neural network unit program.

The number of neural processing units 126 can be increased according to demand, and the width and depth of the random storage memory 124 and the data random access memory 122 can be adjusted and expanded accordingly. For a preferred embodiment, the weight random access memory 124 is larger than the data random access memory 122, because there are many links in a typical neural network layer, so a larger storage space is needed The weight of each neuron. This document discloses many embodiments regarding the size of data and weight text, the size of weight random access memory 124 and data random access memory 122, and the number of different neural processing units 126. In one embodiment, the neural network unit 121 has a data random access memory 122 with a size of 64KB (8192 bits x 64 rows) and a weight random access memory with a size of 2MB (8192 bits x 2048 rows). 124, and 512 neural processing units 126. This neural network unit 121 is manufactured by Taiwan Semiconductor Manufacturing Corporation (TSMC) in a 16-nanometer process, and its area is about 3.3 mm. square.

The sequencer 128 fetches instructions from the program memory 129 and executes them. The operations performed by the sequencer 128 include generating an address and a control signal to the data random access memory 122, a weight random access memory 124, and a neural processing unit 126. The sequencer 128 generates a memory address 123 and a read command and provides it to the data random access memory 122, so as to select one of the N data characters in the D rows and provide it to the N neural processing units 126. The sequencer 128 also generates a memory address 125 and a read command and provides them to the weighted random access memory 124, so as to select one of the W weighted N weighted text and provide it to the N neural processing units 126. The sequence of the addresses 123, 125 generated by the sequencer 128 and provided to the neural processing unit 126 determines the "connection" between the neurons. The sequencer 128 also generates a memory address 123 and a write command to the data random access memory 122, so that one of the N data characters in D rows is selected for writing by the N neural processing units 126. Into. The sequencer 128 also generates a memory address 125 and a write command to provide the weighted random access memory 124, so that one of the N weighted text in W columns is selected for writing by the N neural processing units 126. Into. The sequencer 128 also generates a memory address 131 to the program memory 129 to select a neural network unit instruction provided to the sequencer 128. This section will be described in the subsequent sections. The memory address 131 corresponds to a program counter (not shown). The sequencer 128 generally increments the program counter according to the position sequence of the program memory 129, unless the sequencer 128 encounters a control instruction, such as a loop instruction (Please refer to Figure 26A). In this case, the sequencer 128 will update the program counter to the target address of the control instruction. Sequencer 128 will also produce The control signal is sent to the neural processing unit 126, instructing the neural processing unit 126 to perform various operations or functions, such as initialization, arithmetic / logical operations, rotation / shift operations, start functions, and write-back operations. Related examples are in Subsequent chapters (please refer to the micro-operation 3418 shown in Figure 34) will be explained in more detail.

The N neural processing units 126 generate N result texts 133, and these result texts 133 can be written back to a row of the weight random access memory 124 or the data random access memory 122. In a preferred embodiment, the weight random access memory 124 and the data random access memory 122 are directly coupled to the N neural processing units 126. Further, the weight random access memory 124 and the data random access memory 122 are transferred to these neural processing units 126 and are not shared with other execution units 112 in the processor 100. These neural processing units 126 can continuously A row is obtained and completed from one or both of the weighted random access memory 124 and the data random access memory 122 in each time-frequency cycle. For a preferred embodiment, it can be processed in a pipeline manner. In one embodiment, each of the data random access memory 122 and the weighted random access memory 124 can provide 8192 bits to the neural processing unit 126 in each time-frequency period. These 8192 bits can be treated as 512 16-bytes or 1024 8-bytes, as described later.

The size of the data group processed by the neural network unit 121 is not limited to the sizes of the weighted random access memory 124 and the data random access memory 122, but is only limited by the size of the system memory. This is because the data And weights can be transferred between system memory and weight random access memory 124 and data random access memory 122 via MTNN and MFNN The use of instructions (eg, through the media register 118) moves. In one embodiment, the data random access memory 122 is given a dual port, so that when the data random access memory 122 reads data text or writes data text to the data random access memory 122, Write data text to data random access memory 122. In addition, the large memory hierarchy of the memory subsystem 114 including the cache memory can provide a very large data bandwidth for data transmission between the system memory and the neural network unit 121. In addition, in a preferred embodiment, the memory subsystem 114 includes a hardware data pre-fetcher, which tracks memory access patterns, such as neural data and weights loaded by the system memory, and caches the data. The hierarchical structure performs data pre-fetching to facilitate high-bandwidth and low-latency transmission during transmission to the weighted random access memory 124 and data random access memory 122.

Although in the embodiment herein, one of the operands provided by the weight memory to each neural processing unit 126 is labeled as a weight, this term is commonly used in neural networks, but it should be understood that these operands can also be other and Calculate related types of data, and their speed can be increased by these devices.

The second diagram is a block diagram of the neural processing unit 126, one of the first diagrams. As shown in the figure, the operation of the neural processing unit 126 can perform many functions or operations. In particular, the neural processing unit 126 can operate as a neuron or node in an artificial neural network to perform a typical multiply-accumulate-plus function or operation. That is to say, in general, the neural network unit 126 (neuron) is used to: (1) receive an input value from each neuron that has a connection with it, this connection usually will But not necessarily from the previous layer in the artificial neural network; (2) each output value is multiplied by a corresponding weight value associated with one of its links to produce a product; (3) all products are added to produce a total; (4) Perform a start function on this total to generate neuron output. However, unlike traditional methods that need to perform all multiplication operations associated with all connected inputs and sum their products, each neuron of the present invention can perform weighted multiplication operations associated with one of the connected inputs within a given time-frequency period. And the product is added (accumulated) to the accumulated value of the product of the connected inputs performed in the time-frequency period before the time point. Assume that there are a total of M links connected to this neuron. After the M products are summed up (approximately M time-frequency cycles are required), this neuron will execute a start function on this accumulated number to generate an output or result. The advantage of this method is that it can reduce the number of multipliers required, and only a small, simpler and faster adder circuit (such as an adder using two inputs) is needed in the neuron, without the need for Use an adder required to add up the product of all linked inputs or even a subset of them. This method is also conducive to using a large number (N) of neurons (neural processing unit 126) in the neural network unit 121. In this way, after about M time-frequency cycles, the neural network unit 121 can generate this large number (N) Neuron output. Finally, for a large number of different connection inputs, the neural network unit 121 composed of these neurons can be effectively implemented as an artificial neural network layer. In other words, if the number of M in different layers increases or decreases, the number of time-frequency cycles required to generate memory cell output will increase or decrease accordingly, and resources (such as multipliers and accumulators) will be fully utilized. In contrast, for the smaller M value of the traditional design, some parts of the multiplier and adder cannot be used. use. Therefore, according to the number of connected outputs of the neural network unit, the embodiment described herein has both advantages of flexibility and efficiency, and can provide extremely high performance.

The neural processing unit 126 includes a register 205, a dual-input multiplexing register 208, an arithmetic logic unit (ALU) 204, an accumulator 202, and an activation function unit (AFU) 212. The register 205 receives a weighted text 206 from the weighted random access memory 124 and provides its output 203 in a subsequent time-frequency period. The multiplexer register 208 selects one of the two inputs 207, 211 to store in its register and provides it to its output 209 in a subsequent time-frequency period. Input 207 receives the data text from the data random access memory 122. The other input 211 receives the output 209 from the neighboring neural processing unit 126. The neural processing unit 126 shown in the second figure is labeled as the neural processing unit J among the N neural processing units shown in the first figure. That is, the neural processing unit J is a representative example of the N neural processing units 126. For a preferred embodiment, the input 211 of the multiplexing register 208 of the J example of the neural processing unit 126 receives the output 209 of the multiplexing register 208 of the J-1 example of the neural processing unit 126, and the neural The output 209 of the multiplexing register 208 of the processing unit J is provided to the input 211 of the multiplexing register 208 of the J + 1 example of the neural processing unit 126. In this way, the multiplexing registers 208 of the N neural processing units 126 can work together, such as the spinner or cyclic shifter of the same N characters, which will be described in more detail in the subsequent third figure. The multiplexer register 208 uses a control input 213 to control which one of the two inputs is selected by the multiplexer register 208 to be stored in its register and subsequently provided to the output 209.

The arithmetic logic unit 204 has three inputs. One of the inputs receives the weight text 203 from the register 205. Another input received Output 209 of the multiplexer register 208. The other input receives the output 217 of the accumulator 202. The arithmetic logic unit 204 performs arithmetic and / or logic operations on its input to generate a result to provide its output. For a preferred embodiment, the arithmetic and / or logical operations performed by the arithmetic logic unit 204 are specified by instructions stored in the program memory 129. For example, the multiply-accumulate instruction in the fourth figure specifies a multiply-accumulate operation, that is, the result 215 will be the product of the accumulator 202 value 217 and the weight text 203 and the multiplexer register 208 output 209 data text total. However, other operations can also be specified. These operations include but are not limited to: result 215 is the value passed by the multiplexer register 209; result 215 is the value passed by the weight text 203; result 215 is the zero value; result 215 is the accumulator 202 The sum of the value 217 and the weight 203; the result 215 is the sum of the value 217 of the accumulator 202 and the multiplexer output 209; the result 215 is the maximum of the value 217 of the accumulator 202 and the weight 203; the result 215 is the accumulator 202 is the maximum value of 217 and multiplexer register 209.

The arithmetic logic unit 204 provides its output 215 to the accumulator 202 for storage. The arithmetic logic unit 204 includes a multiplier 242 to perform a multiplication operation on the weight text 203 and the data text output from the multiplexer register 208 to generate a product 246. In one embodiment, the multiplier 242 multiplies two 16-bit operands to produce a 32-bit result. The arithmetic logic unit 204 also includes an adder 244 that adds a product 246 to the output 217 of the accumulator 202 to generate a total, which is the result 215 of the accumulation operation stored in the accumulator 202. In one embodiment, the adder 244 is a 41-bit value 217 of the accumulator 202 and a 32-bit result of the multiplier 242 to generate a 41-bit result. So, in many cases During the period of the frequency period, by using the rotator characteristics of the multiplexer register 208, the neural processing unit 126 can achieve the product sum operation of the neurons required by the neural network. The arithmetic logic unit 204 may also include other circuit elements to perform other arithmetic / logical operations as described above. In one embodiment, the second adder subtracts the weight text 203 from the data text output 209 of the multiplexer register 208 to generate a difference, and the adder 244 adds the difference to the output 217 of the accumulator 202. Value to produce a result 215, which is the accumulated result in the accumulator 202. In this way, during a plurality of time-frequency periods, the neural processing unit 126 can achieve the sum of the difference values. In a preferred embodiment, although the weight text 203 and the data text 209 have the same size (in bits), they may also have different positions of the decimal point, as described later. In a preferred embodiment, the multiplier 242 and the adder 244 are integer multipliers and adders. Compared with an arithmetic logic unit using floating-point arithmetic, the arithmetic logic unit 204 has low complexity, small size, and fast speed. And the advantages of low energy consumption. However, in other embodiments of the present invention, the arithmetic logic unit 204 may also perform floating-point operations.

Although only one multiplier 242 and an adder 244 are shown in the arithmetic logic unit 204 of the second figure, according to a preferred embodiment, the arithmetic logic unit 204 further includes other components to perform the aforementioned different operations. For example, the arithmetic logic unit 204 may include a comparator (not shown) to compare the accumulator 202 with a data / weight text, and a multiplexer (not shown) to select from two values specified by the comparator. The larger (maximum) value is stored in the accumulator 202. In another example, the arithmetic logic unit 204 includes selection logic (not shown), using a data / weight text to skip the multiplier 242, and the adder 224 in the accumulator 202 The value 217 is added to the data / weight text to generate a total and stored in the accumulator 202. These additional operations will be explained in more detail in subsequent chapters such as Figures 18 to 29A, and these operations are also helpful for the implementation of convolution operations and common source operations.

The start function unit 212 receives the output 217 of the accumulator 202. The startup function unit 212 executes a startup function on the output of the accumulator 202 to generate a result 133 of the first figure. Generally speaking, the activation function in the neurons of the intermediary layer of the artificial neural network can be used to standardize the total after multiplying and accumulating, especially in a non-linear manner. In order to "normalize" the cumulative total, the activation function of the current neuron will generate a result value within the range of values expected to be received by other neurons connected to the current neuron. (The standardized result is sometimes called "startup". In this article, startup is the output of the current node, and the receiving node will multiply this output by a weight associated with the connection between the output node and the receiving node to produce a product. , And this product is added to the product of other inputs associated with this receiving node.) For example, if the receiving / connected neuron expects to receive an input value between 0 and 1, the output neuron The cumulative total beyond the range of 0 and 1 may need to be squeezed and / or adjusted non-linearly (eg, shifted up to convert negative values to positive values) to fall within this expected range. Therefore, the operation performed by the activation function unit 212 on the value 217 of the accumulator 202 will bring the result 133 to a known range. The results 133 of the N neural execution units 126 can all be written back to the data random access memory 122 or the weight random access memory 124 simultaneously. In a preferred embodiment, the startup function unit 212 is used to execute multiple startup functions. For example, an input from the control register 127 selects one of these startup functions to be executed on the output 217 of the accumulator 202. . These activation functions may include, but are not limited to, step functions, correction functions, sigmoid functions, hyperbolic tangent functions, and soft addition functions (also known as smooth correction functions). The analytic formula of the soft addition function is f (x) = ln (1 + e ^x ), which is the natural logarithm of the sum of 1 and e ^x , where "e" is Euler's number, and x is this Function input 217. In a preferred embodiment, the start-up function may also include a pass-through function, which directly passes the value 217 or a part of the accumulator 202, as described later. In one embodiment, the circuit of the activation function unit 212 executes the activation function in a single time-frequency period. In an embodiment, the activation function unit 212 includes a plurality of forms, which receive an accumulated value and output a value. For some activation functions, such as an S-type function, a double-taken tangent function, and a soft addition function, the value will be approximately The value provided by the real startup function.

For a preferred embodiment, the width (in bits) of the accumulator 202 is greater than the width of the output 133 of the activation function 212. For example, in an embodiment, the width of the accumulator is 41 bits, to avoid the situation of accumulating up to a maximum of 512 32-bit products (this part in the subsequent chapters corresponds to the position shown in Figure 30). There will be a more detailed explanation) loss of accuracy, and the width of the result 133 is 16 bits. In an embodiment, in the subsequent time-frequency cycle, the activation function unit 212 passes the other unprocessed parts of the accumulator 202 output 217, and writes these parts back to the data random access memory 122 or weight random storage. Take the memory 124. This part will be explained in more detail in the subsequent chapter corresponding to the eighth figure. In this way, the value of the unprocessed accumulator 202 can be loaded back to the media register 118 through the MFNN instruction, so that the instructions executed by the other execution units 112 of the processor 100 can perform the complex startup that the function unit 212 cannot perform. A function, such as the common softmax function, is also called a normalized exponential function. In an embodiment, the instruction set architecture of the processor 100 includes executing an instruction of this exponential function, which is usually expressed as e ^x or exp (x). This instruction can be used by other execution units 112 of the processor 100 to enhance the soft start Function execution speed.

In one embodiment, the neural processing unit 126 is a pipeline design. For example, the neural processing unit 126 may include a register of the arithmetic logic unit 204, such as a register located between the multiplier and the adder and / or other circuits of the arithmetic logic unit 204. The neural processing unit 126 may further include A register that loads the output of the start function function 212. Other embodiments of the neural processing unit 126 will be described in subsequent chapters.

The third diagram is a block diagram showing the use of N multiplexing registers 208 of N neural processing units 126 of the neural network unit 121 of the first diagram to obtain data from the random access memory 122 of the data of the first diagram One row of data characters 207 performs the operation of a rotator or a circular shifter of the same N characters. In the embodiment of the third figure, N is 512. Therefore, the neural network unit 121 has 512 multiplexing registers 208, labeled 0 to 511, corresponding to 512 neural processing units 126, respectively. Each multiplexer register 208 receives a corresponding data text 207 on one of the D rows of the data random access memory 122. In other words, multiplexer register 0 will receive data text 0 from 122 rows of data random access memory, multiplexer register 1 will receive data text 1 from 122 rows of data random access memory, and multiplex temporarily Device 2 will randomly store data Take the memory 122 rows to receive the data text 2 and so on, the multiplexer register 511 will receive the data characters 511 from the data random access memory 122 rows. In addition, multiplexer register 1 will receive output 209 of multiplexer register 0 as another input 211, and multiplexer register 2 will receive output 209 of multiplexer register 1 as another input 211. Register 3 will receive output 209 of multiplex register 2 as another input 211, and so on, multiplex register 511 will receive output 209 of multiplex register 510 as another input 211, and more The working register 0 will receive the output 209 of the multiplexing register 511 as the other input 211. Each multiplexer register 208 receives a control input 213 to control whether it selects data text 207 or cyclic input 211. In one mode of this operation, the control input 213 controls each multiplexing register 208 to select the data text 207 to be stored in the register in a first time-frequency period and provided to the arithmetic logic unit 204 in the subsequent steps. In the subsequent time-frequency cycles (such as the aforementioned M-1 time-frequency cycles), the control input 213 controls each multiplexing register 208 to select the cyclic input 211 to store in the register and provide it to the subsequent steps. Arithmetic logic unit 204.

Although in the embodiment described in the third diagram (and subsequent seventh and nineteenth diagrams), multiple neural processing units 126 can be used to rotate the values of the multiplex registers 208/705 to the right, that is, by the nerves. The processing unit J moves toward the neural processing unit J + 1, but the present invention is not limited thereto. In other embodiments (for example, the embodiments corresponding to the twenty-fourth to twenty-sixth figures), a plurality of neural processing units 126 It can be used to rotate the value of the multiplex register 208/705 to the left, that is, from the neural processing unit J to the neural processing unit J-1. In addition, in other embodiments of the present invention, the neural processing units 126 may selectively store multiplexer registers. The value of 208/705 is rotated to the left or right. For example, this selection can be specified by a neural network unit instruction.

The fourth diagram is a table showing a program stored in the program memory 129 of the neural network unit 121 of the first diagram and executed by the neural network unit 121. As mentioned earlier, this example program performs calculations related to one layer of an artificial neural network. The table in Figure 4 shows four columns and three rows. Each row corresponds to an address marked in the first row in the program memory 129. The second line specifies the corresponding instruction, and the third line indicates the number of time-frequency cycles associated with this instruction. In a preferred embodiment, the foregoing number of time-frequency cycles refers to a valid number of time-frequency cycles per instruction time-frequency cycle value in the embodiment executed by the pipeline, rather than an instruction delay. As shown in the figure, because the neural network unit 121 has the nature of pipeline execution, each instruction has an associated time-frequency cycle. The exception is the instruction at address 2, which actually repeats itself 511 times. Therefore, 511 time-frequency cycles are required, as described later.

All neural processing units 126 process each instruction in the program in parallel. In other words, all N neural processing units 126 will execute the first row of instructions in the same time-frequency cycle, and all N neural processing units 126 will execute the second row of instructions in the same time-frequency cycle. analogy. However, the present invention is not limited to this. In other embodiments of the subsequent chapters, some instructions are executed in a partially parallel partial sequence manner. For example, as described in the embodiment of FIG. In the embodiment where the unit 126 shares a startup function unit, the startup function and the output instructions at addresses 3 and 4 are executed in this manner. The example in Figure 4 assumes a layer with 512 neurons (God After processing unit 126), each neuron has 512 connection inputs from 512 neurons in the previous layer, for a total of 256K connections. Each neuron receives a 16-bit data value from each link input, and multiplies this 16-bit data value by an appropriate 16-bit weight value.

The first row at address 0 (can also be assigned to other addresses) will specify an instruction to initialize the neural processing unit. This initialization instruction clears the value of accumulator 202 to zero. In an embodiment, the initialization instruction may also load a row of the data random access memory 122 or the weight random access memory 124 into the accumulator 202, and the corresponding text specified by the instruction. This initialization instruction will also load the configuration value into the control register 127. This part will be explained in more detail in the subsequent figures 29A and 29B. For example, the width of the data text 207 and the weight text 209 can be loaded for use by the arithmetic logic unit 204 to confirm the size of the operation performed by the circuit. This width also affects the result 215 stored in the accumulator 202. In one embodiment, the neural processing unit 126 includes a circuit to fill the output 215 before the output 215 of the arithmetic logic unit 204 is stored in the accumulator 202, and the initialization instruction will load a configuration value into the circuit, and the configuration value will Affects the aforementioned fill operation. In an embodiment, it may also be specified in an arithmetic logic unit function instruction (such as a multiply accumulate instruction at address 1) or an output instruction (such as a write start function unit output instruction at address 4) so that the accumulator 202 is cleared to zero.

The second row at address 1 specifies a multiply-accumulate instruction to instruct the 512 neural processing units 126 to load a corresponding data text from one row of the data random access memory 122 and from the weight random access memory 124. One column contains a corresponding weight text, and The data text input 207 and the weight text input 206 perform a first multiply-accumulate operation, that is, add the zero value of the initialized accumulator 202. Further, this instruction instructs the sequencer 128 to generate a value at the control input 213 to select the data text input 207. In the example of the fourth figure, the designated row of the data random access memory 122 is row 17 and the designated row of the weight random access memory 124 is row 0, so the sequencer is instructed to output the value 17 as a data random The memory address 123 is accessed, and the value 0 is output as a weight of the random access memory address 125. Therefore, 512 data words from row 17 of data random access memory 122 are provided as corresponding data inputs 207 of 512 neural processing units 126, and 512 weights from row 0 of weight random access memory 124 The text provides the corresponding weight input 206 as 512 neural processing units 126.

The third column at address 2 specifies a multiply accumulate rotation instruction, which has a count of 511 to instruct the 512 neural processing units 126 to perform 511 multiply accumulate operations. This instruction instructs the 512 neural processing units 126 to input the data text 209 of the arithmetic logic unit 204 in each operation of the 511 multiply-accumulate operations as the rotation value 211 from the neighboring neural processing unit 126. That is, this instruction instructs the sequencer 128 to generate a value at the control input 213 to select the rotation value 211. In addition, this instruction instructs the 512 neural processing units 126 to load a corresponding weight value of each of the 511 multiply-accumulate operations into the "next" column of the weight random access memory 124. That is, this instruction will instruct the sequencer 128 to increase the weight random access memory address 125 by one from the previous time-frequency period. In this example, the first time-frequency period of the instruction is column 1, and the next time The frequency period is column 2. Below One time-frequency period is column 3, and so on, and the 511th time-frequency period is column 511. In each of these 511 multiply-accumulate operations, the product of the rotation input 211 and the weighted text input 206 is added to a value before the accumulator 202. The 512 neural processing units 126 will perform the 511 multiply-accumulate operations in 511 time-frequency cycles. Each neural processing unit 126 will respond to different data text from column 17 of data random access memory 122-that is, Adjacent neural processing units 126 performed data texts in the previous time-frequency cycle, and performed a multiply-accumulate operation on texts with different weights associated with the data texts, which are conceptually different inputs for different connections of neurons. This example assumes that each neural processing unit 126 (neuron) has 512 link inputs, and therefore involves the processing of 512 data words and 512 weight words. After the multiply accumulate and rotate instruction of column 2 is repeated for the last iteration, the accumulator 202 stores the sum of the products of the 512 linked inputs. In one embodiment, the instruction set of the neural processing unit 126 includes an "execute" instruction to instruct the arithmetic logic unit 204 to perform one of the arithmetic logic unit operations specified by the instruction of the initialization neural processing unit, such as the arithmetic of figure 29A The logic unit function 2926 specifies a separate instruction for each different type of arithmetic logic operation (such as the aforementioned multiply-accumulate, accumulator, and maximum values of weights, etc.).

The fourth column at address 3 specifies a start function instruction. This startup function instruction instructs the startup function unit 212 to execute the specified startup function on the value of the accumulator 202 to produce a result 133. An example of the startup function will be described in more detail in the subsequent sections.

The fifth row at address 4 specifies a write start function The number unit output instruction instructs the 512 neural processing units 216 to write the output of their activation function unit 212 as a result 133 back to a row of the data random access memory 122, which is 16 in this example. That is, this instruction instructs the sequencer 128 to output the value 16 as the data random access memory address 123 and a write command (corresponding to the read command specified by the multiply-accumulate instruction at address 1). In a preferred embodiment, because of the characteristics of pipeline execution, the write start function unit output instruction can be executed simultaneously with other instructions, so the write start function unit output instruction can actually be executed in a single time-frequency cycle.

In a preferred embodiment, each neural processing unit 126 serves as a pipeline, and the pipeline has various functional elements, such as the multiplexer register 208 (and the multiplexer register 705 in the seventh figure), arithmetic The logic unit 204, the accumulator 202, the start function unit 212, the multiplexer 802 (see FIG. 8), the column buffer 1104, and the start function unit 1112 (see FIG. 11), etc., some of the elements themselves are Can be executed in pipeline. In addition to the data text 207 and the weight text 206, this pipeline also receives instructions from the program memory 129. These instructions flow along the pipeline and control multiple functional units. In another embodiment, the program does not include a start function instruction. Instead, the initialization neural processing unit instruction specifies the start function to be executed on the accumulator 202 value 217, indicating that one of the specified start function values is stored in a configuration. The register is used by the startup function unit 212 of the pipeline after generating the final value 217 of the accumulator 202, that is, after the multiply accumulate rotation instruction at address 2 is repeated for the last execution. In a preferred embodiment, in order to save energy consumption, the startup function unit 212 of the pipeline arrives at the write startup function unit and the output instruction arrives. The former will be in a non-starting state. When the instruction arrives, the startup function unit 212 will start and execute the startup function on the accumulator 202 output 217 specified by the initialization instruction.

The fifth diagram is a timing chart showing that the neural network unit 121 executes the program of the fourth diagram. Each column of this timing diagram corresponds to the continuous time-frequency period indicated by the first row. The other lines correspond to different neural processing units 126 of the 512 neural processing units 126 and indicate their operations. The figure only shows the operations of the neural processing unit 0,1,511 to simplify the description.

At time-frequency period 0, each of the 512 neural processing units 126 executes the initialization instruction of the fourth figure, and in the fifth figure, a zero value is assigned to the accumulator 202.

In the time-frequency cycle 1, each of the 512 neural processing units 126 executes a multiply-accumulate instruction at address 1 in the fourth figure. As shown in the figure, the neural processing unit 0 adds the value of accumulator 202 (that is, zero) to the text 0 of row 17 of data random access memory 122 and the text 0 of row 0 of weight random access memory 124 Product; the neural processing unit 1 adds the value of accumulator 202 (ie, zero) to the product of text 1 in row 17 of data random access memory 122 and text 1 in row 0 of weight random access memory 124; By analogy, the neural processing unit 511 multiplies the value (ie zero) of the accumulator 202 by the text 511 of row 17 of data random access memory 122 and the text 511 of row 0 of weight random access memory 124.

In the time-frequency cycle 2, each of the 512 neural processing units 126 performs the address 2 in the fourth figure. The first iteration of the multiply accumulate rotation instruction. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 to the rotated data text 211 received by the multiplexer register 208 output 209 of the neural processing unit 511 (that is, the data received by the data random access memory 122). Data text 511) multiplied by text 0 of column 1 of weighted random access memory 124; neural processing unit 1 adds the value of accumulator 202 to the rotation received by output 209 of multiplex register 208 of neural processing unit 0 The product of data text 211 (data text 0 received by data random access memory 122) and text 1 of row 1 of weight random access memory 124; and so on, the neural processing unit 511 sets the value of accumulator 202 Add the rotation data text 211 (i.e., the data text 510 received by the data random access memory 122) received by the multiplexing register 208 output 209 of the neural processing unit 510 and the weight random access memory 124 in line 1. Product of text 511.

At time-frequency period 3, each of the 512 neural processing units 126 performs a second iteration of the multiply-accumulate rotation instruction at address 2 in the fourth figure. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 to the rotated data text 211 received by the multiplexer register 208 output 209 of the neural processing unit 511 (that is, the data received by the data random access memory 122). (Data text 510) multiplied by text 0 of weight 2 of column 2 of random access memory 124; neural processing unit 1 adds the value of accumulator 202 to the rotation received by output 209 of multiplex register 208 of neural processing unit 0 Product of data text 211 (data text 511 received by data random access memory 122) and text 1 of row 2 of weighted random access memory 124; and so on, the neural processing unit 511 sets the value of accumulator 202 Plus multi-stage temporary storage by neural processing unit 510 The processor 208 outputs a product of the rotated data text 211 (ie, the data text 509 received by the data random access memory 122) received by the 209 and the text 511 of column 2 of the weighted random access memory 124. As shown in the fifth figure, the 509 time-frequency cycles will continue in this manner until the time-frequency cycle 512.

In the time-frequency cycle 512, each of the 512 neural processing units 126 performs the 511th iteration of the multiply-accumulate rotation instruction at address 2 in the fourth figure. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 to the rotated data text 211 received by the multiplexer register 208 output 209 of the neural processing unit 511 (that is, the data received by the data random access memory 122). Data text 1) Multiplied by text 0 of weighted random access memory 124 column 511; neural processing unit 1 adds the value of accumulator 202 to the rotation received by output 209 of multiplex register 208 of neural processing unit 0 The product of data text 211 (data text 2 received by data random access memory 122) and text 1 of row 511 of weight random access memory 124; and so on, the neural processing unit 511 sets the value of accumulator 202 Add the rotated data text 211 (ie, the data text 0 received by the data random access memory 122) received by the multiplexer register 208 output 209 of the neural processing unit 510 and the weighted random access memory 124 row 511 Product of text 511. In one embodiment, multiple time-frequency cycles are required to read data text and weight text from the data random access memory 122 and the weight random access memory 124 to execute the multiply-accumulate instruction at address 1 in the fourth figure; however, The data random access memory 122, the weight random access memory 124, and the neural processing unit 126 are configured by pipelines, so that after the first multiply-accumulate operation starts (as shown in the fifth figure) Frequency cycle 1), subsequent multiply-accumulate operations (shown as time-frequency cycle 2-512 in the fifth figure) will begin to be executed in subsequent time-frequency cycles. According to a preferred embodiment, according to the architecture instructions, such as MTNN or MFNN instructions (which will be described in the following fourteenth and fifteenth drawings), the data random access memory 122 and / or weight random access memory These neural processing units 126 are temporarily put on hold by the access operation of the body 124 or the micro instructions translated from the architectural instructions.

In the time-frequency period 513, the activation function unit 212 of each of the 512 neural processing units 126 executes the activation function at address 3 in the fourth figure. Finally, in the time-frequency cycle 514, each of the 512 neural processing units 126 will execute the fourth by writing the result 133 back to the corresponding text in column 16 of the data random access memory 122. In the figure, the write start function unit of address 4 outputs an instruction, that is, the result 133 of the neural processing unit 0 will be written into the data 0 of the random access memory 122, and the result 133 of the neural processing unit 1 will be written. The text 1 of the data random access memory 122 is entered, and so on, the result 133 of the neural processing unit 511 is written into the text 511 of the data random access memory 122. The corresponding block diagram corresponding to the operation of the aforementioned fifth figure is shown in the sixth A figure.

The sixth diagram A is a block diagram showing that the neural network unit 121 of the first diagram executes the program of the fourth diagram. The neural network unit 121 includes 512 neural processing units 126, a data random access memory 122 that receives an address input 123, and a weight random access memory 124 that receives an address input 125. When the time-frequency cycle is 0, the 512 neural processing units 126 execute initialization instructions. This operation is not shown in the figure. Such as As shown in the figure, at time-frequency period 1, 512 16-bit data characters of row 17 are read from the data random access memory 122 and provided to the 512 neural processing units 126. During the time-frequency period from 1 to 512, the 512 16-bit weight characters of columns 0 to 511 will be read from the weight random access memory 122 and provided to the 512 neural processing units 126. At time-frequency period 1, the 512 neural processing units 126 perform their corresponding multiply-accumulate operations on the loaded data text and weight text. This operation is not shown in the figure. During the time-frequency cycle from 2 to 512, the multiplex register 208 of 512 neural processing units 126 will operate as a rotator with 512 16-bit characters, and will previously use data random access memory The data text loaded in column 17 of 122 is rotated to the adjacent neural processing unit 126, and these neural processing units 126 execute the corresponding data text after rotation and the corresponding weight text loaded by the weight random access memory 124. Multiply and accumulate operations. At the time-frequency period 513, the 512 startup function units 212 execute the startup instruction. This operation is not shown in the figure. At the time-frequency period 514, the 512 neural processing units 126 write their corresponding 512 16-bit results 133 back to the 16th row of the data random access memory 122.

As shown in the figure, the number of time-frequency cycles required to generate the result text (neuronal output) and write back the data random access memory 122 or weight random access memory 124 is roughly the data received by the current layer of the neural network Enter (square) the root of the number. For example, if the current layer has 512 neurons and each neuron has 512 connections from the previous layer, the total number of these connections is 256K, and the number of time-frequency cycles required to generate the current layer results will be slightly greater than 512. So the neural network Unit 121 can provide extremely high performance in neural network computing.

The sixth diagram B is a flowchart showing that the processor 100 of the first diagram executes a structural program to perform a typical multiplication and accumulation start function operation of a neuron associated with a hidden layer of an artificial neural network using the neural network unit 121 The operation is the same as that performed by the program of the fourth figure. The example in Figure 6B assumes that there are four hidden layers (variable NUM_LAYERS marked in the initialization step 602), each hidden layer has 512 neurons, and each neuron is connected to all 512 neurons in the previous layer (through the fourth Figure program). However, it should be understood that the selection of the number of these layers and the number of neurons is to illustrate the invention of this case. The neural network unit 121 can apply similar calculations to the embodiment of different numbers of hidden layers, each layer has a different number of neurons. Embodiments, or embodiments in which neurons are not all connected. In one embodiment, a weight value for a non-existent neuron or a non-existent neuron connection in this layer is set to zero. In a preferred embodiment, the architecture program writes the first set of weights into the weight random access memory 124 and activates the neural network unit 121. When the neural network unit 121 is performing calculations associated with the first layer This architecture program will write the second set of weights into the weight random access memory 124. In this way, once the neural network unit 121 completes the calculation of the first hidden layer, the neural network unit 121 can start the calculation of the second layer. In this way, the architecture program will go back and forth between the two regions of the weight random access memory 124 to ensure that the neural network unit 121 can be fully utilized. This process starts at step 602.

In step 602, as described in the relevant section of FIG. 6A, the processor 100 executing the architecture program writes input values into the data randomly. The current hidden layer of neurons in the access memory 122 is the row 17 of the random access memory 122 in which data is written. These values may also be located in the row 17 of the data random access memory 122 as the operation result 133 (such as convolution, common source or input layer) of the neural network unit 121 for the previous layer. Second, the framework program initializes the variable N to the value 1. The variable N represents the current layer in the hidden layer to be processed by the neural network unit 121. In addition, the framework program initializes the variable NUM_LAYERS to the value 4 because there are four hidden layers in this example. The flow then proceeds to step 604.

In step 604, the processor 100 writes the weight text of the layer 1 into the weight random access memory 124, for example, the columns 0 to 511 shown in FIG. 6A. The flow then proceeds to step 606.

In step 606, the processor 100 uses the MTNN instruction 1400 that specifies a function 1432 to write the program memory 129, and writes a multiply-accumulate activation function program (as shown in the fourth figure) into the neural network unit 121 program memory. 129. The processor 100 then uses a MTNN instruction 1400 to start the neural network unit program, which instruction specifies a function 1432 to start executing the program. The flow then proceeds to step 608.

In decision step 608, the architecture program determines whether the value of the variable N is less than NUM_LAYERS. If yes, the flow proceeds to step 612; otherwise, it proceeds to step 614.

In step 612, the processor 100 writes the weighted text of layer N + 1 into the weighted random access memory 124, for example, columns 512 to 1023. Therefore, the architecture program can write the weight text of the next layer into the weight random access memory 124 when the neural network unit 121 performs the calculation of the hidden layer of the current layer, thereby completing the calculation of the current layer, that is, writing data After the random access memory 122, the neural network unit 121 can immediately start the hidden layer calculation of the next layer. Next, proceed to step 614.

In step 614, the processor 100 confirms whether the executing neural network unit program (in the case of layer 1, the execution is started in step 606, and in the case of layers 2 to 4, it is executed in step 618) whether the execution has been completed. . In a preferred embodiment, the processor 100 reads the state register 127 of the neural network unit 121 by executing an MFNN instruction 1500 to confirm whether the execution has been completed. In another embodiment, the neural network unit 121 generates an interrupt, indicating that the multiplication and accumulation start function layer program has been completed. The flow then proceeds to decision step 616.

In decision step 616, the architecture program determines whether the value of the variable N is less than NUM_LAYERS. If yes, the flow proceeds to step 618; otherwise, it proceeds to step 622.

In step 618, the processor 100 updates the multiply accumulate activation function program to enable execution of the hidden layer calculation of the layer N + 1. Further, the processor 100 updates the data random access memory 122 row value of the multiply accumulate instruction at address 1 in the fourth figure to the row in which the calculation result of the previous layer in the data random access memory 122 is written ( For example, update to column 16) and update the output column (for example, update to column 15). The processor 100 then starts updating the neural network unit program. In another embodiment, the program of the fourth figure specifies the same row of the output instruction at address 4 as the row specified by the multiply-accumulate instruction at address 1 (that is, the row read by the data random access memory 122). ). In this embodiment, the current column of input data text will be overwritten (because this column of data text has been read into the multiplexer register 208 and rotated between these neural processing units 126 through the N text rotator, as long as This column of text does not need to be used for other purposes, such processing is allowed). In this case, the neural network unit program does not need to be updated in step 618, but only needs to be restarted. The flow then proceeds to step 622.

In step 622, the processor 100 reads the result of the neural network unit program of the layer N from the data random access memory 122. However, if these results are only used in the next layer, the architecture program does not need to read these results from the data random access memory 122, but can retain them in the data random access memory 122 for the next hidden layer calculation. Use. The flow then proceeds to step 624.

In decision step 624, the architecture program determines whether the value of the variable N is less than NUM_LAYERS. If yes, the flow proceeds to step 626; otherwise, the flow is terminated.

In step 626, the framework program increases the value of N by one. The flow then returns to decision step 608.

As shown in the example of Figure 6B, roughly every 512 time-frequency cycles, these neural processing units 126 perform a read and a write on the data random access memory 122 (through the neural network of Figure 4) Effect of the path unit program). In addition, these neural processing units 126 read the weight random access memory 124 for each time-frequency cycle to read a list of weight text. Therefore, the entire bandwidth of the weighted random access memory 124 will be consumed because the neural network unit 121 performs hidden layer operations in a mixed manner. In addition, it is assumed that there is a write and read buffer in an embodiment, such as the buffer 1704 of the seventeenth figure, while the neural processing unit 126 reads, the processor 100 The weight random access memory 124 performs writing, so that the buffer 1704 performs writing to the weight random access memory 124 approximately every 16 time-frequency cycles to write the weight text. Therefore, in the embodiment in which the weight random access memory 124 is a single port (as described in the corresponding section of FIG. 17), these neural processing units 126 temporarily suspend the weighting for approximately every 16 time-frequency cycles. The read by the random access memory 124 enables the buffer 1704 to write to the weighted random access memory 124. However, in the dual-port weight random access memory 124 embodiment, these neural processing units 126 need not be left unused.

The seventh diagram is a block diagram showing another embodiment of the neural processing unit 126 of the first diagram. The neural processing unit 126 in the seventh figure is similar to the neural processing unit 126 in the second figure. However, the neural processing unit 126 in the seventh figure additionally has a dual-input multiplexer register 705. The multiplexer register 705 selects one of the inputs 206 or 711 to be stored in its register, and provides it to its output 203 in the subsequent time-frequency cycle. Input 206 receives weighted text from weighted random access memory 124. The other input 711 receives the output 203 of the second multiplexer register 705 of the neighboring neural processing unit 126. In a preferred embodiment, the input 711 of the neural processing unit J receives the output 203 of the multiplexer register 705 arranged in the neural processing unit 126 of J-1, and the output 203 of the neural processing unit J provides Input 711 to multiplex register 705 arranged in neural processing unit 126 of J + 1. In this way, the multiplexer registers 705 of the N neural processing units 126 can work together. For example, the rotation of the same N characters is similar to the way shown in the third figure above, but it is used for weighted characters. Non-data text. The multiplexer register 705 uses a control input 213 to control the two Which one of the inputs is selected by the multiplexer register 705 to be stored in its register and subsequently provided to the output 203.

The use of the multiplexer register 208 and / or the multiplexer register 705 (and the multiplexer register in other embodiments as shown in Figures 18 and 23) actually constitutes a large rotator By rotating the data / weights from one row of the data random access memory 122 and / or the weight random access memory 124, the neural network unit 121 does not need to randomly store data in the data random access memory 122 and / or the weights. A very large multiplexer is used between the memory 124 to provide the required data / weight text to the appropriate neural network unit.

Write back the accumulator value in addition to the start function result

For some applications, the processor 100 receives (eg, the media register 118 via the MFNN instruction in Figure 15) the unprocessed accumulator 202 value 217 to provide for execution to other execution units 112 The instructions for performing calculations have their usefulness. For example, in one embodiment, the startup function unit 212 is not configured for the execution of the soft maximum startup function to reduce the complexity of the startup function unit 212. Therefore, the neural network unit 121 outputs the unprocessed accumulator 202 value 217 or one of its subsets to the data random access memory 122 or the weight random access memory 124, and the architecture program can be randomized by the data in the subsequent steps. The access memory 122 or the weighted random access memory 124 reads and calculates this unprocessed value. However, the application of the unprocessed accumulator 202 value 217 is not limited to performing soft maximum operations, other applications are also covered by the present invention.

The eighth diagram is a schematic block diagram showing another embodiment of the neural processing unit 126 of the first diagram. The neural processing unit 126 of the eighth figure is similar to the neural processing unit 126 of the second figure. However, the neural processing unit 126 of the eighth figure includes a multiplexer 802 in the activation function unit 212, and the activation function unit 212 has a control input 803. The width (in bits) of the accumulator 202 is greater than the width of the data text. The multiplexer 802 has a plurality of inputs to receive the data text width portion of the accumulator 202 output 217. In an embodiment, the width of the accumulator 202 is 41 bits, and the neural processing unit 126 can output a 16-bit result text 133. Thus, for example, the multiplexer 802 (or the thirty-third figure) The multiplexer 3032 and / or the multiplexer 3037) has three inputs respectively receiving bits [15: 0], bits [31:16], and bits [47:32] of the output 217 of the accumulator 202. For a preferred embodiment, output bits (such as bits [47:41]) not provided by the accumulator 202 are forced to be set to zero-valued bits.

The sequencer 128 generates a value at the control input 803, and controls the multiplexer 802 to select one of the text (such as 16 bits) of the accumulator 202 to write an accumulator instruction corresponding to one, for example, the ninth figure below Write accumulator instructions at addresses 3 to 5. For a preferred embodiment, the multiplexer 802 has one or more inputs to receive the output of a startup function circuit (such as components 3022, 3024, 3026, 3018, 3014, and 3016 in the thirty figure), and The width of the output produced by these startup function circuits is equal to one data text. The sequencer 128 will generate a value at the control input 803 to control the multiplexer 802 to choose one of these startup function circuit outputs, instead of choosing one of the words in the accumulator 202, in response to the position in the fourth figure The start function unit of address 4 outputs the instruction.

The ninth figure is a table showing a program stored in the program memory 129 of the neural network unit 121 of the first figure and executed by the neural network unit 121. The example program in Figure 9 is similar to the program in Figure 4. In particular, the instructions at addresses 0 to 2 are exactly the same. However, the instructions at addresses 3 and 4 in the fourth figure are replaced by the write accumulator instruction in the ninth figure. This instruction instructs 512 neural processing units 126 to write their accumulator 202 output 217 as the result 133 and write it back. The three rows of data random access memory 122, in this example, rows 16 to 18. In other words, the write accumulator instruction instructs the sequencer 128 to output a data random access memory address 123 with a value of 16 in the first time-frequency period and a write command, and output a data command in the second time-frequency period. The data random access memory address 123 with a value of 17 and a write command are outputted in the third time-frequency period with a data random access memory address 123 with a value of 18 and a write command. In a preferred embodiment, the execution time of the write accumulator instruction can overlap with other instructions, so that the write accumulator instruction can actually be executed in these three time-frequency cycles, each of which A cycle is written into one row of the data random access memory 122. In one embodiment, the user specifies the value of the start function 2934 and the output command 2956 column of the control register 127 (Figure 29A) to write the required part of the accumulator 202 into the data for random access. Memory 122 or weight random access memory 124. In addition, the write accumulator instruction may selectively write back a subset of the accumulator 202, instead of writing back the entire contents of the accumulator 202. In one embodiment, the standard type accumulator 202 can be written back. This section will be explained in more detail in the subsequent chapters corresponding to the twenty-ninth to thirty-first figures.

The tenth diagram shows that the neural network unit 121 performs the ninth The timing diagram of the program. The timing diagram of the tenth diagram is similar to the timing diagram of the fifth diagram, in which the time-frequency periods 0 to 512 are the same. However, in the time-frequency period 513-515, the start function unit 212 of each of the 512 neural processing units 126 will execute one of the write accumulator instructions at addresses 3 to 5 in the ninth figure . In particular, in the time-frequency period 513, each of the 512 neural processing units 126 will output bit 217 [15: 0] of the accumulator 202 as its result 133 and write it back to the data random access memory 122 Corresponding text in column 16; in the time-frequency cycle 514, each of the 512 neural processing units 126 will output bit 217 [31:16] of accumulator 202 as the result 133 and write it back to the data Corresponding text in column 17 of random access memory 122; and in the time-frequency cycle 515, each of the 512 neural processing units 126 will output 217 bits of accumulator 202 [40:32] As a result 133, the corresponding text in column 18 of the data random access memory 122 is written back. In a preferred embodiment, the bits [47:41] are forced to be set to zero.

Shared startup function unit

The eleventh figure is a block diagram showing an embodiment of the neural network unit 121 in the first figure. In the embodiment of FIG. 11, a neuron system is divided into two parts, that is, an activation function unit part and an arithmetic logic unit part (this part also includes a shift register part), and each activation function unit part is composed of multiple Part of the arithmetic logic unit is shared. In the eleventh figure, the arithmetic logic unit part refers to the neural processing unit 126, and the shared activation function unit part refers to the activation function unit 1112. Compared with the embodiment shown in the second figure, each neuron includes its own activation function unit 212. Accordingly, in an example of the embodiment in the eleventh figure, the neural processing unit 126 (the arithmetic logic unit part) may include the accumulator 202, the arithmetic logic unit 204, the multiplexer register 208 and the register in the second figure. 205, but does not include the start function unit 212. In the embodiment of the eleventh figure, the neural network unit 121 includes 512 neural processing units 126, but the present invention is not limited thereto. In the example of FIG. 11, the 512 neural processing units 126 are divided into 64 groups, which are labeled as groups 0 to 63 in FIG. 11, and each group has eight neural processing units 126.

The neural network unit 121 includes a column buffer 1104 and a plurality of shared activation function units 1112. These activation function units 1112 are coupled between the neural processing unit 126 and the column buffer 1104. The width (in bits) of the row buffer 1104 is the same as one row of the data random access memory 122 or the weight random access memory 124, for example, 512 characters. Each group of neural processing units 126 has an activation function unit 1112, that is, each activation function unit 1112 corresponds to a group of neural processing units 126; thus, in the embodiment of FIG. 11, there are 64 The activation function unit 1112 corresponds to a group of 64 neural processing units 126. The eight neural processing units 126 of the same group share the activation function unit 1112 corresponding to this group. The invention can also be applied to embodiments having different numbers of activation function units and different numbers of neural processing units in each group. For example, the present invention can also be applied to an embodiment in which two, four, or sixteen neural processing units 126 share the same activation function unit 1112 in each group.

Shared activation function unit 1112 helps reduce nerves The size of the network unit 121. Size reduction will sacrifice performance. That is, depending on the sharing rate, additional time-frequency cycles will be required to generate the result 133 of the entire neural processing unit 126 array. For example, as shown in the twelfth figure below, the sharing rate at 8: 1 In this case, seven additional time-frequency cycles are required. However, in general, compared with the number of time-frequency cycles required to generate the cumulative total (for example, for each neuron with one layer of 512 connections, 512 time-frequency cycles are required), the aforementioned additional time The number of frequency cycles (for example, 7) is relatively small. Therefore, the effect of the shared activation function unit on the performance is very small (for example, an increase of about one percent of the calculation time), which may be a cost effective for reducing the size of the neural network unit 121.

In one embodiment, each neural processing unit 126 includes an activation function unit 212 to execute a relatively simple activation function. These simple activation function units 212 have a small size and can be included in each of the neural processing units 126. Conversely, the shared complex startup function unit 1112 executes a relatively complex startup function, and its size will be significantly larger than the simple startup function unit 212. In this embodiment, only when the complex startup function is specified and needs to be executed by the shared complex startup function unit 1112, an additional time-frequency cycle is required. In the case where the designated startup function can be executed by the simple startup function unit 212, This extra time-frequency period is not needed.

The twelfth and thirteenth diagrams are timing charts showing that the neural network unit 121 of the eleventh diagram executes the program of the fourth diagram. The timing diagram of the twelfth figure is similar to the timing diagram of the fifth figure, and the time-frequency periods 0 to 512 are the same. However, the operation in the time-frequency period 513 is not the same, because The neural processing unit 126 of the eleventh figure will share the activation function unit 1112; that is, the neural processing unit 126 of the same group will share the activation function unit 1112 associated with this group, and the eleventh figure shows this Shared architecture.

Each column of the timing chart of the thirteenth figure corresponds to a continuous time-frequency period marked on the first row. The other lines correspond to different start function units 1112 of the 64 start function units 1112 and indicate their operations. The figure only shows the operations of the neural processing units 0,1,63 to simplify the description. The time-frequency cycle of the thirteenth figure corresponds to the time-frequency cycle of the twelfth figure, but the operation of the neural processing unit 126 sharing the activation function unit 1112 is shown in different ways. As shown in the thirteenth figure, in the time-frequency period of 0 to 512, the 64 startup function units 1112 are in a non-starting state, and the neural processing unit 126 executes an instruction to initialize the neural processing unit, a multiply accumulate instruction, and a multiply accumulate rotation instruction. .

As shown in the twelfth and thirteenth figures, in the time-frequency period 513, the activation function unit 0 (the activation function unit 1112 associated with the group 0) starts performing the specified activation on the value 217 of the accumulator 202 of the neural processing unit 0 Function, the neural processing unit 0 is the first neural processing unit 216 in the group 0, and the output of the activation function unit 1112 will be stored in the character 0 of the column register 1104. Also in the time-frequency period 513, each activation function unit 1112 starts to execute the specified activation function on the value 217 of the accumulator 202 of the first neural processing unit 126 in the corresponding neural processing unit 216 group. Therefore, as shown in the thirteenth figure, in the time-frequency period 513, the activation function unit 0 starts to execute the specified activation function on the accumulator 202 of the neural processing unit 0 to generate a text that will be stored in the column register 1104. The result of word 0; the activation function unit 1 starts executing the specified activation function on the accumulator 202 of the neural processing unit 8 to produce the result of the word 8 which will be stored in the register 1104; and so on, the function unit 63 is activated The execution of the specified activation function on the accumulator 202 of the neural processing unit 504 is started to produce a result of the text 504 which will be stored in the column register 1104.

In the time-frequency period 514, the activation function unit 0 (the activation function unit 1112 associated with group 0) starts to execute the specified activation function on the accumulator 202 value 217 of the neural processing unit 1. The neural processing unit 1 is in group 0. The second neural processing unit 216, and the output of the activation function unit 1112 will be stored in the text 1 of the column register 1104. Also in the time-frequency period 514, each activation function unit 1112 starts to execute the specified activation function on the value 217 of the accumulator 202 of the second neural processing unit 126 in the corresponding neural processing unit 216 group. Therefore, as shown in the thirteenth figure, in the time-frequency period 514, the activation function unit 0 starts to execute the specified activation function on the accumulator 202 of the neural processing unit 1 to generate the text 1 that will be stored in the column register 1104. As a result, the activation function unit 1 starts executing the specified activation function on the accumulator 202 of the neural processing unit 9 to produce a result of the text 9 which will be stored in the column register 1104; and so on, the activation function unit 63 starts to The accumulator 202 of the neural processing unit 505 executes the designated activation function to generate a result of the text 505 which will be stored in the column register 1104. Such processing will continue until the time-frequency period 520, and the activation function unit 0 (the activation function unit 1112 associated with group 0) starts to execute the specified activation function on the accumulator 202 value 217 of the neural processing unit 7, the neural processing unit 7 That is, the eighth (last) neural processing unit 216 in group 0, and the output of the activation function unit 1112 will be Character 7 stored in the column register 1104. Also in the time-frequency period 520, each activation function unit 1112 starts to execute the specified activation function on the value 217 of the accumulator 202 of the eighth neural processing unit 126 in the corresponding neural processing unit 216 group. Therefore, as shown in the thirteenth figure, in the time-frequency period 520, the activation function unit 0 starts to execute the specified activation function on the accumulator 202 of the neural processing unit 7 to generate a character 7 that will be stored in the column register 1104. As a result, the activation function unit 1 starts executing the specified activation function on the accumulator 202 of the neural processing unit 15 to produce a result of the character 15 which will be stored in the register 1104; and so on, the activation function unit 63 starts The accumulator 202 of the neural processing unit 511 executes the designated activation function to generate a result of the text 511 which will be stored in the column register 1104.

In the time-frequency cycle 521, once all 512 results of the 512 neural processing units 126 have been generated and written to the column register 1104, the column register 1104 will begin to write its contents into the data random access memory. 122 or weight random access memory 124. In this way, the activation function unit 1112 of each group of neural processing units 126 executes a part of the activation function instruction at address 3 in the fourth figure.

The embodiment in which the activation function unit 1112 is shared in the group of the arithmetic logic unit 204 as shown in FIG. 11 is particularly helpful for the use with the integer arithmetic logic unit 204. This part will be explained in the subsequent chapters if they correspond to the figures 29A to 33.

MTNN and MFNN architecture instructions

The fourteenth figure is a block diagram showing a move to The neural network (MTNN) architecture instruction 1400 and its operation corresponding to the neural network unit 121 of the first figure. The MTNN instruction 1400 includes an execution code field 1402, an src1 field 1404, a src2 field, a gpr field 1408, and an immediate field 1412. The MTNN instruction is an architecture instruction, that is, the instruction is included in the instruction set architecture of the processor 100. For a preferred embodiment, the instruction set architecture uses a preset value in the execution code field 1402 to distinguish the MTNN instruction 1400 from other instructions in the instruction set architecture. The execution code 1402 of the MTNN instruction 1400 may include a prefix that is common in the x86 architecture and the like, or may not be included.

The immediate field 1412 provides a value to specify a function 1432 to the control logic 1434 of the neural network unit 121. For a preferred embodiment, this function 1432 is an immediate operand of one of the microinstructions 105 in the first figure. These functions 1432 that can be executed by the neural network unit 121 include write data random access memory 122, write weight random access memory 124, write program memory 129, write control register 127, and start execution The program in the program memory 129, the execution of the program in the program memory 129 is suspended, the notification request (such as interruption) after the execution of the program in the program memory 129 is completed, and the neural network unit 121 is reset, but is not limited thereto. According to a preferred embodiment, the neural network unit instruction set includes an instruction, and the result of the instruction indicates that the neural network unit program has been completed. In addition, the neural network unit instruction set includes an explicitly generated interrupt instruction. In a preferred embodiment, the operation of resetting the neural network unit 121 includes removing the data from the neural network unit 121 in addition to the data random access memory 122, the weight random access memory 124, and the program memory 129. Information will remain intact and effectively enforce Revert to a reset state (for example, empty the internal state machine and set it to idle state). In addition, internal registers, such as the accumulator 202, will not be affected by the reset function and must be explicitly cleared, for example, using the instruction of initializing the neural processing unit at address 0 in the fourth figure. In an embodiment, the function 1432 may include a direct execution function, and a first source register thereof includes a micro operation (for example, refer to the micro operation 3418 in FIG. 34). This direct execution function instructs the neural network unit 121 to directly execute the specified micro-operation. In this way, the architecture program can directly control the neural network unit 121 to perform operations, instead of writing instructions to the program memory 129 and subsequently instructing the neural network unit 121 to execute the instructions located in the program memory 129 or via MTNN instructions 1400 (or MFNN instruction 1500 in the fifteenth figure). FIG. 14 shows an example of the function of the write data random access memory 122.

This gpr field specifies one of the general purpose registers in the general purpose register file 116. In one embodiment, each general purpose register is 64 bits. The universal register file 116 provides the value of the selected universal register to the neural network unit 121, as shown in the figure, and the neural network unit 121 uses the value as the address 1422. This address 1422 selects a row of memory specified in function 1432. For data random access memory 122 or weight random access memory 124, an additional data block is selected at this address 1422, which is twice the size of the media register location in this selected row (for example, 512 Bit). For a preferred embodiment, this location is located on a 512-bit boundary. In one embodiment, the multiplexer will select the address 1422 (or the address 1422 in the case of the MFNN instruction 1400 described below) or the address 123/125/131 from the sequencer 128. Provided to data random access memory 124 / weight random access memory 124 / program memory 129. In one embodiment, the data random access memory 122 has dual ports, so that the neural processing unit 126 can use the media register 118 to read / write the data random access memory 122 and read / write at the same time. Enter this data into the random access memory 122. In an embodiment, for similar purposes, the weight random access memory 124 also has dual ports.

The src1 field 1404 and the src2 field 1406 in the figure both specify a media register of the media register file 118. In one embodiment, each media register 118 is 256 bits. The media register file 118 provides the connected data (e.g., 512 bits) from the selected media register to the data random access memory 122 (or weight random access memory 124 or program memory). 129) The selected row 1428 designated by write address 1422 and the position designated by address 1422 in the selected row 1428 are shown in the figure. Through the execution of a series of MTNN instructions 1400 (and the MFNN instruction 1500 described below), the architecture program executed on the processor 100 can fill 122 rows of data random access memory and 124 rows of weight random access memory and A program is written into the program memory 129, for example, the program described in this article (such as the programs shown in Figures 4 and 9) enables the neural network unit 121 to perform operations on data and weights at a very fast speed to complete this manual operation. Neural network. In one embodiment, the architecture program directly controls the neural network unit 121 instead of writing the program into the program memory 129.

In one embodiment, the MTNN instruction 1400 specifies an initial source register and the number of source registers, that is, Q, rather than specifying two source registers (such as those specified in fields 1404 and 1406). This The form of MTNN instruction 1400 instructs the processor 100 to write the media register 118 designated as the starting source register and the next Q-1 consecutive media registers 118 to the neural network unit 121, that is, The designated data random access memory 122 or weight random access memory 124 is entered. For a preferred embodiment, the instruction translator 104 translates the MTNN instruction 1400 into the micro-instructions required to write all Q designated media registers 118. For example, in one embodiment, when the MTNN instruction 1400 designates the register MR4 as the starting source register and Q is 8, the instruction translator 104 translates the MTNN instruction 1400 into four micro instructions, where The first microinstruction is written into the registers MR4 and MR5, the second microinstruction is written into the registers MR6 and MR7, the third microinstruction is written into the registers MR8 and MR9, and the fourth microinstruction is written into the registers MR8 and MR9. The instructions are written into the registers MR10 and MR11. In another embodiment, the data path from the media register 118 to the neural network unit 121 is 1024 bits instead of 512 bits. In this case, the instruction translator 104 translates the MTNN instruction 1400 into two Micro instructions. The first micro instruction is written into the registers MR4 to MR7, and the second micro instruction is written into the registers MR8 to MR11. The present invention can also be applied to the embodiment where the MFNN instruction 1500 specifies an initial destination register and the number of destination registers, so that each MFNN instruction 1500 can access the data random access memory 122 or weight random access memory. One row of 124 reads data blocks larger than the single media register 118.

The fifteenth figure is a block diagram showing the operation of a move to neural network (MTNN) architecture instruction 1500 and its portion corresponding to the neural network unit 121 of the first figure. This MFNN instruction 1500 includes An execution code field 1502, a dst field 1504, a gpr field 1508, and an immediate field 1512. The MFNN instruction is an architecture instruction, that is, the instruction is included in the instruction set architecture of the processor 100. For a preferred embodiment, the instruction set architecture uses a preset value in the execution code field 1502 to distinguish the MFNN instruction 1500 from other instructions in the instruction set architecture. The execution code 1502 of the MFNN instruction 1500 may include a prefix commonly used in the x86 architecture and the like, or may not be included.

The immediate field 1512 provides a value to specify a function 1532 to the control logic 1434 of the neural network unit 121. For a preferred embodiment, this function 1532 is an immediate operand of one of the microinstructions 105 in the first figure. The functions 1532 that these neural network units 121 can execute include read data random access memory 122, read weight random access memory 124, read program memory 129, and read status register 127, but not Limited to this. The example in FIG. 15 shows a function 1532 for reading data from the random access memory 122.

This gpr field 1508 specifies one of the general purpose registers in the general purpose register file 116. This universal register file 116 provides the value of the selected universal register to the neural network unit 121, as shown in the figure, and the neural network unit 121 uses this value as the address 1522 and resembles tenth. The four-picture address 1422 performs calculations to select a row of memory specified in function 1532. For data random access memory 122 or weight random access memory 124, an additional data block is selected at this address 1522, and its size is the size of the media register (such as 256 bits) in the selected row. position. For a preferred embodiment, this location is on a 256-bit boundary.

The dst field 1504 specifies a media register in a media register file 118. As shown in the figure, the media register file 118 receives data (such as 256 bits) from the data random access memory 122 (or weight random access memory 124 or program memory 129) to the selected media Register, this data is read from the selected row 1528 specified by the address 1522 in the data receiving and the location specified by the address 1522 in the selected row 1528.

Port allocation of internal random access memory in neural network unit

The sixteenth diagram is a block diagram showing an embodiment of the data random access memory 122 of the first diagram. The data random access memory 122 includes a memory array 1606, a read port 1602, and a write port 1604. The memory array 1606 is loaded with data characters. For a preferred embodiment, these data are arranged in an array of N characters in D rows as described above. In one embodiment, the memory array 1606 includes an array of 64 horizontally arranged static random access memory cells, where each memory cell has a width of 128 bits and a height of 64 bits. A 64 KB data random access memory 122 is provided, which has a width of 8192 bits and has 64 rows. The die area used by this data random access memory 122 is approximately 0.2 mm 2. However, the present invention is not limited to this.

In a preferred embodiment, the write port 1602 is multiplexed to the neural processing unit 126 and the media register 118. Further, these media registers 118 can be coupled to the read port through a result bus, and the result bus is also used to provide data to the rearrangement buffer. And / or the result transfer bus is provided to other execution units 112. The neural processing unit 126 and the media register 118 share the read port 1602 to read the data random access memory 122. In addition, in a preferred embodiment, the write port 1604 is also multiplexed to the neural processing unit 126 and the media register 118. The neural processing units 126 and the media register 118 share the write port 1604 to write the data to the random access memory 122. In this way, the media register 118 can write to the data random access memory 122 while the neural processing unit 126 reads the data random access memory 122, and the neural processing unit 126 can also temporarily store the data in the media. While the device 118 is reading the data random access memory 122, it is writing into the data random access memory 122. This way of doing things can improve performance. For example, these neural processing units 126 can read the data random access memory 122 (for example, continuously perform calculations), and at the same time, the media register 118 can write more data text into the data random access memory 122 . In another example, the neural processing units 126 can write the calculation results into the data random access memory 122, and at the same time, the media register 118 can read the calculation results from the data random access memory 122. In one embodiment, the neural processing unit 126 can write a row of calculation results into the data random access memory 122, and also read a row of data characters from the data random access memory 122. In one embodiment, the memory array 1606 is configured as a memory bank. When the neural processing unit 126 accesses the data random access memory 122, all the memory blocks will be activated to access a complete row of the memory array 1606; however, the media register 118 accesses the data randomly When accessing memory 122, only the specified memory block will Was started. In one embodiment, the width of each memory block is 128 bits, and the width of the media register 118 is 256 bits. Thus, for example, each time the media register 118 is accessed, Need to start two memory blocks. In one embodiment, one of the ports 1602/1604 is a read / write port. In one embodiment, these ports 1602/1604 are read / write ports.

The advantage of having these neural processing units 126 with the ability of a rotator as described herein is that compared to ensuring that the neural processing unit 126 can be fully utilized, the architecture program (through the media register 118) can continuously provide data to The data random access memory 122 and the memory array required to retrieve the results from the data random access memory 122 while the neural processing unit 126 performs calculations, this capability helps reduce the amount of data in the random access memory 122. The number of rows of the memory array 1606 can be reduced.

Internal random access memory buffer

The seventeenth figure is a block diagram showing an embodiment of the weight random access memory 124 and a buffer 1704 of the first figure. The weighted random access memory 124 includes a memory array 1706 and a port 1702. This memory array 1706 is loaded with weighted texts. For a preferred embodiment, these weighted texts are arranged in an array of N texts in W rows as described above. In one embodiment, the memory array 1706 includes an array of 128 horizontally arranged static random access memory cells, where each memory cell has a width of 64 bits and a height of 2048 bits. Provide a 2MB weight random access memory 124 with a width of 8192 bits and 2048 rows, and this weight is random The die area used for the access memory 124 is approximately 2.4 mm 2. However, the present invention is not limited to this.

In a preferred embodiment, the port 1702 is multiplexed to the neural processing unit 126 and the buffer 1704. The neural processing unit 126 and the buffer 1704 read and write the weight random access memory 124 through the port 1702. The buffer 1704 is coupled to the media register 118 of the first figure. In this way, the media register 118 can read and write the weight random access memory 124 through the buffer 1704. The advantage of this method is that when the neural processing unit 126 is reading or writing the weight random access memory 124, the media register 118 can also write to or read from the buffer 118 (but if it is The neural processing unit 126 is being executed, and in a better case, these neural processing units 126 are shelved to avoid accessing the weight random access memory 124 when the buffer 1704 accesses the weight random access memory 124). This method can improve performance, especially because the reading and writing of the weighted random access memory 124 by the media register 118 is relatively significantly smaller than the reading and writing of the weighted random access memory 124 by the neural processing unit 126. . For example, in one embodiment, the neural processing unit 126 reads / writes 8192 bits (one column) at a time. However, the width of the media register 118 is only 256 bits, and each MTNN instruction 1400 only Two media registers 118 are written, that is, 512 bits. Therefore, when the architecture program executes sixteen MTNN instructions 1400 to fill the buffer 1704, the conflict time between the neural processing unit 126 and the architecture program of the access weight random access memory 124 will be less than approximately the entire time Six percent. In another embodiment, the instruction translator 104 translates one MTNN instruction 1400 into two microinstructions 105, and Each microinstruction writes a single data register 118 into the buffer 1704. In this way, the frequency of conflicts between the neural processing unit 126 and the architecture program when accessing the random access memory 124 is further reduced.

In the embodiment including the buffer 1704, writing the weighted random access memory 124 using a framework program requires multiple MTNN instructions 1400. One or more MTNN instructions 1400 specify a function 1432 to write the data block specified in the buffer 1704, and then an MTNN instruction 1400 specifies a function 1432 to instruct the neural network unit 121 to write the contents of the buffer 1704 to the weight random access One of the memory 124 is selected. The size of a single data block is twice the number of bits in the media register 118, and these data blocks are naturally aligned in the buffer 1704. In one embodiment, the MTNN instruction 1400 of each designated function 1432 to write a designated block of data to the buffer 1704 includes a bit mask having a bit corresponding to each data block of the buffer 1704. The data from the two designated source registers 118 are written into the data block of the buffer 1704, and the corresponding bits in the bit mask are the respective data blocks that are set. This embodiment facilitates the case where duplicate data values exist in one row of the weighted random access memory 124. For example, in order to reset buffer 1704 (and the next row of weighted random access memory 124) to zero, the programmer can load a zero value into the source register and set all bits of the bit mask. . In addition, the bit mask can also allow the programmer to write only the selected data block in the buffer 1704, while keeping the other data blocks in their previous data state.

In the embodiment including the buffer 1704, reading the weighted random access memory 124 using a framework program requires multiple MFNN instructions 1500. The initial MFNN instruction 1500 specifies a function 1532 to weight the A designated column of the machine access unit 124 is loaded into the buffer 1704, and then one or more MFNN instructions 1500 specify a function 1532 to read a designated data block of the buffer 1704 to the destination register. The size of a single data block is the number of bits in the media register 118, and these data blocks are naturally aligned in the buffer 1704. The technical features of the present invention can also be applied to other embodiments. For example, the weight random access memory 124 has multiple buffers 1704. By increasing the accessible number of the architecture program when the neural processing unit 126 is executed, the neural processing unit can be further reduced. The conflict between 126 and the structure program due to the access weight random storage memory 124 is increased. In the time-frequency period of the neural processing unit 126 without accessing the weight random access memory 124, it is stored by the buffer 1704 instead. Take the possibility.

The sixteenth figure illustrates a dual-port data random access memory 122, but the present invention is not limited thereto. The technical features of the present invention can also be applied to other embodiments in which the weight random access memory 124 is also a dual-port design. In addition, FIG. 17 illustrates a buffer used with the weighted random access memory 124, but the present invention is not limited thereto. The technical features of the present invention can also be applied to the embodiment in which the data random access memory 122 has a corresponding buffer similar to the buffer 1704.

Dynamically configurable neural processing unit

The eighteenth figure is a block diagram showing a dynamically configurable neural processing unit 126, one of the first figures. The neural processing unit 126 of the eighteenth figure is similar to the neural processing unit 126 of the second figure. However, the neural processing unit 126 of Fig. 18 can be dynamically configured to operate in one of two different configurations. In the first configuration, the operation of the neural processing unit 126 of Fig. 18 is similar to that of the neural processing unit of Fig. 2. 126. That is, in the first configuration, which is marked here as a "wide" configuration or a "single" configuration, the arithmetic logic unit 204 of the neural processing unit 126 performs a single wide data text and a single wide weight text (E.g., 16 bits) performs an operation to produce a single wide result. In contrast, in the second configuration, which is marked as a “narrow” configuration or a “double” configuration, the neural processing unit 126 performs two narrow data texts and two narrow weight texts (for example, eight (Bits) performing operations each yield two narrow results. In one embodiment, the configuration (wide or narrow) of the neural processing unit 126 is achieved by an instruction to initialize the neural processing unit (eg, an instruction located at address 0 in the twentieth graph). In addition, this configuration can also be achieved by an MTNN instruction with a configuration (wide or narrow) specified by the function 1432 to set the settings of the neural processing unit. In a preferred embodiment, the program memory 129 instructions or MTNN instructions that determine the configuration (wide or narrow) will fill the configuration register. For example, the output of the configuration register is provided to the arithmetic logic unit 204, the start function unit 212, and the logic for generating the multiplexer register control signal 213. Basically, the elements of the neural processing unit 126 of the eighteenth figure perform similar functions as the elements of the same number in the second figure, and reference can be obtained from it to understand the embodiment of the eighteenth figure. The following is a description of the embodiment of the eighteenth figure including the differences from the second figure.

The neural processing unit 126 of the eighteenth figure includes two registers 205A and 205B, two three-input multiplexed registers 208A and 208B, an arithmetic logic unit 204, two accumulators 202A and 202B, and two activations. Function units 212A and 212B. Registers 205A / 205B have half the width of register 205 in the second figure (e.g. 8 bits yuan). The registers 205A / 205B respectively receive a corresponding narrow weight text 206A / B206 (for example, 8 bits) from the weight random access memory 124 and provide the output 203A / 203B to the arithmetic logic in a subsequent time-frequency cycle. The operand selection logic of unit 204 is 1898. When the neural processing unit 126 is in the wide configuration, the registers 205A / 205B will work together to receive a wide weight text 206A / 206B (for example, 16 bits) from the weight random access memory 124, similar to the second The register 205 in the embodiment of the figure; when the neural processing unit 126 is in a narrow configuration, the registers 205A / 205B will actually operate independently, each receiving a narrow weight text from the weight random access memory 124 206A / 206B (for example, 8 bits). In this way, the neural processing unit 126 is actually equivalent to two narrow neural processing units operating independently. However, regardless of the configuration of the neural processing unit 126, the same output bits of the weight random access memory 124 are coupled and provided to the registers 205A / 205B. For example, register 205A of neural processing unit 0 receives byte 0, register 205B of neural processing unit 0 receives byte 1, register 205A of neural processing unit 1 receives byte 2, neural Register 205B of processing unit 1 receives byte 3, and so on, and register 205B of neural processing unit 511 receives byte 1023.

The multiplex register 208A / 208B has half the width (eg, 8 bits) of the register 208 of the second figure, respectively. The multiplexer register 208A will select one of the inputs 207A, 211A, and 1811A to store in its register, and will be provided by the output 209A in the subsequent time-frequency cycle. The multiplex register 208B will select from the inputs 207B, 211B, and 1811B. One is stored in its register and provided by output 209B to the operand selection logic in subsequent time-frequency cycles Series 1898. Input 207A receives a narrow data character (for example, 8 bits) from the data random access memory 122, and input 207B receives a narrow data character from the data random access memory 122. When the neural processing unit 126 is in a wide configuration, the multiplexer registers 208A / 208B actually work together to receive one wide data character 207A / 207B (e.g., 16 bits) from the data random access memory 122. ), Similar to the multiplex register 208 in the embodiment of the second figure; when the neural processing unit 126 is in a narrow configuration, the multiplex register 208A / 208B will actually operate independently, each receiving random data from One of the narrow data characters 207A / 207B (for example, 8 bits) of the memory 122 is accessed. In this way, the neural processing unit 126 is actually equivalent to two narrow neural processing units operating independently. However, regardless of the configuration of the neural processing unit 126, the same output bits of the data random access memory 122 will be coupled and provided to the multiplexer registers 208A / 208B. For example, multiplexer register 208A of neural processing unit 0 receives byte 0, multiplexer register 208B of neural processing unit 0 receives multiplexer register 208A of byte 1, neural processing unit 1 receives The multiplexer register 208B of byte 2 and neural processing unit 1 receives byte group 3, and so on, and the multiplexer register 208B of neural processing unit 511 receives byte group 1023.

Input 211A receives the output 209A of the multiplexer register 208A of the neighboring neural processing unit 126, and input 211B receives the output 209B of the multiplexer register 208B of the neighboring neural processing unit 126. Input 1811A receives the output 209B of the multiplexer register 208B adjacent to the neural processing unit 126, and input 1811B receives the output 209A of the multiplexer register 208A adjacent to the neural processing unit 126. The neural processing unit shown in Figure 18 126 belongs to one of the N neural processing units 126 shown in the first figure and is designated as a neural processing unit J. That is, the neural processing unit J is a representative example of these N neural processing units. For a preferred embodiment, the input 211A of the multiplexing register 208A of the neural processing unit J will receive the output 209A of the multiplexing register 208A of the neural processing unit 126 of Example J-1, and the number of the neural processing unit J is as large as 211A. The input 1811A of the industrial register 208A will receive the output 209B of the multiplex register 208B of the neural processing unit 126 of Example J-1, and the output 209A of the multiple register 208A of the neural processing unit J will be provided to the example J + 1 at the same time. The multiplexer register 208A of the neural processing unit 126 inputs 211A and the multiplexer register 208B of the neural processing unit 126 inputs 211B of the example J; the input 211B of the multiplexer register 208B of the neural processing unit J receives the example J The multiplexer register 208B of the neural processing unit 126 of -1 outputs 209B, and the input 1811B of the multiplexer register 208B of the neural processing unit J will receive the multiplexer register 208A of the neural processing unit 126 of example J to output 209A Moreover, the output 209B of the multiplexing register 208B of the neural processing unit J will be provided to the input 1811A of the multiplexing register 208A of the neural processing unit 126 of example J + 1 and the neural processing unit 126 of example J + 1. The multiplexer register 208B inputs 211B.

The control input 213 controls each of the multiplexing registers 208A / 208B, and selects one of the three inputs to store in its corresponding register, and provides the corresponding output 209A / 209B in the subsequent steps. When the neural processing unit 126 is instructed to load a row from the data random access memory 122 (for example, a multiply-accumulate instruction at address 1 in the twentieth figure, as described later), whether the neural processing unit 126 is in a wide configuration Or narrow configuration, the control input 213 controls the multiplexer register Each multiplex register in 208A / 208B selects a corresponding narrow data character 207A / 207B (for example, 8 bits) from a corresponding narrow character in a selected row of the data random access memory 122.

When the neural processing unit 126 receives an instruction that it is necessary to rotate the previously received data column value (for example, the multiply accumulate rotation instruction of address 2 in the twentieth figure, as described later), if the neural processing unit 126 is in a narrow configuration, control Input 213 will control each multiplexer register 208A / 208B to select the corresponding input 1811A / 1811B. In this case, the multiplexing registers 208A / 208B will actually operate independently so that the neural processing unit 126 will actually act like two independent narrow neural processing units. In this way, the multiplex registers 208A and 208B of the N neural processing units 126 work together as the same 2N narrow-text rotator, which is explained in more detail in the subsequent part corresponding to the nineteenth figure.

When the neural processing unit 126 receives an instruction that it is necessary to rotate the value of the previously received data column, if the neural processing unit 126 is in a wide configuration, the control input 213 controls each of the multiplexing registers in the multiplexing registers 208A / 208B. Select the corresponding input 211A / 211B. In this case, the multiplexer registers 208A / 208B work together as if the neural processing unit 126 is a single wide neural processing unit 126. In this way, the multiplexing registers 208A and 208B of the N neural processing units 126 work together as the same N wide text rotators, similar to the way described in the third figure.

Arithmetic logic unit 204 includes operand selection logic 1898, a wide multiplier 242A, a narrow multiplier 242B, and a wide multiplier 242B. A dual input multiplexer 1896A, a narrow dual input multiplexer 1896B, a wide adder 244A and a narrow adder 244B. In fact, the arithmetic logic unit 204 can be understood to include operand selection logic, a wide arithmetic logic unit 204A (including the aforementioned wide multiplier 242A, the aforementioned wide multiplexer 1896A, and the aforementioned wide adder 244A) and a narrow arithmetic logic unit 204B (including the aforementioned narrow multiplier 242B, the aforementioned narrow multiplexer 1896B, and the aforementioned narrow adder 244B). In a preferred embodiment, the wide multiplier 242A can multiply two wide characters, similar to the multiplier 242 of the second figure, such as a 16-bit multiplier by 16-bit multiplier. The narrow multiplier 242B can multiply two narrow characters, such as an 8-bit by 8-bit multiplier to produce a 16-bit result. When the neural processing unit 126 is in a narrow configuration, with the assistance of the operator selection logic 1898, the wide multiplier 242A can be fully used as a narrow multiplier to multiply two narrow texts, so the neural processing unit 126 will be as Two effectively functioning narrow neural processing units. For a preferred embodiment, the wide adder 244A adds the output of the wide multiplexer 1896A and the output 217A of the wide accumulator 202A to generate a total 215A for the wide accumulator 202A. Its operation is similar to that of the first Adder 244 of the second figure. The narrow adder 244B adds the output of the narrow multiplexer 1896B and the output of the narrow accumulator 202B to 217B to generate a total 215B for the narrow accumulator 202B. In one embodiment, the narrow accumulator 202B has a width of 28 bits to avoid losing accuracy when performing an accumulation operation of up to 1024 16-bit products. When the neural processing unit 126 is in a wide configuration, the narrow multiplier 244B, the narrow accumulator 202B, and the narrow start function unit 212B are preferably in a non-start state to reduce energy consumption.

Operand selection logic 1898 will select from 209A, 209B, The other elements in 203A and 203B are provided to the other elements of the arithmetic logic unit 204, as described later. For a preferred embodiment, the operand selection logic 1898 also has other functions, such as performing sign extension of signed numeric data text and weight text. For example, if the neural processing unit 126 is in a narrow configuration, the operator selection logic 1898 will extend the symbols of the narrow data text and weight text to the width of the wide text, and then provide it to the wide multiplier 242A. Similarly, if the arithmetic logic unit 204 accepts an instruction to pass a narrow data / weight text (using the wide multiplexer 1896A to skip the wide multiplier 242A), the operator selection logic 1898 will extend the sign of the narrow data text and weight text to The width of the wide text is then supplied to the wide adder 244A. For a preferred embodiment, the logic for performing the symbol extension function also exists in the arithmetic logic operation 204 of the neural processing unit 126 in the second figure.

The wide multiplexer 1896A receives the output of the wide multiplier 242A and an operand from the operand selection logic 1898, and selects one of these inputs to provide to the wide adder 244A. The narrow multiplexer 1896B receives the narrow multiplier 242B. The output and one of the operands from the operand selection logic 1898 selects one of these inputs and supplies it to the narrow adder 244B.

Operand selection logic 1898 provides operands according to the configuration of the neural processing unit 126 and the arithmetic and / or logical operations to be performed by the arithmetic logic unit 204. This arithmetic / logical operation is based on the functions specified by the instructions executed by the neural processing unit 126. Decide. For example, if the instruction instructs the arithmetic logic unit 204 to perform a multiply-accumulate operation and the neural processing unit 126 is in a wide configuration, the operand selection logic 1898 A wide text formed by concatenating the outputs 209A and 209B is provided to one input of the wide multiplier 242A, and a wide text formed by the concatenation of outputs 203A and 203B is provided to the other input, while the narrow multiplier 242B is not Start up, so that the neural processing unit 126 functions as a single wide neural processing unit 126 similar to the neural processing unit 126 of the second figure. However, if the instruction instructs the arithmetic logic unit to perform a multiply-accumulate operation and the neural processing unit 126 is in a narrow configuration, the operand selection logic 1898 provides an extended or expanded version of the narrow data text 209A to one of the wide multipliers 242A Input, and the extended version of the narrow weight text 203A is provided to another input; in addition, the operator selection logic 1898 provides the narrow data text 209B to one of the narrow multipliers 242B input, and the narrow weight text 203B to another One input. In order to achieve the operation of extending or expanding the narrow text as described above, if the narrow text has a sign, the operand selection logic 1898 will extend the narrow text; if the narrow text has no sign, the operand selection logic 1898 Will add the top zero bit to the narrow text.

In another example, if the neural processing unit 126 is in a wide configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation operation on a weight literal, the wide multiplier 242A will be skipped, and the operand selection logic 1898 will output 203A In series with 203B, a wide multiplexer 1896A is provided to provide a wide adder 244A. However, if the neural processing unit 126 is in a narrow configuration and the instruction instructs the arithmetic logic unit 204 to perform a weighted text accumulation operation, the wide multiplier 242A will be skipped, and the operand selection logic 1898 will output an extended version 203A is provided to a wide multiplexer 1896A to be provided to a wide adder 244A; in addition, The narrow multiplier 242B will be skipped, and the operand selection logic 1898 will provide the output 203B of the extended version to the narrow multiplexer 1896B for the narrow adder 244B.

In another example, if the neural processing unit 126 is in a wide configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation operation on a data literal, the wide multiplier 242A will be skipped, and the operand selection logic 1898 will output 209A In series with 209B, a wide multiplexer 1896A is provided to a wide adder 244A. However, if the neural processing unit 126 is in a narrow configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation operation on the data text, the wide multiplier 242A will be skipped, and the operand selection logic 1898 will output an extended version 209A is provided to the wide multiplexer 1896A to be provided to the wide adder 244A; in addition, the narrow multiplier 242B will be skipped, and the operand selection logic 1898 will provide the output of the extended version 209B to the narrow multiplexer 1896B to provide Narrow adder 244B. Accumulation of weights / data text is helpful for averaging. The averaging can be used as a pooling layer in some artificial neural network applications such as image processing.

In a preferred embodiment, the neural processing unit 126 further includes a second wide multiplexer (not shown) for skipping the wide adder 244A, so as to facilitate the configuration of a wide data / weight text in a wide configuration. Or one of the extended narrow data / weight text in the narrow configuration is loaded into the wide accumulator 202A, and a second narrow multiplexer (not shown) is used to skip the narrow adder 244B to facilitate the narrow configuration. The next narrow data / weight text is loaded into the narrow accumulator 202B. For a preferred embodiment, the arithmetic logic unit 204 further includes a wide / narrow comparator / multiplexer combination (not shown). The comparator / multiplexer combination receives the corresponding accumulator value 217A / 217B and the corresponding multiplexer 1896A / 1896B output, so that the accumulator value 217A / 217B and a data / weight text 209A / 209B / 203A / The maximum value is selected between 203B. The pooling layer of some artificial neural network applications uses this operation. This part will be described in more detail in the subsequent chapters, for example, corresponding to the figures 27 and 28. Instructions. In addition, the operand selection logic 1898 is used to provide an operand with a value of zero (for addition operations to add zeros or to clear the accumulator), and an operand for a value of one (for multiplications by multiplication by one).

The narrow start function unit 212B receives the output 217B of the narrow accumulator 202B and executes a start function on it to produce a narrow result 133B. The wide start function unit 212A receives the output 217A of the wide accumulator 202A and executes a start function on it to generate a Wide result 133A. When the neural processing unit 126 is in a narrow configuration, the wide start function unit 212A will understand the output 217A of the accumulator 202A according to this configuration and execute a start function on it to generate a narrow result, such as 8 bits. More detailed descriptions are provided in the 29th to 30th drawings.

As mentioned before, a single neural processing unit 126 can actually operate as two narrow neural processing units when in a narrow configuration, so for smaller text, compared to the wide configuration, it can roughly provide Up to twice the processing power. For example, suppose the neural network layer has 1024 neurons, and each neuron receives 1024 narrow inputs (and has narrow weighted text) from the previous layer, which results in one million connections. For the neural network unit 121 with 512 neural processing units 126, in a narrow configuration (equivalent to 1024 narrow neural processing Unit), although it is processing narrow text instead of wide text, it can process four times as many links (one million link pairs as 256K links), and it takes about half the time (Approximately 1026 time-frequency cycles versus 514 time-frequency cycles).

In an embodiment, the dynamically configured neural processing unit 126 of FIG. 18 includes a three-input multiplex register similar to the multiplex registers 208A and 208B to replace the registers 205A and 205B to form a rotator. Processing the weight character string received by the weight random access memory 124. This operation part is similar to the method described in the embodiment of FIG. 7 but applied to the dynamic configuration described in FIG. 18.

The nineteenth figure is a block diagram showing the embodiment according to the eighteenth figure, using the 2N multiplexing registers 208A / 208B of the N neural processing units 126 of the neural network unit 121 of the first figure. The data random access memory 122 of the first figure obtains a row of data characters 207 and performs the same operation as the same rotator. In the embodiment of the nineteenth figure, N is 512, and the neural network unit 121 has 1024 multiplexing registers 208A / 208B, labeled 0 to 511, corresponding to 512 neural processing units 126 and 1024 in practice, respectively. Narrow nerve processing unit. The two narrow neural processing units in the neural processing unit 126 are labeled A and B, respectively. In each multiplexing register 208, the corresponding narrow neural processing units are also labeled. Further, the multiplex register 208A of the neural processing unit 126 labeled as 0 is labeled as 0-A, and the multiplex register 208B of the neural processing unit 126 labeled as 0 is labeled as 0-B and labeled as The multiplex register 208A of the neural processing unit 126 of 1 is labeled 1-A, and the multiplex register 208B of the neural processing unit 126 of 1 is labeled 1-B, labeled as The multiplex register 208A of the neural processing unit 126 of 511 is labeled 511-A, and the multiplex register 208B of the neural processing unit 126 of 511 is labeled 511-B, and its value also corresponds to the subsequent The narrow nerve processing unit described in the twenty-first figure.

Each multiplexer register 208A receives its corresponding narrow data text 207A in one of the D rows of data random access memory 122, and each multiplexer register 208B stores data in the data random access memory One of the D columns of 122 receives its corresponding narrow data text 207B. That is, the multiplexer register 0-A receives the narrow data text 0 of 122 rows of the random access memory, and the multiplex register 0-B receives the narrow data text 1 of 122 rows of the random access memory. Multiplex register 1-A receives narrow data text 2 in 122 rows of random access memory, multiplex register 1-B receives narrow data text 3 in 122 rows of random access memory, and so on, Multiplex register 511-A receives narrow data characters 1022 of 122 rows of random access memory, and multiplex register 511-B receives narrow data characters 1023 of 122 rows of random access memory. In addition, multiplexer 1-A receives the output 209A of multiplexer 0-A as its input 211A, and multiplexer 1-B receives the output 209B of multiplexer 0-B as its input 211B, and so on, multiplexer register 511-A receives the output of multiplexer register 510-A 209A as its input 211A, multiplexer register 511-B receives the output of multiplexer register 510-B 209B as its input 211B, and multiplexer register 0-A receives the output of multiplexer register 511-A 209A as its input 211A, multiplexer register 0-B receives the multiplexer register 511-1B Output 209B is used as its input 211B. Each multiplexer register 208A / 208B receives control input 213 to control its selection data The words 207A / 207B are either input 211A / 211B after rotation or 1811A / 1811B after rotation. Finally, multiplexer 1-A receives the output 209B of multiplexer 0-B as its input 1811A, and multiplexer 1-B receives the output 209A of multiplexer 1-A as its input 1811B, and so on, multiplexer register 511-A receives the output of multiplexer register 510-B 209B as its input 1811A, multiplexer register 511-B receives the output of multiplexer register 511-A 209A is used as its input 1811B, and multiplex register 0-A receives the output of multiplex register 511-B 209B is used as its input 1811A, multiplex register 0-B receives the multiplex register 0-A Output 209A as its input 1811B. Each multiplexer register 208A / 208B will receive control input 213 to control its selection of data text 207A / 207B or input 211A / 211B after rotation or input 1811A / 1811B after rotation. In an operation mode, in the first time-frequency period, the control input 213 controls each multiplexer register 208A / 208B to select the data text 207A / 207B and store it in the register for subsequent supply to the arithmetic logic unit 204; For subsequent time-frequency cycles (such as the aforementioned M-1 time-frequency cycle), the control input 213 will control each multiplexer register 208A / 208B after selecting rotation and input 1811A / 1811B to store in the register for subsequent supply to the arithmetic logic unit 204, this part will be explained in more detail in subsequent chapters.

The twentieth chart is a table showing a program stored in the program memory 129 of the neural network unit 121 of the first chart and executed by the neural network unit 121. The neural processing unit 126 is shown in the embodiment of the figure. The example program in Figure 20 is similar to the program in Figure 4. The differences are explained below. The initial neural processing unit located at address 0 is designated by the instruction The processing unit 126 will enter a narrow configuration. In addition, as shown in the figure, the multiply-accumulate rotation instruction at address 2 specifies a count value of 1023 and requires 1023 time-frequency cycles. This is because the example in Figure 20 assumes that there are actually 1024 narrow (e.g. 8-bit) neurons (i.e., neural processing units) in a layer, and each narrow neuron has 1024 neurons from the previous layer Yuan's link input, so there are 1024K links in total. Each neuron receives an 8-bit data value from each link input and multiplies this 8-bit data value by an appropriate 8-bit weight value.

The twenty-first diagram is a timing chart showing a neural network unit 121 executing the program of the twentieth diagram. The neural network unit 121 has the neural processing unit 126 shown in the eighteenth diagram executed in a narrow configuration. The timing chart of the twenty-first figure is similar to the timing chart of the fifth figure. The differences are explained below.

In the timing diagram of the twenty-first figure, these neural processing units 126 will be in a narrow configuration, because the initialization neural processing unit instruction at address 0 initializes them to a narrow configuration. Therefore, these 512 neural processing units 126 actually work like 1024 narrow neural processing units (or neurons). These 1024 narrow neural processing units are connected with neural processing units 0-A and neural processing units in the column. 0-B (two narrow neural processing units of the neural processing unit 126 labeled 0), neural processing unit 1-A and neural processing unit 1-B (two narrow neural processing units of the neural processing unit 126 labeled 1) ), And so on until the neural processing unit 511-A and the neural processing unit 511-B (two narrow neural processing units of the neural processing unit 126 labeled 511) are specified. In order to simplify the description, only the operations of the narrow neural processing units 0-A, 0-B, and 511-B are shown in the figure. Because the count value specified by the multiply accumulate rotation instruction at address 2 is 1023, which requires 1023 time-frequency cycles to operate, the number of columns in the timing chart of Figure 21 includes up to 1026 time-frequency cycles.

At time-frequency period 0, each of the 1024 neural processing units will execute the initialization instruction of the fourth figure, that is, the operation of assigning a zero value to the accumulator 202 shown in the fifth figure.

At time-frequency period 1, each of the 1024 narrow neural processing units executes the multiply-accumulate instruction at address 1 in the twentieth figure. As shown in the figure, the narrow neural processing unit 0-A adds the value of accumulator 202A (ie, zero) to the column 17 of the data random access unit 122 and the narrow text 0 and the weight 0 of the random access unit 124 to the narrow text 0. Product: Narrow nerve processing unit 0-B multiplies the value of accumulator 202B (ie, zero) plus data random access unit 122 column 17 narrow text 1 and weight random access unit 124 column 0 narrow text 1; By analogy, the narrow neural processing unit 511-B multiplies the value of accumulator 202B (ie, zero) by the product of column 17 narrow text 1023 of data random access unit 122 and column 0 narrow text 1023 of weight random access unit 124.

In time-frequency cycle 2, each of the 1024 narrow neural processing units executes the first iteration of the multiply-accumulate rotation instruction at address 2 in the twentieth figure. As shown in the figure, the narrow neural processing unit 0-A adds the value 217A of the accumulator 202A plus the rotated narrow data text 1811A received by the multiplexer register 208B output 209B of the narrow neural processing unit 511-B (that is, The product of the narrow data text 1023 received by the data random access memory 122 and the narrow text 0 of column 1 of the weight random access unit 124; the narrow neural processing unit 0-B adds the value 217B of the accumulator 202B by The multiplexing register 208A of the narrow neural processing unit 0-A outputs the rotated narrow data text 1811B (that is, the narrow data text 0 received by the data random access memory 122) received by 209A and the weight random access unit The product of column 1 of the narrow text 1 of 124; and so on, until the narrow neural processing unit 511-B adds the value of the accumulator 202B to 217B plus the value received by the multiplexer register 208A of the narrow neural processing unit 511-A and outputs 209A. The product of the rotated narrow data text 1811B (that is, the narrow data text 1022 received by the data random access memory 122) and the narrow text 1023 of column 1 of the weight random access unit 124.

At time-frequency period 3, each of the 1024 narrow neural processing units will execute the second iteration of the multiply-accumulate rotation instruction at address 2 in the twentieth figure. As shown in the figure, the narrow neural processing unit 0-A adds the value 217A of the accumulator 202A plus the rotated narrow data text 1811A received by the multiplexer register 208B output 209B of the narrow neural processing unit 511-B (ie, The product of the narrow data text 1022) received by the data random access memory 122 and the narrow text 0 of column 2 of the weight random access unit 124; the narrow neural processing unit 0-B adds the value 217B of the accumulator 202B by the narrow nerve The multiplexing register 208A of the processing unit 0-A outputs the narrow data text 1811B (that is, the narrow data text 1023 received by the data random access memory 122) received by the 209A and the weight random access unit 124. Column 2 is the product of the narrow text 1; and so on, until the narrow neural processing unit 511-B adds the value of the accumulator 202B 217B to the rotation received by the multiplexer register 208A of the narrow neural processing unit 511-A and outputs 209A Narrow data text 1811B (i.e., narrow data text 1021 received by data random access memory 122) and weighted random access unit 124 narrow text Product of words 1023. As shown in the twenty-first figure, this operation will continue in the subsequent 1021 time-frequency cycles until the time-frequency cycle 1024 described below.

In a time-frequency cycle of 1024, each of the 1024 narrow neural processing units will execute the 1023th iteration of the multiply-accumulate rotation instruction at address 2 in the twentieth figure. As shown in the figure, the narrow neural processing unit 0-A adds the value 217A of the accumulator 202A plus the rotated narrow data text 1811A received by the multiplexer register 208B of the narrow neural processing unit 511-B to output 209B (that is, Product of narrow data text received by data random access memory 122 1) and weighted random access unit 124 1023 narrow text 0; product of narrow neural processing unit 0-B accumulator 202B value 217B plus narrow nerve The multiplexing register 208A of the processing unit 0-A outputs the narrow data text 1811B (that is, the narrow data text 2 received by the data random access memory 122) received by the 209A and the weight random access unit 124. Column 1023 the product of the narrow text 1; and so on, until the narrow neural processing unit 511-B adds the accumulator 202B value 217B plus the rotation received by the multiplexer register 208A of the narrow neural processing unit 511-A to output 209A The product of the narrow data text 1811B (that is, the narrow data text 0 received by the data random access memory 122) and the narrow text 1023 of the row 1023 of the weight random access unit 124.

In the time-frequency cycle 1025, the activation function unit 212A / 212B of each of the 1024 narrow neural processing units executes the activation function instruction at address 3 in the twentieth figure. Finally, in the time-frequency cycle 1026, each of the 1024 narrow neural processing units will write its narrow result 133A / 133B back to the corresponding narrow text in column 16 of the data random access memory 122 to execute the second Write enable at address 4 in the ten figure Function unit instructions. That is, the narrow result 133A of the neural processing unit 0-A will be written into the narrow text 0 of the data random access memory 122, and the narrow result 133B of the neural processing unit 0-B will be written into the data random access memory 122 Narrow text 1, and so on, until the narrow result 133B of the neural processing unit 511-B is written into the narrow text 1023 of the data random access memory 122. The twenty-second figure is a block diagram showing the aforementioned operations corresponding to the twenty-first figure.

The twenty-second figure is a block diagram showing the neural network unit 121 of the first figure. The neural network unit 121 has a neural processing unit 126 as shown in the eighteenth figure to execute the program of the twentieth figure. The neural network unit 121 includes 512 neural processing units 126, that is, 1024 narrow neural processing units, a data random access memory 122, and a weight random access memory 124. The data random access memory 122 receives its bits. The address input 123 and the weight random access memory 124 receive its address input 125. Although it is not shown in the figure, at a time-frequency period of 0, the 1024 narrow neural processing units will execute the initialization instructions of the twentieth figure. As shown in the figure, in time-frequency period 1, 1024 8-bit data characters of row 17 are read from the data random access memory 122 and provided to the 1024 narrow neural processing units. In the time-frequency period of 1 to 1024, 1024 8-bit weight characters of columns 0 to 1023 are read from the weight random access memory 124 and provided to the 1024 narrow neural processing units. Although it is not shown in the figure, at time-frequency period 1, the 1024 narrow neural processing units perform corresponding multiply-accumulate operations on the loaded data text and weight text. In the time-frequency period of 2 to 1024, the multiplex register 208A / 208B of the 1024 narrow neural processing units operates like a 1024 8-bit text. The word rotator will rotate the data text previously loaded into the data random access memory 122 row 17 to the adjacent narrow neural processing unit, and these narrow neural processing units will correspondingly rotate the data text and the weight by The corresponding narrow weight text loaded in the random access memory 124 performs a multiply-accumulate operation. Although not shown in the figure, the 1024 narrow start function units 212A / 212B will execute the start instruction in the time-frequency cycle 1025. In the time-frequency cycle 1026, the 1024 narrow neural processing units write their corresponding 1024 8-bit results 133A / 133B back to the 16th row of the data random access memory 122.

It can be found that, compared to the embodiment of the second figure, the embodiment of the eighteenth figure gives the programmer flexibility to use wide data and weighted text (such as 16 bits) and narrow data and weighted text (such as 8-bit) to perform calculations in response to the need for accuracy in a particular application. From an aspect, for the application of narrow data, the embodiment of Fig. 18 can provide twice the performance compared to the embodiment of Fig. 2. However, additional narrow components (such as multiplexed temporary storage) must be added. 208B, register 205B, narrow arithmetic logic unit 204B, narrow accumulator 202B, and narrow start function unit 212B). At the cost of these additional narrow elements, the area of the neural processing unit 126 is increased by about 50%.

Trimodal nerve processing unit

The twenty-third figure is a schematic block diagram showing another embodiment of the dynamically configurable neural processing unit 126, one of the first figures. The neural processing unit 126 of the twenty-third figure can be used not only in a wide configuration and a narrow configuration, but also in a third configuration, which is referred to herein as a "funnel" Configuration. The neural processing unit 126 of the twenty-third figure is similar to the neural processing unit 126 of the eighteenth figure. However, the wide adder 244A in the eighteenth figure is replaced by a three-input wide adder 2344A in the neural processing unit 126 in the twenty-third figure. The three-input wide adder 2344A receives a third adder 2399. It is an extended version of the output of the narrow multiplexer 1896B. The program executed by the neural network unit having the neural processing unit of the twenty-third diagram is similar to that of the twentieth diagram. However, the initialization neural processing unit instruction at address 0 initializes these neural processing units 126 to a funnel configuration instead of a narrow configuration. In addition, the count value of the multiply accumulate rotation instruction at address 2 is 511 instead of 1023.

In the funnel configuration, the operation of the neural processing unit 126 is similar to that in the narrow configuration. When the multiply-accumulate instruction at address 1 in the twentieth figure is executed, the neural processing unit 126 will receive two narrow data characters 207A / 207B and Two narrow weight texts 206A / 206B; wide multiplier 242A will multiply data text 209A and weight text 203A to produce a product 246A selected by wide multiplexer 1896A; narrow multiplier 242B will phase data text 209B and weight text 203B Multiply by 246B to produce the selection of the narrow multiplexer 1896B. However, the wide adder 2344A adds the product 246A (selected by the wide multiplexer 1896A) and the product 246B / 2399 (selected by the wide multiplexer 1896B) to the wide accumulator 202A output 217A, and the narrow adder 244B and The narrow accumulator 202B is not activated. In addition, when in the funnel configuration and performing the multiply-accumulate rotation instruction as shown in Figure 20, the control input 213 causes the multiplexer register 208A / 208B to rotate two narrow characters (such as 16 bits), that is, Say, the multiplexer register 208A / 208B will choose its corresponding input 211A / 211B, just like in wide configuration. kind. However, the wide multiplier 242A will multiply the data text 209A and the weight text 203A to produce the product 246A selected by the wide multiplexer 1896A; the narrow multiplier 242B will multiply the data text 209B and the weight text 203B to produce a narrow multiplexer. The product 246B selected by 1896B; and the wide adder 2344A adds the product 246A (selected by the wide multiplexer 1896A) and the product 246B / 2399 (selected by the wide multiplexer 1896B) to the wide accumulator 202A output 217A. The narrow adder 244B and the narrow accumulator 202B are not activated as described above. Finally, when the start function instruction at address 3 is executed in the funnel configuration, the wide start function unit 212A executes the start function on the total number of results 215A to produce a narrow result 133A, and the narrow start function unit 212B is Does not start. In this way, only the narrow neural processing unit labeled A will produce a narrow result 133A, and the narrow neural processing unit labeled B will produce a narrow result 133A. Therefore, the column of the write-back result (such as column 16 indicated by the instruction of address 4 in the twentieth figure) will contain holes, because only the narrow result 133A is valid, and the narrow result 133B is invalid. Therefore, conceptually, in each time-frequency period, each neuron (the neural processing unit of the twenty-third figure) will perform two input of connected data, that is, multiply two narrow data characters by their corresponding weights and By adding these two products, in comparison, the embodiments of the second graph and the eighteenth graph perform only one connection data input in each time-frequency period.

In the embodiment of the twenty-third figure, it can be found that the number of result text (neuron output) generated and written back to the data random access memory 122 or the weight random access memory 124 is the received data input (link) The square root of the number is half, and the result is written back to the column with holes, that is, every other narrow text result is invalid, more precisely, the standard The results of the narrow neural processing unit shown as B are not meaningful. Therefore, the embodiment of the twenty-third figure is particularly efficient for a neural network with two consecutive layers. For example, the number of neurons in the first layer is twice that of the second layer (for example, the first layer has 1024 The neurons are fully connected to the 512 neurons in the second layer). In addition, other execution units 122 (for example, media units, such as x86 advanced vector expansion unit) may perform a pack operation on a scattered result column (that is, have a hole) to make it compact (that is, have no hole) when necessary. . Subsequently, when the neural processing unit 121 performs other calculations related to the other rows of the data random access memory 122 and / or the weight random access memory 124, the processed data row may be used for calculation.

Hybrid Neural Network Unit Operations: Convolution and Common Source Computing Capabilities

The advantage of the neural network unit 121 according to the embodiment of the present invention is that the neural network unit 121 can operate in a manner similar to that of a coprocessor executing its own internal program at the same time and executes all operations in a processing unit similar to a processor. Issued architectural instructions (or micro-instructions translated from architectural instructions). The architectural instructions are contained in an architectural program executed by a processor having a neural network unit 121. In this way, the neural network unit 121 can operate in a mixed manner, and the high utilization rate of the neural processing unit 121 can be maintained. For example, the twenty-fourth to twenty-sixth graphs show the operation of the neural network unit 121 to perform the convolution operation. Among them, the neural network unit is fully utilized, and the twenty-seventh to twenty-eighth graphs show the nerves. The network unit 121 performs a common source operation. Convolutional layers, common source layers, and other digital data calculation applications, such as image processing (such as edge detection, Sharpening, blurring, identification / classification) require these operations. However, the hybrid operation of the neural processing unit 121 is not limited to performing convolution or common source operations. This hybrid feature can also be used to perform other operations, such as the traditional neural network multiply-accumulate operations and activation functions described in Figures 4-13. Operation. That is, the processor 100 (more precisely, the reserved station 108) will issue MTNN instructions 1400 and MFNN instructions 1500 to the neural network unit 121. In response to the issued instructions, the neural network unit 121 will write data into the memory The body 122/124/129 reads the result from the memory 122/124 written by the neural network unit 121. At the same time, in order to execute the program written by the processor 100 (via the MTNN1400 instruction) into the program memory 129 The neural network unit 121 will read and write the memory 122/124/129.

The twenty-fourth figure is a block diagram showing an example of a data structure used by the neural network unit 121 of the first figure to perform a convolution operation. The block diagram includes a convolution kernel 2402, a data array 2404, and the data random access memory 122 and the weight random access memory 124 of the first figure. In a preferred embodiment, the data array 2404 (for example, corresponding to an image pixel) is loaded in a system memory (not shown) connected to the processor 100 and is loaded into the nerve by the processor 100 by executing the MTNN instruction 1400 The weight of the network unit 121 is a random access memory 124. The convolution operation is to convolve a first array and a second array, and this second array is the convolution kernel described herein. As described herein, a convolution kernel is a matrix of coefficients. These coefficients can also be called weights, parameters, elements, or values. According to a preferred embodiment, the convolution kernel 2042 is static data of a framework program executed by the processor 100.

This data array 2404 is a two-dimensional array of data values Rows, and the size of each data value (such as an image pixel value) is the size of the text in the data random access memory 122 or the weight random access memory 124 (such as 16 bits or 8 bits). In this example, the data value is a 16-bit text, and the neural network unit 121 is configured with 512 wide-configured neural processing units 126. In addition, in this embodiment, the neural processing unit 126 includes a multiplexing register to receive the weight text 206 from the weight random access memory 124, such as the multiplexing register 705 of the seventh figure, so as to randomize the weights. The access memory 124 receives a row of data values and performs a collective rotator operation, which will be described in more detail in subsequent sections. In this example, the data array 2404 is a pixel array of 2560 rows x 1600 columns. As shown in the figure, when the architecture program performs a convolution calculation on the data array 2404 and the convolution kernel 2402, the data array 2402 is divided into 20 data blocks, and each data block is a 512x400 data array 2406.

In this example, the convolution kernel 2402 is a 3x3 array of coefficients, weights, parameters, or elements. The first column of these coefficients is labeled C0,0; C0,1; and C0,2; the second column of these coefficients is labeled C1,0; C1,1; and C1,2; the third column of these coefficients is labeled C2,0; C2,1; and C2,2. For example, a convolution kernel with the following coefficients can be used to perform edge detection: 0,1,0,1, -4,1,0,1,0. In another embodiment, a convolution kernel with the following coefficients can be used to perform a Gaussian fuzzy operation: 1,2,1,2,4,2,1,2,1. In this example, a division is usually performed on the value after the final accumulation. The divisor is the sum of the absolute values of the elements of the convolution kernel 2042, which is 16 in this example. In another example, the divisor may be the number of elements of the convolution kernel 2042. In yet another example, the divisor can be a compression of a convolution operation to a target range of values. The value used, this divisor is determined by the element value of the convolution kernel 2042, the target range, and the range of the input value array to perform the convolution operation.

Please refer to the twenty-fourth figure and the twenty-fifth figure in which the details are detailed. The architecture program writes the coefficients of the convolution kernel 2042 into the data random access memory 122. In a preferred embodiment, all text on each row of the nine consecutive rows of data random access memory 122 (the number of elements in the convolution kernel 2402) will use different elements of the convolution kernel 2402 to The columns are written in their main order. That is, as shown in the figure, each character in the same column is written with a first coefficient C0,0; the next column is written with a second coefficient C0,1; the next column is written with a third coefficient C0,2 is written; the next column is written with a fourth coefficient C1,0; and so on until each character in the ninth column is written with a ninth coefficient C2,2. In order to perform a convolution operation on the data matrix 2406 of the data block segmented by the data array 2404, the neural processing unit 126 repeatedly reads the nine rows of the coefficients of the convolution kernel 2042 loaded in the random access memory 122 according to the sequence. This part In the subsequent chapters, especially the part corresponding to Figure 26A, there will be more detailed explanation.

Please refer to the twenty-fourth figure and the twenty-fifth figure in which the details are detailed. The architecture program writes the values of the data matrix 2406 into the weight random access memory 124. When the neural network unit program performs a convolution operation, it writes the resulting array back to the weight random access memory 124. In a preferred embodiment, the architecture program writes a first data matrix 2406 into the weight random access memory 124 and starts the neural network unit 121 to operate. When the neural network unit 121 is When the 2406 and the convolution kernel 2402 perform a convolution operation, the framework program will convert a second data matrix 2406 is written into the weight random access memory 124. In this way, after the neural network unit 121 completes the convolution operation of the first data matrix 2406, it can start performing the convolution operation of the second data matrix 2406. This part corresponds to the following. A more detailed explanation is provided at the twenty-fifth figure. In this way, the framework program will go back and forth between the two regions of the weight random access memory 124 to ensure that the neural network unit 121 is fully used. Therefore, the example in the twenty-fourth figure shows a first data matrix 2406A and a second data matrix 2406B. The first data matrix 2406A corresponds to the first data block in the row 0 to 399 in the occupied random access memory 124. The second data matrix 2406B corresponds to the second data blocks of columns 500 to 899 in the occupied random access memory 124. In addition, as shown in the figure, the neural network unit 121 writes the result of the convolution operation back to columns 900-1299 and 1300-1699 of the weight random access memory 124, and then the architecture program will randomly access the memory from the weights. The body 124 reads these results. The data values of the data matrix 2406 loaded in the weight random access memory 124 are marked as "Dx, y", where "x" is the number of rows of the weight random access memory 124, and "y" is the weight random access memory Text, or number of lines. For example, the data text 511 in column 399 is labeled D399,511 in the twenty-fourth figure, and this data text is received by the multiplexer register 705 of the neural processing unit 511.

The twenty-fifth figure is a flowchart showing that the processor 100 of the first figure executes a framework program to perform the convolution operation of the convolution kernel 2042 on the data array 2404 of the twenty-fourth figure by using the neural network unit 121. This process starts at step 2502.

In step 2502, the processor 100, that is, the processor 100 executing a framework program, converts the convolution kernel 2402 of the twenty-fourth figure to The data random access memory 122 is written in the manner shown and described in the twenty-fourth figure. In addition, the framework program initializes a variable N to a value of 1. The variable N indicates a data block being processed by the neural network unit 121 in the data array 2404. In addition, the framework program initializes a variable NUM_CHUNKS to the value 20. The flow then proceeds to step 2504.

In step 2504, as shown in FIG. 24, the processor 100 writes the data matrix 2406 of the data block 1 into the weight random access memory 124 (such as the data matrix 2406A of the data block 1). The flow then proceeds to step 2506.

In step 2506, the processor 100 uses a MTNN instruction 1400 that specifies a function 1432 to write the program memory 129, and writes a convolution program to the program memory 129 of the neural network unit 121. The processor 100 then uses a MTNN instruction 1400 that specifies a function 1432 to start executing the program to start the neural network unit convolution program. An example of a neural network unit convolution program is explained in more detail at the position corresponding to the twenty-sixth A figure. The flow then proceeds to step 2508.

In decision step 2508, the architecture program determines whether the value of the variable N is less than NUM_CHUNKS. If yes, the flow advances to step 2512; otherwise, it advances to step 2514.

In step 2512, as shown in FIG. 24, the processor 100 writes the data matrix 2406 of the data block N + 1 into the weight random access memory 124 (such as the data matrix 2406B of the data block 2). Therefore, when the neural network unit 121 is performing a convolution operation on the current data block, the architecture program can write the data matrix 2406 of the next data block into the random weights. The memory 124 is accessed. In this way, after the convolution operation of the current data block is completed, that is, after the weighted random access memory 124 is written, the neural network unit 121 can immediately start performing a convolution operation on the next data block.

At step 2514, the processor 100 confirms whether the executing neural network unit program (for data block 1 but from step 2506 and for data block 2-20 from step 2518) has completed execution. In a preferred embodiment, the processor 100 reads the state register 127 of the neural network unit 121 by executing an MFNN instruction 1500 to confirm whether the execution has been completed. In another embodiment, the neural network unit 121 generates an interrupt, indicating that the convolution routine has been completed. The flow then proceeds to decision step 2516.

In decision step 2516, the architecture program determines whether the value of the variable N is less than NUM_CHUNKS. If yes, the flow proceeds to step 2518; otherwise, it proceeds to step 2522.

In step 2518, the processor 100 updates the convolution program to execute on the data block N + 1. More precisely, the processor 100 updates the column value of the initialization neural processing unit instruction corresponding to the address 0 in the weight random access memory 124 to the first column of the data matrix 2406 (for example, to the data matrix 2406A). Column 0 or column 500 of data matrix 2406B), and the output column is updated (for example, to column 900 or 1300). The processor 100 will then execute the updated neural network unit convolution program. The flow then proceeds to step 2522.

In step 2522, the processor 100 reads the execution result of the neural network unit convolution program of the data block N from the weighted random access memory 124. The flow then proceeds to decision step 2524.

In decision step 2524, the architecture program determines whether the value of the variable N is less than NUM_CHUNKS. If yes, the flow advances to step 2526; otherwise, it ends.

In step 2526, the framework program increases the value of N by one. The flow then returns to decision step 2508.

Figure 26A is a program list of a neural network unit program. This neural network unit program uses the convolution kernel 2402 of figure 24 to perform a convolution operation of a data matrix 2406 and writes it back. Weight random access memory 124. This program loops a certain number of instruction loops consisting of the instructions at addresses 1 to 9. The initialization neural processing unit instruction at address 0 specifies the number of times each neural processing unit 126 executes this instruction. The example in Figure 26A has a loop count value of 400, corresponding to the twenty-fourth. The number of columns in the data matrix 2406 of the figure, and the loop instruction (located at address 10) located at the loop terminal will decrement the current loop count value. If the result is non-zero, it will return to the command loop. Top (ie the instruction to return to address 1). The instruction to initialize the neural processing unit also clears the accumulator 202 to zero. For a preferred embodiment, the loop instruction at address 10 also clears the accumulator 202 to zero. In addition, the multiply-accumulate instruction at address 1 can also clear the accumulator 202 to zero.

For each execution of the instruction loop in the program, the 512 neural processing units 126 will simultaneously perform 512 3x3 convolution kernels and 512 corresponding 3x3 sub-matrix convolution operations of the data matrix 2406. The convolution operation is the sum of the nine products calculated from the elements of the convolution kernel 2042 and the corresponding elements in the corresponding sub-matrix. Figure in the twenty-sixth In the embodiment, the origin (central element) of each of the 512 corresponding 3x3 sub-matrixes is the data text Dx + 1, y + 1 in the twenty-fourth figure, where y (row number) is the neural processing Unit 126 is numbered, and x (row number) is the row number read by the current weight random access memory 124 by the multiply-accumulate instruction at address 1 in the program of Figure 26A The instruction for initializing the neural processing unit at address 0 is also initialized and will be incremented when the multiply-accumulate instruction at addresses 3 and 5 is executed, and it will also be updated by the decrement instruction at address 9.) Thus, in each loop of the program, the 512 neural processing units 126 calculate 512 convolution operations and write the results of the 512 convolution operations back to the instruction line of the weight random access memory 124. In this article, edge handling is omitted to simplify the description, but it should be noted that using the collective rotation features of these neural processing units 126 will cause the data matrix 2406 (for the image processor, the data matrix of the image). Wrapping occurs in two rows of the multi-line data from the vertical edge on one side to the other vertical edge (for example, from the left edge to the right edge and vice versa). The instruction loop will now be described.

Address 1 is a multiply accumulate instruction. This instruction will specify row 0 of data random access memory 122 and secretly use the current weight of random access memory 124. This row is preferably loaded in sequencer 128 (and It is initialized to zero by the instruction at address 0 to perform the operation of the first instruction loop pass). That is, the instruction at address 1 will cause each neural processing unit 126 to read its corresponding text from column 0 of the data random storage memory 122 and read its corresponding from the current weight random access memory 124 column. Text, and perform a multiply-accumulate operation on the two texts. Such as So, for example, the neural processing unit 5 multiplies C0,0 and Dx, 5 (where "x" is the current weight random access memory 124 columns), adds the result to the accumulator 202 value 217, and adds the total Write back to accumulator 202.

Address 2 is a multiply accumulate instruction. This instruction specifies that the row of data random access memory 122 is incremented (ie, increased to 1), and then the row is read from the address of the data random access memory 122 after incrementing. This instruction also specifies that the value in the multiplex register 705 of each neural processing unit 126 is rotated to the neighboring neural processing unit 126. In this example, it is the random access memory from the weight corresponding to the instruction at address 1. The row of data matrix 2406 values read by volume 124. In the embodiments of the twenty-fourth to twenty-sixth figures, these neural processing units 126 are used to rotate the values of the multiplex register 705 to the left, that is, from the neural processing unit J to the neural processing unit J- 1, instead of rotating from the neural processing unit J to the neural processing unit J + 1 as described in the third, seventh, and nineteenth figures. It is worth noting that in the embodiment where the neural processing unit 126 rotates to the right, the architecture program will write the convolution kernel 2042 coefficient values into the data random access memory 122 (for example, rotate around its center row) in different orders to achieve The purpose of similar convolution results. In addition, the architecture program can perform additional pre-processing of the convolution kernel (such as transposition) when needed. In addition, the instruction specifies a count value of two. Therefore, the instruction at address 2 will cause each neural processing unit 126 to read its corresponding text from row 1 of the data random access memory 122, receive the rotated text to the multiplexer register 705, and Each literal performs a multiply-accumulate operation. Because the count value is 2, this instruction also causes each neural processing unit 126 to repeat the aforementioned operation. That is, the sequencer 128 increments the data random access memory 122 row address 123 (ie, increases to 2), and Each neural processing unit 126 reads its corresponding text from the column 2 of the data random access memory 122 and receives the rotated text to the multiplexing register 705, and performs a multiply-accumulate operation on the two texts. So, for example, assuming that the current weight random access memory 124 is column 27, after executing the instruction of address 2, the neural processing unit 5 will multiply the product of C0,1 and D27,6 by C0,2 and D27, The multiplication of 7 is added to its accumulator 202. In this way, after completing the instructions of address 1 and address 2, the product of C0,0 and D27,5, the product of C0,1 and D27,6 and C0,2 and D27,7 will be added to accumulator 202, add All other accumulated values from the previously passed instruction loop.

The operations performed by the instructions at addresses 3 and 4 are similar to the instructions at addresses 1 and 2, using the effect of increasing the index of the 124 rows of random access memory. These instructions will Operations are performed, and these instructions perform operations on the next three rows of data random access memory 122, that is, rows 3 to 5. That is, taking the neural processing unit 5 as an example, after completing the instructions of addresses 1 to 4, the product of C0,0 and D27,5, the product of C0,1 and D27,6, and the product of C0,2 and D27,7 The product, the product of C1,0 and D28,5, the product of C1,1 and D28,6, and the product of C1,2 and D28,7 are accumulated to accumulator 202, and all other loops from the previously passed instruction loop are added. Cumulative value.

The instructions at addresses 5 and 6 perform operations similar to the instructions at addresses 3 and 4. These instructions will load the next row of random access memory 124 and the next three rows of data random access memory 122. That is, columns 6 to 8 are calculated. That is, taking the neural processing unit 5 as an example, after completing the instructions of addresses 1 to 6, the product of C0,0 and D27,5, the product of C0,1 and D27,6, and the product of C0,2 and D27,7 Product, multiplication of C1,0 and D28,5 Product of C1,1 and D28,6, product of C1,2 and D28,7, product of C2,0 and D29,5, product of C2,1 and D29,6, and product of C2,2 and D29,7 It is accumulated to the accumulator 202, adding all other accumulated values from the previously passed instruction loop. That is, after completing the instructions at addresses 1 to 6, it is assumed that at the beginning of the instruction loop, the weight random access memory 124 is listed as 27. Taking the neural processing unit 5 as an example, the convolution kernel 2042 will be used for the following 3x3 The submatrix performs the convolution operation: D27,5 D27,6 D27,7 D28,5 D28,6 D28,7 D29,5 D29,6 D29,7 In general, after completing the instructions of addresses 1 to 6, these 512 Each of the neural processing units 126 has used the convolution kernel 2042 to perform convolution operations on the following 3x3 submatrices: Dr, n Dr, n + 1 Dr, n + 2 Dr + 1, n Dr + 1, n + 1 Dr + 1 , n + 2 Dr + 2, n Dr + 2, n + 1 Dr + 2, n + 2 where r is the value of the column address of the random access memory 124 at the beginning of the instruction loop, and n is the neural processing Number of unit 126.

The instruction at address 7 passes the value 217 of the accumulator 202 through the activation function unit 121. This transfer function will pass a text whose size (in bits) is equivalent to the text read by the data random access memory 122 and the weight random access memory 124 (16 bits in this example) . In a preferred embodiment, the user can specify an output format, such as how many bits in the output bits are fractional bits. This section will be described in more detail in subsequent sections. In addition, this means You can specify a division start function instead of a pass-through start function. This division start function divides the accumulator 202 value 217 by a divisor, as described in this article corresponding to the 29th A and 30th figures, for example Use one of the "dividers" 3014/3016 in Figure 30. For example, for a convolution kernel 2042 with coefficients, such as the aforementioned Gaussian fuzzy kernel with a coefficient of one-sixteenth, the instruction at address 7 will specify a division start function (for example, divide by 16), and Unspecified a transfer function. In addition, the structure program can perform the operation of dividing the convolution kernel 2042 coefficient by 16 before writing the convolution kernel coefficient into the data random access memory 122, and adjust the decimal point of the convolution kernel 2042 value accordingly. For example, use the data decimal point 2922 of the 29th figure as shown below.

The instruction at address 8 will write the output of the activation function unit 212 into the row specified by the current value of the output register in the weight random access memory 124. This current value is initialized by the instruction at address 0, and the value is incremented by the increment indicator within the instruction every time a loop is passed.

As shown in the examples of the twenty-fourth to twenty-sixth figures with a 3x3 convolution kernel 2402, the neural processing unit 126 reads the weight random access memory 124 to read the data matrix 2406 approximately every three time-frequency cycles. One column, and the convolution kernel result matrix is written into the weight random access memory 124 approximately every twelve time-frequency periods. In addition, it is assumed that in one embodiment, there is a write and read buffer as one of the buffers 1704 in FIG. The memory 124 is accessed for reading and writing, and the buffer 1704 randomly accesses the weights approximately every sixteen time-frequency cycles The volume performs a read and write operation to read the data matrix and write the convolution kernel result matrix respectively. Therefore, about half of the bandwidth of the weighted random access memory 124 is consumed by the convolution kernel operation performed by the neural network unit 121 in a hybrid manner. This example contains a 3x3 convolution kernel 2042, but the invention is not limited to this. Convolution kernels of other sizes, such as 2x2, 4x4, 5x5, 6x6, 7x7, 8x8, etc., can also be applied to different neural networks. Unit program. In the case of using a larger convolution kernel, because the rotated version of the multiply accumulate instruction (such as the instructions at address 2, 4, and 6 of Figure 26A, larger convolution kernels will need to use these instructions). With a larger count value, the proportion of time for the neural processing unit 126 to read the weighted random access memory 124 will decrease, so the bandwidth usage ratio of the weighted random access memory 124 will also decrease.

In addition, the structural program can make the neural network unit program overwrite the rows that are no longer needed in the input data matrix 2406 instead of writing the convolution operation results back to different rows of the weight random access memory 124 (such as row 900 -1299 and 1300-1699). For example, for a 3x3 convolution kernel, the architecture program can write the data matrix 2406 into the weighted random access memory 124 rows 2-401 instead of the rows 0-399, and the neural processing unit program The result of the convolution operation is written from column 0 of the weight random access memory 124, and the number of columns is incremented after each instruction loop is passed. In this way, the neural network unit program will only overwrite those that are no longer needed. For example, after the first pass through the instruction loop (or more precisely, after the instruction at address 1 is executed, its load weight random access memory 124 row 0), the data of row 0 can be overwritten Write, however, the data of columns 1-3 need to be left for the second pass after the instruction returns The operation of the circle cannot be overwritten. Similarly, after the second pass through the instruction loop, the data in column 1 can be overwritten, but the data in columns 2-4 need to be left for the third pass through the instruction loop. Operations cannot be overridden; and so on. In this embodiment, the height (for example, 800 rows) of each data matrix 2406 (data block) can be increased, so fewer data blocks can be used.

In addition, the structure program can make the neural network unit program write the results of the convolution operation back to the data random access memory 122 rows above the convolution kernel 2402 (for example, above the row 8), instead of writing the convolution operation results back. The weight random access memory 124, when the neural network unit 121 writes the result, the framework program can read the result from the data random access memory 122 (for example, using the data random access memory 122 in the twenty-sixth figure). Address 2606 was recently written). This configuration is suitable for the embodiment with a port weight random access memory 124 and a dual port data random access memory.

According to the calculations of the neural network unit 121 in the embodiments of the twenty-fourth to twenty-sixth figures A, it can be found that each execution of the program of the twenty-sixth figure A requires about 5000 time-frequency cycles. Thus, the second The convolution operation of the entire 2560x1600 data array 2404 in the fourteenth figure requires about 100,000 time-frequency cycles, which is significantly less than the number of time-frequency cycles required to perform the same task in a traditional manner.

FIG. 26B is a block diagram showing an embodiment of some fields of the control register 127 of the neural network unit 121 in the first diagram. The status register 127 includes a field 2602 indicating the address of the row in the weight random access memory 124 that was recently written by the neural processing unit 126; a field 2606 indicating the most recent data in the random access memory 122. The address of the column recently written by the neural processing unit 126; a field 2604 indicating the address of the row in the weight random access memory 124 that was recently read by the neural processing unit 126; and a field 2608 indicating the data The address in the random access memory 122 that was recently read by the neural processing unit 126. In this way, the architecture program running on the processor 100 can confirm the processing progress of the neural network unit 121, and read and / or write data from the data random access memory 122 and / or the weight random access memory 124. Time. Utilize this ability, plus the above option to overwrite the input data matrix (or write the result to the data random access memory 122 as described above), as shown in the following example, the data array 2404 of Figure 24 It can be implemented as 5 512x1600 data blocks instead of 20 512x400 data blocks. The processor 100 writes the first 512x1600 data block starting from column 2 of the weight random access memory 124 and starts the neural network unit program (this program has a cycle count of 1600 and randomly weights The output column of the access memory 124 is initialized to 0). When the neural network unit 121 executes the neural network unit program, the processor 100 monitors the output position / address of the weighted random access memory 124 to read the weighted random access memory by (1) (using the MFNN instruction 1500) The volume 124 has a column of valid convolution operation results written by the neural network unit 121 (starting from column 0); and (2) overwriting the second 512x1600 data matrix 2406 (starting from column 2) to the Read the valid convolution operation results, so when the neural network unit 121 completes the neural network unit program for the first 512x1600 data block, the processor 100 can immediately update the neural network unit program and restart the neural network when necessary. Path unit program to run on the second 512x1600 data block. This process will be repeated three more times. The next three 512x1600 data blocks, so that the neural network unit 121 can be fully used.

In one embodiment, the starting function unit 212 has the ability to effectively perform an effective division operation on the value 217 of the accumulator 202, which is particularly corresponding to the figures in the subsequent chapters, 29A, 29B, and 30. There will be more detailed instructions. For example, a start function neural network unit instruction that divides the value of the accumulator 202 by a division of 16 can be used for a Gaussian fuzzy matrix described below.

The convolution kernel 2402 used in the example in Figure 24 is a small static convolution kernel applied to the entire data matrix 2404. However, the present invention is not limited to this. The convolution kernel can also be a large matrix. Having specific weights correspond to different data values of the data array 2404, such as convolution kernels commonly used in convolutional neural networks. When the neural network unit 121 is used in this way, the architecture program will interchange the positions of the data matrix and the convolution kernel, that is, place the data matrix in the data random access memory 122 and place the convolution kernel in random weights. The memory 124 is accessed, and the number of rows required to execute the neural network unit program is relatively small.

The twenty-seventh figure is a block diagram showing an example of the weighted random access memory 124 filled with input data in the first figure. This input data is performed by the neural network unit 121 in the first figure. pooling operation). The common source operation is performed by a common source layer of an artificial neural network. By obtaining a sub-region or sub-matrix of the input matrix and calculating the maximum or average value of the sub-matrix as a result matrix, that is, the common source matrix, the input data matrix is reduced ( (Such as an image or a convolution image) Size (dimension). In the examples of Figures 27 and 28, the common source operation calculates the maximum value of each sub-matrix. Common source computing is particularly useful for artificial neural networks such as performing object classification or detection. In general, the common source operation can actually reduce the factor of the input matrix to the number of elements of the detected sub-matrix, in particular, reduce the dimensions of the input matrix to the number of elements in the corresponding dimension of the sub-matrix. In the example in the twenty-seventh figure, the input data is a 512x1600 matrix of wide text (such as 16 bits), which is stored in rows 0 to 1599 of the weight random access memory 124. In the twenty-seventh figure, these characters are indicated by their column positions. For example, the characters at column 0 and row 0 are labeled D0,0; the characters at column 0 and row 1 are labeled D0,1; The text in column 0, row 2 is labeled D0,2; and so on, the text in column 0, row 511 is labeled D0,511. Similarly, the text on column 1 row 0 is labeled D1,0; the text on column 1 row 1 is labeled D1,1; the text on column 1 row 2 is labeled D1,2; and so on, the column The text on line 511 is marked as D1,511; and so on, the text on line 1599 is marked as D1599,0 on line 0; the text on line 1599 is marked on D1599,1 is located on line 1599 on line 2 The text is labeled D1599,2; and so on, the text at column 1599, line 511 is labeled D1599,511.

The twenty-eighth figure is a program list of a neural network unit program. The neural network unit program performs the common source operation of the input data matrix of the twenty-seventh figure and writes it back to the weight random access memory 124. . In the example in Figure 28, the common source operation calculates the maximum value of each 4x4 sub-matrix in the input data matrix. This program will execute the instruction loop consisting of instructions 1 to 10 multiple times. Initialized Neural Processing at Address 0 The unit instruction specifies the number of instruction loops performed by each neural processing unit 126. In the example in Figure 28, the loop count value is 400, and the loop instruction at the end of the loop (at address 11) Decrement the current loop count value, and if the result is a non-zero value, return it to the top of the instruction loop (ie, return to the instruction at address 1). The input data matrix in the weight random access memory 124 will be substantially regarded by the neural network unit program as 400 mutually exclusive groups consisting of four adjacent rows, that is, rows 0-3, 4-7, and 4 8-11, and so on, until column 1596-1599. Each group consisting of four adjacent columns includes 128 4x4 sub-matrixes. These sub-matrixes are 4x4 sub-matrixes formed by the elements at the intersection of the four columns of the group and four adjacent rows. Adjacent lines are lines 0-3, 4-7, lines 8-11, and so on until lines 508-511. Of the 512 neural processing units 126, each four is a group of calculations. The fourth neural processing unit 126 (a total of 128) performs a common source operation on a corresponding 4x4 submatrix, while the other three neural processing units 126 is not used. More precisely, the neural processing unit 0, 4, 8, and so on up to the neural processing unit 508 will perform a common source operation on its corresponding 4x4 submatrix, and the leftmost row number of this 4x4 submatrix corresponds to The number of the neural processing unit, and the lower row corresponds to the value of the current weight random access memory 124. This value will be initialized to zero by the initialization instruction at address 0 and will increase by 4 after each instruction cycle. Some will be explained in more detail in subsequent chapters. The repeated operations of the 400 instruction loops correspond to the number of 4x4 sub-matrix groups in the input data matrix of the twenty-seventh figure (that is, the 1,600 columns of the input data matrix divided by 4). Initializing the neural processing unit instructions also clears the accumulator 202 to zero. For a preferred embodiment, address 11 The loop instruction also clears the accumulator 202 to zero. In addition, the maxwacc instruction at address 1 specifies to clear accumulator 202 to zero.

Each time the instruction loop of the program is executed, the 128 used neural processing units 126 perform 128 common source operations on 128 individual 4x4 sub-matrices in the current four-row group of the input data matrix. Further, the common source operation will confirm the maximum element among the 16 elements of the 4x4 sub-matrix. In the embodiment of the twenty-eighth figure, for each of the 128 neural processing units 126 used, the left element below the 4x4 sub-matrix is the element Dx in the twenty-seventh figure , y, where x is the number of rows of the current weight random access memory 124 at the beginning of the instruction loop, and this row of data is read by the maxwacc instruction at address 1 in the program in Figure 28 (the number of rows is also It will be initialized by the initialization neural processing unit instruction at address 0 and incremented each time the maxwacc instruction at addresses 3, 5 and 7 is executed). Therefore, for each loop of this program, the 128 used neural processing units 126 write back the weighted random access memory 124 of the maximum element of the corresponding 128 4x4 sub-matrix of the current column group. Of the specified column. The following is a description of this instruction loop.

The maxwacc instruction at address 1 implicitly uses the current weighted random access memory 124 rows. This row is preferably loaded in the sequencer 128 (and initialized to zero by the instruction at address 0 to execute the first time). Pass the operation of the instruction loop). The instruction at address 1 will cause each neural processing unit 126 to read its corresponding text from the current row of the weight random access memory 124, compare this text with the value 217 of the accumulator 202, and add the two values to The largest is stored in the accumulator 202. So for example In other words, the neural processing unit 8 confirms the maximum value of the value 217 of the accumulator 202 and the data text Dx, 8 (where “x” is the current weight random access memory 124 rows) and writes it back to the accumulator 202.

Address 2 is a maxwacc instruction. This instruction specifies that the value in the multiplexer register 705 of each neural processing unit 126 is rotated to the nearest to the neural processing unit 126. This is the address corresponding to the instruction of address 1. The weight random access memory 124 reads a row of input data array values. In the embodiments of Figures 27 to 28, the neural processing unit 126 is used to rotate the value of the multiplexer 705 to the left, that is, from the neural processing unit J to the neural processing unit J-1, as described above. Corresponding to the chapters in figures 24 to 26. In addition, this instruction specifies a count value of 3. In this way, the instruction at address 2 will cause each neural processing unit 126 to receive the rotated text to the multiplexer register 705 and confirm the maximum value between the rotated text and the accumulator 202 value, and then repeat this operation twice more. . In other words, each neural processing unit 126 performs three operations of receiving the rotated text to the multiplexer register 705 and confirming the maximum value between the rotated text and the value of the accumulator 202. So, for example, suppose that at the start of this instruction loop, the current weight random access memory 124 is 36, and the neural processing unit 8 is taken as an example. After executing the instructions of addresses 1 and 2, the neural processing unit 8 will The accumulator 202 and the four weighted random access memories 124 at the beginning of the loop will store the maximum value of the characters D36, 8, D36, 9, D36, 10, and D36, 11 in the accumulator 202.

The operations performed by the maxwacc instructions at addresses 3 and 4 are similar to the instructions at address 1. The use of the weighted random access memory with 124 rows of increasing indexes has the effect. The instructions at addresses 3 and 4 randomly access the weights. Executed in a row below the memory 124. In other words, assuming that the current weight random access memory 124 column is 36 at the beginning of the instruction loop, the neural processing unit 8 is taken as an example. After completing the instructions at addresses 1 to 4, the neural processing unit 8 will accumulate the instructions. The accumulator 202 and the eight weighted random access memories 124 at the beginning of the loop are stored in the memory 202. The maximum value of D37,11.

The operations performed by the maxwacc instructions at addresses 5 to 8 are similar to the instructions at addresses 1 to 4. The instructions at addresses 5 to 8 are executed on the two rows below the weight random access memory 124. That is, assuming that the current weight random access memory 124 column is 36 at the beginning of the instruction loop, the neural processing unit 8 is taken as an example. After completing the instructions at the addresses 1 to 8, the neural processing unit 8 will accumulate the instructions. The accumulator 202 and sixteen weighted random access memories 124 at the beginning of the loop are stored in the memory 202. The text D36, 8, D36, 9, D36, 10, D36, 11, D37, 8, D37, 9, D37, 10 , D37,11, D38,8, D38,9, D38,10, D38,11, D39,8, D39,9, D39,10 and D39,11. That is, assuming that the current weight random access memory 124 column is 36 at the beginning of the instruction loop, the neural processing unit 8 is taken as an example. After completing the instructions at addresses 1 to 8, the neural processing unit 8 will complete the confirmation of the following Maximum of 4x4 sub-matrix: D36,8 D36,9 D36,10 D36,11 D37,8 D37,9 D37,10 D37,11 D38,8 D38,9 D38,10 D38,11 D39,8 D39,9 D39 , 10 D39,11 Basically, after completing the instructions at addresses 1 to 8, each of the 128 neural processing units 126 used will complete the confirmation. Find the maximum of the following 4x4 sub-matrix: Dr, n Dr, n + 1 Dr, n + 2 Dr, n + 3 Dr + 1, n Dr + 1, n + 1 Dr + 1, n + 2 Dr + 1, n + 3 Dr + 2, n Dr + 2, n + 1 Dr + 2, n + 2 Dr + 2, n + 3 Dr + 3, n Dr + 3, n + 1 Dr + 3, n + 2 Dr + 3, n + 3, where r is the column address value of the current weight random access memory 124 at the beginning of the instruction loop, and n is the number of the neural processing unit 126.

The instruction at address 9 passes the value 217 of the accumulator 202 through the enable function unit 212. This transfer function will pass a text whose size (in bits) is equivalent to the text read by the weight random access memory 124 (in this example, 16 bits). In a preferred embodiment, the user can specify an output format, such as how many bits in the output bits are fractional bits. This section will be described in more detail in subsequent sections.

The instruction at address 10 will write the value 217 of the accumulator 202 into the weighted random access memory 124 in the row specified by the current value of the output register. This current value will be initialized by the instruction at address 0, and Use the increment indicator in the instruction to increment this value after each pass through the loop. Further, the instruction at the address 10 writes a wide text (such as 16 bits) of the accumulator 202 into the weight random access memory 124. In a preferred embodiment, this instruction will write the 16 bits according to the output binary decimal point 2916. This part will be shown in the following figures corresponding to the 29th and 29th figures. More detailed explanation.

As mentioned above, the iterative instruction loop write weight random access memory 124 row contains holes with invalid values. That is That said, the wide characters 1 to 3, 5 to 7, 9 to 11, and so on of the result 133 are all invalid or unused until the wide characters 509 to 511. In one embodiment, the startup function unit 212 includes a multiplexer that enables the results to be merged into the adjacent text of the column buffer, such as the column buffer 1104 of FIG. 11 to write back the output weight random access memory. 124 columns. In a preferred embodiment, the start function instruction specifies the number of characters in each hole, and the number of characters in the hole controls the multiplexer merge result. In one embodiment, the number of holes can be specified as a value from 2 to 6 to combine the outputs of the 3x3, 4x4, 5x5, 6x6, or 7x7 sub-matrix of the common source. In addition, the architecture program running on the processor 100 reads the generated sparse (ie, has a hole) result row from the weighted random access memory 124 and uses other execution units 112, such as the media unit using the architecture merge instruction, such as The x86 single instruction multiple data flow extension (SSE) instruction performs a merge function. In a manner similar to the foregoing, and utilizing the hybrid nature of the neural network unit 121, the architecture program executing on the processor 100 can read the status register 127 to monitor the most recently written row of the random access memory 124 ( (For example, column 26602 in Figure 26B) to read one of the sparse result rows, merge them and write them back to the same row of the weight random access memory 124, so that the preparation is completed and it can be used as an input data. The matrix is provided for use by the next layer of the neural network, such as a convolutional layer or a traditional neural network layer (ie, a multiply-accumulate layer). In addition, the embodiment described herein uses a 4x4 sub-matrix to perform common source operations, but the present invention is not limited to this. The neural network unit program of Figure 28 can be adjusted, and sub-matrixes of other sizes, such as 3x3 , 5x5, 6x6, or 7x7 to perform common source operations.

As can be found before, write weight random access memory The number of result rows of the volume 124 is one quarter of the number of rows of the input data matrix. Finally, the data random access memory 122 is not used in this example. However, the data random access memory 122 may be used instead of the weight random access memory 124 to perform common source operations.

In the embodiments of the twenty-seventh and twenty-eighth figures, the common source operation calculates the maximum value of the sub-region. However, the program in the twenty-eighth figure can be adjusted to calculate the average value of the sub-region. It is beneficial to replace the maxwacc instruction with the sumwacc instruction (summing the weight text with the accumulator 202 value 217) and starting the address 9 The function instruction is modified to divide the accumulation result by the number of elements in each subregion (preferably through the inverse multiplication operation described below), which is sixteen in this example.

It can be found from the calculation of the twenty-seventh and twenty-eighth graphs by the neural network unit 121 that each execution of the program of the twenty-eighth graph requires about 6000 time-frequency cycles to perform the calculations shown in the twenty-seventh graph. The entire 512x1600 data matrix performs a common source operation. The number of time-frequency cycles used in this operation is significantly less than the number of time-frequency cycles required to perform similar tasks in a traditional manner.

In addition, the structural program can make the neural network unit program write the results of the common source operation back to the data random access memory 122 rows instead of writing the results back to the weight random access memory 124. When the neural network unit 121 writes the results When the data random access memory 122 is written (for example, using the data random access memory 122 in Figure 26B, the address of the most recently written row 2606), the architecture program will read from the data random access memory 122 result. This configuration is applicable to the embodiment having the port weight random access memory 124 and the dual port data random access memory 122.

Fixed-point arithmetic operations, with users providing binary decimal points, full-precision fixed-point accumulation, user-specified reciprocal values, random rounding of accumulator values, and optional start / output functions

Generally speaking, a hardware unit that performs arithmetic operations in a digital computing device is an integer or a floating-point number according to its object of performing arithmetic operations, and can generally be divided into "integer" units and "floating point" units. Floating point numbers have a magnitude (or mantissa) and an exponent, and usually have a sign. The index is an indicator of the position of the radix point (usually a binary decimal point) relative to the value. In contrast, integers have no exponent, but only have a value, and usually have a sign. The floating-point unit allows programmers to obtain the number they want to use for their work from a very large range of different values, while the hardware is responsible for adjusting the index value of this number when needed without the programmer's handling. For example, suppose two floating-point numbers 0.111 x 10 ^{29 are} multiplied by 0.81 x 10 ³¹ . (Although floating-point units usually work with 2-based floating-point numbers, decimal decimals or 10-based floating-point numbers are used in this example.) Floating-point units are automatically responsible for multiplying the mantissa and exponential And then normalize the result to the value .8911 x 10 ⁵⁹ . In another example, suppose the same two floating-point numbers are added. Floating-point units are automatically responsible for aligning the mantissa's decimal points before adding to produce a total of .81111 x 10 ³¹ .

However, it is well known that such a complicated operation will result in an increase in the size of the floating-point unit, an increase in power consumption, an increase in the number of time-frequency cycles required per instruction, and / or an increase in cycle time. For this reason, many devices (such as embedded processors, microcontrollers, and Power microprocessor) does not have a floating point unit. As can be seen from the foregoing example, the complex structure of a floating-point unit contains logic for performing exponential calculations related to floating-point addition and multiplication / divide The adder subtracts the operand exponent to confirm that the decimal point of the floating-point addition is aligned with the offset of the offset). It includes an offsetr for achieving the decimal point alignment of the mantissa in floating-point addition. Contains an offsetter that normalizes floating-point results. In addition, the flow usually requires the logic of performing rounding operations on floating-point results, performing conversions between integer and floating-point formats, and conversions between different floating-point formats (such as augmented precision, double precision, single precision, and half precision). Logic for leading, zero and leading one detectors, and logic for handling special floating-point numbers, such as outliers, non-numeric values, and infinite values.

In addition, the verification of the accuracy of floating-point units will greatly increase the complexity of the design due to the increase in the numerical space that needs to be verified, which will extend the product development cycle and time to market. In addition, as mentioned above, floating-point arithmetic operations need to store and use the mantissa and exponent fields of each floating-point number used for calculation, respectively, and will increase the required storage space and / or in a given storage space to Reduced accuracy when storing integers. Many of these disadvantages can be avoided by performing arithmetic operations on integer units.

Programmers usually need to write a program that handles decimals, and decimals are incomplete numbers. This program may need to be executed on a processor without a floating-point unit, or the processor may have a floating-point unit, but it is faster to execute integer instructions from the processor's integer unit. In order to take advantage of the performance advantages of integer processors, programmers Traditional fixed-point arithmetic operations are used on fixed-point numbers. Such a program would include instructions that execute on integer units to process integers or integer data. Software knows that data is decimal. This software does not include instructions to perform operations on integer data and processing this data is actually a decimal problem, such as aligning the offset. Basically, fixed-point software can manually perform functions that some or all floating-point units can perform.

In this article, a "fixed-point" number (or value or operand or input or output) is a number whose storage bits are understood to include bits to represent a fractional part of this fixed-point number. This bit is called here Is "decimal places". The storage point of a fixed-point number is contained in a memory or a register, such as an 8-bit or 16-bit text in the memory or the register. In addition, all the storage bits of a fixed-point number are used to express a value, and in some cases, one of the bits is used to express a symbol. However, there is no fixed-point number to store the bit. index. In addition, the fixed-point number or binary decimal point position is specified in a storage space different from the fixed-point storage bit, and the number or decimal of the decimal point is indicated in a shared or universal manner. The position of the decimal point is shared with a fixed-point number set containing the fixed-point number, such as a set of input operands, accumulated values, or output results of an array of processing units.

In the embodiment described here, the arithmetic logic unit is an integer unit, but the startup function unit includes hardware assistance or acceleration of floating-point arithmetic. This can make the part of the arithmetic logic unit smaller and faster, which facilitates the use of more arithmetic logic units on a given chip space. This also means that more gods can be set per unit of chip space. The warp element is particularly beneficial to the neural network unit.

In addition, compared to each floating-point number requiring exponential storage bits, the fixed-point numbers in the embodiments described herein use one index to express the number of storage bits that belong to the decimal place in the entire set of numbers, however, This indicator is located in a single, shared storage space and broadly indicates all numbers of the entire set, such as the input set of a series of operations, the set of cumulative numbers of a series of operations, and the set of outputs, among which the number of decimal places. In a preferred embodiment, the user of the neural network unit can specify the number of decimal storage bits for this number set. Therefore, it can be understood that although in many cases (such as general mathematics), the term "integer" refers to a signed complete number, that is, a number without a decimal part, but in the context of this article, " The term "integer" can mean a number with a decimal part. In addition, in the context of this article, the term "integer" is used to distinguish from floating-point numbers. For floating-point numbers, some bits in their respective storage space are used to express the exponent of floating-point numbers. Similarly, integer arithmetic operations, such as integer multiplication or addition or comparison operations performed by integer units, assume that there is no exponent in the operands. Therefore, integer components of integer units, such as integer multipliers, integer adders, integer comparators, There is no need to include logic to handle the exponents, for example, it is not necessary to move the mantissa to align the decimal points for addition or comparison operations, and it is not necessary to add the exponents for multiplication operations.

In addition, the embodiments described herein include a large hardware integer accumulator to accumulate a large series of integer operations (such as 1000 multiply-accumulate operations) without losing accuracy. This prevents the neural network unit from processing floating-point numbers, while maintaining the accumulated numbers Full precision without saturating it or producing inaccurate results due to overflow. Once the result of this series of integer operations is added to the full-accuracy accumulator, the fixed-point hardware assist will perform the necessary scaling and saturation operations to use the user-specified number of decimal places for the accumulated value and the output value required. The number of decimal places, this full-precision accumulative value is converted into an output value, this part will be explained in more detail in subsequent chapters.

When the accumulated value needs to be compressed from the full-precision form for input to one of the startup functions or used for transfer, in a preferred embodiment, the startup function unit can selectively perform a random rounding operation on the accumulated value. This section will be explained in more detail in subsequent chapters. Finally, depending on the different needs of a given layer of one of the neural networks, the neural processing unit can selectively accept instructions to use different activation functions and / or output many different forms of accumulated values.

Figure 29A is a block diagram showing an embodiment of the control register 127 of the first figure. The control register 127 may include a plurality of control registers 127. As shown in the figure, the control register 127 includes the following fields: configuration 2902, signed data 2912, signed weight 2914, data binary decimal point 2922, weighted binary decimal point 2924, arithmetic logic unit function 2926, Round control 2932, start function 2934, countdown 2942, offset 2944, output random access memory 2952, output binary decimal point 2954, and output command 2956. The value of the control register 127 can be written using MTNN instruction 1400 and NNU program instructions, such as start instruction.

The configuration 2902 value specifies whether the neural network unit 121 belongs to a narrow configuration, a wide configuration, or a funnel configuration, as described above. Configuration 2902 The size of the input text received by the data random access memory 122 and the weight random access memory 124 is also set. In the narrow and funnel configurations, the size of the input text is narrow (for example, 8-bit or 9-bit), but in the wide configuration, the size of the input text is wide (for example, 12-bit or 16-bit) ). In addition, the configuration 2902 also sets the size of the output result 133 the same as the size of the input text.

When the signed data value 2912 is true, it means that the data text received by the data random access memory 122 is a signed value. If it is false, it means that these data texts are unsigned values. When the signed weight value 2914 is true, it means that the weight text received by the weight random access memory 122 is a signed value. If it is false, it means that these weight text are unsigned values.

The data binary decimal point 2922 value indicates the binary decimal point position of the data text received by the data random access memory 122. For a preferred embodiment, for the position of the binary decimal point, the value of the data binary decimal point 2922 represents the number of bit positions calculated from the right side of the binary decimal point. In other words, the data decimal point 2922 represents the number of decimal places in the least significant digits of the data text, that is, the number of digits to the right of the binary decimal point. Similarly, the weight binary decimal point 2924 value indicates the position of the binary decimal point of the weight text received by the weight random access memory 124. In a preferred embodiment, when the arithmetic logic unit function 2926 is a multiplication and accumulation or output accumulation, the neural processing unit 126 determines the number of digits to the right of the decimal point of the binary value of the accumulator 202 as the data two The total of the decimal point 2922 and the weighted decimal point 2924 are added up. So, for example, if The value of the binary decimal point 2922 is 5 and the weighted binary decimal point 2924 is 3, and the value in the accumulator 202 will have 8 bits to the right of the binary decimal point. When the arithmetic logic unit function 2926 is a total / maximum accumulator and data / weight text or passed data / weight text, the neural processing unit 126 will place the number of digits to the right of the decimal point of the binary value of the accumulator 202 Decided as data / weight binary decimal point 2922/2924 respectively. In another embodiment, a single accumulator binary decimal point 2923 is specified instead of specifying an individual data decimal point 2922 and a weighted decimal point 2924. This section will be explained in more detail in the subsequent section corresponding to Figure 29B.

The arithmetic logic unit function 2926 specifies a function to be executed by the arithmetic logic unit 204 of the neural processing unit 126. As mentioned above, the arithmetic logic unit function 2926 may include the following operations but is not limited to: multiplying the data text 209 by the weight text 203 and adding this product to the accumulator 202; adding the accumulator 202 and the weight text 203; adding up Adder 202 and data text 209; maximum value of accumulator 202 and data text 209; maximum value of accumulator 202 and weight text 209; output accumulator 202; pass data text 209; pass weight text 209; output zero value. In one embodiment, the arithmetic logic unit function 2926 is specified by a neural network unit initialization instruction and used by the arithmetic logic unit 204 to respond to an execution instruction (not shown). In one embodiment, the arithmetic logic unit function 2926 is specified by individual neural network unit instructions, such as the aforementioned multiply accumulate and maxwacc instructions.

The rounding control 2932 specifies (Figure 30) the form of the rounding operation used by the rounder 3004. In one embodiment, you can specify Rounding modes include, but are not limited to: no rounding, rounding to the nearest value, and random rounding. For a preferred embodiment, the processor 100 includes a random bit source 3003 (refer to Figure 30) to generate random bits 3005. These random bits 3005 are sampled to perform random rounding to reduce The possibility of rounding bias. In one embodiment, when the rounded bit 3005 is one and the sticky bit is zero, if the sampled random bit 3005 is true, the neural processing unit 126 will round up. If it is the sampled random bit 3005 is false, the neural processing unit 126 will not round up. In one embodiment, the random bit source 3003 is sampled based on the random electronic characteristics of the processor 100 to generate random bits 3005. These random electronic characteristics are thermal noise of semiconductor diodes or resistors. Not limited to this.

The activation function 2934 specifies a function for the value 217 of the accumulator 202 to generate the output 133 of the neural processing unit 126. As described herein, the start function 2934 includes, but is not limited to: S-type function; hyperbolic tangent function; soft addition function; correction function; division by a specified power of two; multiplication by a user-specified inverse value to achieve, etc. Divide by effect; pass the entire accumulator; and pass the accumulator in standard size, which will be explained in more detail in the following sections. In one embodiment, the activation function is specified by a neural network unit activation function instruction. In addition, the startup function can also be specified by an initialization instruction and used in response to an output instruction. For example, the startup function unit output address at address 4 in the fourth figure. In this embodiment, the startup function at address 3 in the fourth figure Function instructions are included in the output instructions.

Reciprocal 2942 value specifies a value with accumulator 202 Multiply 217 to obtain a value that divides the value 217 of the accumulator 202. In other words, the user-specified inverse 2942 value will be the inverse of the divisor actually desired to be performed. This facilitates the use of convolution or common source operations as described herein. In a preferred embodiment, the user designates the penultimate 2942 value as two parts, which will be described in more detail at a subsequent point corresponding to the twenty-ninth figure C. In one embodiment, the control register 127 includes a field (not shown) so that the user can specify one of a plurality of built-in division values to perform division. The size of these built-in division values is equivalent to a common convolution. The size of the core, such as 9, 25, 36, or 49. In this embodiment, the activation function unit 212 stores the inverses of these built-in divisors for multiplication with the value 217 of the accumulator 202.

The offset 2944 specifies the number of bits to shift the value 217 of the accumulator 202 to the right by one of the shifters of the activation function unit 212 to achieve the operation of dividing it by a power of two. This is conducive to the operation with a convolution kernel with a power of two.

The output random access memory 2952 value will designate one of the data random access memory 122 and the weight random access memory 124 to receive the output result 133.

The output binary point 2954 value indicates the position of the binary point of the output result 133 bis. For a preferred embodiment, for the position of the decimal point of the output 133 bis, outputting the value of the binary point 2954 represents the number of bit positions calculated from the right side. In other words, the output decimal point 2954 represents the number of decimal places in the least significant bit of the output result 133, that is, the number of bits to the right of the binary decimal point. The start function unit 212 will output the decimal point based on the binary The value of 2954 (in most cases, it will also be rounded, compressed, saturated, and converted based on the data binary decimal point 2922, the weight binary decimal point 2924, the activation function 2934, and / or the configuration 2902 value). Operation.

The output command 2956 outputs the result 133 from a number of control-oriented. In one embodiment, the activation function unit 121 uses the concept of a standard size, which is twice the width (in bits) specified by the configuration 2902. So, for example, if the configuration 2902 sets the size of the input text received by the data random access memory 122 and the weight random access memory 124 to 8 bits, the standard size will be 16 bits; in another example If the configuration 2902 sets the size of the input text received by the data random access memory 122 and the weight random access memory 124 to 16 bits, the standard size will be 32 bits. As described in this article, the size of the accumulator 202 is large (for example, the narrow accumulator 202B is 28 bits, and the wide accumulator 202A is 41 bits) to maintain intermediate calculations, such as 1024 and 512 nerves. Network unit multiply accumulate instructions with full precision. In this way, the value 217 of the accumulator 202 will be greater than (in bits) the standard size, and for most of the values of the start function 2934 (except for passing the entire accumulator), the start function unit 212 (for example, the following corresponds to the one in Figure 30) The standard size compressor 3008) described in the paragraph will compress the value 217 of the accumulator 202 to the size of the standard size. The first preset value of the output command 2956 instructs the startup function unit 212 to execute the specified startup function 2934 to generate an internal result and output the internal result as the output result 133. The size of the internal result is equal to the size of the original input text, that is, Half the standard size. The second preset value of output command 2956 will instruct to start The function unit 212 executes a specified start function 2934 to generate an internal result and outputs the lower half of the internal result as an output result 133. The size of the internal result is equal to twice the size of the original input text, that is, the standard size; The third preset value of the command 2956 will instruct the activation function unit 212 to output the upper half of the internal result of the standard size as the output result 133. The fourth preset value of output command 2956 instructs startup function unit 212 to output the unprocessed least significant text of accumulator 202 as output result 133; and the fifth preset value of output command 2956 instructs startup function unit 212 to The unprocessed intermediate valid text of the accumulator 202 is output as the output result 133; the sixth preset value of the output command 2956 will instruct the activation function unit 212 to process the unprocessed highest valid text of the accumulator 202 (the width of which is determined by the configuration (Specified by 2902) is output as the output result 133, which is explained in more detail in the previous section corresponding to the eighth to tenth figures. As described above, outputting the internal result of the size of the accumulator 202 or the standard size is helpful to enable other execution units 112 of the processor 100 to execute startup functions, such as soft maximum startup functions.

The fields described in Figure 29A (and Figures 29B and 29C) are located inside the control register 127. However, the present invention is not limited to this. One or more of the fields It may also be located in other parts of the neural network unit 121. In a preferred embodiment, many of the fields can be included in the neural network unit instruction and decoded by the sequencer 128 to generate a microinstruction 3416 (see Figure 34) to control the arithmetic logic unit. 204 and / or start function unit 212. In addition, these fields can also be included in the micro operation 3414 stored in the media register 118 (please refer to Figure 34) to control the arithmetic logic unit 204 And / or start the function unit 212. This embodiment can reduce the use of the initialization neural network unit instruction, and in other embodiments, the initialization neural network unit instruction can be removed.

As mentioned above, the neural network unit instruction can specify a memory operand (such as text from data random access memory 122 and / or weight random access memory 123) or a rotated operand (such as Registers 208/705) perform arithmetic logic instruction operations. In an embodiment, the neural network unit instruction may further designate an operand as a register output of an activation function (such as the output of the register 3038 in FIG. 30). In addition, as described above, the neural network unit instruction may be designated to increment the current row address of one of the data random access memory 122 or the weight random access memory 124. In one embodiment, the neural network unit instruction may specify an immediate signed integer difference value to be added to the current column to achieve the purpose of increasing or decreasing by a value other than one.

Figure 29B is a block diagram showing another embodiment of the control register 127 of the first figure. The control register 127 of Fig. 29B is similar to the control register 127 of Fig. 29A, but the control register 127 of Fig. 29B includes an accumulator binary decimal point 2923 . The accumulator binary decimal point 2923 indicates the position of the decimal point of the accumulator 202 binary. In a preferred embodiment, the accumulator binary decimal point value of 2923 indicates the number of bit positions from the right of the binary decimal point position. In other words, the accumulator binary decimal point 2923 represents the number of decimal places in the least significant bit of the accumulator 202, that is, the bit located to the right of the binary decimal point. In this embodiment, the accumulator binary decimal point 2923 is clearly indicated, and is not implemented as shown in Figure 29A. Example is to confirm secretly.

Figure 29C is a block diagram showing an example of storing the penultimate 2942 of Figure 29A in two parts. The first part 2962 is an offset value indicating that the user wants to multiply the number 2962 of the true inverse value of the value 217 of the accumulator 202 by the leading zero before being suppressed. The number of leading zeros is the number of zeros arranged consecutively to the right of the decimal point. The second part 2694 is the leading zero suppression inverse value, that is, the true inverse value after removing all leading zeros. In one embodiment, the suppressed leading zero number 2962 is stored in 4 bits, and the leading zero suppression reciprocal value 2964 is stored in an 8-bit unsigned value.

For example, suppose a user wants to multiply the value 217 of the accumulator 202 by the inverse value of the value 49. The reciprocal value of the value 49 is displayed in two dimensions and the 13 decimal places are set to 0.0000010100111, of which there are five leading zeros. In this way, the user will fill the number of suppressed leading zeros 2962 with a value of 5 and the number of leading zeros with a suppression of 2964 into a value of 10100111. After the reciprocal multiplier "divider A" 3014 (refer to Figure 30) multiplies the value 217 of the accumulator 202 with the leading zero suppression reciprocal value 2964, the resulting product is shifted to the right by the suppressed leading zero number 2962. Such an embodiment helps to use a relatively small number of bits to express the inverse 2942 value to achieve the requirement of high accuracy.

The thirtieth figure is a block diagram showing an embodiment of the activation function unit 212 of the second figure. The start function unit 212 includes the control logic 127 of the first figure, a positive type converter (PFC) and an output binary decimal point aligner (OBPA) 3002 to receive the accumulator 202 value 217, and a rounder 3004 to receive Accumulator 202 value 217 and output two An indicator of the number of bits shifted out by the carry decimal aligner 3002, a random bit source 3003 as described above to generate random bits 3005, and a first multiplexer 3006 to receive positive type converters and output binary decimal points Output of aligner 3002 and output of rounder 3004, a standard size compressor (CCS) and saturator 3008 to receive the output of the first multiplexer 3006, a bit selector and saturator 3012 to receive the standard size The output of the compressor and saturator 3008, a corrector 3018 to receive the output of the standard size compressor and saturator 3008, an inverse multiplier 3014 to receive the output of the standard size compressor and saturator 3008, a right shifter 3016 receives the output of the standard size compressor and saturator 3008, a double tangent (tanh) module 3022 receives the output of the bit selector and saturator 3012, and an S-type module 3024 receives the bit selector and The output of the saturator 3012, a soft add module 3026 to receive the output of the bit selector and saturator 3012, a second multiplexer 3032 to receive the double-take tangent module 3022, the S-module 3024, the soft Module 3026, corrector 3018, reciprocal multiplier 3014 and right shifter 3016 output and standard size output 3028 passed by standard size compressor and saturator 3008, a symbol restorer 3034 to receive the second multiplexer 3032 output, a size converter and saturator 3036 to receive the output of the symbol restorer 3034, a third multiplexer 3037 to receive the output of the size converter and saturator 3036 and the accumulator output 217, and an output buffer The multiplexer 3038 receives the output of the multiplexer 3037, and the output is the result 133 in the first figure.

The positive type converter and output binary point aligner 3002 receives the accumulator 202 value 217. For a preferred embodiment, as mentioned above, the accumulator 202 value 217 is a full precision value. That is, the accumulator 202 has a sufficient number of storage bits to load an accumulative number. The accumulative number is the total number generated by the integer adder 244 adding a series of products generated by the integer multiplier 242, and this operation None of the individual products of the multiplier 242 or the individual totals of the adder are discarded to maintain accuracy. For a preferred embodiment, the accumulator 202 has at least a sufficient number of bits to load the maximum number of multiply-accumulate additions that can be generated by a neural network unit 121 that is programmed. For example, please refer to the program in the fourth figure. In a wide configuration, the maximum number of multiply-accumulate sums that can be generated by the neural network unit 121 by programmatic execution is 512, and the accumulative number is 202 bits wide. In another example, please refer to the program in the twentieth chart. In a narrow configuration, the maximum number of multiply-accumulate additions that can be generated by the neural network unit 121 by programmatic execution is 1024, and the accumulative number 202-bit width is 28. Basically, the full precision accumulator 202 has at least Q bits, where Q is the sum of M and log ₂ P, where M is the bit width of the integer product of multiplier 242 (for example, for narrow multiplier 242 16 bits, 32 bits for wide multiplier 242), and P is the maximum allowable number of products that accumulator 202 can accumulate. In a preferred embodiment, the maximum number of multiplications is specified according to the program specifications of the programmer of the neural network unit 121. In one embodiment, it is assumed that a previous multiply accumulate instruction is used to load data / weight random access memory 122/124 from the data / weight text 206/207 rows (such as the instruction at address 1 in the fourth figure). In the above, the sequencer 128 will execute a multiply accumulate neural network unit instruction (such as the instruction at address 2 in the fourth figure). The maximum count is, for example, 511.

Make use of one with sufficient bit width The accumulator 202, which performs one of the accumulating operations on one of the largest number of full-precision accumulative values, can simplify the design of the arithmetic logic unit 204 of the neural processing unit 126. In particular, this process can alleviate the need to use logic to perform saturation operations on the totals produced by the integer adder 244, because the integer adder 244 can cause a small accumulator to overflow, and it needs to keep track of the decimal places of the accumulator Click on the position to see if an overflow occurs to see if a saturation operation needs to be performed. For example, for a design with a non-full-precision accumulator but with saturation logic to handle the overflow of the non-full-precision accumulator, assume the following.

(1) The range of data text values is between 0 and 1 and all storage bits are used to store decimal places. The weight text value ranges from -8 to +8 and all storage bits except three are used to store decimal places. The accumulative value of the input as a hyperbolic tangent activation function is between -8 and 8, and all storage bits except three are used to store decimal places.

(2) The bit width of the accumulator is not full precision (such as only the bit width of the product).

(3) Assuming that the accumulator is full precision, the final accumulative value will also be between -8 and 8 (such as +4.2); however, the product before "point A" in this sequence will produce a positive value more frequently , And the product after point A will produce negative values more frequently. In this case, incorrect results (such as results other than +4.2) may be obtained. This is because at some points in front of point A, when the accumulator needs to reach a value that exceeds its saturation maximum +8, such as +8.2, an extra 0.2 will be lost. The accumulator will even maintain the remaining multiply-accumulate results At saturation values, more positive values are lost. Therefore, the final value of the accumulator may be smaller than the value calculated using the accumulator with full precision bit width (ie, less than +4.2).

The positive type converter 3004 converts the accumulator 202 to a positive type when the value 217 of the accumulator 202 is negative, and generates an extra bit to indicate the positive or negative value of the original value. This bit is passed down to the start function unit 212 along with this value. Pipeline. Converting a negative number to a positive type can simplify the operation of the subsequent activation function unit 121. For example, after this processing, only positive values will be input to the hyperbolic tangent module 3022 and the S-type module 3024, so the design of these modules can be simplified. In addition, the rounder 3004 and the saturator 3008 can be simplified.

The output binary point aligner 3002 shifts or scales this positive type value to the right to align it with the output binary point 2954 specified in the control register 127. For a preferred embodiment, the output decimal point aligner 3002 calculates the number of decimal places of the value 217 of the accumulator 202 (for example, specified by the accumulator binary decimal point 2923 or the data binary decimal point 2922). And the weighted binary decimal point (total of 2924) minus the output decimal place number (eg, specified by the output binary decimal point 2954) as the offset. So, for example, if the accumulator 202 has a binary decimal point 2923 of 8 (that is, the above embodiment) and the output binary decimal point 2954 is 3, the output binary decimal point aligner 3002 will right the positive type value. The 5 bits are shifted to produce a result provided to the multiplexer 3006 and the rounder 3004.

The rounder 3004 performs a rounding operation on the accumulator 202 value 217. For a preferred embodiment, the rounder 3004 The type converter and the output binary point aligner 3002 generate a positive type value to generate a rounded version, and provide this rounded version to the multiplexer 3006. The rounder 3004 performs a rounding operation according to the aforementioned rounding control 2932. As described herein, the aforementioned rounding control includes random rounding using a random bit 3005. The multiplexer 3006 will control 2932 according to the rounding (as described in this article, which may include random rounding), and select one of its multiple inputs, that is, from the positive type converter and the output decimal point aligner 3002 The positive type value is also a rounded version from the rounder 3004, and the selected value is provided to the standard size compressor and saturator 3008. In a preferred embodiment, if the rounding control specifies no rounding, the multiplexer 3006 will select the output of the positive type converter and the output binary decimal aligner 3002, otherwise the rounder will be selected. 3004 output. In other embodiments, an additional rounding operation may be performed by the startup function unit 212. For example, in one embodiment, when the bit selector 3012 compresses the output (as described below) of the standard size compressor and the saturator 3008, the bit selector 3012 will base on the missing low order bits Performs rounding. In another example, the product of the inverse multiplier 3014 (described below) is rounded. In another example, the size converter 3036 needs to convert an appropriate output size (as described later). This conversion may involve dropping some low order bits used to determine rounding, and then perform a rounding operation.

The standard size compressor 3008 compresses the output value of the multiplexer 3006 to a standard size. Therefore, for example, if the neural processing unit 126 is in a narrow configuration or a funnel configuration 2902, the standard size compressor 3008 can compress the 28-bit multiplexer 3006 output value to 16 bits; and If the neural processing unit 126 is in a wide configuration 2902, the standard size compressor 3008 can compress the output value of the 41-bit multiplexer 3006 to 32-bit. However, before compression to the standard size, if the value before compression is greater than the maximum value that can be expressed by the standard pattern, the saturator 3008 will fill this value before compression to the maximum value that can be expressed by the standard pattern. For example, if any bit in the pre-compression value that is to the left of the most significant pre-compression value bit is the value 1, the saturator 3008 will fill up to the maximum value (for example, if it fills all 1).

In a preferred embodiment, the hyperbolic tangent module 3022, the S-shaped module 3024, and the soft-plus module 3026 all include look-up tables, such as a programmable logic array (PLA), read-only memory (ROM) , Combined logic gates, and so on. In an embodiment, in order to simplify and reduce the size of these modules 3022/3024/3026, the input value provided to these modules has a type of 3.4, that is, three integer bits and four decimal places, that is, The input value has four bits to the right of the decimal point and three bits to the left of the decimal point. These values are chosen because the output value will approach the minimum / maximum value asymptotically at the extremes of the input value range (-8, +8) of type 3.4. However, the present invention is not limited to this, and the present invention can also be applied to other embodiments in which a binary decimal point is placed at a different position, such as a 4.3 type or a 2.5 type. The bit selector 3012 selects the bit that meets the 3.4 type specification among the bits output by the standard size compressor and saturator 3008. This involves compression processing, that is, some bits are lost, because the standard type has more The number of bits. However, before selecting / compressing the output values of the standard size compressor and saturator 3008, if the value before compression is greater than the maximum value that can be expressed by the 3.4 type, the saturation The device 3012 will fill the value before compression to the maximum value that the 3.4 type can express. For example, if any bit on the left side of the most significant 3.4 type bit in the pre-compression value is the value 1, the saturator 3012 will be filled to the maximum value (for example, filled to all 1).

The hyperbolic tangent module 3022, the S-type module 3024 and the soft-add module 3026 perform a corresponding activation function (as described above) on the 3.4-type values output by the standard size compressor and the saturator 3008 to produce a result. In a preferred embodiment, the hyperbolic tangent module 3022 and the S-shaped module 3024 produce a 0.7-bit 7-bit result, that is, zero integer bits and seven decimal places, that is, input The value has seven digits to the right of the decimal point. In a preferred embodiment, the soft add module 3026 produces a 7-bit result of the 3.4 type, that is, the type is the same as the input type of the module 3026. For a preferred embodiment, the output of the hyperbolic tangent module 3022, the S-shaped module 3024 and the soft plus module 3026 will be extended to the standard type (for example, leading zeros are added if necessary) and aligned so that The binary decimal point is specified by the output binary decimal point 2954 value.

The corrector 3018 generates a corrected version of one of the output values of the standard size compressor and the saturator 3008. That is to say, if the output values of the standard size compressor and saturator 3008 (the symbols are shifted down the pipeline as described above) are negative, the corrector 3018 will output a zero value; otherwise, the corrector 3018 will output its input value . In a preferred embodiment, the output of the corrector 3018 is a standard type and has a decimal point specified by the output binary point 2954 value.

Reciprocal multiplier 3014 inverts the output of the standard size compressor and saturator 3008 with the user-specified inverse value 2942 Multiply the values to produce a standard size product. This product is actually the output value of the standard size compressor and saturator 3008. The reciprocal of the inverse of the inverse value 2942 is used as the divisor. In a preferred embodiment, the output of the reciprocal multiplier 3014 is a standard type and has a decimal point specified by the output binary point 2954 value.

The right shifter 3016 shifts the output of the standard size compressor and the saturator 3008 by the user-specified number of bits specified at the offset value 2944 to generate a standard size quotient. According to a preferred embodiment, the output of the right shifter 3016 is a standard type and has a decimal point specified by the output binary point 2954 value.

The multiplexer 3032 selects the appropriate input specified by the value of the start function 2934 and provides the selection to the symbol restorer 3034. If the original accumulator 202 value 217 is negative, the symbol restorer 3034 outputs the multiplexer 3032 A positive type value is converted to a negative type, such as a two's complement type.

The size converter 3036 converts the output of the symbol restorer 3034 to an appropriate size according to the value of the output command 2956 as shown in FIG. 29A. For a preferred embodiment, the output of the sign restorer 3034 has a decimal point specified by the output binary point 2954 value. For a preferred embodiment, for the first preset value of the output command, the size converter 3036 discards the upper half of the output of the symbol restorer 3034. In addition, if the output of the symbol restorer 3034 is positive and exceeds the maximum value that can be expressed by the text size specified by the configuration 2902, or if the output is negative and less than the minimum value that can be expressed by the text size, the saturator 3036 will output it Fill up to this text size separately It can express the maximum / minimum value. For the second and third preset values, the size converter 3036 passes the output of the symbol restorer 3034.

The multiplexer 3037 selects one of the data converter and saturator 3036 output and the accumulator 202 output 217 according to the output command 2956 to provide to the output register 3038. Further, for the first and second preset values of the output command 2956, the multiplexer 3037 selects the lower text of the output of the size converter and the saturator 3036 (the size is specified by the configuration 2902). For the third preset value, the multiplexer 3037 selects the text above the output of the size converter and the saturator 3036. For the fourth preset value, the multiplexer 3037 selects the text below the value 217 of the unprocessed accumulator 202; for the fifth preset value, the multiplexer 3037 selects the middle of the value 217 of the unprocessed accumulator 202 Text; and for the sixth preset value, the multiplexer 3037 selects the text above the value 217 of the unprocessed accumulator 202. As mentioned above, in a preferred embodiment, the activation function unit 212 adds a zero value upper azimuth element to the text above the value 217 of the unprocessed accumulator 202.

The thirty-first figure is an example of the operation of the activation function unit 212 of the thirty-first figure. As shown in the figure, the configuration 2902 of the neural processing unit 126 is set to a narrow configuration. In addition, the signed data 2912 and the signed weight 2914 are true. In addition, the data binary decimal point 2922 value indicates that for the data random access memory 122 text, there are 7 bits to the right of the binary decimal point position, and an example value of the first data text received by the neural processing unit 126 The line appears to be 0.1001110. In addition, the weight binary decimal point 2924 value indicates that for the weight random access memory 124 text, there are 3 bits to the right of the binary decimal point position. An example value of the first weight text received by the processing unit 126 is presented as 00001.010.

The 16-bit product of the first data and the weight text (this product will be added to the initial zero value of the accumulator 202) is presented as 000000.1100001100. Because the data decimal point 2912 is 7 and the weighted decimal point 2914 is 3, for the implied accumulator 202 binary decimal point, there will be 10 digits to the right. In the case of a narrow configuration, as shown in this embodiment, the accumulator 202 has a width of 28 bits. For example, after completing all arithmetic logic operations (such as all 1024 multiply-accumulate operations in the twentieth graph), the value 217 of the accumulator 202 will be 000000000000000001.1101010100.

The output 2954 decimal point value indicates that there are 7 digits to the right of the output binary decimal point. Therefore, after passing the output binary decimal aligner 3002 and the standard size compressor 3008, the value 217 of the accumulator 202 is scaled, rounded, and compressed to the standard type value, that is, 000000001.1101011. In this example, the output binary decimal point address represents 7 decimal places, and accumulator 202 binary decimal point position represents 10 decimal places. Therefore, the output binary aligner 3002 calculates the difference 3 and scales the accumulator 202 value 217 by 3 bits to the right. In the thirty-first figure, it is shown that the value 217 of the accumulator 202 will lose the 3 least significant bits (100 binary digits). In addition, in this example, the rounding control 2932 value means that random rounding is used, and in this example, the sampling random bit 3005 is assumed to be true. In this way, as mentioned above, the least significant bit will be rounded up. This is because of the rounding bit of the accumulator 202 value 217 (these three are because the accumulator 202 value 217 The most significant bit of the bits removed by the scaling operation is one, and the two least significant bits of the sticky bits (the three bits removed by the scaling operation of the accumulator 202 value 217) The Bollinger or operation result) is zero.

In this example, the activation function 2934 indicates that an S-type function is used. In this way, the bit selector 3012 selects the bits of the standard type value so that the input of the S-type module 3024 has three integer bits and four decimal places, as described above, that is, the value 001.1101 shown. The output value of the S-type module 3024 will be put into the standard type, that is, the value 000000000.1101110 shown.

The output command 2956 of this example specifies the first preset value, that is, the text size indicated by the output configuration 2902, in this case, the narrow text (8 bits). In this way, the size converter 3036 converts the standard S-type output value into an 8-bit quantity, which has an implicit binary decimal point, that is, 7 bits to the right of the binary decimal point, and produces an output The value 01101110, as shown in the figure.

The thirty-second figure is a second example showing the operation of the activation function unit 212 of the thirty figure. The example in the thirty-second figure describes the operation of the activation function unit 212 when the activation function 2934 indicates that the value 217 of the accumulator 202 is passed in a standard size. As shown in the figure, this configuration 2902 is set as a narrow configuration of the neural processing unit 216.

In this example, the width of the accumulator 202 is 28 bits, and there are 10 bits to the right of the position of the decimal point of the accumulator 202 (this is because in one embodiment, the data decimal point 2912 and the weight binary bit The decimal point 2914 adds up to 10, or the accumulator in another embodiment The binary decimal point 2923 is explicitly specified as having the value 10). For example, after performing all the arithmetic logic operations, the value 217 of the accumulator 202 shown in the thirty-second figure is 000001100000011011.1101111010.

In this example, the output of the decimal point 2954 value means that for the output, there are 4 digits to the right of the decimal point. Therefore, after passing the output binary aligner 3002 and the standard size compressor 3008, the value 217 of the accumulator 202 will saturate and compress to the standard type value 111111111111.1111 shown. This value is received by the multiplexer 3032. Pass the value 3028 as a standard size.

Two output commands 2956 are shown in this example. The first output command 2956 specifies the second preset value, that is, the text below the standard type size is output. Because the size indicated by the configuration 2902 is narrow text (8 bits), the standard size will be 16 bits, and the size converter 3036 will select the 8 bits below the standard size transfer value 3028 to generate the figure The 8-bit value is 11111111. The second output command 2956 specifies the third preset value, that is, the text above the standard size is output. In this way, the size converter 3036 selects 8 bits above the standard size transfer value 3028 to generate the 8-bit value 11111111 as shown in the figure.

The thirty-third figure is a third example showing the operation of the activation function unit 212 of the thirty-third figure. The example in FIG. 33 shows the operation of the activation function unit 212 when the activation function 2934 indicates that the entire unprocessed accumulator 202 value 217 is to be passed. As shown in the figure, this configuration 2902 is set to a wide configuration of the neural processing unit 126 (for example, a 16-bit input text).

In this example, the width of the accumulator 202 is 41 bits There are 8 digits to the right of the decimal point of the accumulator 202 (this is because the sum of the data decimal point 2912 and the weighted binary decimal point 2914 in one embodiment is 8 or in another implementation The accumulator binary decimal point 2923 is explicitly specified as having the value 8). For example, after performing all arithmetic logic operations, the value 217 of the accumulator 202 shown in Figure 33 is 001000000000000000001100000011011.11011110.

Three output commands 2956 are shown in this example. The first output command specifies the fourth preset value, that is, the text below the value of the unprocessed accumulator 202 is output; the second output command specifies the fifth preset value, which is the middle of the value of the unprocessed accumulator 202 Text; and the third output command specifies a sixth preset value, that is, the text above the value of the unprocessed accumulator 202 is output. Because the size indicated by configuration 2902 is wide text (16 bits), as shown in Figure 33, in response to the first output command 2956, the multiplexer 3037 will select the 16-bit value 0001101111011110; in response to the second output command 2956 The multiplexer 3037 will select the 16-bit value 0000000000011000; in response to the third output command 2956, the multiplexer 3037 will select the 16-bit value 0000000001000000.

As mentioned above, the neural network unit 121 can be executed on integer data instead of floating-point data. In this way, it helps to simplify each of the neural processing units 126, or at least the arithmetic logic unit 204 thereof. For example, the arithmetic logic unit 204 does not need to include an adder that is used to add the exponents of the multipliers in the floating-point operation for the multiplier 242. Similarly, the arithmetic logic unit 204 does not need to be included in the floating-point operation for the adder 234 to be used to align the decimal point of the decimal place of the addend. Shifter. Those skilled in the art can understand that floating-point units are often very complex; therefore, the examples described in this article are only simplified for the arithmetic logic unit 204. The use of hardware fixed-point assistance allows users to specify related two The integer embodiment of the decimal point can also be used to simplify other parts. Compared with the floating-point embodiment, using an integer unit as the arithmetic logic unit 204 can generate a smaller (and faster) neural processing unit 126, which is advantageous for integrating a large array of neural processing units 126 into a neural network. Unit 121. The part of the activation function unit 212 can process the scaling and saturation operations of the value 217 of the accumulator 202 based on the user specified, the number of decimal places required by the accumulated number and the number of decimal places required by the output value, and the better is based on the use Designator. Any additional complexity and accompanying increase in size, and energy and / or time consumption in the fixed-point hardware assistance of the activation function unit 212 can be shared by sharing the activation function unit 212 among the arithmetic logic unit 204. This is because, as shown in the embodiment in FIG. 11, the embodiment adopting the sharing method can reduce the number of activation function units 1112.

The embodiments described herein can enjoy many advantages of using integer arithmetic units to reduce hardware complexity (compared to using floating-point arithmetic units), and can also be used for arithmetic operations on decimals, that is, with binary decimal digital. The advantage of floating-point arithmetic is that it can provide data arithmetic operations to individual values of the data falling within a very wide range of values (actually only limited by the size of the exponential range, so it will be a very large range). That is, each floating-point number has its potentially unique exponential value. However, the embodiments described herein understand and utilize In some applications, the input data is highly parallel and falls within a relatively narrow range, so that all parallel data have the same "index" characteristic. As such, these embodiments allow the user to assign binary decimal point positions to all input values and / or accumulated values at once. Similarly, by understanding and utilizing the characteristics of parallel outputs with similar ranges, these embodiments allow the user to assign the binary decimal point position to all output values at once. Artificial neural networks are one example of such applications, but embodiments of the present invention can also be applied to perform calculations for other applications. By assigning the decimal point position to multiple inputs at one time rather than to individual input numbers, the embodiment of the present invention can use the memory space more efficiently (if less memory is required) than using floating-point arithmetic Volume) and / or increase accuracy with a similar amount of memory, because the bits of the exponent used for floating-point arithmetic can be used to increase numerical accuracy.

In addition, the embodiments of the present invention understand that accuracy may be lost when performing accumulation on a large series of integer operations (such as overflow or loss of less significant decimal places), so a solution is provided, mainly by using a large enough Accumulator to avoid loss of accuracy.

Direct execution of micro-operations in neural network units

The thirty-fourth figure is a block diagram showing some details of the processor 100 and the neural network unit 121 in the first figure. The neural network unit 121 includes a pipeline stage 3401 of the neural processing unit 126. Each pipeline stage 3401 is distinguished by a stage register and includes combinational logic to achieve the operations of the neural processing unit 126 in this article, such as a Bollinger logic gate, a multiplexer, an adder, a multiplier, a comparator, and so on. Pipeline stage 3401 slave multiplexer 3402 Receive a micro operation 3418. The micro-operation 3418 flows down to the pipeline stage 3401 and controls its combination logic. Micro operation 3418 is a set of bits. In a preferred embodiment, the micro-operation 3418 includes data random access memory 122, memory address 123, weight random access memory 124, memory address 125, and program memory 129. Bit 131, multiplex register 208/705 control signal 213/713, and many fields of control register 217 (such as the control registers of Figures 29A to 29C). In one embodiment, the micro-operation 3418 includes approximately 120 bits. The multiplexer 3402 receives micro-operations from three different sources and selects one of them as the micro-operation 3418 provided to the pipeline stage 3401.

One micro-computing source of the multiplexer 3402 is the sequencer 128 of the first figure. The sequencer 128 decodes the neural network unit instructions received by the program memory 129 and generates a micro-operation 3416 to provide the first input to the multiplexer 3402.

The second micro-computing source of the multiplexer 3402 is a decoder 3404 that receives micro-instructions 105 from the reservation station 108 in the first figure and receives operands from the general-purpose register 116 and the media register 118. In a preferred embodiment, as mentioned above, the microinstruction 105 is generated by the instruction translator 104 in response to the translation of the MTNN instruction 1400 and the MFNN instruction 1500. The micro-instruction 105 may include an immediate field to specify a specific function (specified by an MTNN instruction 1400 or an MFNN instruction 1500), such as the start and stop of execution of a program in the program memory 129, and execution of a program directly from the media register 118 Micro-computing, or one of the read / write memories of the neural network unit. The decoder 3404 decodes the micro instruction 105 and generates a micro operation 3412 to provide a second input to the multiplexer. For a preferred embodiment In terms of some functions 1432/1532 of MTNN instruction 1400 / MFNN instruction 1500, decoder 3404 does not need to generate a micro-operation 3412 and send it down to pipeline 3401, such as writing to control register 127 and starting program execution Programs in the memory 129, suspend execution of the programs in the program memory 129, wait for the programs in the program memory 129 to finish executing, read from the state register 127, and reset the neural network unit 121.

The third micro-computing source of the multiplexer 3402 is the media register 118 itself. For a preferred embodiment, as described above corresponding to the fourteenth figure, the MTNN instruction 1400 may specify a function to instruct the neural network unit 121 to directly execute a first step provided by the media register 118 to the multiplexer 3402 Three-input micro operation 3414. Directly performing the micro-operations 3414 provided by the architecture media register 118 is beneficial for testing the neural network unit 121, such as a built-in self-test (BIST), or a debugging action.

For a preferred embodiment, the decoder 3404 generates a mode indicator 3422 to control the selection of the multiplexer 3402. When MTNN instruction 1400 specifies a function to start executing a program from program memory 129, the decoder 3404 will generate a mode indicator 3422 value to cause the multiplexer 3402 to select the micro-operation 3416 from the sequencer 128 until an error occurs or until decoding The processor 3404 encounters an MTNN instruction 1400 specifying a function to stop executing a program from the program memory 129. When the MTNN instruction 1400 specifies a function to instruct the neural network unit 121 to directly perform a micro-operation 3414 provided by the media register 118, the decoder 3404 will generate a mode indicator 3422 value to enable the multiplexer 3402 to select the media from the specified medium. The micro-operation of the register 118 is 3414. Otherwise, the decoder 3404 will generate a mode indicator 3422 value to cause the multiplexer 3402 to select the micro-op from the decoder 3404. Count 3412.

Variable rate neural network unit

In many cases, after executing the program, the neural network unit 121 enters an idle state and waits for the processor 100 to process something that needs to be processed before executing the next program. For example, if it is in a situation similar to that described in Figures 3 to 6A, the neural network unit 121 accumulates a multiplying activation function program (also known as a feed forward neural network layer program (feed forward neural network layer program)) is performed two or more times in a row. Compared with the time taken by the neural network unit 121 to execute the program, the processor 100 obviously takes a longer time to write the 512KB weight value into the weight random access memory 124 for the next neural network unit program. . In other words, the neural network unit 121 executes the program in a short time, and then enters the standby state until the processor 100 writes the next weight value into the weight random access memory 124 for the next program execution. In this case, please refer to Figure 36A, which will be described in detail later. In this case, the neural network unit 121 can run at a lower time frequency to extend the execution time of the program, so that the energy consumption required to execute the program is spread over a longer time range, so that the neural network unit 121 and even The entire processor 100 is maintained at a relatively low temperature. This situation is called a relaxation mode, and you can refer to Figure 36B, which will be described in detail later.

The thirty-fifth figure is a block diagram showing a processor 100 having a variable rate neural network unit 121. The processor 100 is similar to the processor 100 in the first figure, and the components with the same reference numerals in the figure are similar. The processor 100 in the thirty-fifth figure has time-frequency generating logic 3502, which is a functional unit coupled to the processor 100. These functional units are instruction fetches. Unit 101, instruction cache 102, instruction translator 104, rename unit 106, reservation station 108, neural network unit 121, other execution units 112, memory subsystem 114, general purpose register 116, and media register 118 . The time-frequency generating logic 3502 includes a time-frequency generator, such as a phase-locked loop (PLL), to generate a time-frequency signal having a main time-frequency or time-frequency. For example, the main frequency can be 1GHz, 1.5GHz, 2GHz, and so on. The time frequency represents the number of cycles per second, such as the number of oscillations of the time frequency signal between high and low states. Preferably, the time-frequency signal has a duty cycle, that is, one half of the period is high and the other half is low. In addition, the time-frequency signal may also have an unbalanced period, which is time-frequency A signal is in a high state longer than it is in a low state, and vice versa. Preferably, the phase-locked loop is used to generate a main time-frequency signal with multiple time-frequency. Preferably, the processor 100 includes a power management module that automatically adjusts the main time frequency based on a variety of factors, including dynamic detection of operating temperature, utilization of the processor 100, and system software (such as an operating system). , Basic Input Output System (BIOS) command to indicate required performance and / or energy saving indicators. In one embodiment, the power management module includes microcode of the processor 100.

The time-frequency generation logic 3502 includes a time-frequency dispersion network, or a time-frequency tree. The time-frequency tree will distribute the main time-frequency signal to the functional unit of the processor 100. As shown in Figure 35, this distribution action is to transmit the time-frequency signal 3506-1 to the instruction fetch unit 101 and the time-frequency signal 3506. -2 to instruction cache 102, time-frequency signal 3506-10 to instruction translator 104, time-frequency signal 3506-9 to rename unit 106, and time-frequency signal 3506-8 to reserved stations 108, transmitting the time-frequency signal 3506-7 to the neural network unit 121, transmitting the time-frequency signal 3506-4 to other execution units 112, transmitting the time-frequency signal 3506-3 to the memory subsystem 114, and transmitting the time-frequency signal 3506-5 is transmitted to the general-purpose register 116 and the time-frequency signal 3506-6 is transmitted to the media register 118. These signals are collectively referred to as the time-frequency signal 3506. The time-frequency tree has nodes or lines to transmit the main time-frequency signal 3506 to its corresponding functional unit. In addition, preferably, the time-frequency generation logic 3502 may include a time-frequency buffer, when a cleaner time-frequency signal is needed and / or a voltage level of the main time-frequency signal needs to be raised, especially for a distant node, The time-frequency buffer can regenerate the main time-frequency signal. In addition, each functional unit has its own child time-frequency tree, and when necessary, reproduces and / or boosts the voltage level of the corresponding main time-frequency signal 3506 received.

The neural network unit 121 includes time-frequency reduction logic 3504. The time-frequency reduction logic 3504 receives a relaxation index 3512 and a main time-frequency signal 3506-7 to generate a second time-frequency signal. The second time-frequency signal has a time-frequency. At this time, if the frequency is not the same as the main time frequency, it is in a relaxation mode. A value is reduced from the main time frequency to reduce heat generation. This value is programmed to the relaxation index 3512. The time-frequency reduction logic 3504 is similar to the time-frequency generation logic 3502. It has a time-frequency distribution network, or time-frequency tree, to distribute the second time-frequency signal to various functional blocks of the neural network unit 121. This distribution action is to divide the time The frequency signal 3508-1 is transmitted to the neural processing unit array 126, and the time-frequency signal 3508-2 is transmitted to the sequencer 128 to transmit the time-frequency signal 3508-3 to the interface logic 3514. These signals are collectively referred to as the second time-frequency signal 3508. Preferably, these neural processing units 126 include a plurality of pipeline stages 3401, as shown in Figure 34. The line stage 3401 includes a pipeline hierarchical register for receiving a second time-frequency signal 3508-1 from the time-frequency reduction logic 3504.

The neural network unit 121 also has interface logic 3514 to receive the main time-frequency signal 3506-7 and the second time-frequency signal 3508-3. The interface logic 3514 is coupled between the lower part of the front end of the processor 100 (such as the reservation station 108, the media register 118 and the general purpose register 116) and the various functional blocks of the neural network unit 121. These functional blocks are reduced frequently. Logic 3504, data random access memory 122, weight random access memory 124, program memory 129, and sequencer 128. The interface logic 3514 includes a data random access memory buffer 3522, a weighted random access memory buffer 3524, a decoder 3404 of the thirty-fourth figure, and a relaxation index 3512. The relaxation indicator 3512 is loaded with a value that specifies how slowly the neural processing unit array 126 will execute the neural network unit program instructions. Preferably, the relaxation index 3512 specifies a division value N, and the time-frequency reduction logic 3504 divides the main time-frequency signal 3506-7 by the division value to generate a second time-frequency signal 3508. Thus, at the time of the second time-frequency signal, The frequency will be 1 / N. Preferably, the value of N can be programmed into any one of a plurality of different preset values, which can cause the time-frequency reduction logic 3504 to correspondingly generate a plurality of second time-frequency signals 3508 with different time frequencies. These The time frequency is smaller than the main time frequency.

In one embodiment, the time-frequency reduction logic 3504 includes a time-frequency divider circuit for dividing the main time-frequency signal 3506-7 by the value of the relaxation index 3512. In one embodiment, the time-frequency reduction logic 3504 includes a time-frequency gate (such as an AND gate). The time-frequency gate can gate the main time-frequency signal 3506-7 through a start signal. The start signal is at every N of the main time-frequency signal. Each True values are generated only once in the cycle. Taking a circuit including a counter to generate a start signal as an example, this counter can count up to N. When the accompanying logic circuit detects that the output of the counter matches N, the logic circuit will generate a true value pulse on the second time-frequency signal 3508 and reset the counter. Preferably, the value of the mitigation index 3512 can be stylized by a framework instruction, such as the MTNN instruction 1400 of the fourteenth figure. Preferably, before the architecture program instructs the neural network unit 121 to start executing the neural network unit program, the architecture program running on the processor 100 programs the relaxation value to the relaxation index 3512, which corresponds to the thirtieth in the following. There will be a more detailed explanation at the seventh figure.

The weighted random access memory buffer 3524 is coupled between the weighted random access memory 124 and the media register 118 as a buffer for data transmission therebetween. Preferably, the weighted random access memory buffer 3524 is one or more embodiments similar to the buffer 1704 of FIG. Preferably, the part of the weighted random access memory buffer 3524 receiving data from the media register 118 uses the main time-frequency signal 3506-7 with the main time-frequency as the time-frequency, and the weight random-access memory buffer 3524 from The part of the data received by the weighted random access memory 124 uses the second time-frequency signal 3508-3 with the second time frequency as the time frequency. The second time frequency can be adjusted from the main time frequency according to the value programmed in the relaxation index 3512. Decrease or not, that is, decrease or not according to the execution of the neural network unit 121 in the relaxation or normal mode. In an embodiment, the weight random access memory 124 is a port. As described in the seventeenth figure, the weight random access memory 124 can be buffered by the media register 118 through the weight random access memory 3524. And buffered by the neural processing unit 126 or the eleventh figure 1104. Arbitrated fashion access. In another embodiment, the weight random access memory 124 is a dual port. As described in the sixteenth figure above, each port can be buffered by the media register 118 through the weight random access memory buffer 3524 and by the neural processing unit 126. OR column buffer 1104 is accessed in parallel.

Similar to the weighted random access memory buffer 3524, the data random access memory buffer 3522 is coupled between the data random access memory 122 and the media register 118 as a buffer for data transmission therebetween. Preferably, the data random access memory buffer 3522 is one or more embodiments similar to the buffer 1704 of FIG. Preferably, the part of the data random access memory buffer 3522 that receives data from the media register 118 uses the main time-frequency signal 3506-7 with the main time frequency as the time frequency, and the data random access memory buffer 3522 from The part of data received by the data random access memory 122 uses the second time-frequency signal 3508-3 with the second time frequency as the time frequency. The second time frequency can be adjusted from the main time frequency according to the value programmed in the relaxation index 3512. Decrease or not, that is, decrease or not according to the execution of the neural network unit 121 in the relaxation or normal mode. In one embodiment, the data random access memory 122 is a port. As described in the seventeenth figure, the data random access memory 122 can be buffered by the media register 118 through the data random access memory 3522. And it is buffered 1104 by the neural processing unit 126 or the column of the eleventh figure, and accessed in a stretched manner. In another embodiment, the data random access memory 122 is a dual port. As described in the sixteenth figure, each port can be buffered by the media register 118 through the data random access memory buffer 3522 and by the neural processing unit 126. OR column buffer 1104 is accessed in parallel.

Preferably, regardless of whether the data random access memory 122 and / or the weight random access memory 124 are port or dual port, the interface logic 3514 will include a data random access memory buffer 3522 and a weight random access memory buffer. 3524 synchronizes the main time-frequency domain with the second time-frequency domain. Preferably, the data random access memory 122, the weight random access memory 124 and the program memory 129 all have a static random access memory (SRAM), which contains individual read enable signals and write enable Enable signal and memory select enable signal.

As described above, the neural network unit 121 is an execution unit of the processor 100. The execution unit is a micro instruction translated from the execution of the architectural instruction in the processor or a functional unit executed by the execution of the architectural instruction itself. The execution unit receives operands from a general purpose register of the processor, such as from the general purpose register 116 and the media register 118. After the execution unit executes the micro instruction or the architecture instruction, a result is generated, and the result is written into the general purpose register. The MTNN instruction 1400 and MFNN instruction 1500 described in the fourteenth and fifteenth figures are examples of the architecture instruction 103. Micro instructions are used to implement architectural instructions. More precisely, the collective execution of one or more micro-instructions translated by the architectural instruction by the execution unit is to perform the operation specified by the architectural instruction on the input specified by the architectural instruction to produce the result defined by the architectural instruction.

Figure 36A is a timing diagram showing the processor 100 having an operation example in which the neural network unit 121 operates in one of the general modes, and this general mode operates at the main time frequency. In the timing diagram, the progress of time is from left to right. The processor 100 executes the frame at the main frequency Program. More precisely, the front end of the processor 100 (for example, the instruction fetch unit 101, the instruction cache 102, the instruction translator 104, the renaming unit 106, and the reservation station 108) is fetched at the main time frequency, decodes and issues architectural instructions to the nerve Network unit 121 and other execution units 112.

Initially, the framework program executes a framework instruction (such as MTNN instruction 1400). The processor front-end 100 issues this framework instruction to the neural network unit 121 to instruct the neural network unit 121 to start executing the neural network in its program memory 129. Unit program. Previously, the architecture program would execute a framework instruction to write a value specifying the main time frequency into the relaxation indicator 3512, even if the neural network unit was in normal mode. More precisely, the value programmed to the relaxation indicator 3512 causes the time-frequency reduction logic 3504 to generate the second time-frequency signal 3508 at the main time-frequency of the main time-frequency signal 3506. Preferably, in this example, the time-frequency buffer of the time-frequency reduction logic 3504 simply raises the voltage level of the main time-frequency signal 3506. In addition, before, the architecture program executes the architecture instruction to write data to the random access memory 122, weights the random access memory 124, and writes the neural network unit program to the program memory 129. In response to the neural network unit program MTNN instruction 1400, the neural network unit 121 will start to execute the neural network unit program at the main time frequency, because the relaxation index 3512 is programmed with the main time frequency value. After the execution of the neural network unit 121, the architecture program will continue to execute the architecture instructions at the main time frequency, including mainly using the MTNN instruction 1400 to write and / or read data random access memory 122 and weight random access memory 124 To complete the preparation of the next instance of the neural network unit program, or invocation or run.

In the example of Figure 36A, compared with the time it takes for the architecture program to complete the writing / reading of the data random access memory 122 and the weight random access memory 124, the neural network unit 121 can Significantly less time (such as a quarter of the time) to complete the execution of the neural network unit program. For example, in the case of operating at the main time frequency, the neural network unit 121 takes about 1,000 time-frequency cycles to execute the neural network unit program, but the architecture program takes about 4000 time-frequency cycles. In this way, the neural network unit 121 will be in a standby state for the rest of the time. In this example, this is a relatively long time, such as about 3000 major time frequency cycles. As shown in the example of Figure 36A, depending on the size and configuration of the neural network, the aforementioned mode will be executed again, and may continue to be executed many times. Because the neural network unit 121 is a relatively large and transistor-intensive functional unit in the processor 100, the operation of the neural network unit 121 will generate a large amount of thermal energy, especially when it operates at the main frequency.

Figure 36B is a timing diagram showing that the processor 100 has an operation example in which the neural network unit 121 operates in one of the relaxation modes. The frequency of the relaxation mode is lower than the main time frequency. The timing diagram of Figure 36B is similar to Figure 36A. In Figure 36A, the processor 100 executes an architecture program at the main time frequency. This example assumes that the architecture program and the neural network unit program in Figure 36B are the same as the architecture program and the neural network unit program in Figure 36A. However, before starting the neural network unit program, the framework program will execute a MTNN instruction 1400 to program the relaxation index 3512 with a value, which will cause the time-frequency reduction logic 3504 to generate a second time frequency that is less than the main time frequency. Two time-frequency signals 3508. In other words, the structure program will put the neural network unit 121 in the relaxation mode of the thirty-sixth figure B, instead of the general mode of the thirty-sixth figure A. In this way, the neural processing unit 126 executes the neural network unit program at the second time frequency. In the relaxation mode, the second time frequency is less than the main time frequency. In this example, it is assumed that the relaxation index 3512 is stylized with a value that specifies the second time frequency as a quarter of the main time frequency. In this way, the time taken by the neural network unit 121 to execute the neural network unit program in the mitigation mode will be four times the time it spends in the normal mode, as shown in Figures 36A and 36B. Comparing these two figures, it can be seen that the length of time that the neural network unit 121 is in the standby state is significantly shortened. In this way, the duration of the energy consumed by the neural network unit 121 in executing the program of the neural network unit 121 in FIG. 36B is approximately four times that of the execution of the program in the normal mode by the neural network unit 121 in FIG. 36A. Therefore, the thermal energy generated by the neural network unit 121 executed by the neural network unit 121 in the thirty-sixth figure in a unit time will be about a quarter of that in the thirty-sixth figure A, and has the advantages described herein.

The thirty-seventh figure is a flowchart showing the operation of the processor 100 in the thirty-fifth figure. The operation described in this flowchart is similar to the operation corresponding to the thirty-fifth, thirty-sixth A, and thirty-sixth B drawings. This process starts at step 3702.

In step 3702, the processor 100 executes the MTNN instruction 1400 to write weights into the weight random access memory 124 and write data into the data random access memory 122. The flow then proceeds to step 3704.

In step 3704, the processor 100 executes the MTNN instruction. Let 1400 program the relaxation index 3512 with a value that specifies a time frequency lower than the main time frequency, even if the neural network unit 121 is in the relaxation mode. The flow then proceeds to step 3706.

In step 3706, the processor 100 executes the MTNN instruction 1400 to instruct the neural network unit 121 to start executing the neural network unit program, which is similar to the manner shown in FIG. 36B. The flow then proceeds to step 3708.

In step 3708, the neural network unit 121 starts executing the neural network unit program. At the same time, the processor 100 executes the MTNN instruction 1400 and writes new weights into the weight random access memory 124 (may also write new data into the data random access memory 122), and / or executes the MFNN instruction 1500 The results are read from the data random access memory 122 (the results may also be read from the weight random access memory 124). The flow then proceeds to step 3712.

In step 3712, the processor 100 executes the MFNN instruction 1500 (for example, the read status register 127) to detect that the neural network unit 121 has finished program execution. Assuming that the framework program selects a good value of the mitigating index 3512, the time taken by the neural network unit 121 to execute the neural network unit program will be the same as that of the processor 100 executing part of the framework program to access the weight random access memory 124 and / Or the time taken by the data random access memory 122 is shown in FIG. 36B. The flow then proceeds to step 3714.

In step 3714, the processor 100 executes the MTNN instruction 1400 and uses a value to program the relaxation index 3512. This value specifies the main time frequency, even if the neural network unit 121 is in the general mode. Take over Come to step 3716.

In step 3716, the processor 100 executes the MTNN instruction 1400 to instruct the neural network unit 121 to start executing the neural network unit program, which is similar to the manner shown in FIG. 36A. The flow then proceeds to step 3718.

In step 3718, the neural network unit 121 starts executing the neural network unit program in a general mode. This process ends at step 3718.

As mentioned above, compared with executing the neural network unit program in the general mode (that is, the main frequency of the processor is executed), the execution in the relaxation mode can disperse the execution time and avoid high temperature. Further, when the neural network unit executes a program in a relaxation mode, the neural network unit generates thermal energy at a lower time frequency, and this thermal energy can smoothly pass through the neural network unit (such as a semiconductor device, a metal layer and a lower layer). The substrate) is discharged from the surrounding packages and cooling mechanisms (such as heat sinks, fans). Therefore, the devices (such as transistors, capacitors, and wires) in the neural network unit are more likely to operate at lower temperatures. Overall, operating in mitigation mode also helps reduce device temperature in other parts of the processor die. The lower operating temperature, especially for the junction temperature of these devices, can reduce the generation of leakage current. In addition, because the amount of current flowing in per unit time is reduced, the inductance noise and IR voltage drop noise are also reduced. In addition, the temperature decrease has a positive impact on the negative bias temperature instability (NBTI) and positive bias instability (PBSI) of the metal-oxide-semiconductor field-effect transistor (MOSFET) in the processor, which can improve reliability and / Or device and processor life. Reduced temperature and ease Joule heating and electromigration effects in the metal layer of the controller.

Communication mechanism between architecture program and non-architecture program of neural network unit shared resources

As mentioned above, in the examples of the twenty-fourth to twenty-eight and thirty-five to thirty-seven figures, the resources of the data random access memory 122 and the weight random access memory 124 are shared. The neural processing unit 126 and the front end of the processor 100 share a data random access memory 122 and a weight random access memory 124. More precisely, the neural processing unit 126 and the front end of the processor 100, such as the media register 118, will both read and write the data random access memory 122 and the weight random access memory 124. In other words, the architecture program running on the processor 100 and the neural network unit program running on the neural network unit 121 share data random access memory 122 and weight random access memory 124, and in some cases As mentioned before, the flow between the framework program and the neural network unit program needs to be controlled. The resources of the program memory 129 are also shared to a certain extent, because the framework program will write them and the sequencer 128 will read them. The embodiment described herein provides a high-performance solution to control the flow of accessing shared resources between the architecture program and the neural network unit program.

In the embodiment described herein, the neural network unit program is also referred to as non-architecture program, the neural network unit instruction is also referred to as non-architecture instruction, and the neural network unit instruction set (also referred to as neural processing as described above) Unit instruction set) is also called non-architecture instruction set. Non-architecture instruction sets are different from architectural instruction sets. In the embodiment where the processor 100 includes the instruction translator 104 to translate the architectural instructions into micro instructions, the non-architecture instruction set is not included. Same as micro instruction set.

The thirty-eighth figure is a block diagram showing the sequencer 128 of the neural network unit 121 in detail. The sequencer 128 provides a memory address to the program memory 129 to select non-architecture instructions provided to the sequencer 128, as described above. As shown in Figure 38, the memory address is loaded into a program counter 3802 of the sequencer 128. The sequencer 128 generally increments in sequence from the address of the program memory 129, unless the sequencer 128 encounters a non-architecture instruction, such as a loop or branch instruction, in which case the sequencer 128 will The program counter 3802 is updated to the target address of the control instruction, that is, the address of the non-framework instruction located at the target of the control instruction. Therefore, the address 131 loaded in the program counter 3802 specifies the address in the program memory 129 of the non-framework instruction of the non-framework program currently being fetched for execution by the neural processing unit 126. The value of the program counter 3802 can be obtained by the framework program through the neural network unit program counter field 3912 of the state register 127, as described in the following figure 39. In this way, based on the progress of the non-architecture program, the architecture program can determine the position where the data random storage memory 122 and / or the weight random access memory 124 read / write data.

The sequencer 128 also includes a loop counter 3804. This loop counter 3804 will operate with a non-framework loop instruction, such as the loop to address 1 instruction at address 10 and the twenty-eighth in the 26th A diagram. In the figure, the loop from address 11 to instruction 1. In the examples of the twenty-sixth A and the twenty-eighth figures, the loop counter 3804 contains the value specified by the non-architecture initialization instruction at address 0, for example, the value 400 is loaded. Each time the sequencer 128 encounters a loop instruction and jumps to the target instruction (as shown in Figure 26A) The multiply accumulate instruction at address 1 or the maxwacc instruction at address 1 in the twenty-eighth figure), sequencer 128 will decrement the loop counter 3804. Once the loop counter 3804 is reduced to zero, the sequencer 128 moves to the next non-architecture instruction that is ordered. In another embodiment, when a loop instruction is first encountered, a loop count value specified in the loop instruction is loaded into the loop counter to eliminate the need to initialize the loop counter 3804 with a non-architecture initialization instruction. Therefore, the value of the loop counter 3804 indicates the number of times that the loop group of the non-schema program has to be executed. The value of the loop counter 3804 can be obtained by the framework program through the loop count field 3914 of the state register 127, as shown in the subsequent thirty-ninth figure. In this way, based on the progress of the non-architecture program, the architecture program can determine the position where the data random storage memory 122 and / or the weight random access memory 124 read / write data. In one embodiment, the sequencer includes three additional loop counters to match the nested loops in the non-architecture program. The values of the three loop counters can also be read through the state register 127. There is a bit in the loop instruction to indicate which of the four loop counters is provided for the current loop instruction.

The sequencer 128 also includes a number of iterations counter 3806. The iteration counter 3806 is matched with non-architectural instructions, such as the multiply-accumulate instruction of address 2 in the fourth, ninth, twenty, and twenty-sixth A and the maxwacc instruction of address 2 in the twenty-eighth illustration. It will be referred to as the "execute" instruction hereinafter. In the foregoing example, each execution instruction specifies an execution count 511, 511, 1023, 2 and 3. When the sequencer 128 encounters an execution instruction specifying a non-zero iteration count, the sequencer 128 loads the iteration number counter 3806 with the specified value. In addition, The sequencer 128 generates an appropriate micro-operation 3418 to control the execution of logic in the pipeline stage 3401 of the neural processing unit 126 in the thirty-fourth figure, and decrements the iteration counter 3806. If the number of iterations counter 3806 is greater than zero, the sequencer 128 will again generate an appropriate micro operation 3418 to control the logic in the neural processing unit 126 and decrement the number of iterations counter 3806. The sequencer 128 continues to operate in this manner until the value of the number of iterations counter 3806 returns to zero. Therefore, the value of the iteration number counter 3806 is the number of operations to be executed specified in the non-architecture execution instruction (such as multiply accumulating the accumulated value and a data / weight text, take the maximum value, add the total operation, etc.). The value of the iteration count counter 3806 can be obtained by the framework program through the iteration count field 3916 of the state register 127, as described in the following figure 39. In this way, based on the progress of the non-architecture program, the architecture program can determine the position where the data random storage memory 122 and / or the weight random access memory 124 read / write data.

The thirty-ninth figure is a block diagram showing the fields of the control and status register 127 of the neural network unit 121. These fields include the address 2602 of the weighted random access memory row recently written by the neural processing unit 126 executing the non-structural program, and the position of the weighted random access memory row recently read by the neural processing unit 126 executing the non-structural program. Address 2604, the address 2606 of the random access memory row of the data recently written by the non-structural program executed by the neural processing unit 126, and the address of the random access memory row of the data recently read by the neural processing unit 126 executed by the non-architecture program 2608, as shown in the aforementioned twenty-sixth B figure. In addition, these fields also include a neural network unit program counter 3912 field, a loop counter 3914 field, and an iteration number counter 3916 field. As mentioned before, The framework program can read the data in the state register 127 to the media register 118 and / or the general register 116, for example, through the MFNN instruction 1500 to read the program including the neural network unit program counter 3912, the loop counter 3914 and The value of the number of iterations counter 3916. The value of the program counter field 3912 reflects the value of the program counter 3802 in the thirty-eighth figure. The value of the lap counter field 3914 reflects the value of the lap counter 3804. The value of the iteration number counter field 3916 reflects the value of the iteration number counter 3806. In one embodiment, the sequencer 128 updates the program counter field 3912, the loop counter field 3914, and the iteration number counter field each time the program counter 3802, the loop counter 3804, or the iteration number counter 3806 is adjusted. The value of bit 3916, so when the framework program reads the value of these fields will be the current value. In another embodiment, when the neural network unit 121 executes an architectural instruction to read the state register 127, the neural network unit 121 only obtains the values of the program counter 3802, the loop counter 3804 and the iteration number counter 3806 and It provides back-to-architecture instructions (eg, to media register 118 or general purpose register 116).

From this, it can be found that the value of the field in the state register 127 of the thirty-ninth figure can be understood as the information of the execution progress of the non-architecture instruction executed by the neural network unit. Some specific aspects of the execution progress of non-architecture programs, such as the value of the program counter 3802, the value of the loop counter 3804, the value of the iteration counter 3806, and the weight of the most recent read / write random access memory 124 address 125 2602/2604, and the recently read / written data random access memory 122 address 123 field 2606/2608 have been described in the previous chapter Described. The architecture program running on the processor 100 can read the progress value of the non-architecture program in the thirty-ninth figure from the state register 127 and use this information to make decisions, for example, through architecture instructions such as comparison and branch instructions. For example, the architecture program may decide to read / write data / weight rows for the data random access memory 122 and / or the weight random access memory 124 to control the data random access memory 122 or the weight The inflow and outflow of the data in the random access memory 124 is especially for the large data sets and / or the overlapping execution of different non-architecture instructions. These examples of making decisions using the framework program can refer to the description in the previous and subsequent chapters of this article.

For example, as described in Figure 26A above, the structural program sets the non-structural program to write the result of the convolution operation back to the row above the convolution kernel 2402 in the data random access memory 122 (as above row 8) When the neural network unit 121 writes the result using the address of the recently written data random access memory 122 row 2606, the architecture program reads the result from the data random access memory 122.

In another example, as described in Figure 26B above, the framework program uses the information from the status register 127 field of Figure 38 to confirm that the non-schema program will use the data array 2404 of Figure 24 Divide into 5 512 x 1600 data blocks to perform the progress of the convolution operation. The architecture program writes the first 512 x 1600 data block of this 2560 x 1600 data array to the weight random access memory 124 and starts the non-architecture program. Its loop count is 1600 and the weight random access memory 124 initializes the output. Column is 0. When the neural network unit 121 executes the non-architecture program, the architecture program reads the status register 127 to confirm the weight of the recent write row 2602 of the random access memory 124, so that the architecture program can read the non-architecture program. Write the valid convolution operation result, and overwrite the effective convolution operation result with the next 512 x 1600 data block after reading. In this way, complete the non-architecture program for the first 512 x 1600 in the neural network unit 121 After the data block is executed, the processor 100 can immediately update the non-architecture program and start the non-architecture program again to execute the next 512 x 1600 data block when necessary.

In another example, suppose the architecture program causes the neural network unit 121 to execute a series of typical neural network multiply-accumulate activation functions, wherein the weights are stored in the weight random access memory 124 and the results are written back to the data random Access memory 122. In this case, after the architecture program reads a row of the weight random access memory 124, it will not read it again. In this way, after the current weight has been read / used by the non-architecture program, the architecture program can be used to start overwriting the new weight with the weight on the random access memory 124 to provide the next example of the non-architecture program ( Such as the next neural network layer). In this case, the architecture program reads the state register 127 to obtain the address of the most recent read row 2604 of the weighted random access memory to determine the write of the new weighted reassembly in the weighted random access memory 124. position.

In another example, suppose the architecture program knows that the non-architecture program includes an execution instruction with a large iteration count, such as the non-architecture multiply accumulate instruction at address 2 in the twentieth figure. In this case, the architecture program needs to know the number of iteration counts 3916 to know how many time-frequency cycles are needed to complete this non-architecture instruction to determine which of the two or more actions the architecture program will take next. . For example, if it takes a long time to complete execution, the framework program You give up control to another framework program, such as the operating system. Similarly, it is assumed that the architecture program knows that the non-architecture program includes a loop group with a considerable loop count, such as the non-architecture program of FIG. In this case, the architecture program needs to know the loop count of 3914 to know how many time-frequency cycles are needed to complete this non-architecture instruction to determine which of two or more actions to take next.

In another example, suppose that the structural program causes the neural network unit 121 to perform a common source operation similar to that described in Figures 27 and 28, where the data to be common source is stored in the weight random access memory 124 The result is written back to the weight random access memory 124. However, unlike the examples in figures 27 and 28, it is assumed that the results of this example will be written back to the top 400 rows of the weighted random access memory 124, such as rows 1600 to 1999. In this case, after the non-architecture program finishes reading the four rows of weighted random access memory 124 data it wants to share in common, the non-architecture program will not read it again. Therefore, once the current four rows of data have been read / used by the non-schema program, you can use the framework program to start the new data (such as the weight of the next example of the non-schema program, for example, perform a typical The multiplication and accumulation start function calculation is a non-structural program) overwriting the data of the weight random access memory 124. In this case, the architecture program will read the status register 127 to obtain the address of the most recent read row 2604 of the weighted random access memory to determine the new weight to be written to the position of the weighted random access memory 124. .

Time recurrent neural network acceleration

Traditional feed-forward neural networks do not have memory to store the network's previous inputs. Feedforward neural networks are often used to perform tasks The input to the network over time is independent of each other, and the same is true for multiple outputs. In contrast, time-recurrent neural networks often help perform tasks where the order of input to the neural network over time is important in the task. (The sequence here is often referred to as a time step.) Therefore, a time-recurrent neural network includes a conceptual memory or internal state to load information generated by the network in response to calculations performed by previous inputs in the sequence The output of the time recurrent neural network is related to this internal state and the input of the next time step. The following tasks, such as speech recognition, language models, text generation, language translation, image description generation, and some forms of handwriting recognition, are good examples of how time-recurrent neural networks can perform.

Three examples of conventional time recurrent neural networks are Elman time recurrent neural network, Jordan time recurrent neural network and long short-term memory (LSTM) neural network. The Elman time recurrent neural network contains content nodes to memorize the hidden layer state of the time recurrent neural network in the current time step. This state will be used as the input to the hidden layer in the next time step. Jordan time recurrent neural network is similar to Elman time recurrent neural network, except that the content nodes in it will remember the state of the output layer of the time recurrent neural network instead of the state of the hidden layer. The long-term and short-term memory neural network includes a long-term and short-term memory layer composed of long-term and short-term memory cells. Each long-term and short-term memory cell has a current state and a current output in one of the current time steps, and a new state and a new output in a new or subsequent time step. The long-term and short-term memory cells include an input gate, an output gate, and a forget gate. The forget gate can cause a neuron to lose its memorized state. These three types of temporal recurrent neural networks will be described in more detail in subsequent chapters. description.

As described in this article, for a time-recurrent neural network, such as Elman or Jordan time-recurrent neural network, each time the neural network unit executes, it uses a time step to obtain a set of input layer node values, and performs the necessary calculations so that It is propagated through a time-recursive neural network to generate output layer node values and hidden layer and content layer node values. Therefore, the input layer node values will be related to the time steps of calculating hidden, output and content layer node values; and hidden, output and content layer node values will be related to the time steps that generate these node values. The input layer node values are sample values of the system simulated by the time-recurrent neural network, such as images, voice samples, and snapshots of business market data. For long-term and short-term memory neural networks, each execution of the neural network unit uses a time step to obtain a set of memory cell input values and perform the necessary calculations to generate memory cell output values (as well as memory cell states and input gates, Forget gate and output gate values), which can also be understood as the transmission of memory cell input values through long and short-term memory cells. Therefore, the input value of the memory cell will be related to the time step of calculating the state of the memory cell and the values of the input gate, the forget gate and the output gate; and the memory state and the input gate, the value of the forget gate and the output gate will be related to the time of generating these node values. step.

The content layer node value, also called the state node, is the state value of the neural network. This state value is based on the input layer node value associated with the previous time step, not only the input layer node value associated with the current time step. The calculation performed by the neural network unit for the time step (for example, the calculation of the hidden layer node value of the Elman or Jordan time recurrent neural network) is a function of the content layer node value generated by the previous time step number. Therefore, the network state value (content node value) at the beginning of the time step will affect the output layer node value generated during the time step. In addition, the network status value at the end of the time step will be affected by the input node value of this time step and the network status value at the beginning of the time step. Similarly, for long-term and short-term memory cells, the memory cell state value is related to the memory cell input value of the previous time step, rather than only the memory cell input value of the current time step. Because the calculation performed by the neural network unit on the time step (such as the calculation of the next memory cell state) is a function of the memory cell state value generated by the previous time step, the network state value (memory cell state value) at the beginning of the time step Affects the output value of the memory cell in this time step, and the network state value at the end of this time step will be affected by the memory cell input value and the previous network state value at this time step.

Figure 40 is a block diagram showing an example of an Elman time recurrent neural network. The Elman time recurrent neural network of the fortieth figure includes input layer nodes, or neurons, labeled D0, D1 to Dn, collectively referred to as multiple input layer nodes D and individually collectively referred to as input layer nodes D; hidden layer nodes / Neurons, labeled Z0, Z1 to Zn, collectively referred to as multiple hidden layer nodes Z and individually referred to as hidden layer nodes Z; output layer nodes / neurons, labeled Y0, Y1 to Yn, collectively referred to as multiple outputs The layer node Y is collectively referred to as the output layer node Y; and the content layer nodes / neurons are labeled C0, C1 to Cn, collectively referred to as a plurality of content layer nodes C, and individually referred to as the content layer nodes C. In the example of the Elman time recurrent neural network of the fortieth figure, each hidden layer node Z has an input connected to the output of each input layer node D, and has an input connected to each The output of the content layer node C; each output layer node Y has an input connected to the output of each hidden layer node Z; and each content layer node C has an input connected to an output of a corresponding hidden layer node Z.

In many ways, Elman time recurrent neural networks operate similarly to traditional feedforward artificial neural networks. That is, for a given node, each input link of this node will have an associated weight; the value received by a node at an input link will be multiplied by the associated weight to produce a product; this node will Adding the products associated with all input links to generate a total (the total may also include an offset term); generally, an activation function is performed on the total to generate the node's output value, which Sometimes called the start value of this node. For traditional feedforward networks, data always flows from the input layer to the output layer. That is, the input layer provides a value to the hidden layer (usually there will be multiple hidden layers), and the hidden layer will generate its output value to provide to the output layer, and the output layer will generate an output that can be accessed.

However, unlike the traditional feedforward network, the Elman time recurrent neural network also includes some feedback links, that is, the link from the hidden layer node Z to the content layer node C in the fortieth figure. The Elman time recurrent neural network works as follows. When the input layer node D provides an input value to the hidden layer node Z at a new time step, the content node C provides a value to the hidden layer node Z. This value is the hidden layer node Z. The output value corresponding to the previous input, that is, the current time step. In this sense, the content node C of the Elman time recurrent neural network is a memory based on the input value of the previous time step. The forty-first and forty-two graphs will execute the Elman time recurrent neural network associated with the forty graphs. An operation example of the computational neural network unit 121 will be described.

To illustrate the present invention, the Elman time recurrent neural network is a time recurrent neural network including at least one input node layer, a hidden node layer, an output node layer and a content node layer. For a given time step, the content node layer stores the results generated by the hidden node layer in the previous time step and fed back to the content node layer. The result of the feedback to the content layer may be the execution result of the startup function or the result of the hidden node layer performing the accumulation operation without executing the startup function.

The forty-first diagram is a block diagram showing the random access memory 122 and weights of the data of the neural network unit 121 when the neural network unit 121 performs the calculation of the Elman time recurrent neural network associated with the forty diagram. An example of data configuration in the random access memory 124 is. In the example of Figure 41, it is assumed that the Elman time recurrent neural network of Figure 40 has 512 input nodes D, 512 hidden nodes Z, 512 content nodes C, and 512 output nodes Y. In addition, it is also assumed that this Elman time recurrent neural network is fully connected, that is, all 512 input nodes D are connected to each hidden node Z as input, all 512 content nodes C are connected to each hidden node Z as input, and all 512 The hidden node Z is connected to each output node Y as an input. In addition, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, such as adopting a wide configuration. Finally, this example assumes that the weights associated with the links from the content node C to the hidden node Z are all values 1, so there is no need to store these one weight values.

As shown in the figure, the 512 rows (columns 0 to 511) below the weighted random access memory 124 are loaded and associated with the input node D and the hidden node. The weight of the connection between the hidden nodes Z. More precisely, as shown in the figure, column 0 is the weight associated with the input link from the input node D0 to the hidden node Z, that is, the text 0 will load the link associated with the input node D0 and the hidden node Z0. Text 1 will load the weight associated with the connection between input node D0 and hidden node Z1, text 2 will load the weight associated with the connection between input node D0 and hidden node Z2, and so on, text 511 will load association The weight of the link between the input node D0 and the hidden node Z511; column 1 contains the weight associated with the input link from the input node D1 to the hidden node Z, that is, the text 0 will load the input node D1 and the hidden node Z0 The weight of the link between texts, text 1 will load the weight associated with the link between input node D1 and hidden node Z1, the text 2 will load the weight of the link associated with input node D1 and hidden node Z2, and so on, text 511 Will load the weights associated with the link between input node D1 and hidden node Z511; up to column 511, column 511 will load the weights associated with the input link from input node D511 to hidden node Z, also That is, text 0 will be loaded with the weight associated with the connection between the input node D511 and the hidden node Z0, text 1 will be loaded with the weight associated with the connection between the input node D511 and the hidden node Z1, and text 2 will be loaded with the input node D511 and the hidden node The weight of the link between the hidden nodes Z2, and so on, the text 511 will load the weight associated with the link between the input node D511 and the hidden node Z511. This configuration and use is similar to the embodiment described above corresponding to the fourth to sixth A figures.

As shown in the figure, the subsequent 512 columns (columns 512 to 1023) of the weight random storage memory 124 load the weights associated with the connection between the hidden node Z and the output node Y in a similar manner.

The data random access memory 122 is loaded with Elman time recurrent neural network node values for a series of time steps. Further, the data random access memory 122 loads the node values of a given time step in groups of three rows. As shown in the figure, a data random access memory 122 with 64 rows is taken as an example. The data random access memory 122 can load node values for 20 different time steps. In the example of the forty-first figure, columns 0 to 2 are loaded with node values for time step 0, columns 3 to 5 are loaded with node values for time step 1, and so on, and columns 57 to 59 are loaded for time steps 19 node value used. The first column in each group is the value of the input node D at this time step. The second column in each group contains the value of the hidden node Z at this time step. The third column in each group contains the value of output node Y at this time step. As shown in the figure, each row of the data random access memory 122 is loaded with the node value of its corresponding neuron or neural processing unit 126. That is, row 0 loads the node values associated with nodes D0, Z0, and Y0, and its calculation is performed by the neural processing unit 0; row 1 loads the node values associated with nodes D1, Z1, and Y1. The calculation is Executed by the neural processing unit 1; and so on, row 511 loads the node values associated with the nodes D511, Z511 and Y511, and the calculation is performed by the neural processing unit 511. This part corresponds to the forty-second figure in the following. There will be more detailed instructions.

As indicated in the forty-first figure, for a given time step, the value of the hidden node Z located in the second column of each group of three columns of memory will be the value of the content node C of the next time step. In other words, the value of node Z calculated and written by the neural processing unit 126 in a time step will become the neural processing unit 126 in the next time step The value of node C (together with the value of input node D for this next time step) used to calculate the value of node Z. The initial value of content node C (the value of node C used to calculate the value of node Z in column 1 at time step 0) is assumed to be zero. This will be explained in more detail in the subsequent sections of the non-structural program corresponding to Figure 42.

Preferably, the value of the input node D (the values of columns 0, 3 in the example in FIG. 41, and so on to the value of column 57) is written by the architecture program running on the processor 100 through the MTNN instruction 1400 / The data random access memory 122 is filled in, and is read / used by a non-architecture program running on the neural network unit 121, such as the non-architecture program of FIG. 42. Conversely, the values of the hidden / output nodes Z / Y (columns 1 and 2, 4 and 5 in the example in Figure 41, and so on to the values of columns 58 and 59) are executed by the neural network The non-architecture program of the unit 121 writes / fills in the data random access memory 122, and is read / used by the architecture program running on the processor 100 through the MFNN instruction 1500. The example in Figure 41 assumes that the framework program will perform the following steps: (1) For 20 different time steps, fill the value of the input node D into the data random access memory 122 (rows 0, 3, according to (Similarly to column 57); (2) start the non-architecture program in Figure 42; (3) detect whether the non-architecture program is completed; (4) read out the output node Y from the data random access memory 122 Values (columns 2, 5, and so on to column 59); and (5) repeat steps (1) to (4) several times until the task is completed, such as the calculation required to identify the words of a mobile phone user.

In another execution method, the framework program will perform the following steps: (1) For a single time step, fill in the value of the input node D Enter the data random access memory 122 (such as column 0); (2) Start the non-architecture program (a modified version of one of the non-architecture programs in Figure 42 without looping, and only access the data random memory 122 in a single group of three rows); (3) detecting whether the non-structural program has been executed; (4) reading the value of the output node Y from the data random access memory 122 (such as row 2); and (5) repeating Steps (1) to (4) several times until the task is completed. Which of these two methods is optimal depends on the sampling method of the input value of the time recurrent neural network. For example, if this task allows the input to be sampled at multiple time steps (for example, about 20 time steps) and perform calculations, the first method is ideal because this method may bring more computing resources efficiency and / Or better performance, but if this task only allows sampling in a single time step, you need to use the second method.

The third embodiment is similar to the foregoing second method, but different from the second method using a single set of three rows of data random access memory 122, the non-structural program of this method uses multiple sets of three rows of memory, that is, in the Each time step uses a different set of three columns of memory, this section is similar to the first method. In this third embodiment, preferably, the architecture program includes a step before step (2). In this step, the architecture program will update the non-architecture program before it is started, for example, in the address 1 instruction. The 122 rows of data random access memory are updated to point to the next set of three rows of memory.

Figure 42 is a table showing a program stored in the program memory 129 of the neural network unit 121. This program is executed by the neural network unit 121 and uses data and weights according to the configuration of figure 41 To achieve Elman time recurrent neural network. Figure 42 And the forty-fifth, forty-eight, fifty-one, fifty-four, and fifty-seven diagrams) of the non-architecture program are as described above (for example, MULT-ACCUM), loop (LOOP), initialization ( INITIALIZE) instruction, the following paragraphs assume that these instructions are consistent with the previous description, unless there is a different description.

The example program in Figure 42 contains 13 non-framework instructions, located at addresses 0 to 12, respectively. The instruction at address 0 (INITIALIZE NPU, LOOPCNT = 20) clears the accumulator 202 and initializes the loop counter 3804 to the value 20 to execute the loop group 20 times (the instructions at addresses 4 to 11). Preferably, this initialization instruction will also make the neural network unit 121 in a wide configuration. In this way, the neural network unit 121 will be configured as 512 neural processing units 126. As described in the subsequent chapters, during the execution of instructions at addresses 1 to 3 and addresses 7 to 11, these 512 neural processing units 126 operate as 512 corresponding hidden layer nodes Z, and at address 4 During the execution of instructions from 6 to 6, these 512 neural processing units 126 operate as 512 corresponding output layer nodes Y.

The instructions at addresses 1 to 3 do not belong to the loop group of the program and will only be executed once. These instructions calculate the initial value of node Z in the hidden layer and write it into the data random access memory 122 in column 1 for the first execution of the instructions at addresses 4 to 6, to calculate the first time step (time step 0) output node Y. In addition, the values of these hidden layer nodes Z calculated by the instructions 1 to 3 and written into the data random access memory 122 row 1 will become the values of the content layer node C for the 7th and 8th instructions. Use it once to calculate the value of node Z in the hidden layer for the second time step (time step 1).

During the execution of the instructions at addresses 1 and 2, each of the 512 neural processing units 126 will perform 512 multiplication operations, and will place 512 input nodes located at 122 rows 0 of the data random access memory. The value of D is multiplied by the weight corresponding to the row of the neural processing unit 126 in the columns 0 to 511 of the weighted random access memory 124 to generate 512 multiply accumulations and add the accumulator 202 to the corresponding neural processing unit 126. During the execution of the instruction at the address 3, the values of the 512 accumulators 202 of the 512 neural processing units will be transferred and written into the data random access memory 122 row 1. In other words, the output instruction at address 3 writes the value of the accumulator 202 of each of the 512 neural processing units 512 into the data random access memory 122 column 1. This value is the initial hidden layer Z value. Subsequently, this instruction clears the accumulator 202.

The operations performed by the instructions at addresses 1 to 2 of the non-architecture program in Figure 42 are similar to the operations performed by the instructions at addresses 1 to 2 of the non-architecture instruction in Figure 4. Further, the instruction at address 1 (MULT_ACCUM DR ROW 0) will instruct each of the 512 neural processing units 126 to read the corresponding text in column 0 of the data random access memory 122 The work register 208 reads the corresponding text in column 0 of the weight random access memory 124 into its multiplex register 705, multiplies the data text and the weight text to generate a product, and adds the product to the accumulator 202. The instruction at address 2 (MULT-ACCUM ROTATE, WR ROW +1, COUNT = 511) instructs each of the 512 neural processing units 126 to transfer the text from the adjacent neural processing unit 126 into its multiplex Register 208 (using a 512-character rotator composed of the collective operation of 512 multiplexing registers 208 of the neural network unit 121, These registers are the instructions of address 1 to instruct the data random access memory 122 to be read into the register), and the corresponding text in the row below the weight random access memory 124 is read into its multiplexing. The register 705 multiplies the data text and the weight text to generate a product, adds the product to the accumulator 202, and performs the foregoing operation 511 times.

In addition, a single non-architecture output instruction (OUTPUT PASSTHRU, DR OUT ROW 1, CLR ACC) at address 3 in the forty-second figure will write the operation of the start function instruction and the write output at addresses 3 and 4 in the fourth figure Instruction merging (although the program in Figure 42 passes the value of accumulator 202, the program in Figure 4 performs an activation function on the value of accumulator 202). That is to say, in the program of the forty-second figure, the start function executed on the value of the accumulator 202, if any, is specified in the output instruction (also specified in the output instructions at addresses 6 and 11), and It is not specified in the program of the fourth figure in a different non-framework startup function instruction. Another embodiment of the non-architecture program in the fourth figure (and twentieth, twenty-sixth A, and twenty-eight) is to start the operation of the function instruction and write the output instruction (such as address 3 in the fourth figure). And 4) combined into a single non-architecture output instruction as shown in FIG. 42 also belongs to the scope of the present invention. The example in Figure 42 assumes that the nodes of the hidden layer (Z) will not execute the start function on the accumulator value. However, embodiments in which the hidden layer (Z) executes the start function on the accumulator value are also within the scope of the present invention. These embodiments can use instructions at addresses 3 and 11 to perform operations, such as S-type, hyperbolic tangent, correction functions, etc .

Compared to addresses 1 to 3, the instructions are executed only once, and instructions 4 to 11 are located in the program loop and will be executed several times. The number is specified by the loop count (for example, 20). The first nineteen executions of the instructions at addresses 7 to 11 calculate the value of the hidden layer node Z and write it to the data random access memory 122 for the second to twenty executions of the instructions at addresses 4 to 6 To calculate the output layer node Y for the remaining time steps (time steps 1 to 19). (The last / twentieth execution of the instructions at addresses 7 to 11 is to calculate the value of the hidden layer node Z and write it to the data random access memory row 122, 61. However, these values are not used.)

In the first execution of the instructions at addresses 4 and 5 (MULT-ACCUM DR ROW +1, WR ROW 512 and MULT-ACCUM ROTATE, WR ROW +1, COUNT = 511), this corresponds to time step 0. Each of the 512 neural processing units 126 performs 512 multiplication operations to randomize the data to the values of 512 hidden nodes Z in row 1 of the memory 122 (these values are from addresses 1 to 3) Instructions are generated and written in a single execution) Multiplied by the weights of rows 512 to 1023 of the random access memory 124 corresponding to the trip of this neural processing unit 126 to generate 512 multiplying accumulations and adding to the corresponding neural processing unit 126 The accumulator 202. In the first execution of the instruction at address 6 (OUTPUT ACTIVATION FUNCTION, DR OUT ROW +1, CLR ACC), an activation function will be executed for these 512 accumulated values (for example, S-type, hyperbolic tangent, correction function) To calculate the value of node Y in the output layer, the execution result will be written into row 2 of the data random access memory 122.

In the second execution of the instructions at addresses 4 and 5 (corresponding to time step 1), each of the 512 neural processing units 126 will perform 512 multiplication operations to randomly access the data in memory. The values of 512 hidden nodes Z in column 4 of 122 (these values are determined by Generated and written for the first execution of the instructions at addresses 7 to 11) Multiplied by the weight of the rows 512 to 1023 of the random access memory 124 corresponding to the row of this neural processing unit 126 to generate 512 multiplied accumulations plus In the accumulator 202 corresponding to the neural processing unit 126, in the second execution of the instruction at the address 6, a start function is executed for the 512 accumulated values to calculate the value of the output layer node Y. The result is written Into the data random access memory 122 column 5; in the third execution of the instructions at addresses 4 and 5 (corresponding to time step 2), each of the 512 neural processing units 126 will execute 512 multiplication operations, multiplying the values of 512 hidden nodes Z of row 7 of data random access memory 122 (these values are generated and written by the second execution of instructions at addresses 7 to 11) by the weight The weights of the rows 512 to 1023 of the random access memory 124 corresponding to the trip of this neural processing unit 126 to generate 512 multiply-accumulate and add to the accumulator 202 of the corresponding neural processing unit 126. In three executions, the 512 accumulated values will be Execute a startup function to calculate the value of node Y in the output layer. The result is written into column 8 of the data random access memory 122; and so on, during the twentieth execution of the instructions at addresses 4 and 5 (corresponding to At time step 19), each of the 512 neural processing units 126 will perform 512 multiplication operations to randomize the data to the values of the 512 hidden nodes Z of the 58 rows in the memory 122 (these values are Generated and written by the nineteenth execution of the instructions at addresses 7 to 11) Multiplied by the weight of columns 512 to 1023 of the random access memory 124 corresponding to the row of this neural processing unit 126 to generate 512 The multiply-accumulate is added to the accumulator 202 of the corresponding neural processing unit 126, and in the twentieth execution of the instruction at address 6, a start is performed for the 512 accumulated values The function is calculated to calculate the value of node Y in the output layer, and the execution result is written into row 59 of data random access memory 122.

In the first execution of the instructions at addresses 7 and 8, each of the 512 neural processing units 126 accumulates the values of the 512 content nodes C in the random access memory 122 in column 1 to it. Accumulator 202. These values are generated by a single execution of the instructions at addresses 1 to 3. Further, the instruction at address 7 (ADD_D_ACC DR ROW +0) will instruct each of the 512 neural processing units 126 to randomly access data in the current row of the memory 122 (in the first execution process) The corresponding text in column 0) is read into its multiplexer register 208, and this text is added to the accumulator 202. The instruction at address 8 (ADD_D_ACC ROTATE, COUNT = 511) instructs each of the 512 neural processing units 126 to transfer the text from the adjacent neural processing unit 126 into its multiplexing register 208 (using the The 512 multiplex register 208 of the neural network unit 121 is a 512-character rotator composed of collective operations. These multiplex registers are instructions at address 7 to read data into the random access memory 122 row. Register), add this text to the accumulator 202, and perform the aforementioned operation 511 times.

In the second execution of the instructions at addresses 7 and 8, each of the 512 neural processing units 126 will randomly accumulate the value of the 512 content nodes C of the memory 4 in row 4 of the memory 122. To its accumulator 202, these values are generated and written by the first execution of instructions at addresses 9 to 11; in the third execution of instructions at addresses 7 and 8, the 512 neural processing units 126 Each of the neural processing units 126 will randomly fetch the data from the 512 contents of the 7th memory 122 The value of node C is accumulated to its accumulator 202. These values are generated and written by the second execution of the instructions at addresses 9 to 11; and so on, the twentieth of the instructions at addresses 7 and 8 are analogous. During execution, each of the 512 neural processing units 126 will accumulate the data of the 512 content nodes C of the memory 122 in the random rows of memory 122 to its accumulator 202. These values are determined by the address Generated and written in the nineteenth execution of instructions 9 to 11.

As mentioned above, the example of the forty-second figure assumes that the weight of the link associated with the content node C to the hidden layer node Z has a value of one. However, in another embodiment, these links in the Elman time recurrent neural network have non-zero weight values. These weights are placed in the weight random access memory 124 before the program in Figure 42 is executed. (For example, rows 1024 to 1535), the program command at address 7 is MULT-ACCUM DR ROW + 0, WR ROW 1024, and the program command at address 8 is MULT-ACCUM ROTATE, WR ROW + 1, COUNT = 511. Preferably, the instruction of address 8 does not access the weight random access memory 124, but the instruction of rotating the address 7 reads the value of the multiplexer register 705 from the weight random access memory 124. Without accessing the weight random access memory 124 during the time-frequency period of the 511 execution address 8 instructions, more bandwidth can be reserved for use by the framework program access weight random access memory 124.

In the first execution of the instructions at addresses 9 and 10 (MULT-ACCUM DR ROW +2, WR ROW 0 and MULT-ACCUM ROTATE, WR ROW +1, COUNT = 511) (corresponding to time step 1), this Each of the 512 neural processing units 126 126 will perform 512 multiplication operations, multiply the value of 512 input nodes D of row 3 of data random access memory 122 by weights of rows 0 to 511 of random access memory 124 corresponding to the row of this neural processing unit 126 Weights to generate 512 products, together with the accumulation operation performed by the instructions at addresses 7 and 8 on the 512 content node C values, and accumulate the accumulator 202 of the corresponding neural processing unit 126 to calculate the value of the hidden layer node Z, In the first execution of the instruction at address 11 (OUTPUT PASSTHRU, DR OUT ROW +2, CLR ACC), the values of 512 accumulators 202 of the 512 neural processing units 126 are transferred and written into the data random access memory The body 122 is listed in column 4, and the accumulator 202 is cleared. In the second execution of the instructions at addresses 9 and 10 (corresponding to time step 2), each of the 512 neural processing units 126 is processed. 512 multiplication operations will be performed to multiply the values of 512 input nodes D of row 6 of data random access memory 122 by weights of rows 0 to 511 of random access memory 124 corresponding to the row of this neural processing unit 126 Weights to produce 512 products together with addresses The accumulation operations performed by the 7 and 8 instructions on the value of 512 content nodes C are accumulated in the accumulator 202 of the corresponding neural processing unit 126 to calculate the value of the hidden layer node Z. The second execution of the instruction at address 11 In the 512 neural processing units 126, the values of the 512 accumulators 202 are passed and written into the data random access memory 122, column 7, and the accumulator 202 is cleared; and so on, at addresses 9 and In the nineteenth execution of the instruction of 10 (corresponding to time step 19), each of the 512 neural processing units 126 will perform 512 multiplication operations to randomize the data to the rank 122 of the memory 57 The value of 512 input nodes D is multiplied by weights in columns 0 to 511 of random access memory 124 Corresponds to the weight of the trip of the neural processing unit 126 to generate 512 products, and the accumulation operation performed by the addresses 7 and 8 on 512 content node C values is accumulated in the accumulator 202 of the corresponding neural processing unit 126 To calculate the value of node Z in the hidden layer, in the nineteenth execution of the instruction at address 11, the values of the 512 accumulators 202 of the 512 neural processing units 126 are passed and written into the data random access memory 122 Column 58, and accumulator 202 is cleared. As mentioned above, the hidden layer node Z value generated and written in the twentieth execution of the instructions at addresses 9 and 10 will not be used.

The instruction at address 12 (LOOP 4) will decrement the loop counter 3804 and return to the instruction at address 4 if the new loop counter 3804 value is greater than zero.

The forty-third diagram is a block diagram showing an example of Jordan time recurrent neural network. The Jordan Time Recurrent Neural Network in Figure 43 is similar to the Elman Time Recurrent Neural Network in Figure 40. It has an input layer node / neuron D, a hidden layer node / neuron Z, and an output layer node / neuron Y. , With content layer node / neuron C. However, in the Jordan time recursive neural network of Figure 43, the content layer node C uses the output feedback from its corresponding output layer node Y as its input link, instead of the Elman time recursion as shown in Figure 40. In the neural network, the output from the hidden layer node Z is used as its input link.

To illustrate the present invention, the Jordan time-recurrent neural network is a time-recurrent neural network including at least one input node layer, a hidden node layer, an output node layer and a content node layer. At the beginning of a given time step, the content node layer stores the output The result of the node layer generated in the previous time step and fed back to the content node layer. The result fed back to the content layer may be the result of the activation function or the output node layer performs an accumulation operation without executing the activation function.

The forty-fourth diagram is a block diagram showing that when the neural network unit 121 performs the calculation of the Jordan time recurrent neural network associated with the forty-third diagram, the data of the neural network unit 121 and the random access memory 122 and An example of data allocation in the weighted random access memory 124 is. In the example of Figure 44, it is assumed that the Jordan time recurrent neural network of Figure 43 has 512 input nodes D, 512 hidden nodes Z, 512 content nodes C, and 512 output nodes Y. In addition, it is also assumed that this Jordan time recurrent neural network is fully connected, that is, all 512 input nodes D are connected to each hidden node Z as input, all 512 content nodes C are connected to each hidden node Z as input, and all 512 The hidden node Z is connected to each output node Y as an input. The example of the Jordan time recursive neural network shown in Figure 44 applies an activation function to the value of the accumulator 202 to generate the value of the output node Y. However, this example assumes that the accumulation before the activation function is applied. The value of the router 202 is passed to the content layer node C, instead of the actual output layer node Y value. In addition, the neural network unit 121 is provided with 512 neural processing units 126, or neurons, for example, adopting a wide configuration. Finally, this example assumes that the weights associated with the links from the content node C to the hidden node Z have a value of one; therefore, it is not necessary to store these one weight values.

As in the example of the forty-first figure, as shown in the figure, the 512 rows (rows 0 to 511) below the weight random access memory 124 will be loaded with the weight values associated with the connection between the input node D and the hidden node Z ,and The subsequent 512 rows (columns 512 to 1023) of the weight random access memory 124 will be loaded with weight values associated with the connection between the hidden node Z and the output node Y.

Data random access memory 122 is loaded with Jordan time recurrent neural network node values for a series of time steps similar to the example in Figure 41; however, the example in Figure 44 uses a set of four The memory loads listed provide the node values for a given time step. As shown in the figure, in an embodiment with 64 rows of data random access memory 122, data random access memory 122 can be loaded with the node values required for 15 different time steps. In the example of Figure 44, columns 0 to 3 are loaded with node values for time step 0, columns 4 to 7 are loaded with node values for time step 1, and so on, and 60 to 63 are loaded. Node value for time step 15. The first row of the four-row set of memories is the value of the input node D loaded with this time step. The second row of the four-row set of memories is the value of the hidden node Z that is loaded at this time step. The third column of the four-column set of memory is the value of the content node C that contains this time step. The fourth column of the four-column set of memory is the value of the output node Y loaded with this time step. As shown in the figure, each row of the data random access memory 122 is loaded with the node value of its corresponding neuron or neural processing unit 126. That is, row 0 loads the node values associated with nodes D0, Z0, C0, and Y0, and the calculation is performed by the neural processing unit 0; row 1 loads the node values associated with nodes D1, Z1, C1, and Y1. The calculation is performed by the neural processing unit 1; and so on, the row 511 loads the node values associated with the nodes D511, Z511, C511, and Y511, and the calculation is performed by the neural processing unit 511. This part corresponds to the 44th figure in the following. There will be more detailed instructions.

The value of the content node C at a given time step in Figure 44 is generated during this time step and used as the input for the next time step. That is, the value of node C calculated and written by the neural processing unit 126 in this time step will become the value of node C used by this neural processing unit 126 to calculate the value of node Z in the next time step ( With the value of input node D for this next time step). The initial value of the content node C (that is, the value of the node C used for calculating the value of the node 1 in the time step 0) is assumed to be zero. This part will be explained in more detail in the subsequent chapter of the non-structural program corresponding to Figure 45.

As described in the forty-first figure above, preferably, the value of the input node D (the values of columns 0 and 4 in the example of the forty-fourth figure, and so on to the value of column 60) is executed by the processor 100. The architecture program is written / filled into the data random access memory 122 through the MTNN instruction 1400, and is read / used by a non-architecture program running on the neural network unit 121, such as the non-architecture program of FIG. 45. Conversely, the values of hidden node Z / content node C / output node Y (in the example in Figure 44 are columns 1/2/3, 5/6/7, and so on to column 61/62/63 The value) is written / filled into the data random access memory 122 by a non-architecture program executed on the neural network unit 121, and is read / used by an architecture program executed on the processor 100 through the MFNN instruction 1500. The example in Figure 44 assumes that the framework program will perform the following steps: (1) For 15 different time steps, fill the value of the input node D into the data random access memory 122 (rows 0, 4, according to And so on to column 60); (2) start the fourth The non-structural program of the fifteenth figure; (3) detecting whether the non-structural program has been executed; (4) reading the value of the output node Y from the data random access memory 122 (rows 3, 7, and so on to row 63) ); And (5) repeat steps (1) to (4) several times until the task is completed, such as the calculation required to identify the words of the mobile phone user.

In another implementation method, the architecture program will perform the following steps: (1) for a single time step, fill the data random access memory 122 (such as row 0) with the value of the input node D; (2) start the non-architecture Program (a modified version of one of the non-architecture programs in Figure 45 without looping, and only accesses a single set of four rows of random storage memory 122); (3) detects whether the non-architecture program has been executed (4) read the value of the output node Y from the data random access memory 122 (such as column 3); and (5) repeat steps (1) to (4) several times until the task is completed. Which of these two methods is optimal depends on the sampling method of the input value of the time recurrent neural network. For example, if this task allows the input to be sampled over multiple time steps (for example, about 15 time steps) and perform calculations, the first method is ideal because this method can bring more computing resources efficiency and / Or better performance, but if this task only allows sampling in a single time step, you need to use the second method.

The third embodiment is similar to the foregoing second method, but different from the second method using a single set of four rows of random access memory 122 rows, the non-structural program of this method uses multiple sets of four rows of memory, that is, Use different sets of four columns of memory at various time steps, this section is similar to the first way. In this third embodiment, preferably, the framework program includes a step before step (2). In this step, the framework program will Update the non-architecture program before starting, for example, update the 122 rows of data random access memory in the instruction at address 1 to point to the next set of four rows of memory.

The forty-fifth diagram is a table showing a program stored in the program memory 129 of the neural network unit 121. This program is executed by the neural network unit 121 and uses data and weights according to the configuration of the forty-fourth diagram. To achieve Jordan time recurrent neural network. The non-structural program in Figure 45 is similar to the non-structural program in Figure 42. For the differences between the two, refer to the descriptions in the relevant sections of this article.

The example program in Figure 45 includes 14 non-framework instructions, which are located at addresses 0 to 13. The instruction at address 0 is an initialization instruction that clears the accumulator 202 and initializes the loop counter 3804 to a value of 15 to execute the loop group 15 times (instructions at addresses 4 to 12). Preferably, the initialization instruction does not place the neural network unit 121 in a wide configuration and configures 512 neural processing units 126. As described in this article, during the execution of instructions at addresses 1 to 3 and addresses 8 to 12, the 512 neural processing units 126 correspond to and operate as 512 hidden layer nodes Z, and at addresses 4, 5 During the execution of the instruction of 7, these 512 neural processing units 126 correspond to and operate as 512 output layer nodes Y.

The instructions at addresses 1 to 5 and 7 are the same as the instructions at addresses 1 to 6 in Figure 42 and have the same functions. The instructions at addresses 1 to 3 calculate the initial value of node Z in the hidden layer and write it to the data in random access memory 122 in column 1 for the first execution of instructions at addresses 4, 5 and 7, to calculate The output layer node Y at the first time step (time step 0).

The first execution of the output instruction at address 6 The 512 accumulator 202 values generated by the accumulation of the instructions at addresses 4 and 5 (these values will be used by the output instruction at address 7 to calculate and write the value of node Y in the output layer) will be passed and Write data to random access memory 122, column 2. These values are the content layer node C values generated in the first time step (time step 0) and used in the second time step (time step 1); in place During the second execution of the output instruction at address 6, the 512 accumulator 202 values generated by the accumulation of the instructions at addresses 4 and 5 (Next, these values will be used by the output instruction at address 7 to calculate and The values written to the output layer node Y) will be passed and written into the data random access memory 122 column 6, these values are the content layer node C values generated in the second time step (time step 1) and Used in the three time steps (time step 2); and so on, during the fifteenth execution of the output instruction at address 6, the 512 accumulator 202 values generated by the accumulation of the instructions at addresses 4 and 5 (These values will be used by the output instruction at address 7 to calculate and The values written to the output layer node Y) will be passed and written into the data random access memory 122 column 58. These values are the content layer node C values (and Read by instruction at address 8 but not used).

The instructions at addresses 8 to 12 are roughly the same as the instructions at addresses 7 to 11 in the forty-second figure, and they have the same functions, with only one difference. The difference is that the instruction at address 8 (ADD_D_ACC DR ROW +1) in the forty-fifth figure will increase the number of rows in the data random access memory 122 by one, while the instruction at address seven in the forty-second figure (ADD_D_ACC DR ROW +0) makes data random access memory The number of columns increased to zero. This difference is due to the difference in the data configuration in the data random access memory 122. In particular, the four-column configuration in Figure 44 includes an independent row for the content layer node C value (such as row 2 , 6, 10, etc.), and the three-column configuration in the forty-first figure does not have this independent column, but the value of the content layer node C and the value of the hidden layer node Z share the same column (such as column 1, 4, 7 etc.). Fifteen executions of the instructions at addresses 8 to 12 calculate the value of the hidden layer node Z and write it to the data random access memory 122 (write to columns 5, 9, 13, and so on until column 57) for The second to sixteen executions of the instructions at addresses 4, 5, and 7 are used to calculate the output layer node Y for the second to fifteenth time steps (time steps 1 to 14). (The last / fifteenth execution of the instructions at addresses 8 to 12 is to calculate the value of the hidden layer node Z and write it to the data random access memory row 122, but these values are not used.)

The loop instruction at address 13 decrements the loop counter 3804 and returns to the instruction at address 4 if the new loop counter 3804 value is greater than zero.

In another embodiment, the design of the Jordan time recurrent neural network uses the content node C to load the activation function value of the output node Y, and the activation function value is the cumulative value after the activation function is executed. In this embodiment, because the value of the output node Y is the same as the value of the content node C, the non-framework instruction at address 6 is not included in the non-framework program. Therefore, the number of rows used in the data random access memory 122 can be reduced. More precisely, none of the columns (for example, columns 2, 6, 59) of the values of each load content node C in the forty-fourth figure exist in this embodiment. In addition, each time step of this embodiment requires only three of the data random access memory 122 It will be matched with 20 time steps instead of 15, and the address of the non-structural program instruction in Figure 45 will be adjusted appropriately.

Long short-term memory cells

The use of long short-term memory cells for temporal recurrent neural networks is a concept well known in the art. For example, Long Short-Term Memory, Sepp Hochreiter and Jürgen Schmidhuber, Neural Computation, November 15, 1997, Vol. 9, No. 8, Pages 1735-1780; Learning to Forget: Continental Prediction with LSTM, Felix A. Gers , Jürgen Schmidhuber, and Fred Cummins, Neural Computation, October 2000, Vol. 12, No. 10, Pages 2451-2471; these documents can be obtained from the MIT Press Journals. Long and short-term memory cells can be constructed in many different types. The long-term and short-term memory cell 4600 of the forty-sixth figure described below is based on the URL http://deeplearning.net/tutorial/lstm.html and is titled as the LSTM Networks for Sentiment Analysis. The short and long-term memory cells described in the tutorial are models. A copy of this tutorial was downloaded on October 19, 2015 (hereinafter referred to as the "long-and-short-term memory tutorial") and provided in the US application information disclosure report of this case. This long short-term memory cell 4600 can be used to generally describe the ability of the embodiment of the neural network unit 121 described herein to effectively perform calculations related to long-term and short-term memory. It is worth noting that the embodiments of these neural network units 121, including the embodiment described in FIG. 49, can effectively perform other long-term and short-term events other than the long-term and short-term memory cells described in FIG. 46. Calculation of memory cells.

Preferably, the neural network unit 121 may be configured to perform calculations on a time-recurrent neural network having a long-term short-term memory cell layer connected to other layers. For example, in this long-short-term memory tutorial, the network includes a mean-source layer to receive the output of the long-term and short-term memory cells from the long-term and short-term memory layer (H), and a logistic regression layer to receive the output of the mean-source source .

The forty-sixth figure is a block diagram showing one embodiment of a long-term short-term memory cell 4600.

As shown in the figure, the long-term and short-term memory cell 4600 includes a memory cell input (X), a memory cell output (H), an input gate (I), an output gate (O), and a forget gate (F). A memory cell state (C) and a candidate memory cell state (C '). The input gate (I) can gate the signal transmission from the memory cell input (X) to the memory cell state (C), and the output gate (O) can gate the signal transmission from the memory cell state (C) to the memory cell output (H). . This memory cell state (C) is fed back as a candidate memory cell state (C ') at a time step. The forget gate (F) can gate this candidate memory cell state (C '). This candidate memory cell state will reverse and become the memory cell state (C) at the next time step.

The embodiment of Figure 46 uses the following equations to calculate the aforementioned various values: (1) I = SIGMOID (Wi * X + Ui * H + Bi)

(2) F = SIGMOID (Wf * X + Uf * H + Bf)

(3) C ’= TANH (Wc * X + Uc * H + Bc)

(4) C = I * C ’+ F * C

(5) O = SIGMOID (Wo * X + Uo * H + Bo)

(6) H = O * TANH (C)

Wi and Ui are weight values associated with the input gate (I), and Bi is an offset value associated with the input gate (I). Wf and Uf are weight values associated with forget gate (F), and Bf is an offset value associated with forget gate (F). Wo and Uo are weight values associated with the output gate (O), and Bo is an offset value associated with the output gate (O). As before, equations (1), (2) and (5) calculate the input gate (I), the forget gate (F) and the output gate (O), respectively. Equation (3) calculates the candidate memory cell state (C '), and equation (4) calculates the candidate memory cell state (C') with the current memory cell state (C) as the input. The current memory cell state (C) is Memory cell status at the current time step (C). Equation (6) calculates the memory cell output (H). However, the present invention is not limited to this. Embodiments of calculating the input gate, forget gate, output gate, candidate memory cell state, memory cell state, and long-term and short-term memory cell output using other methods are also covered by the present invention.

To illustrate the present invention, the long-term and short-term memory cells include a memory cell input, a memory cell output, a memory cell state, a candidate memory cell state, an input gate, an output gate, and a forget gate. For each time step, the state of the input gate, output gate, forget gate and candidate memory cell is a function of the memory cell input of the current time step and the memory cell output and related weights of the previous time step. The state of the memory cell at this time step is a function of the state of the memory cell at the previous time step, the state of the candidate memory cell, the input gate and the output gate. In this sense, the state of the memory cell is fed back to calculate the state of the memory cell for the next time step. The memory cell output at this time step is the state and output of the memory cell calculated at this time step. Opening function. A long-term short-term memory neural network is a neural network with a long-term short-term memory cell.

The forty-seventh diagram is a block diagram showing the data of the neural network unit 121 when the neural network unit 121 performs the calculation of the long-term and short-term memory cells of the long-term and short-term memory neural network associated with the forty-sixth diagram. An example of the data arrangement in the random access memory 122 and the weighted random access memory 124. In the example in Figure 47, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, such as a wide configuration. However, there are only 128 neural processing units 126 (such as neural processing units 0 to 127). The value generated by) will be used because the long-term and short-term memory layer in this example has only 128 long-term and short-term memory cells 4600.

As shown in the figure, the weight random access memory 124 loads the weight values, offset values, and median values of the corresponding neural processing units 0 to 127 of the neural network unit 121. Rows 0 to 127 of the weight random access memory 124 load the weight values, offset values, and intermediate values of the corresponding neural processing units 0 to 127 of the neural network unit 121. Each of the columns 0 to 14 is loaded with 128 values corresponding to the aforementioned equations (1) to (6) to be provided to the neural processing unit 0 to 127. These values are: Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, C ', TANH (C), C, Wo, Uo, Bo. Preferably, the weight and offset values-Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, Wo, Uo, Bo (located in columns 0 to 8 and columns 12 to 14)-are performed by The architecture program on the processor 100 is written / filled into the weight random access memory 124 through the MTNN instruction 1400, and is read / used by a non-architecture program executed on the neural network unit 121. Framework program. Preferably, the intermediate value-C ', TANH (C), C (located in columns 9 to 11)-is performed by God The weighted random access memory 124 is written / filled by the non-framework program of the network unit 121 and read / used, as described later.

As shown in the figure, the data random access memory 122 is loaded with input (X), output (H), input gate (I), forget gate (F) and output gate (O) values for a series of time steps. Further, the memory is loaded with a set of five values of X, H, I, F, and O for a given time step. Take a data random access memory 122 with 64 rows as an example. As shown in the figure, the data random access memory 122 can be loaded with memory cell values for 12 different time steps. In the example in Figure 47, columns 0 to 4 are the values of memory cells for time step 0, columns 5 to 9 are the values of memory cells for time step 1, and so on, 55 to 59. Load the memory cell value for time step 11. The first column of the five-column set of memory contains the X value for this time step. The second column of the five-column set contains the H value for this time step. The third column of the five-column set contains the I value for this time step. The fourth column of the five-column set contains the F value for this time step. The fifth column in the five-column set contains the value of O for this time step. As shown in the figure, each row in the data random access memory 122 is loaded with a value for use by a corresponding neuron or neural processing unit 126. That is, row 0 loads the value associated with long-term and short-term memory cells 0, and its calculation is performed by the neural processing unit 0; row 1 loads the value associated with long-term and short-term memory cells 1, and its calculation is performed by the nerve Executed by the processing unit 1; and so on, line 127 is loaded with the value associated with the long-term and short-term memory cells 127, and the calculation is performed by the neural processing unit 127, as described in the subsequent figure 48.

Preferably, the X value (located in rows 0, 5, 9, and so on to row 55) is written / filled into the data random access memory 122 by the architecture program running on the processor 100 through the MTNN instruction 1400, and The non-framework program executed by the neural network unit 121 reads / uses the non-framework program shown in FIG. 48. Preferably, the I value, the F value and the O value (located in columns 2/3/4, 7/8/9, 12/13/14, and so on to columns 57/58/59) are performed by neural processing The non-schema program of the unit 121 writes / fills in the data random access memory 122, as described later. Preferably, the H value (located in columns 1, 6, 10, and so on to column 56) is written / filled into the data random access memory 122 and read by a non-architecture program executed on the neural processing unit 121. / Use, and read by the framework program running on the processor 100 through the MFNN instruction 1500.

The example in Figure 47 assumes that the framework program will perform the following steps: (1) For 12 different time steps, fill the value of the input X into the data random access memory 122 (rows 0, 5, and so on) (Analog to column 55); (2) Start the non-architecture program of Figure 48; (3) detect whether the non-architecture program has been executed; (4) read the value of output H from the data random access memory 122 ( (Columns 1, 6, and so on to column 59); and (5) Repeat steps (1) to (4) several times until the task is completed, such as the calculation required to recognize the words of the mobile phone user.

In another execution method, the architecture program will perform the following steps: (1) For a single time step, fill the data random access memory 122 (such as row 0) with the value of X; (2) start the non-schema program (Figure 48 is a modified version of one of the non-architecture programs, which does not require loops, and only accesses a single set of five rows of random storage memory 122); (3) detection Test whether the non-structural program has been executed; (4) read the value of the output H from the data random access memory 122 (such as column 1); and (5) repeat steps (1) to (4) several times until the task is completed. Which of these two methods is optimal may depend on the sampling method of the input X value of the long-term and short-term memory layers. For example, if this task allows the input to be sampled at multiple time steps (for example, about 12 time steps) and perform calculations, the first method is ideal because this method may bring more computing resources efficiency and / Or better performance, but if this task only allows sampling in a single time step, you need to use the second method.

The third embodiment is similar to the foregoing second method, but different from the second method using a single set of five rows of data random access memory 122, the non-structural program of this method uses multiple sets of five rows of memory, that is, in Each time step uses a different set of five columns of memory, this part is similar to the first way. In this third embodiment, preferably, the architecture program includes a step before step (2). In this step, the architecture program will update the non-architecture program before it is started, for example, in the address 0 instruction. The 122 rows of data random access memory are updated to point to the next set of five rows of memory.

The forty-eighth figure is a table showing a program stored in the program memory 129 of the neural network unit 121. This program is executed by the neural network unit 121 and uses data and weights according to the configuration of the forty-seventh figure. To achieve the calculations related to the long-term and short-term memory cells. The example program in Figure 48 includes 24 non-framework instructions at addresses 0 to 23. The instruction at address 0 (INITIALIZE NPU, CLR ACC, LOOPCNT = 12, DR IN ROW = -1, DR OUT ROW = 2) will The accumulator 202 is cleared and the loop counter 3804 is initialized to a value of 12 to execute the loop group 12 times (instructions at addresses 1 to 22). This initialization command initializes the row to be read of the data random access memory 122 to a value of -1, and after the first execution of the instruction at address 1, this value will increase to zero. This initialization command also initializes the data random access memory 122 to be written into the row (for example, the register 2606 of the twenty-sixth and thirty-ninth figures) as the row two. Preferably, the initialization command does not place the neural network unit 121 in a wide configuration. Thus, the neural network unit 121 is configured with 512 neural processing units 126. As described in the subsequent chapters, during the execution of the instructions at addresses 0 to 23, 128 of the 512 neural processing units 126 correspond to and operate as 128 long-term and short-term memory cells 4600.

In the first execution of the instructions at addresses 1 to 4, each of the 128 neural processing units 126 (ie, neural processing units 0 to 127) will target the first long-term short-term memory cell 4600. Time step (time step 0) calculates the value of the input gate (I) and writes the value of I into the corresponding text in column 2 of the data random access memory 122; in the second execution of the instructions at addresses 1 to 4, Each of the 128 neural processing units 126 calculates the I value for the second time step (time step 1) corresponding to the long-term short-term memory cell 4600 and writes the I value into the data random access memory 122. Corresponding text in column 7; and so on, in the twelfth execution of the instructions at addresses 1 to 4, each of the 128 neural processing units 126 will target the corresponding long-term and short-term memory cells 4600. The twelfth time step (time step 11) calculates the I value and writes the I value into the data random access memory 122 row. The corresponding text of 57 is shown in Figure 47.

Further, the multiply-accumulate instruction at address 1 reads the next row behind the current row of the random access memory 122 (row 0 in the first execution, row 5 in the second execution, and so on, etc.) In the twelfth execution, it is column 55). This column contains the memory cell input (X) value associated with the current time step. This instruction will read the weight 0 in the random access memory 124 which contains the Wi value. And the aforementioned read values are multiplied to generate a first multiplying accumulation to be added to the accumulator 202 which has just been cleared by the initialization instruction at address 0 or the instruction at address 22. Subsequently, the multiply-accumulate instruction at address 2 reads the next row of random access memory 122 rows (row 1 in the first execution, row 6 in the second execution, and so on, in the twelfth execution This is column 56). This column contains the memory cell output (H) value associated with the current time step. This instruction will read column 1 containing the Ui value in the weight random access memory 124 and compare the aforementioned value with Multiplying produces a second multiplying accumulation and adds it to the accumulator 202. The H value associated with the current time step is read by the instruction at address 2 (and the instructions at addresses 6, 10, and 18) by the data random access memory 122, generated at the previous time step, and by the address 22 The output instruction writes to the data random access memory 122; however, in the first execution, the instruction at address 2 will write an initial value to the data random access memory row 1 as the H value. Preferably, the structured program will write the initial H value into the data random access memory 122 column 1 (for example, using MTNN instruction 1400) before starting the non-structured program shown in Figure 48; however, the present invention is not limited to this. Therefore, other embodiments that include an initialization instruction to write the initial H value into the data random access memory 122 in the non-framework program also belong to the scope of the present invention. In one embodiment, the initial H value is zero. Next, the instruction (ADD_W_ACC WR ROW 2) at the address 3 to add the weight text to the accumulator reads the column 2 containing the Bi value in the weight random access memory 124 and adds it to the accumulator 202. Finally, the output instruction at address 4 (OUTPUT SIGMOID, DR OUT ROW +0, CLR ACC) executes an S-shaped start function on the value of accumulator 202 and writes the execution result to the current output row of the data random access memory 122 (Column 2 in the first execution, column 7 in the second execution, and so on, and column 57 in the twelfth execution) and the accumulator 202 is cleared.

In the first execution of the instructions at addresses 5 to 8, each of the 128 neural processing units 126 calculates the first time step (time step 0) of the corresponding long-term short-term memory cell 4600. Forget the gate (F) value and write the F value into the corresponding text in column 3 of the data random access memory 122; in the second execution of the instructions at addresses 5 to 8, the 128 neural processing units 126 Each neural processing unit 126 calculates its forgetting gate (F) value for the second time step (time step 1) corresponding to the long-term short-term memory cell 4600 and writes the F value into the 8th row of the data random access memory 122. Corresponding text; and so on, in the twelfth execution of the instructions at addresses 5 to 8, each of the 128 neural processing units 126 will target the tenth corresponding to the short-term memory cell 4600. The second time step (time step 11) calculates its forget gate (F) value and writes the F value into the corresponding text in column 58 of the data random access memory 122, as shown in FIG. 47. The instructions for addresses 5 to 8 calculate the F value similarly to the instructions for addresses 1 to 4, except that the instructions for addresses 5 to 7 will access random access memory 124 from row 3, row 4, and row respectively. 5 Read Wf, Uf and Bf values to perform multiplication and / or Addition.

In the twelve executions of the instructions at addresses 9 to 12, each of the 128 neural processing units 126 will calculate its candidate memory cell state for the corresponding time step corresponding to the long-term short-term memory cell 4600 ( C ') value and write the C' value into the corresponding text in column 9 of the weight random access memory 124. The instructions for addresses 9 to 12 calculate the C 'value similarly to the instructions for addresses 1 to 4 above, but the instructions for addresses 9 to 11 will access random access memory 124 from row 6, row 7, and Column 8 reads Wc, Uc and Bc values to perform multiplication and / or addition operations. In addition, the output instruction at address 12 will execute the hyperbolic tangent start function instead of (such as the output instruction at address 4) the S-type start function.

Further, the multiply-accumulate instruction at address 9 reads the current row of data random access memory 122 (row 0 in the first execution, row 5 in the second execution, and so on. The twelfth execution is column 55). This current column contains the memory cell input (X) value associated with the current time step. This instruction also reads column 6 containing the Wc value in the random access memory 124. And multiplying the aforementioned values to generate a first multiplying accumulation and adding to the accumulator 202 just cleared by the instruction at address 8. Next, the multiply-accumulate instruction at address 10 reads the next row of the data random access memory 122 (row 1 in the first execution, row 6 in the second execution, and so on. Twelve executions is column 56). This column contains the memory cell output (H) value associated with the current time step. This instruction also reads column 7 containing the Uc value in the weight random access memory 124, and The aforementioned values are multiplied to produce a second multiplied accumulation and added to the accumulator 202. Next, the instruction to add weight text to the accumulator at address 11 will The read weight random access memory 124 includes a column 8 of Bc values and adds it to the accumulator 202. Finally, the output instruction at address 12 (OUTPUT TANH, WR OUT ROW 9, CLR ACC) performs a hyperbolic tangent start function on the value of accumulator 202 and writes the execution result to column 9 of weight random access memory 124. And the accumulator 202 is cleared.

In the twelve executions of the instructions at addresses 13 to 16, each of the 128 neural processing units 126 will calculate a new memory cell state for the corresponding time step corresponding to the long-term short-term memory cell 4600 ( C) value and write the new C value into the corresponding text in column 11 of the weighted random access memory 122. Each neural processing unit 126 also calculates tanh (C) and writes it into the weighted random access memory. Corresponding text in column 10 of 124. Further, the multiply-accumulate instruction at address 13 reads the next row behind the current row of the random access memory 122 (row 2 in the first execution and row 7 in the second execution, and so on) By analogy, in the twelfth execution, it is column 57). This column contains the input gate (I) value associated with the current time step. This instruction reads the weight of the random access memory 124 and contains the state of the candidate memory cell (C ') Value column 9 (just written by the instruction at address 12), and multiplying the aforementioned values to generate a first multiplication accumulation is added to the accumulator 202 just cleared by the instruction at address 12. Next, the multiply accumulate instruction at address 14 reads the next row of the data random access memory 122 (row 3 in the first execution, row 8 in the second execution, and so on. Twelve executions are column 58). This column contains the value of the forget gate (F) associated with the current time step. This instruction reads the weight of the current memory cell included in the random access memory 124 calculated in the previous time step. State (C) value (from the most recent instruction at address 15 Column 11 is performed, and the aforementioned values are multiplied to generate a second product and added to the accumulator 202. Next, the output instruction at address 15 (OUTPUT PASSTHRU, WR OUT ROW 11) will pass the value of this accumulator 202 and write it into the column 11 of the weight random access memory 124. It should be understood that the C value read by the instruction at address 14 from column 11 of the data random access memory 122 is the C value generated and written by the instructions at addresses 13 to 15 in the most recent execution. The output instruction at address 15 does not clear the accumulator 202, so its value can be used by the instruction at address 16. Finally, the output instruction at address 16 (OUTPUT TANH, WR OUT ROW 10, CLR ACC) executes a hyperbolic tangent start function on the value of accumulator 202 and writes its execution result to column 10 of weight random access memory 124 Used by instruction at address 21 to calculate memory cell output (H) value. The instruction at address 16 clears accumulator 202.

In the first execution of the instructions at addresses 17 to 20, each of the 128 neural processing units 126 calculates the first time step (time step 0) of the corresponding long-term short-term memory cell 4600. Output the gate (O) value and write the O value into the corresponding text in column 4 of the data random access memory 122; in the second execution of the instructions at addresses 17 to 20, the 128 neural processing units 126 Each neural processing unit 126 calculates the output gate (O) value for the second time step (time step 1) corresponding to the long-term and short-term memory cell 4600 and writes the O value into the data random access memory 122 column 9 Corresponding text; and so on. In the twelfth execution of the instructions at addresses 17 to 20, each of the 128 neural processing units 126 will target the tenth corresponding to the short-term memory cell 4600. Two time steps (time step 11) Calculate the output gate (O) value and write the O value to the corresponding text in column 58 of the data random access memory 122, as shown in Figure 47. The instructions for addresses 17 to 20 calculate the O value similar to the instructions for addresses 1 to 4 above. However, the instructions for addresses 17 to 19 will access random access memory 124 from row 12, row 13, and row respectively. 14 Read Wo, Uo and Bo values to perform multiplication and / or addition.

In the first execution of the instructions at addresses 21 to 22, each of the 128 neural processing units 126 calculates the first time step (time step 0) of the corresponding long-term short-term memory cell 4600. The memory cell outputs the (H) value and writes the H value into the corresponding text in column 6 of the data random access memory 122; in the second execution of the instructions at addresses 21 to 22, the 128 neural processing units 126 Each of the neural processing units 126 calculates the memory cell output (H) value for the second time step (time step 1) corresponding to the long-term short-term memory cell 4600 and writes the H value into the data random access memory 122 column. The corresponding text of 11; and so on, in the twelfth execution of the instructions at addresses 21 to 22, each of the 128 neural processing units 126 will target the corresponding long-term and short-term memory cells 4600. The twelfth time step (time step 11) calculates the memory cell output (H) value and writes the H value into the corresponding text of column 60 of the data random access memory 122, as shown in FIG. 47.

Further, the multiply-accumulate instruction at address 21 reads the third row behind the current row of the data random access memory 122 (row 4 in the first execution and row 9 in the second execution. And so on, in the twelfth execution is column 59), this column contains the step related to the current time The output gate (O) value. This instruction reads the weight 10 of the random access memory 124 containing the tanh (C) value in the column 10 (written by the instruction at address 16), and multiplies the foregoing values to generate a The multiplied accumulation is added to the accumulator 202 which has just been cleared by the instruction at address 20. Subsequently, the output instruction at address 22 will pass the value of accumulator 202 and write it to the data in random access memory 122. The second output row 11 (in the first execution, it will be row 6, in the second execution The execution is column 11 and so on, and the twelfth execution is column 61), and the accumulator 202 is cleared. It should be understood that the value of H in row 122 of the random access memory is written by the instruction at address 22 (in the first execution, it is row 6, in the second execution, it is row 11, and so on. The twelfth execution is column 61) which is the H value consumed / read in the subsequent execution of the instructions at addresses 2, 6, 10, and 18. However, the value of H written in column 61 during the twelfth execution will not be consumed / read by the execution of the instructions at addresses 2, 6, 10, and 18; for a preferred embodiment, this value will be It is consumed / read by the framework program.

The instruction at address 23 (LOOP 1) decrements the loop counter 3804 and returns to the instruction at address 1 if the new loop counter 3804 value is greater than zero.

The forty-ninth figure is a block diagram showing an embodiment of a neural network unit 121. The neural processing unit group in this embodiment has output buffer masking and feedback capabilities. The forty-ninth figure shows a single neural processing unit group 4901 composed of four neural processing units 126. Although the forty-ninth figure only shows a single neural processing unit group 4901, it should be understood that each neural processing unit 126 in the neural network unit 121 is included in a neural processing unit group 4901. Therefore, a There will be a total of N / J neural processing unit groups 4901, where N is the number of neural processing units 126 (for example, 512 for a wide configuration and 1024 for a narrow configuration) and J is a single group The number of neural processing units 126 in the group 4901 (for example, four for the embodiment of FIG. 49). In the forty-ninth figure, the four neural processing units 126 in the neural processing unit group 4901 are referred to as neural processing unit 0, neural processing unit 1, neural processing unit 2, and neural processing unit 3.

Each neural processing unit in the embodiment of the forty-ninth figure is similar to the neural processing unit 126 of the aforementioned seventh figure, and the components with the same reference numerals in the figure are also similar. However, the multiplex register 208 is adjusted to include four additional inputs 4905, the multiplex register 705 is adjusted to include four additional inputs 4907, and the selection input 213 is adjusted to be able to recover from the original input The selection input 211 and 207 and the additional input 4905 are provided to the output 209, and the selection input 713 is adjusted to provide the selection 203 from the original inputs 711 and 206 and the additional input 4907.

As shown in the figure, the column buffer 1104 in the eleventh figure is the output buffer 1104 in the forty-ninth figure. Further, the characters 0, 1, 2 and 3 of the output buffer 1104 shown in the figure receive the corresponding outputs of the four activation function units 212 associated with the neural processing units 0, 1, 2 and 3. The output buffer 1104 in this section contains N characters corresponding to a neural processing unit group 4901. These characters are called an output buffer text group. In the embodiment of the forty-ninth figure, N is four. The four characters of the output buffer 1104 are fed back to the multiplexer registers 208 and 705, and are received by the multiplexer register 208 as four additional inputs 4905. And as four additional inputs 4907 are received by the multiplexer register 705. The output buffer text group feeds back to the corresponding action of its corresponding neural processing unit group 4901, so that the arithmetic instructions of the non-architecture program can read the text from the output buffer 1104 associated with the neural processing unit group 4901 (that is, the output buffer text group Group) to select one or two characters as its input. For an example, refer to the non-framework program in the following fifty-first figure, as shown in the instructions at addresses 4, 8, 11, 12, and 15. That is, the text in the output buffer 1104 specified in the non-architecture instruction will confirm the value generated by selecting the input 213/713. This capability actually enables the output buffer 1104 to be used as a scratch pad memory, which enables non-architectural programs to reduce the amount of data written to the random access memory 122 and / or the weighted random access memory 124 and subsequent slaves. The number of readings, such as reducing the number of intervening generations and uses during the process. Preferably, the output buffer 1104, or column buffer 1104, includes a one-dimensional register array for storing 1024 narrow characters or 512 wide characters. Preferably, the reading to the output buffer 1104 can be performed in a single time-frequency cycle, and the writing to the output buffer 1104 can also be performed in a single time-frequency cycle. Unlike data random access memory 122 and weighted random access memory 124, which can be accessed by architecture programs and non-architecture programs, the output buffer 1104 cannot be accessed by architecture programs, but can only be stored by non-architecture programs. take.

The output buffer 1104 is adjusted to receive a mask input 4903. Preferably, the mask input 4903 includes four characters corresponding to four characters of the output buffer 1104, and the four characters are related to the four neural processing units of the neural processing unit group 4901. 126. Preferably, if the mask input 4903 corresponding to the text of the output buffer 1104 is true, the text of the output buffer 1104 will maintain its current value; otherwise, the text of the output buffer 1104 will be The output of the activation function unit 212 is updated. That is, if the mask input 4903 corresponding to the text of the output buffer 1104 is false, the output of the activation function unit 212 will be written into the text of the output buffer 1104. In this way, the output instruction of the non-structural program can selectively write the output of the activation function unit 212 into some characters of the output buffer 1104 and keep the current values of other characters of the output buffer 1104 unchanged. For an example, please refer to The following instructions of the non-framework program in the fifty-first figure are the instructions of addresses 6, 10, 13 and 14 in the figure. That is, the text specified in the output buffer 1104 in the non-architecture program is generated from the value of the mask input 4903.

To simplify the description, the input 1811 of the multiplexer register 208/705 is not shown in the forty-ninth figure (as shown in the eighteenth, nineteenth, and twenty-third). However, embodiments that support the feedback / masking of the dynamically configurable neural processing unit 126 and the output buffer 1104 are also within the scope of the present invention. Preferably, in these embodiments, the output buffer text group can be dynamically configured correspondingly.

It should be understood that although the number of neural processing units 126 in the neural processing unit group 4901 in this embodiment is four, the present invention is not limited to this, and the number of neural processing units 126 in the group is larger or smaller. The examples belong to the scope of the present invention. In addition, for an embodiment having a shared activation function unit 1112, as shown in FIG. 52, the neural processing unit 126 in a neural processing unit group 4901 The number has a synergistic effect with the number of neural processing units 126 in a group of activation function units 212. The shielding and feedback capabilities of the output buffer 1104 in the neural processing unit group are particularly helpful to improve the computational efficiency associated with long-term and short-term memory cells 4600, as described in the subsequent figures 50 and 51.

Figure 50 is a block diagram showing the neural network of Figure 49 when the neural network unit 121 performs a one-level calculation associated with 128 long and short-term memory cells 4600 in Figure 46. The data random access memory 122, the weight random access memory 124, and the data buffer in the output buffer 1104 of the unit 121 are an example of data configuration. In the example in FIG. 50, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, for example, a wide configuration is adopted. As in the examples of the forty-seventh and forty-eight diagrams, the long-short-term memory layer in the examples of the fifty-first and fifty-one diagrams has only 128 long-short-term memory cells 4600. However, in the example in Figure 50, the values generated by all 512 neural processing units 126 (such as neural processing units 0 to 127) will be used. When the non-architecture program of the fifty-first figure is executed, each neural processing unit group 4901 collectively operates as a long-term and short-term memory cell 4600.

As shown in the figure, the data random access memory 122 is loaded with memory cell input (X) and output (H) values for a series of time steps. Further, for a given time step, there will be a pair of two rows of memory loaded with X and H values, respectively. Taking a data random access memory 122 with 64 rows as an example, as shown in the figure, the memory cell value loaded by the data random access memory 122 can be used in 31 different time steps. In the example in Figure 50, rows 2 and 3 contain the values for time step 0, rows 4 and 5 contain the values for time step 1, and so on. 62 and 63 load values for time step 30. The first of the two rows of memory contains the X value of this time step, and the second of the two rows contains the H value of this time step. As shown in the figure, each group of four rows in the data random access memory 122 corresponds to the memory system of the neural processing unit group 4901, which is loaded with values for its corresponding long-term and short-term memory cells 4600. That is, rows 0 to 3 are loaded with the values associated with long-term and short-term memory cells 0, and the calculation is performed by the neural processing unit 0-3, that is, the neural processing unit group 0; rows 4 to 7 are loaded with the long The calculation of the short-term memory cell 1 is performed by the neural processing unit 4-7, that is, the neural processing unit group 1. By analogy, lines 508 to 511 are the values associated with the long-term and short-term memory cell 127. It is executed by the neural processing units 508-511, that is, the neural processing unit group 127, as shown in the following fifty-first figure. As shown in the figure, column 1 is not used, and column 0 is loaded with the initial memory cell output (H) value. For a preferred embodiment, a zero value can be filled in by a structural program. However, the present invention is not limited to this. Therefore, it is also within the scope of the present invention to fill the initial memory cell output (H) value of column 0 with non-architectural program instructions.

Preferably, the X value (located in rows 2, 4, 6 and so on to row 62) is written / filled into the data random access memory 122 by the architecture program running on the processor 100 through the MTNN instruction 1400, and Non-architecture programs executed on the neural network unit 121 are read / used, such as the non-architecture program shown in FIG. 50. Preferably, the H value (located in columns 3, 5, 7 and so on to column 63) is written / filled into the data random access memory 122 and read by a non-architecture program executed on the neural network unit 121. / Use, as described below. Preferably, the value of H is not executed by the processor 100 The framework program is read by MFNN instruction 1500. It should be noted that the non-architecture program of the fifty-first figure assumes that each group of four rows of memory corresponding to the neural processing unit group 4901 (such as rows 0-3, rows 4-7, rows 5-8, and so on) By analogy to rows 508-511), the four X values in a given column are filled with the same value (for example, by a framework program). Similarly, the non-architecture program of the fifty-first figure calculates and writes the same value to the four H values in a given column in the four rows of memory corresponding to the neural processing unit group 4901.

As shown in the figure, the weight random access memory 124 is a weight, offset and memory cell state (C) value required by the neural processing unit loaded with the neural network unit 121. In each of the four rows of memory corresponding to the neural processing unit group 121 (eg, rows 0-3, rows 4-7, rows 5-8, and so on to rows 508-511): (1) line number divided by Rows with the remainder of 4 equal to 3 will be loaded with the values of Wc, Uc, Bc, and C in columns 0, 1, 2, and 6, respectively. (2) The row with the remainder divided by 4 equal to 2 Columns 3, 4 and 5 carry the values of Wo, Uo, and Bo, respectively; (3) The row where the remainder of the row number divided by 4 equals 1 will load the values of Wf, Uf, and Bf in columns 3, 4, and 5, respectively; And (4) The line whose number is divided by 4 and whose remainder is equal to 0 will have the values of Wi, Ui, and Bi in columns 3, 4, and 5, respectively. Preferably, these weights and offset values-Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, Wo, Uo, Bo (in columns 0 to 5)-are executed by the processor 100 The architecture program is written / filled into the weight random access memory 124 through the MTNN instruction 1400, and is read / used by a non-architecture program executed on the neural network unit 121, as shown in the non-architecture program of FIG. 51. Preferably, the intermediate C value is written / filled by a non-framework program executed on the neural network unit 121. Re-random access memory 124 is read and used as described later.

The example in Figure 50 assumes that the framework program will perform the following steps: (1) For 31 different time steps, fill the value of the input X into the data random access memory 122 (rows 2, 4, and so on) (Column 62); (2) start the non-schema program of the fifty-first figure; (3) detect whether the non-schema program has been executed; (4) read the value of output H from the data random access memory 122 (column 3) , 5, and so on to column 63); and (5) Repeat steps (1) to (4) several times until the task is completed, such as the calculation required to identify the speech of the mobile phone user.

In another implementation method, the architecture program will perform the following steps: (1) for a single time step, fill the data random access memory 122 (such as column 2) with the value of X; (2) start the non-schema program (Figure 51 is a revised version of one of the non-architecture programs, which does not require loops, and only accesses a single pair of two rows of the random storage memory 122); (3) detects whether the non-architecture program has completed execution; (4) Read the value of the output H from the data random access memory 122 (such as column 3); and (5) repeat steps (1) to (4) several times until the task is completed. Which of these two methods is optimal may depend on the sampling method of the input X value of the long-term and short-term memory layers. For example, if this task allows the input to be sampled at multiple time steps (for example, about 31 time steps) and perform calculations, the first method is ideal because this method may bring more computing resources efficiency and / Or better performance, but if this task only allows sampling in a single time step, you need to use the second method.

The third embodiment is similar to the foregoing second method, but different from the second method using a single random access record of two rows of data Memory 122. The non-architecture program in this way uses multiple pairs of memory rows, that is, different pairs of memory rows are used at each time step. This part is similar to the first way. Preferably, the architecture program of this third embodiment includes a step before step (2). In this step, the architecture program will update the non-architecture program before it is started, for example, the instruction in the address 1 is updated. The 122 rows of data random access memory are updated to point to the next two rows of memory.

As shown in the figure, for the neural processing units 0 to 511 of the neural network unit 121, the output buffer 1104 is loaded with the memory cell output (H ), Candidate memory cell state (C '), input gate (I), forget gate (F), output gate (O), intermediate value of memory cell state (C) and tanh (C), each output buffers text group In the group (for example, the output buffer 1104 corresponds to the group of four characters of the neural processing unit group 4901, such as the characters 0-3, 4-7, 5-8 and so on to 508-511), the text number is divided by Characters with a remainder of 4 being 3 are represented as OUTBUF [3], characters with a remainder of 4 divided by 4 are represented by OUTBUF [2], characters with a remainder of 4 divided by 4 are represented as OUTBUF [ 1], and the character whose remainder is 0 when the character number is divided by 4 is represented as OUTBUF [0].

As shown in the figure, after the instruction of address 2 in the non-architecture program of the fifty-first figure is executed, for each neural processing unit group 4901, all four characters of the output buffer 1104 are written correspondingly. The initial memory cell output (H) value of the long-term and short-term memory cell 4600. After the instruction at address 6 is executed, for each neural processing unit group 4901, the word OUTBUF [3] in the output buffer 1104 is written into the candidate memory cell state (C ') value corresponding to the long-term and short-term memory cell 4600. While output buffer 1104 The other three texts will retain their previous values. After the instruction at address 10 is executed, for each neural processing unit group 4901, the OUTBUF [0] characters of the output buffer 1104 will be written into the input gate (I) value corresponding to the long-term short-term memory cell 4600, OUTBUF [ 1] The text will write the value of the forget gate (F) corresponding to the long short-term memory cell 4600, the OUTBUF [2] text will write the value of the output gate (O) corresponding to the long short-term memory cell 4600, and the OUTBUF [3] text Is to maintain its previous value. After the instruction at address 13 is executed, for each neural processing unit group 4901, the word OUTBUF [3] in the output buffer 1104 will be written into the new memory cell state (C) value corresponding to the short-term memory cell 4600. (For the output buffer 1104, the value of C including slot 3 is written in column 6 of the weight random access memory 124, as described in the following fifty-first figure), and the output buffer 1104 The other three texts maintain their previous values. After the instruction at address 14 is executed, for each neural processing unit group 4901, the word OUTBUF [3] in the output buffer 1104 is written into the tanh (C) value corresponding to the long-term short-term memory cell 4600, and the output buffer The other three characters of the device 1104 maintain their previous values. After the instruction at address 16 is executed, for each neural processing unit group 4901, all four characters of the output buffer 1104 are written into the new memory cell output (H) value corresponding to the long-term and short-term memory cell 4600. The aforementioned execution flow of addresses 6 to 16 (that is, the execution of address 2 is excluded, because address 2 is not part of the program loop) will be repeated 30 times, and the program returned to address 3 as address 17 Loop.

The fifty-first figure is a table showing a program stored in the program memory 129 of the neural network unit 121. The neural network unit 121 of Fig. 19 executes and uses data and weights according to the configuration of Fig. 50 to achieve calculations related to the long-term and short-term memory cells. The example program in Figure 51 contains 18 non-framework instructions at addresses 0 to 17, respectively. The instruction at address 0 is an initialization instruction that clears the accumulator 202 and initializes the loop counter 3804 to a value of 31 to execute the loop group 31 times (instructions at addresses 1 to 17). This initialization command also initializes the to-be-written row of the data random access memory 122 (for example, the register 2606 of the twenty-sixth / thirty-nine figure) to the value 1, and the first in the address 16 instruction After the first execution, this value will increase to 3. Preferably, the initialization command does not place the neural network unit 121 in a wide configuration. Thus, the neural network unit 121 is configured with 512 neural processing units 126. As described in the subsequent chapters, during the execution of instructions at addresses 0 to 17, the 128 neural processing unit group 4901 composed of the 512 neural processing units 126 operates as 128 corresponding long-term and short-term memory cells 4600.

The instructions at addresses 1 and 2 do not belong to the loop group of the program and will only be executed once. These instructions generate the initial memory cell output (H) value (for example, 0) and write it to all text in the output buffer 1104. The instruction at address 1 reads the initial H value from row 0 of the data random access memory 122 and places it in the accumulator 202 cleared by the instruction at address 0. The instruction at address 2 (OUTPUT PASSTHRU, NOP, CLR ACC) will pass the value of accumulator 202 to the output buffer 1104, as shown in Figure 50. The "NOP" mark in the output instruction at address 2 (and other output instructions in Figure 51) indicates that the output value will only be written to the output buffer 1104, and will not be written to memory, that is, it will not The data is written into the random access memory 122 or the weighted random access memory 124. The instruction at address 2 will be cleared Dividing the accumulator 202.

The instructions at addresses 3 to 17 are located in the loop group, and the number of executions is the value of the loop count (such as 31).

Each execution of the instructions at addresses 3 to 6 will calculate the tanh (C ') value of the current time step and write it into the text OUTBUF [3]. This text will be used by the instruction at address 11. More precisely, the multiply-accumulate instruction at address 3 reads the memory associated with this time step from the current read row of data random access memory 122 (such as row 2, 4, 6, and so on to row 62). The cell inputs the (X) value, reads the value of Wc from column 0 of the weighted random access memory 124, and multiplies the aforementioned values to generate a product that is added to the accumulator 202 cleared by the instruction at address 2.

The multiply accumulate instruction at address 4 (MULT-ACCUM OUTBUF [0], WR ROW 1) will read the H value from the text OUTBUF [0] (that is, all four neural processing units 126 of the neural processing unit group 4901), from Column 1 of the weighted random access memory 124 reads the value of Uc, and multiplies the aforementioned values to generate a second product and adds it to the accumulator 202.

The add weight instruction to the accumulator instruction (ADD_W_ACC WR ROW 2) at address 5 reads the value of Bc from column 2 of the weight random storage memory 124 and adds it to the accumulator 202.

The output instruction at address 6 (OUTPUT TANH, NOP, MASK [0: 2], CLR ACC) performs a hyperbolic tangent start function on the value of accumulator 202, and only writes the execution result to the text OUTBUF [3] (also That is, only the neural processing unit 126 with the remainder of the number divided by 4 in the neural processing unit group 4901 will write this result), and the accumulator 202 will be cleared. In other words, the output instruction at address 6 will obscure the text OUTBUF [0], OUTBUF [1], and OUTBUF [2] (as indicated by the instruction term MASK [0: 2]) and maintain their current values, as shown in Figure 50. In addition, the output instruction at address 6 is not written into the memory (as indicated by the instruction term NOP).

Each execution of the instructions at addresses 7 to 10 will calculate the input gate (I) value, forget gate (F) value and output gate (O) value at the current time step and write them into the text OUTBUF [0], OUTBUF respectively. [1], and OUTBUF [2], these values will be used by instructions at addresses 11, 12, and 15. More precisely, the multiply accumulate instruction at address 7 reads the memory associated with this time step from the current read row of data random access memory 122 (such as row 2, 4, 6 and so on to row 62). The cell input (X) value reads Wi, Wf, and Wo values from column 3 of the weighted random access memory 124, and multiplies the foregoing values to generate a product that is added to the accumulator 202 cleared by the instruction at address 6. More precisely, in the neural processing unit group 4901, the neural processing unit 126 with the remainder of the number divided by 4 is 0 and calculates the product of X and Wi, and the neural processing unit 126 with the remainder of the number divided by 4 is 1 calculates X and The product of Wf, and the neural processing unit 126 with a remainder of 2 divided by 4 calculates the product of X and Wo.

The multiply-accumulate instruction at address 8 reads the H value from the text OUTBUF [0] (that is, all four neural processing units 126 of the neural processing unit group 4901), and reads Ui from column 4 of the weight random access memory 124. , Uf and Uo values, and multiplying the aforementioned values to generate a second product is added to the accumulator 202. More precisely, in the neural processing unit group 4901, the neural processing unit 126 with the remainder of the number divided by 4 is 0 will calculate the product of H and Ui, and the neural processing unit with the remainder of the number divided by 4 is 1. The element 126 calculates the product of H and Uf, and the neural processing unit 126 with a remainder of 2 divided by 4 calculates the product of H and Uo.

The instruction of adding weight text to the accumulator at address 9 (ADD_W_ACC WR ROW 2) will read the values of Bi, Bf and Bo from column 5 of the weight random storage memory 124 and add them to the accumulator 202. More precisely, in the neural processing unit group 4901, the neural processing unit 126 with the remainder of the number divided by 4 is 0 to perform an addition calculation of the Bi value, and the neural processing unit 126 with the remainder of the number divided by 4 is 1 performs Bf The numerical value is added, and the neural processing unit 126 whose number is divided by 4 and whose remainder is 2 performs the addition of the Bo value.

The output instruction at address 10 (OUTPUT SIGMOID, NOP, MASK [3], CLR ACC) will execute an S-shaped start function on the value of accumulator 202 and write the calculated I, F and O values to the text OUTBUF [0 ], OUTBUF [1] and OUTBUF [2]. This instruction does not clear the accumulator 202 without writing to the memory. In other words, the output instruction at address 10 will obscure the text OUTBUF [3] (as indicated by the instruction term MASK [3]) and maintain the current value of the text (that is, C '), as shown in Figure 50.

Each execution of the instructions at addresses 11 to 13 calculates a new memory cell state (C) value generated by the current time step and writes it into the weight random access memory 124 in column 6 for the next time step (also (It is used for the instruction of address 12 in the next loop execution). More precisely, this value is written in column 6 corresponding to the number 4 in the four lines of text of the neural processing unit group 4901. The remainder is 3. . In addition, each execution of the instruction at address 14 will write the value of tanh (C) OUTBUF [3] is used for the instruction of address 15.

More precisely, the multiply-accumulate instruction at address 11 (MULT-ACCUM OUTBUF [0], OUTBUF [3]) reads the value of the input gate (I) from the text OUTBUF [0], and reads it from the text OUTBUF [3] Candidate memory cell state (C ') values are multiplied to generate a first product and added to the accumulator 202 cleared by the instruction at address 10. More specifically, each of the four neural processing units 126 in the four neural processing units 126 of the neural processing unit group 4901 calculates the first product of the I value and the C 'value.

The multiply-accumulate instruction at address 12 (MULT-ACCUM OUTBUF [1], WR ROW 6) instructs the neural processing unit 126 to read the value of the forget gate (F) from the text OUTBUF [1], and randomly access the memory 124 from the weight. Column 6 reads its corresponding text, and multiplies it to generate a second product and adds the first product in the accumulator 202 to the instruction at address 11. More precisely, for the neural processing unit 126 with the remainder of the label divided by 4 in the neural processing unit group 4901, the text read from column 6 is the current memory cell state (C) value calculated in the previous time step. The sum of the first product and the second product is the new memory cell state (C). However, for the other three neural processing units 126 of the neural processing unit group 4901, the text read from column 6 is a value that is ignored, because the cumulative value generated by these values will not be used, and That is, the instructions at addresses 13 and 14 will not be put into the output buffer 1104 and will be cleared by the instructions at address 14. In other words, only the new memory cell state (C) value generated by the neural processing unit 126 with the remainder of the number 3 divided by 4 in the neural processing unit group 4901 will be used, that is, the instructions at addresses 13 and 14 Make use. For the second to thirty-first executions of the instruction at address 12, the value of C read from row 6 of the weighted random access memory 124 is written by the instruction at address 13 during the previous execution of the loop group Value. However, for the first execution of the instruction at address 12, the value of C in column 6 was written by the framework program before starting the non-schema program in Figure 51 or after an adjusted version of one of the non-schema programs. Into the initial value.

The output instruction at address 13 (OUTPUT PASSTHRU, WR ROW 6, MASK [0: 2]) will only pass the value of accumulator 202, that is, the calculated C value, to the text OUTBUF [3] (that is, only neural processing In the unit group 4901, the neural processing unit 126 whose number is divided by 4 and whose remainder is 3 will write the calculated C value to the output buffer 1104), and the weight 6 of the random access memory 124 is the updated one The output buffer 1104 is written as shown in the fiftieth figure. In other words, the output instruction at address 13 will obscure the characters OUTBUF [0], OUTBUF [1] and OUTBUF [2] and maintain their current values (ie, I, F and O values). As mentioned above, only the value of C in column 4 corresponding to the number 4 in the four lines of the text of the neural processing unit group 4901 and the remainder of 3 will be used, that is, used by the instruction at address 12; therefore, non-architecture The program will ignore the values in row 6-2, row 4-6 in row 6 of weight random access memory 124, and so on to the values in row 508-510, as shown in Figure 50 (that is, I, F, and O value).

The output instruction at address 14 (OUTPUT TANH, NOP, MASK [0: 2], CLR ACC) performs a hyperbolic tangent start function on the value of accumulator 202, and writes the calculated tanh (C) value into the text OUTBUF [3], this instruction does not clear the accumulator 202 without writing to the memory body. The output instruction at address 14 is the same as the output instruction at address 13. It will obscure the characters OUTBUF [0], OUTBUF [1] and OUTBUF [2] and maintain their original values, as shown in Figure 50.

Each execution of the instructions at addresses 15 to 16 will calculate the memory cell output (H) value generated by the current time step and write it into the second row behind the current output row of the data random access memory 122. The value will be Read by the framework program and used for the next time step (that is, used by the instructions at addresses 3 and 7 in the next loop execution). More precisely, the multiply-accumulate instruction at address 15 reads the output gate (O) value from the text OUTBUF [2], reads the tanh (C) value from the text OUTBUF [3], and multiplies it to produce a The product is added to the accumulator 202 cleared by the instruction at address 14. More precisely, each of the four neural processing units 126 of the four neural processing units 126 of the neural processing unit group 4901 calculates the product of the value O and tanh (C).

The output instruction at address 16 will pass the value of accumulator 202 and write the calculated H value in column 3 in the first execution, and write the calculated H value in column 5 in the second execution, and so on. In the thirty-first execution, the calculated H values are written into column 63, as shown in the fifty figure, and these values will be used by the instructions at addresses 4 and 8. In addition, as shown in Figure 50, these calculated H values will be placed in the output buffer 1104 for subsequent use of the instructions at addresses 4 and 8. The output instruction at address 16 will clear the accumulator 202. In one embodiment, the design of the long-term and short-term memory cell 4600 is such that the output instruction at address 16 (and / or the output instruction at address 22 in the forty-eighth figure) has an activation function, such as an S-type or hyperbolic tangent Function instead of passing the accumulator 202 value.

The loop instruction at address 17 decrements the loop counter 3804 and returns to the instruction at address 3 if the new loop counter 3804 value is greater than zero.

It can be found that because of the feedback and shielding capabilities of the output buffer 1104 in the embodiment of the neural network unit 121 in Figure 49, the number of instructions in the loop group of the non-architecture program in Figure 51 is compared. The non-structural instructions in Figure 48 were reduced by approximately 34%. In addition, because of the feedback and shielding capabilities of the output buffer 1104 in the embodiment of the neural network unit 121 in FIG. 49, the memory allocation in the data random access memory 122 of the non-architecture program in FIG. 51 The number of matching time steps is roughly three times that of the forty-eighth picture. The aforementioned improvements are helpful for certain framework program applications that use the neural network unit 121 to perform long-term and short-term memory cell calculations, especially for applications where the number of long-term and short-term memory cells in the long-term and short-term memory cells is less than or equal to 128.

The embodiments of the forty-seventh to fifty-one diagrams assume that the weights and offset values in each time step remain unchanged. However, the present invention is not limited to this, and other embodiments in which the weights and offset values change with time steps are also within the scope of the present invention. The weight random access memory 124 is not as shown in Figures 47 to 50. Fill in a single set of weights and offsets, but fill in different sets of weights and offsets at each time step. The weights of the non-architecture programs in the forty-eighth to fifty-first pictures are stored randomly in memory 124. Of adjustment.

Basically, in the foregoing embodiments of the forty-seventh to fifty-one figures, the weights, offsets, and intermediate values (such as C, C 'values) are stored in the weight random access memory 124, and the input and output values (Such as X, H The value) is stored in the data random access memory 122. This feature is beneficial to the embodiment in which the data random access memory 122 is dual-port and the weight random access memory 124 is a port. This is because the non-architecture program and the architecture program to the data random access memory 122 will have more changes. More traffic. However, because the weight random access memory 124 is relatively large, in another embodiment of the present invention, it is a memory that stores non-architecture and structure program write values interchangeably (that is, the data random access memory 122 and the weight random access memory). Access memory 124). That is, the values of W, U, B, C ', tanh (C) and C are stored in the data random access memory 122 and the values of X, H, I, F and O are stored in the weight random access memory. Body 124 (the modified embodiment of the forty-seventh figure); and W, U, B, and C values are stored in the data random access memory 122 and X and H values are stored in the weight random access memory 124 (the fifty-second embodiment after adjustment). Because the weighted random access memory 124 is larger, these embodiments can process more time steps in a batch. For applications that use the neural network unit 121 to execute calculation-based programs, this feature is beneficial to some applications that can benefit from more time steps and can be designed for the port (such as weight random access memory). 124) Provide sufficient bandwidth.

The fifty-second figure is a block diagram showing an embodiment of a neural network unit 121. The neural processing unit group in this embodiment has output buffer masking and feedback capabilities, and shares an activation function unit 1112. The neural network unit 121 of the fifty-second figure is similar to the neural network unit 121 of the forty-seventh figure, and the components with the same reference numerals in the figure are similar. However, the four activation function units 212 of FIG. 49 are replaced by a single shared activation function unit 1112 in this embodiment. This single start function unit will receive four outputs 217 from four accumulators 202 and generate four outputs to the texts OUTBUF [0], OUTBUF [1], OUTBUF [2] and OUTBUF [3]. The operation mode of the neural network unit 212 in the fifty-second diagram is similar to the embodiment described in the forty-ninth to fifty-one diagrams above, and the operation of the shared activation function unit 1112 is similar to the eleventh to the eleventh to The embodiment described in the thirteenth figure.

The fifty-third diagram is a block diagram showing the neural network of the forty-ninth diagram when the neural network unit 121 performs a calculation related to a hierarchy of 128 long-term and short-term memory cells 4600 in the forty-sixth diagram. The data random access memory 122, the weight random access memory 124, and the data buffer in the output buffer 1104 of the unit 121 are another embodiment of data allocation. The example of Figure 53 is similar to the example of Figure 50. However, in the 53rd figure, the values of Wi, Wf and Wo are in column 0 (instead of the 50th figure in column 3); Ui, Uf and Uo values are in the column 1 (instead of 5th Figure 10 is in column 4); Bi, Bf and Bo values are in column 2 (as opposed to column 50 in Figure 50); value C is in column 3 (not column 50 as in Figure 50) ). In addition, the content of the output buffer 1104 of Figure 53 is similar to that of Figure 50, but because of the difference between the non-structural programs of Figure 54 and Figure 51, the content of the third column (that is, The values of I, F, O, and C ') appear in the output buffer 1104 after the instruction at address 7 is executed (instead of the instruction at address 10 as shown in Figure 50); the contents of the fourth column (that is, I, F, O, and C values) appear in the output buffer 1104 after the instruction at address 10 is executed (instead of the instruction at address 13 as shown in Figure 50); the contents of the fifth column (that is, I, F, O And tanh (C) value) is the instruction at address 11 Appears in the output buffer 1104 after execution (instead of the instruction at address 14 as shown in Figure 50); and the content of the sixth column (the H value) appears in the output buffer 1104 after execution of the instruction at address 13. (Instead of the address 16 as shown in Figure 50), please refer to the details below.

The fifty-fourth figure is a table showing a program stored in the program memory 129 of the neural network unit 121. This program is executed by the neural network unit 121 of the forty-ninth figure and according to the fifty-third figure. Configure usage data and weights to achieve calculations related to long and short-term memory cells. The example program of figure 54 is similar to the program of figure 51. More precisely, the instructions in addresses 54 to 54 are the same as those in address 51 to 51; the instructions at addresses 7 and 8 in the fifty-fourth illustration are the same as those in the fifty-first illustration. The instructions of 10 and 11; and the instructions of addresses 10 to 14 in the fifty-fourth figure are the same as the instructions of addresses 13 to 17 in the fifty-first figure.

However, the instruction at address 6 in the fifty-fourth figure will not clear the accumulator 202 (in contrast, the instruction at address six in the fifty-first figure will clear the accumulator 202). In addition, the instructions at addresses 7 to 9 in the fifty-first figure do not appear in the non-structural program in the fifty-fourth figure. Finally, with regard to the instruction at address 9 in the fifty-fourth figure and the instruction at address 12 in the fifty-first figure, except for the instruction at address ninety-four in the fifty-fourth figure, the read weight random access memory Column 3 of 124 and the instruction at address 12 in the fifty-first figure are read weights of random access memory except column 6, and the other parts are the same.

Because of the difference between the non-structural program in Figure 54 and the non-structural program in Figure 51, the number of rows of weighted random access memory 124 used in the configuration of Figure 53 is reduced by three, and the program returns The number of instructions in the circle will also be reduced by three. Figure 54 of the non-structural program The size of the loop group is substantially only half of the size of the loop group in the non-structural program of Figure 48, and is roughly only 80% of the size of the loop group in the non-structural program of Figure 51.

The fifty-fifth figure is a block diagram showing a part of a neural processing unit 126 according to another embodiment of the present invention. More precisely, for a single neural processing unit 126 of the plurality of neural processing units 126 in the forty-ninth figure, the figure shows the multiplexing register 208 and its associated inputs 207, 211, and 4905, and multiple The work register 705 is associated with inputs 206, 711, and 4907. In addition to the input of the forty-ninth figure, the multiplexer register 208 and the multiplexer register 705 of the neural processing unit 126 each receive a group number (index_within_group) input 5599. In-group number input 5599 indicates the number of a particular neural processing unit 126 within its neural processing unit group 4901. Therefore, for example, taking an embodiment in which each neural processing unit group 4901 has four neural processing units 126 as an example, within each neural processing unit group 4901, one of the neural processing units 126 is numbered in its group and entered 5599 receives the value zero, one of the neural processing units 126 receives the value one in its group number input 5599, one of the neural processing units 126 receives the value two in its group number input 5599, and one of the neural processing units 126 receives the value three in its group number entry 5599. In other words, the number in the group received by the neural processing unit 126 is 5599, which is the remainder of the number of the neural processing unit 126 in the neural network unit 121 divided by J, where J is the number in the neural processing unit group 4901. Number of neural processing units 126. Therefore, for example, the neural processing unit 73 enters the number 5599 in its group to receive the value one, and the neural processing unit 353 in its group The internal number input 5599 receives the value three, and the neural processing unit 6 enters the number 5599 in its group to receive the value two.

In addition, when the control input 213 specifies a preset value, which is represented as "SELF" here, the multiplexer register 208 will select the output buffer 1104 corresponding to the number 5599 in the group to output 4905. Therefore, when a non-architecture instruction is designated to receive data from the output buffer 1104 (indicated as OUTBUF [SELF] in the instructions at addresses 2 and 7 in Figure 57), the value of each neural processing unit 126 The multiplexer register 208 receives its corresponding text from the output buffer 1104. Therefore, for example, when the neural network unit 121 executes the non-architecture instructions at addresses 2 and 7 in the fifty-seventh figure, the multiplexer register 208 of the neural processing unit 73 will select the second among the four inputs 4905. (Number 1) input to receive the text 73 from the output buffer 1104, the multiplexing register 208 of the neural processing unit 353 will select the fourth (number 3) input from the four inputs 4905 to receive from the output buffer The text 353 of 1104, and the multiplexing register 208 of the neural processing unit 6 will select the third (number 2) input among the four inputs 4905 to receive the text 6 from the output buffer 1104. Although it is not used in the non-architecture program in the fifty-seventh figure, the non-architecture instructions can also use the SELF value (OUTBUF [SELF]) to specify the reception of data from the output buffer 1104 and the control input 713 to specify a preset value The multiplexer register 705 of each neural processing unit 126 receives its corresponding text from the output buffer 1104.

The fifty-sixth diagram is a block diagram showing the neural network unit 121 when the neural network unit performs calculations of the Jordan time recursive neural network associated with the forty-third diagram and uses the embodiment of the fifty-fifth diagram. An example of the data arrangement in the data random access memory 122 and the weight random access memory 124 is. The weight allocation in the weight random access memory 124 in the figure is the same as the example in Figure 44. The configuration of the values in the data random access memory 122 in the figure is similar to the example in Figure 44 except that in this example, each time step has a corresponding pair of two rows of memory to load the input layer node D. Value and output layer node Y value instead of using a set of four rows of memory as in the example in Figure 44. That is, in this example, the values of the hidden layer Z and the content layer C are not written into the data random access memory 122. Instead, the output buffer 1104 is used as a type of draft memory that is one of the hidden layer Z value and the content layer C value, as described in the non-architecture program in FIG. 57. The aforementioned feedback characteristics of the OUTBUF [SELF] output buffer 1104 can make non-architecture programs operate faster (this is to perform two writes and two reads to the data random access memory 122 to The output buffer 1104 performs two writes and two reads instead) and reduces the space of the data random access memory 122 used in each time step, so that the data random access memory 122 of this embodiment The loaded data can be used for about twice the time steps of the embodiment of Figures 44 and 45, as shown in the figure, which is 32 time steps.

The fifty-seventh figure is a table showing a program stored in the program memory 129 of the neural network unit 121. This program is executed by the neural network unit 121 and uses data and weights according to the configuration of the fifty-sixth figure. In order to achieve Jordan time recursive neural network. The non-structural program in Figure 57 is similar to the non-structural program in Figure 45. The differences are as follows.

The example program in Figure 57 has twelve non-framework instructions at addresses 0 to 11, respectively. The initialization instruction at address 0 clears the accumulator 202 and initializes the value of the loop counter 3804 to 32, so that the loop group (instructions at addresses 2 to 11) is executed 32 times. The output instruction at address 1 places the zero value of accumulator 202 (cleared by the instruction at address 0) into output buffer 1104. It can be observed that during the execution of the instructions at addresses 2 to 6, the 512 neural processing units 126 correspond to and operate as 512 hidden layer nodes Z, and the execution of the instructions at addresses 7 to 10 In the process, it corresponds to and operates as 512 output layer nodes Y. In other words, the 32 executions of the instructions at addresses 2 to 6 will calculate the hidden layer node Z values of 32 corresponding time steps and put them into the output buffer 1104 for the correspondence of the instructions at addresses 7 to 9. 32 executions are used to calculate the output layer node Y of the 32 corresponding time steps and write them into the data random access memory 122, and provide the corresponding 32 executions of the address 10 instruction to use this The content layer node C of 32 corresponding time steps is put into the output buffer 1104. (The content layer node C placed in the 32nd time step in the output buffer 1104 will not be used.)

In the first execution of instructions 2 and 3 (ADD_D_ACC OUTBUF [SELF] and ADD_D_ACC ROTATE, COUNT = 511), each of the 512 neural processing units 126 will output 512 of the output buffer 1104. The content C values of each content node are accumulated to its accumulator 202. These content node C values are generated and written by the execution of instructions at addresses 0 to 1. In the second execution of the instructions at addresses 2 and 3, each of the 512 neural processing units 126 will accumulate the 512 content node C values of the output buffer 1104 to it. Accumulator 202. These content node C values are generated and written by instruction execution at addresses 7 to 8 and 10. More precisely, the instruction at address 2 will instruct the multiplexer register 208 of each neural processing unit 126 to select its corresponding output buffer 1104 text, as described above, and add it to the accumulator 202; address 3 The instruction will instruct the neural processing unit 126 to rotate the content node C value in a 512-character rotator. The 512-character rotator is collectively operated by the multiplexed register 208 of the 512 neural processing units. So that each neural processing unit 126 can accumulate the 512 content node C values to its accumulator 202. The instruction at address 3 does not clear the accumulator 202, so the instructions at addresses 4 and 5 can add the D value of the input layer node (multiplied by its corresponding weight) to the value accumulated by the instructions at addresses 2 and 3. Content layer node C value.

In each execution of the instructions at addresses 4 and 5 (MULT-ACCUM DR ROW +2, WR ROW 0 and MULT-ACCUM ROTATE, WR ROW +1, COUNT = 511), one of the 512 neural processing units 126 Each neural processing unit 126 performs 512 multiplication operations to associate the row of the data random access memory 122 with the current time step (for example, for time step 0, it is column 0, and for time step 1, it is Column 2, and so on. For time step 31, 512 input node D values are multiplied by weights in columns 0 to 511 of the random access memory 124 corresponding to this neural processing unit 126. Weights to generate 512 products, and together with the accumulated results of the instructions of addresses 2 and 3 for the 512 content node C values, they are added to the accumulator 202 of the corresponding neural processing unit 126 to calculate the hidden value. Node Z layer value.

The instruction at address 6 (OUTPUT PASSTHRU, In each execution of NOP, CLR ACC), the values of the 512 accumulators 202 of the 512 neural processing units 126 are transmitted and written to the corresponding characters in the output buffer 1104, and the accumulators 202 are cleared.

During the execution of instructions at addresses 7 and 8 (MULT-ACCUM OUTBUF [SELF], WR ROW 512 and MULT-ACCUM ROTATE, WR ROW +1, COUNT = 511), each of these 512 neural processing units 126 The neural processing unit 126 will perform 512 multiplication operations, and the 512 hidden node Z values in the output buffer 1104 (produced and written by the corresponding execution of the instructions at addresses 2 to 6) will be multiplied by the random storage. The weights in the columns 512 to 1023 of the memory 124 corresponding to the trip of the neural processing unit 126 are taken to generate 512 multiply-accumulate and add to the accumulator 202 of the corresponding neural processing unit 126.

In each execution of the instruction at address 9 (OUTPUT ACTIVATION FUNCTION, DR OUT ROW +2), a start function (such as hyperbolic tangent function, S-shaped function, correction function) will be executed for the 512 accumulated values to calculate Output node Y value, this output node Y value will be written into the column of the data random access memory 122 corresponding to the current time step (for example, for time step 0, it is column 1 and for time step 1, it is Is column 3, and so on, for time step 31 it is column 63). The instruction at address 9 does not clear accumulator 202.

In each execution of the instruction at address 10 (OUTPUT PASSTHRU, NOP, CLR ACC), the 512 values accumulated by the instructions at addresses 7 and 8 will be put into the output buffer 1104 for the instructions at addresses 2 and 3. The next execution is used and the accumulator 202 is cleared.

The loop instruction at address 11 makes the loop counter 3804 The value is decremented, and if the new loop counter 3804 value is still greater than zero, an instruction to return to address 2 is instructed.

As described in the section corresponding to figure 44, in the example of executing a Jordan time recursive neural network using a non-architecture program of figure 57, although an activation function is applied to the value of accumulator 202 to generate an output Layer node Y value, however, this example assumes that the accumulator 202 value is passed to the content layer node C before the start function is applied, rather than the actual output layer node Y value. However, for the Jordan time recursive neural network that applies the activation function to the value of the accumulator 202 to generate the content layer node C, the instruction at address 10 will be removed from the non-architecture program in Figure 57. In the embodiment described herein, the Elman or Jordan time-recurrent neural network has a single hidden node layer (such as the fortieth and forty-two graphs), but it should be understood that these processors 100 and neural networks The embodiment of unit 121 may use a method similar to that described herein to efficiently perform computations associated with a temporal recurrent neural network with multiple hidden layers.

As described in the previous section corresponding to the second figure, each neural processing unit 126 operates as a neuron in an artificial neural network, and all neural processing units 126 in the neural network unit 121 are processed in a large-scale parallel. This method effectively calculates the neuron output at one level of this network. The parallel processing method of this neural network unit, especially the rotator composed of the multiplex register of the neural processing unit, is not intuitively conceivable in the traditional way of calculating the output of the neuron layer. Furthermore, traditional approaches usually involve calculations that are associated with a single neuron or a very small subset of neurons (for example, using parallel calculations). The arithmetic unit performs multiplication and addition calculations), and then continues to perform calculations associated with a neuron below the same level, and so on in a sequential manner until the calculation of all neurons in this level is completed. In contrast, in the present invention, in each time-frequency period, all the neural processing units 126 (neurons) of the neural network unit 121 execute a small set (e.g. A multiplication and accumulation). After about M time-frequency periods have ended-M is the number of nodes connected in the current level-the neural network unit 121 will calculate the output of all neurons. In many artificial neural network configurations, because there are a large number of neural processing units 126, the neural network unit 121 can calculate its neuron output value for all neurons in the entire hierarchy at the end of M time-frequency cycles. As described in this article, this calculation is efficient for all types of artificial neural network calculations. These artificial neural networks include, but are not limited to, recursive and temporal recurrent neural networks such as Elman, Jordan, and long-term and short-term memory networks. . Finally, although the neural network unit 121 is configured as 512 neural processing units 126 (for example, in a wide text configuration) to perform time-recurrent neural network calculations in the embodiment herein, the present invention is not limited to this. The neural network unit 121 is configured as an embodiment of 1024 neural processing units 126 (for example, in a narrow text configuration) to perform the calculation of the time recursive neural network unit, and as described above, the number of neural processing units 126 other than 512 and 1024 The neural network unit 121 also belongs to the scope of the present invention.

However, the above are only the preferred embodiments of the present invention. When the scope of implementation of the present invention cannot be limited by this, that is, the simple equivalent changes made according to the scope of the patent application and the description of the invention All modifications and modifications are still within the scope of the invention patent. For example, software can perform the functions, manufacturing, shaping, simulation, description, and / or testing of the devices and methods described in the present invention. This can be achieved by general programming languages (such as C, C ++), hardware description language (HDL) including Verilog HDL, VHDL, etc., or other existing programs. This software can be installed on any known computer-usable medium, such as magnetic tape, semiconductor, magnetic disk, optical disk (such as CD-ROM, DVD-ROM, etc.), network connection, wireless or other communication media. Embodiments of the devices and methods described herein may be included in a semiconductor intellectual property core, such as a micro-processing core (such as an implementation in a hardware description language) and converted into hardware through the fabrication of integrated circuits. In addition, the devices and methods described herein may include a combination of hardware and software. Therefore, any embodiments described herein are not intended to limit the scope of the invention. In addition, the present invention can be applied to a microprocessor device of a general-purpose computer. Finally, those skilled in the art can use the concepts and embodiments disclosed in the present invention as a basis to design and adjust different structures to achieve the same purpose without departing from the scope of the present invention.

Claims (21)

A neural network unit includes: a programmable index specifying first and second actions; a first memory containing a first operand; a second memory containing a second operand; a plurality of neural processes A unit (NPU), each of the neural processing units includes: an accumulator; and an arithmetic unit that performs a series of multiplication operations on a plurality of pairs of the first and second operands received by the first and second memories to Generating a series of products, the arithmetic unit and performing a series of addition operations on the series of products to accumulate an accumulative value in the accumulator; and a plurality of start-up units executing a start function on the accumulator value in the accumulator to generate Result; when the indicator specifies the first action, the neural network unit writes the result generated by the plurality of activation units into the first memory; when the indicator specifies the second action, the neural network unit applies the result The accumulated value in the accumulator is written into the first memory. For example, the neural network unit of the first patent application range, wherein the width of the result produced by the activation function unit is the same as the first and second operands, and the width of the accumulator is greater than the first and second operands. And, when the indicator specifies the second action, the neural network unit writes a part of the accumulator into the first memory, and the width of each part is the same as that of the first and second operands. . For example, the neural network unit of the first patent application scope, wherein the neural network unit converts the accumulated value into a standard type, and the width of the standard type is twice that of the first and second operands, and, When the indicator specifies the second action, the neural network unit writes the accumulated value of the standard type into the first memory. For example, the neural network unit of the scope of application for patent No. 1 further includes: the accumulated value is a fixed-point value. For example, the neural network unit of the fourth item of the patent application scope further includes: a temporary register, which can be programmed by using one of the positions of one of the decimal points of the fixed-point accumulation value. For example, the neural network unit of the fourth item of the patent application, wherein the width of the accumulator is sufficient to accommodate the arithmetic unit to perform the series of multiplication operations and addition operations to generate the accumulated value without losing accuracy. For example, the neural network unit of the sixth scope of the application for patent, wherein the series of multiplication operations and addition operations using integer arithmetic to generate the accumulated value is at least 512 integer multiplication operations and additions for each of the neural processing units. The series of operations. For example, the neural network unit of the first patent application range, wherein the first memory includes a first address input for designating a row of characters having the width of the first operand, and the second memory includes a The second address input is used to designate a column of text having the width of the second operand, and the neural network unit further includes a sequencer to generate the first bit provided to the first and second address inputs. Address and second address. For example, the neural network unit of the eighth patent application scope, wherein, in each clock cycle of the plurality of clock cycles, the sequencer generates the first address to designate a different column of the first memory for The neural network unit writes a different part of the accumulated value into the different columns. For example, the neural network unit of the eighth patent application scope further includes: a memory, which can be programmed using instructions executable by the sequencer. A processor includes: a neural network unit including: a programmable index specifying first and second actions; a first memory containing a first operation element; a second memory containing a second operation A plurality of neural processing units (NPUs), each of which includes: an accumulator; and an arithmetic unit that executes a plurality of pairs of the first and second operands received by the first and second memories A series of multiplication operations to generate a series of products, the arithmetic unit and a series of addition operations performed on the series of products to accumulate an accumulated value in the accumulator; and a plurality of activation units to accumulate the accumulation in the accumulator The value executes a start function to produce a result; when the indicator specifies the first action, the neural network unit writes the result generated by the plurality of startup units into the first memory; when the indicator specifies the second action, the The neural network unit writes the accumulated value in the accumulator into the first memory. For example, the processor of the scope of patent application No. 11 further includes: a plurality of architecture registers, which can be accessed by instructions of an instruction set architecture of the processor; and at least one execution unit other than the neural network unit, which is used to: Perform arithmetic operations on the operands loaded in the architecture register; wherein the instruction set architecture of the processor includes an instruction to read the first memory of the neural network unit, and the processor executes A program to read the accumulated value from the first memory and execute an activation function other than the activation functions executed by the activation function unit of the neural network unit on the accumulated value. For example, the processor according to item 12, wherein the activation function other than the activation functions executed by the activation function unit of the neural network unit is a soft maximum activation function. A method for operating a neural network unit. The neural network unit includes a programmable index to specify first and second actions, a first memory to load a first operand, and a second memory to load a second An arithmetic unit, and a plurality of neural network units, each of which includes an accumulator and an arithmetic unit, the method includes: using each arithmetic unit, a plurality of sets of the received by the first and second memories; The first and second operands perform a series of multiplication operations to generate a series of products; each of the arithmetic units performs a series of addition operations on the series of products to accumulate an accumulated value in the accumulator; use a plurality of start A unit that executes a start function on the accumulated value in the accumulator to generate a result; when the index specifies the first action, writes the result generated by the plurality of start units into the first memory; and when the index specifies In the second action, the accumulated value in the accumulator is written into the first memory. For example, the method of claim 14 in the patent scope, wherein the width of the result produced by the activation function unit is the same as the first and second operands, the width of the accumulator is larger than the first and second operands, and, When the indicator specifies the second action, the method writes a part of the accumulator into the first memory, and the width of each part is the same as the first and second operands. For example, the method for applying for item 14 of the patent scope further includes: converting the accumulated value into a standard pattern, and the width of the standard pattern is twice that of the first and second operands; wherein, when the index specifies the second Action, the method writes the accumulated value of the standard type into the first memory. For example, the method of claim 16 in the patent application range, wherein the accumulated value is a fixed-point value. For example, the method of claim 17 of the scope of patent application further includes: using a position of one of the decimal points of the fixed-point accumulation value to program a register. For example, the method of claim 17 in the scope of patent application, wherein the width of the accumulator is sufficient to accommodate the arithmetic unit to perform the series of multiplication operations and addition operations to generate an accumulated value without losing accuracy. For example, the method of claim 19 in the scope of patent application, wherein the series of multiplication and addition operations using integer arithmetic to generate the accumulated value is, for each of the neural processing units, a series of at least 512 integer multiplication and addition operations . For example, the method of claim 14 in the patent application range, wherein the first memory includes a first address input, designating a row of characters having the width of the first operand, and the second memory includes a second address. Input, specifying a column of text with the width of the second operand, and the method further includes using a sequencer of the neural network unit to generate a first address and a second address to the first and Enter the second address.