TWI601062B - Apparatus employing user-specified binary point fixed point arithmetic - Google Patents

Description

Use user-specified two-dimensional fixed-point arithmetic Device

This application claims the international priority of the following U.S. Provisional Application. The entire contents of these priority cases are incorporated herein by reference.

This application is related to the following US provisional application filed concurrently. The entire contents of these related applications are incorporated herein by reference.

In recent years, artificial neural networks (ANN) have re-engaged people's attention. These studies are often referred to as deep learning, computer learning, and the like. The increased computing power of general-purpose processors has also boosted people's interest in artificial neural networks decades later. Recent applications of artificial neural networks include language and image recognition. The need to improve the computational efficiency and efficiency of artificial neural networks seems to be increasing.

In view of the above, the present invention provides an apparatus comprising an array of N processing units, a first memory and a second memory. Each of the processing units in the array of N processing units includes an accumulator, an arithmetic unit, a weight input, and a multiplex register. The accumulator has an output. The arithmetic unit has first, second and third inputs and performs an operation thereon to generate a result stored to the accumulator, the first input receiving the output of the accumulator. The weight input is received by the aforementioned second input to the arithmetic unit. The multiplex register has first and second data inputs, an output and a control input, and more The output of the scratchpad is received by the aforementioned third input to the arithmetic unit, and the control input controls the selection of the first and second data inputs. Wherein, the output of the multiplex register is received by the second data input of the multiplex register of an adjacent processing unit, and when the control input selects the second data input, the multiplex register of the N processing units The rotators that work together as the same N characters. The first memory system loads N weight texts of W columns and provides N weight texts of one of the W columns to the corresponding weight inputs of the N processing units of the processing unit array. The second memory system loads N data words of D columns, and provides N data words of one of the D columns to the corresponding first data of the multiplexed registers of the N processing units of the processing unit array. Input.

The invention also provides a processor. The processor includes an instruction set, an array of N processing units, a first memory and a second memory. The instruction set has architectural instructions to indicate that the processor is operational. Each of the processing units in the array of N processing units includes an accumulator, an arithmetic unit, a weight input, and a multiplex register. The accumulator has an output. The arithmetic unit has first, second and third inputs and performs an operation thereon to generate a result stored to the accumulator, the first input receiving the output of the accumulator. The weight input is received by the aforementioned second input to the arithmetic unit. The multiplex register has first and second data inputs, an output and a control input, and the output of the multiplex register is received by the third input to the arithmetic unit, and the control input controls the first and second data. Enter the choice. Wherein, the output of the multiplex register is received by the second data input of the multiplex register of an adjacent processing unit, and when the control input selects the second data input, the N processing The multiplexed registers of the unit work together as a rotator of the same N characters. The first memory system loads N weight texts of W columns and provides N weight texts of one of the W columns to the corresponding weight inputs of the N processing units of the processing unit array. The second memory system loads N data words of D columns, and provides N data words of one of the D columns to the corresponding first data of the multiplexed registers of the N processing units of the processing unit array. Input.

The present invention also provides a computer program product encoded in at least one non-transitory computer usable medium for use by a computer device. The computer program product includes a computer usable code embodied in the medium for describing a device. The computer can use the code including the first code, the second code and the third code. The first code specifies an array of N processing units. Each processing unit in the array includes an accumulator, an arithmetic unit, a weight input, and a multiplex register. The accumulator has an output. The arithmetic unit has first, second and third inputs and performs an operation thereon to generate a result stored to the accumulator, the first input receiving the output of the accumulator. The weight input is received by the aforementioned second input to the arithmetic unit. The multiplex register has first and second data inputs, an output and a control input, and the output of the multiplex register is received by the third input to the arithmetic unit, and the control input controls the first and second data. Enter the choice. Wherein, the output of the multiplex register is received by the second data input of the multiplex register of an adjacent processing unit, and when the control input selects the second data input, the multiplex register of the N processing units The rotators that work together as the same N characters. The second code specifies a first memory to load N columns of N Weight text, and N weight texts of one of the W columns are provided to the corresponding weight inputs of the N processing units of the processing unit array. The third code specifies a second memory to load N data words of the D columns, and provides N data characters of one of the D columns to the multiplex of the N processing units of the processing unit array. The corresponding data input of the register is the first data input.

The specific embodiments of the present invention will be further described by the following examples and drawings.

100‧‧‧ processor

101‧‧‧Command capture unit

102‧‧‧ instruction cache

103‧‧‧Architecture Instructions

104‧‧‧Instruction Translator

105‧‧‧Microinstructions

106‧‧‧Renaming unit

108‧‧‧Reservation station

112‧‧‧Other execution units

114‧‧‧ memory subsystem

116‧‧‧Universal register

118‧‧‧Media register

121‧‧‧Neural Network Unit

122‧‧‧ Data Random Access Memory

124‧‧‧ weighted random access memory

126‧‧‧Neural Processing Unit

127‧‧‧Control and Status Register

128‧‧‧Sequencer

129‧‧‧Program memory

123,125,131‧‧‧ memory address

133, 215, 133A, 133B, 215A, 215B‧‧‧ Results

202‧‧‧ accumulator

204‧‧‧Arithmetic Logic Unit

203,209,217,203A,203B,209A,209B,217A,217B‧‧‧ Output

205, 205A, 205B‧‧‧ register

206, 207, 211, 1811, 206A, 206B, 207A, 207B, 211A, 211B, 1811A, 1811B, 711‧‧ Input

208,705,208A,208B‧‧‧Multiplex register

213,713,803‧‧‧Control input

212‧‧‧Start function unit

242‧‧‧Multiplier

244‧‧‧Adder

246,246A, 246B‧‧‧ product

802‧‧‧Multiplexer

1104‧‧‧ column buffer

1112‧‧‧Start function unit

1400‧‧‧MTNN Directive

1500‧‧‧MFNN Directive

1432, 1532‧‧‧ function

1402, 1502‧‧‧ Code field

1404‧‧‧src1 field

1406‧‧‧src2 field

1408, 1508‧‧‧gpr field

1412, 1512‧‧‧ immediate field

1422, 1522‧‧‧ address

1424, 1426, 1524‧‧‧ data block

1428, 1528‧‧‧ Selected columns

1434‧‧‧Control logic

1504‧‧dst field

1602‧‧‧Reading

1604‧‧‧written 埠

1606‧‧‧Memory array

1702‧‧‧埠

1704‧‧‧buffer

1898‧‧‧Operator selection logic

1896A‧‧‧Wide multiplexer

1896B‧‧‧Narrow multiplexer

242A‧‧‧Wide Multiplier

242B‧‧‧Narrow multiplier

244A‧‧‧ wide adder

244B‧‧‧Narrow adder

204A‧‧‧wide arithmetic logic unit

204B‧‧‧Narrow arithmetic logic unit

202A‧‧‧Wide accumulator

202B‧‧‧Narrow Accumulator

212A‧‧‧wide start function unit

212B‧‧‧Narrow start function unit

2402‧‧‧Convolution Core

2404‧‧‧Data Array

2406A, 2406B‧‧‧ Data Matrix

2602, 2604, 2606, 2608, 2902, 2912, 2914, 2922, 2924, 2926, 2932, 2934, 2942, 2944, 2952, 2954, 2956, 2923, 2962, 2964‧‧ Field

3003‧‧‧ Random bit source

3005‧‧‧ random bit

3002‧‧‧Positive type converter and output binary decimal point aligner

3004‧‧‧ rounder

3006‧‧‧Multiplexer

3008‧‧‧Standard size compressor and saturator

3012‧‧‧ bit selection and saturator

3018‧‧‧ Corrector

3014‧‧‧Reciprocal multiplier

3016‧‧‧ Right shifter

3028‧‧‧Standard size transfer value

3022‧‧‧Hyper tangent module

3024‧‧‧S type module

3026‧‧‧Soft Plus Module

3032‧‧‧Multiplexer

3034‧‧‧ Symbol Recovery

3036‧‧‧Dimensional Converter and Saturator

3037‧‧‧Multiplexer

3038‧‧‧Output register

3402‧‧‧Multiplexer

3401‧‧‧Neuro Processing Unit Pipeline Level

3404‧‧‧Decoder

3412, 3414, 3418‧‧‧ micro operations

3416‧‧‧ microinstructions

3422‧‧‧ model indicators

3502‧‧‧Time-frequency generation logic

3504‧‧‧Time-frequency reduction logic

3514‧‧‧Interface logic

3522‧‧‧ Data Random Access Memory Buffer

3524‧‧‧ weight random access memory buffer

3512‧‧‧Relief indicator

3802‧‧‧Program Counter

3804‧‧‧Circle counter

3806‧‧‧ Iteration number counter

3912, 3914, 3916‧‧‧ fields

4901‧‧‧Neuro Processing Unit Group

4903‧‧‧ mask

4905, 4907, 5599‧‧‧ Input

The first figure shows a block diagram of a processor including a neural network unit (NNU).

The second figure shows a block diagram of a neural processing unit (NPU) of the first figure.

The third figure is a block diagram showing N multiplex registers of N neural processing units of the neural network unit of the first figure, and one column of data is obtained for the data random access memory of the first figure. The operation of a rotator or a circular shifter such as the same N characters is performed.

The fourth diagram is a table showing a program stored in the program memory of the neural network unit of the first figure and executed by the neural network unit.

The fifth diagram shows the timing diagram of the neural network unit executing the program of the fourth figure.

Figure 6A shows the process of executing the fourth figure of the neural network unit of the first figure. Schematic diagram of the block.

Figure 6B is a flow chart showing that the processor of the first figure executes an architectural program to perform a typical multiply-accumulate start function operation of a neuron associated with a hidden layer of an artificial neural network using a neural network unit As the operation performed by the program of the fourth figure.

Figure 7 is a block diagram showing another embodiment of the neural processing unit of the first figure.

The eighth figure is a block diagram showing still another embodiment of the neural processing unit of the first figure.

The ninth figure is a table showing a program stored in the program memory of the neural network unit of the first figure and executed by the neural network unit.

The tenth figure shows a timing diagram of the execution of the program of the ninth diagram by the neural network unit.

Figure 11 is a block diagram showing one embodiment of a neural network unit of the first figure. In the embodiment of the eleventh diagram, a neuron is divided into two parts, that is, a start function unit part and an arithmetic logic unit part (this part includes a shift register part), and each start function unit part is composed of The arithmetic logic units are partially shared.

Figure 12 is a timing diagram showing the execution of the fourth diagram of the neural network unit of the eleventh figure.

The thirteenth figure is a timing chart showing the execution of the program of the fourth figure by the neural network unit of the eleventh figure.

Figure 14 is a block diagram showing the operation of a mobile-to-neural network (MTNN) architecture command and its portion of the neural network unit corresponding to the first map.

The fifteenth diagram is a block diagram showing the operation of a mobile-to-neural network (MTNN) architecture instruction and its portion of the neural network unit corresponding to the first diagram.

Figure 16 is a block diagram showing an embodiment of the data random access memory of the first figure.

Figure 17 is a block diagram showing an embodiment of a weighted random access memory and a buffer of the first figure.

Figure 18 is a block diagram showing a dynamically configurable neural processing unit of the first figure.

Figure 19 is a block diagram showing the data of the first image by using the NN neural processing units of the neural network unit of the first figure according to the embodiment of the eighteenth embodiment. The random access memory obtains one of the data texts to perform the operation as the same rotator.

Figure 20 is a table showing a program stored in the program memory of the neural network unit of the first figure and executed by the neural network unit, and the neural network unit has the embodiment as shown in the eighteenth The neuroprocessing unit shown.

The twenty-first figure shows a timing diagram of a neural network unit executing the program of the twentieth diagram, the neural network unit having the neural processing unit as shown in the eighteenth diagram executed in a narrow configuration.

Figure 22 is a block diagram showing the neural network unit of the first figure, the neural network unit having a neural processing unit as shown in Fig. 18 to execute the program of the twentieth diagram.

The twenty-third figure shows a dynamically configurable neural processing list of the first figure A block diagram of another embodiment of the element.

The twenty-fourth diagram is a block diagram showing an example of a data structure used by the neural network unit of the first figure to perform a convolution operation.

The twenty-fifth diagram is a flowchart showing that the processor of the first figure executes an architectural program to perform a convolution operation of the convolution kernel according to the data array of the twenty-fourth figure using the neural network unit.

The twenty-sixth A diagram is a list of programs of a neural network unit program that performs a convolution operation on a data matrix using the convolution kernel of the twenty-fourth graph and writes it back to the weight random Access memory.

Figure 26B is a block diagram showing one embodiment of certain fields of the control register of the neural network unit of the first figure.

Figure 27 is a block diagram showing an example of weighted random access memory filled with input data in the first figure. The input data is performed by the neural network unit of the first figure. ).

The twenty-eighth figure is a list of programs of a neural network unit program that performs the common source operation of the input data matrix of the twenty-seventh figure and writes it back to the weighted random access memory.

Figure 29A is a block diagram showing one embodiment of the control register of the first figure.

Figure 29B is a block diagram showing another embodiment of the control register of the first figure.

The twenty-ninth C diagram shows a block diagram of one embodiment of storing a reciprocal of the twenty-ninth A map in two parts.

The thirtieth figure shows the implementation of one of the start function units (AFU) of the second figure. A block diagram of an example.

The thirty-first figure shows an example of the operation of the start function unit of the thirtieth figure.

Figure 32 shows a second example of the operation of the start function unit of Figure 30.

The thirty-third figure shows a third example of the operation of the start function unit of the thirtieth figure.

The thirty-fourth diagram is a block diagram showing a portion of the details of the processor of the first diagram and the neural network unit.

The thirty-fifth diagram is a block diagram showing a processor having a variable rate neural network unit.

The thirty-sixth A diagram is a timing diagram showing an operational example in which a processor having a neural network unit operates in a general mode, which operates at a primary time frequency.

The thirty-sixth B diagram is a timing diagram showing an operation example in which a processor having a neural network unit operates in a mitigation mode, and the mitigation mode operates at a lower frequency than the primary time.

Figure 37 is a flow chart showing the operation of the processor of the thirty-fifth figure.

The thirty-eighth figure is a block diagram showing in detail the sequence of neural network elements.

The thirty-ninth figure is a block diagram showing certain fields of the control and status register of the neural network unit.

The fortieth figure is a block diagram showing an example of an Elman time recurrent neural network (RNN).

The forty-first figure is a block diagram showing when the neural network unit performs the association An example of data configuration in the data random access memory and weight random access memory of the neural network unit in the calculation of the Elman time recurrent neural network in the 40th.

Figure 42 shows a table showing a program memory stored in a neural network unit, which is executed by a neural network unit and uses data and weights according to the configuration of the 41st figure to achieve Elman time recurrent neural network

A forty-third figure is a block diagram showing an example of a Jordan time recurrent neural network.

The forty-fourth figure is a block diagram showing that when the neural network unit performs the calculation of the Jordan time recurrent neural network associated with the forty-third figure, the data random access memory and the weight of the neural network unit are randomly stored. Take an example of data configuration in memory.

The forty-fifth diagram is a table showing a program stored in a neural network unit of a program memory, the program is executed by the neural network unit, and the data and weights are used according to the configuration of the forty-fourth figure to achieve Jordan time recurrent neural network.

The forty-sixth diagram is a block diagram showing one embodiment of a long short term memory (LSTM) cell.

The forty-seventh diagram is a block diagram showing the data random access memory and weight random access of the neural network unit when the neural network unit performs the calculation associated with the long-short-term memory cell layer of the forty-sixth figure. An example of data configuration in memory.

The forty-eighth figure is a table showing a program stored in the neural network unit of the program memory, the program is executed by the neural network unit and The configuration of Figure 47 uses data and weights to achieve calculations associated with long- and short-term memory cells.

A forty-ninth diagram is a block diagram showing an embodiment of a neural network unit having an output buffer masking and feedback capability within the neural processing unit group of this embodiment.

Figure 50 is a block diagram showing the data random access memory of the neural network unit of the forty-ninth figure when the neural network unit performs the calculation of the long-short-term memory cell layer associated with the forty-sixth figure. An example of data configuration in weighted random access memory and output buffers.

The fifty-first figure is a table showing a program memory stored in the neural network unit, which is executed by the neural network unit of the forty-ninth figure and uses the data according to the configuration of the fifty-fifth figure. Weights are calculated to achieve a correlation with the long- and short-term memory cell layers.

A fifty-second diagram is a block diagram showing an embodiment of a neural network unit having an output buffer masking and feedback capability within the group of neural processing units of this embodiment, and sharing a start function unit.

The fifty-third figure is a block diagram showing the data random access memory of the neural network unit of the forty-ninth figure when the neural network unit performs the calculation of the long-short-term memory cell layer associated with the forty-sixth figure. Another embodiment of the configuration of the data in the weighted random access memory and the output buffer.

Figure 54 shows a table showing a program memory stored in a neural network unit. The program is executed by the neural network unit of the forty-ninth figure and is used according to the configuration of the fifty-third figure. And weights to achieve calculations associated with long- and short-term memory cell layers.

Figure 55 is a block diagram showing a portion of another embodiment of the present invention Neural processing unit.

Figure 56 is a block diagram showing the neural network unit when the neural network unit performs the calculation of the Jordan time recurrent neural network associated with the forty-third figure and utilizes the embodiment of the fifty-fifth figure. An example of data configuration in data random access memory and weight random access memory.

Figure 57 shows a table showing a program memory stored in a neural network unit. The program is executed by the neural network unit and uses data and weights according to the configuration of Figure 56 to achieve Jordan. Time recurrent neural network.

Processor with an architectural neural network unit

The first figure shows a block diagram of a processor 100 including a neural network unit (NNU) 121. As shown in the figure, the processor 100 includes an instruction capture unit 101, an instruction cache 102, an instruction translator 104, a rename unit 106, a plurality of reservation stations 108, a plurality of media registers 118, and a plurality of The universal register 116 is a plurality of execution units 112 and a memory subsystem 114 outside the neural network unit 121.

The processor 100 is an electronic device that acts as a central processing unit of the integrated circuit. The processor 100 receives the input digital data, processes the data according to instructions fetched by the memory, and generates a processing result of the operation indicated by the instruction as its output. The processor 100 can be used in a desktop computer, mobile device, or tablet computer, and is used for applications such as computing, word processing, multimedia display, and web browsing. This processor 100 can It is placed in an embedded system to control a variety of devices including devices, mobile phones, smart phones, vehicles, and industrial controllers. The central processing unit executes electronic circuits (ie, hardware) that execute instructions of a computer program (or computer application or application) by performing operations including arithmetic, logic, and input/output on the data. An integrated circuit is a group of electronic circuits that are fabricated on a small semiconductor material, usually germanium. Integrated circuits are also commonly used to represent wafers, microchips or dies.

The instruction fetching unit 101 controls the operation of the system memory 103 (not shown) to retrieve the architecture instruction 103 to the instruction cache 102. The instruction fetch unit 101 provides a fetch address to the instruction cache 102 to specify that the processor 100 fetches the memory address of the cache line of the architectural instruction byte of the cache memory 102. The selection of the retrieved address is based on the current value of the instruction indicator (not shown) of the processor 100 or a program counter. In general, the program counters are incremented according to the instruction size until a control instruction such as a branch, call, or return occurs in the instruction stream, or an exception condition such as an interrupt, a trap, an exception, or an error occurs. The program counter is updated with non-sequential addresses such as branch target address, return address, or exception vector. In summary, the program counter is updated in response to execution of the execution unit 112/121. The program counter may also be updated upon detection of an exception condition, such as instruction instruction 104 encountering instruction 103 that is not defined in the instruction set architecture of processor 100.

The instruction cache 102 is stored from a fabric instruction 103 coupled to the system memory of the processor 100. These architectural instructions 103 include a Move to Neural Network (MTNN) instruction and a Neural Network Removal (MFNN) instruction, as will be described later. In an embodiment, the architectural instructions 103 It is an instruction of the x86 instruction set architecture, and is attached with the MTNN instruction and the MFNN instruction. In the context of this disclosure, an x86 instruction set architecture processor is understood to be a processor that produces the same results as an Intel® 80386® processor at the instruction set architecture level while executing the same mechanical language instructions. However, other instruction set architectures, such as the Advanced Reduced Instruction Set Machine Architecture (ARM), Sun's Scalable Processor Architecture (SPARC), or the Enhanced Reduced Instruction Set Performance Computing Performance Architecture (PowerPC), It can also be used in other embodiments of the invention. Instruction cache 102 provides architectural instructions 103 to instruction translator 104 to translate architectural instructions 103 into microinstructions 105.

Microinstructions 105 are provided to rename unit 106 and ultimately executed by execution unit 112/121. These microinstructions 105 implement architectural instructions. In a preferred embodiment, the instruction translator 104 includes a first portion for translating frequently executed and/or relatively less complex architectural instructions 103 into microinstructions 105. The instruction translator 104 also includes a second portion having a microcode unit (not shown). The microcode unit has a microcode memory loaded microcode instruction to execute complex and/or less useful instructions in the architectural instruction set. The microcode unit includes a microsequencer that provides a non-architected microprogram counter (micro-PC) to the microcode memory. In a preferred embodiment, the microinstructions are translated into microinstructions 105 via a micro-translator (not shown). The selector selects the microinstruction 105 from the first portion or the second portion to provide the rename unit 106 depending on whether the microcode unit currently has control.

Renaming unit 106 renames the architectural register specified by architectural instruction 103 to the physical scratchpad of processor 100. In a preferred embodiment, the processor 100 includes a rearrangement buffer (not shown). The rename unit 106 assigns the items of the rearrangement buffer to the respective microinstructions 105 in accordance with the program order. This allows the processor 100 to remove the microinstructions 105 and their corresponding architectural instructions 103 in accordance with the program sequence. In one embodiment, the media register 118 has a 256-bit width and the general-purpose register 116 has a 64-bit width. In one embodiment, media register 118 is an x86 media register, such as an Advanced Vector Expansion (AVX) register.

In one embodiment, each item of the rearrangement buffer has a storage space to store the results of the microinstructions 105. In addition, the processor 100 includes an architecture register file having a physical register corresponding to each architecture register, such as the media register 118, the general register 116, and other architecture temporary storage. Device. (For a preferred embodiment, for example, the media register 118 and the general register 116 are different in size, and a separate register file can be used to correspond to the two registers.) The instruction source 105 specifies a source operand of an architecture register, and the rename unit fills in the microinstruction 105 by using the reorder buffer directory of the latest microinstruction in the old microinstruction 105 written to the architecture register. The source operand field. When execution unit 112/121 completes execution of microinstruction 105, execution unit 112/121 writes its result to the reorder buffer entry of this microinstruction 105. When the microinstruction 105 is removed, the removal unit (not shown) writes the result from the reorder buffer field of the microinstruction to the scratchpad of the physical scratchpad file associated with the physical scratchpad file Thereby, the scratchpad of the architecture specified by the microinstruction 105 is removed.

In another embodiment, the processor 100 includes a physical scratchpad file having more physical scratchpads than the number of architectural registers, however, the processor 100 does not include an architectural register file. Case, and the result storage space is not included in the rearrangement buffer project. (In a preferred embodiment, because the media register 118 and the general register 116 are different in size, a separate register file can be used to correspond to the two registers.) The processor 100 It includes an indicator table with corresponding indicators for each architecture register. For each operand specified in the microinstruction 105 with an architectural register, the rename unit fills the destination operand field in the microinstruction 105 with an indicator pointing to a free register in the physical scratchpad file. If there is no free register in the physical scratchpad file, the rename unit 106 will temporarily suspend the pipeline. For each source operand in the microinstruction 105 that specifies the architecture register, the rename unit will utilize a pointer to the entity register file and assign the latest microinstruction to the old microinstruction 105 of the write architecture register. The indicator of the register is filled in the source operand field in the microinstruction 105. When execution unit 112/121 completes execution of microinstruction 105, execution unit 112/121 writes the result to the destination operand field of microinstruction 105 in the physical scratchpad file. When the microinstruction 105 is removed, the removal unit copies the destination operand field value of the microinstruction 105 to the indicator associated with the indicator table of the schema destination register specified by the microinstruction 105.

The reservation station 108 will load the microinstructions 105 until they are ready to be issued to the execution unit 112/121 for execution. When all of the source operands of a microinstruction 105 are available and the execution unit 112/121 is also available for execution, the microinstruction 105 is ready for publication. The execution unit 112/121 is configured by the rearrangement buffer or the architecture register file described in the foregoing first embodiment, or the physical register file of the foregoing second embodiment to receive the register source operation unit. In addition, the execution order The element 112/121 can receive the scratchpad source operand directly through the result transfer bus (not shown). Additionally, execution unit 112/121 can receive the immediate operand specified by microinstruction 105 from reservation station 108. The MTNN and MFNN architecture instructions 103 include an immediate operand to specify the functions to be performed by the neural network unit 121, and this functionality is provided by one or more microinstructions 105 generated by the MTNN and MFNN architecture instructions 103, as described below. .

Execution unit 112 includes one or more load/store units (not shown) that are loaded by memory subsystem 114 and store data to memory subsystem 114. In a preferred embodiment, the memory subsystem 114 includes a memory management unit (not shown), which may include, for example, a plurality of lookaside buffers, a table move ( A tablewalk unit, a hierarchy-data cache (with instruction cache 102), a hierarchy 2 unified cache, and a bus interface unit that serves as an interface between processor 100 and system memory. In one embodiment, the processor 100 of the first diagram is represented by one of a plurality of processing cores of a multi-core processor that shares a last-level cache memory. Execution unit 112 may include a plurality of integer units, a plurality of media units, a plurality of floating point units, and a branch unit.

The neural network unit 121 includes a weight random access memory (RAM) 124, a data random access memory 122, N neural processing units (NPU) 126, a program memory 129, and a sequencer 128 and more. Control and status register 127. These neural processing units 126 are conceptually like the functions of neurons in a neural network. The weighted random access memory 124, the data random access memory 122, and the program memory 129 can each be written and read through the MTNN and MFNN architecture instructions 103. The weight random access memory 124 is arranged in W columns, each column of N weight characters, and the data random access memory 122 is arranged in D columns, each column of N data characters. Each of the data texts and each weight text is a plurality of bits. For a preferred embodiment, it may be 8 bits, 9 bits, 12 bits, or 16 bits. Each data text is used as the output value of a neuron in the previous layer of the network (sometimes expressed as a startup value), and each weight text is used as a weight in the network associated with one of the neurons entering the current layer of the network. . Although in many applications of the neural network unit 121, the literal or operand loaded in the weighted random access memory 124 is actually associated with the weight of the link into a neuron, it should be noted that in the neural network In some applications of unit 121, the text loaded in weighted random access memory 124 is not weighted, but because the text is stored in weighted random access memory 124, it is still expressed in terms of "weighted text." For example, in some applications of the neural network unit 121, for example, an example of a convolution operation of the twenty-fourth to twenty-sixth A or an example of a common source operation of the twenty-seventh to twenty-eighth The weighted random access memory 124 loads objects other than weights, such as elements of a data matrix (such as image pixel data). Similarly, although in many applications of the neural network unit 121, the words or operands loaded in the data random access memory 122 are essentially the output values or activation values of the neurons, it should be noted that in the neural network In some applications of the way unit 121, the text loaded in the data random access memory 122 is not the case, but since the words are stored in the data random access memory 122, they are still expressed in terms of "data text". For example, in some applications of neural network unit 121, such as the convolution of the twenty-fourth to twenty-sixth A maps As an example of the operation, the data random access memory 122 will load the output of the non-neuron, such as the elements of the convolution kernel.

In an embodiment, the neural processing unit 126 and the sequencer 128 comprise combinational logic, sequencing logic, state machines, or a combination thereof. The architectural instructions (e.g., MFNN instruction 1500) load the contents of state register 127 into one of the general purpose registers 116 to confirm the state of neural network unit 121, such as neural network unit 121 having completed from program memory 129. The operation of a command or a program, or the neural network unit 121 is free to receive a new command or start a new neural network unit program.

The number of neural processing units 126 may be increased according to requirements, and the width and depth of the random storage memory 124 and the data random access memory 122 may be adjusted to expand. In a preferred embodiment, the weighted random access memory 124 is larger than the data random access memory 122 because there are many links in a typical neural network layer, and thus a larger storage space is required to be associated with The weight of each neuron. A number of embodiments are disclosed herein regarding the size of data and weight text, the size of the weighted random access memory 124 and the data random access memory 122, and the number of different neural processing units 126. In one embodiment, the neural network unit 121 has a data random access memory 122 of 64 KB (8192 bits x 64 columns) and a weighted random access memory of 2 MB (8192 bits x 2048 columns). 124, and 512 neural processing units 126. The neural network unit 121 is manufactured by a 16 nanometer process of Taiwan Integrated Circuit (TSMC), which occupies an area of approximately 3.3 mm square.

The sequencer 128 is fetched and executed by the program memory 129. The execution of the sequencer 128 further includes generating address and control signals for the data random access memory 122, the weight random access memory 124, and the neural processing unit 126. The sequencer 128 generates a memory address 123 and a read command to the data random access memory 122, whereby one of the N data words of the D columns is selected for supply to the N neural processing units 126. The sequencer 128 also generates a memory address 125 and a read command to the weighted random access memory 124, whereby one of the N weight words of the W columns is selected for supply to the N neural processing units 126. The sequence in which the sequencer 128 generates and provides the addresses 123, 125 to the neural processing unit 126 determines the "link" between the neurons. The sequencer 128 also generates a memory address 123 and a write command to the data random access memory 122, whereby one of the N data words of the D columns is selected for writing by the N neural processing units 126. In. The sequencer 128 also generates a memory address 125 and a write command to the weighted random access memory 124, whereby one of the N weight words of the W columns is selected for writing by the N neural processing units 126. In. The sequencer 128 also generates a memory address 131 to the program memory 129 to select one of the neural network unit instructions provided to the sequencer 128, as will be explained in subsequent sections. The memory address 131 corresponds to a program counter (not shown), and the sequencer 128 generally increments the program counter according to the position order of the program memory 129 unless the sequencer 128 encounters a control command, such as a loop command. (Refer to Figure 26A). In this case, sequencer 128 updates the program counter to the target address of this control instruction. The sequencer 128 also generates a control signal to the neural processing unit 126 indicating the neural processing unit 126 Perform a variety of different operations or functions, such as initialization, arithmetic / logic operations, rotation / shift operations, start functions, and write back operations, the relevant examples in the following chapters (please refer to the micro-operations as shown in Figure 34) A more detailed description will be given in 3418).

The N neural processing units 126 generate N result words 133 which can be written back to one of the weighted random access memory 124 or the data random access memory 122. In a preferred embodiment, the weight random access memory 124 and the data random access memory 122 are directly coupled to the N neural processing units 126. Further, the weight random access memory 124 and the data random access memory 122 belong to the neural processing unit 126 and are not shared with other execution units 112 in the processor 100. The neural processing units 126 can continuously A column is obtained and completed from one or both of the weight random access memory 124 and the data random access memory 122 in each time-frequency period. In a preferred embodiment, the pipeline processing is performed. In one embodiment, each of the data random access memory 122 and the weighted random access memory 124 can provide 8192 bits to the neural processing unit 126 in each time-frequency period. These 8192 bits can be treated as 512 16-bits or 1024 8-bits, as will be described later.

The size of the data set processed by the neural network unit 121 is not limited by the weight of the random access memory 124 and the data random access memory 122, but is limited only by the size of the system memory, because the data The weights can be moved between the system memory and the weighted random access memory 124 and the data random access memory 122 by the use of MTNN and MFNN instructions (e.g., via the media register 118). In a In the embodiment, the data random access memory 122 is given a double port, enabling writing in the data random access memory 122 or writing the data to the data random access memory 122. The data is to the data random access memory 122. In addition, the large memory hierarchy of the memory subsystem 114, including the cache memory, provides a very large data bandwidth for data transfer between the system memory and the neural network unit 121. Moreover, in a preferred embodiment, the memory subsystem 114 includes a hardware data prefetcher that tracks memory access modes, such as neural data and weights loaded by system memory, and caches The hierarchical structure performs data prefetching to facilitate high frequency wide and low latency transmissions during transmission to the weighted random access memory 124 and the data random access memory 122.

Although in the embodiments herein, one of the operands provided by the weight memory to each of the neural processing units 126 is labeled as a weight, this term is commonly found in neural networks, but it should be understood that these operands may also be other Calculate the type of data associated with it, and its calculation speed can be improved by these devices.

The second figure shows a block diagram of a neural processing unit 126 of the first figure. As shown in the figure, the operation of the neural processing unit 126 can perform a number of functions or operations. In particular, the neural processing unit 126 can operate as a neuron or node within the artificial neural network to perform a typical multiply-accumulate function or operation. That is to say, in general, the neural network unit 126 (neuron) is used to: (1) receive an input value from each of the neurons with which it is connected, which usually but not necessarily from an artificial neural network Before the middle layer; (2) will lose each The value is multiplied by a corresponding weight value associated with one of its links to produce a product; (3) all products are summed to produce a total; (4) a start function is performed on the total to generate an output of the neuron. However, unlike conventional methods that require performing all multiplication operations associated with all of the concatenated inputs and summing their products, the various neurons of the present invention can perform weight multiplication operations associated with one of the concatenated inputs in a given time-frequency period. The product is added (accumulated) to the accumulated value of the product of the join input executed in the time-frequency period associated with the time point before the point in time. Assuming that a total of M links are connected to this neuron, after the M products are summed (approximately M time-frequency cycles are required), the neuron will perform a start function on this accumulated number to produce an output or result. The advantage of this approach is that it reduces the number of multipliers required, and requires only a smaller, simpler, and faster adder circuit (eg, an adder using two inputs) within the neuron, without the need for Use an adder that is capable of summing the products of all the connected inputs or even adding one of the subsets. This approach also facilitates the use of a very large number (N) of neurons (neural processing unit 126) within neural network unit 121, such that neural network unit 121 can produce this large number after approximately M time-frequency periods. (N) Output of neurons. Finally, for a large number of different connection inputs, the neural network unit 121 composed of these neurons can be effectively implemented as an artificial neural network layer. That is to say, if the number of M in different layers increases or decreases, the number of time-frequency cycles required to generate the memory cell output will correspondingly increase and decrease, and resources (such as multipliers and accumulators) will be fully utilized. In contrast, conventional designs have some of the multipliers and adders that are not utilized for smaller M values. Therefore, in response to the number of connected outputs of the neural network unit, this article describes The embodiment combines the advantages of flexibility and efficiency to provide extremely high performance.

The neural processing unit 126 includes a register 205, a dual input multiplex register 208, an arithmetic logic unit (ALU) 204, an accumulator 202, and an enable function unit (AFU) 212. The register 205 receives a weighted text 206 from the weighted random access memory 124 and provides its output 203 at a subsequent time-frequency period. Multiplex register 208 selects one of the two inputs 207, 211 to store in its register and provide its output 209 for a subsequent time-frequency period. Input 207 receives the data text from data random access memory 122. Another input 211 receives the output 209 of the adjacent neural processing unit 126. The neural processing unit 126 shown in the second figure is labeled as the neural processing unit J among the N neural processing units shown in the first figure. That is, the neural processing unit J is an example of one of the N neural processing units 126. In a preferred embodiment, the input 211 of the multiplex register 208 of the J example of the neural processing unit 126 receives the output 209 of the multiplex register 208 of the J-1 example of the neural processing unit 126, and the neural The output 209 of the multiplex register 208 of the processing unit J is provided to the input 211 of the J+1 paradigm multiplex register 208 of the neural processing unit 126. Thus, the multiplex registers 208 of the N neural processing units 126 can operate together, such as the same N-word rotator or cyclic shifter, which will be described in more detail in the subsequent third figure. The multiplex register 208 utilizes a control input 213 to control which of the two inputs is selected by the multiplex register 208 for storage in its register and for subsequent supply to the output 209.

The arithmetic logic unit 204 has three inputs. One of the inputs is received by the register 205 as a weight text 203. The other input receives the output 209 of the multiplex register 208. Yet another input receive accumulator 202 The output is 217. This arithmetic logic unit 204 performs arithmetic and/or logical operations on its inputs to produce a result for its output. For a preferred embodiment, the arithmetic and/or logical operations performed by arithmetic logic unit 204 are specified by instructions stored in program memory 129. For example, the multiply-accumulate instruction in the fourth figure specifies a multiply-accumulate operation, that is, the result 215 would be the product of the accumulator 202 value 217 and the weight text 203 and the data text of the output 209 of the multiplex register 208. total. However, other operations may be specified, including but not limited to: result 215 is the value passed by multiplex register output 209; result 215 is the value passed by weight text 203; result 215 is a zero value; result 215 is accumulator 202 The sum of the value 217 and the weight 203; the result 215 is the sum of the accumulator 202 value 217 and the multiplex register output 209; the result 215 is the maximum value of the accumulator 202 value 217 and the weight 203; the result 215 is the accumulator The maximum value of 202 value 217 and multiplex register output 209.

Arithmetic logic unit 204 provides its output 215 to accumulator 202 for storage. The arithmetic logic unit 204 includes a multiplier 242 that multiplies the weight text 203 and the data text of the multiplex register 208 output 209 to produce a product 246. In one embodiment, multiplier 242 multiplies two 16-bit operands to produce a 32-bit result. The arithmetic logic unit 204 also includes an adder 244 that adds a product 246 to the output 217 of the accumulator 202 to produce a total, which is the result 215 of the accumulation operation stored in the accumulator 202. In one embodiment, adder 244 is coupled to a 41 bit value 217 of accumulator 202 plus a 32 bit result of multiplier 242 to produce a 41 bit result. Thus, the rotator of the multiplex register 208 is utilized during a plurality of time-frequency periods. The neural processing unit 126 can achieve the product summation operation of the neurons required by the neural network. This arithmetic logic unit 204 may also include other circuit elements to perform other arithmetic/logic operations as previously described. In one embodiment, the second adder subtracts the weight text 203 from the data text output 209 of the multiplex register 208 to generate a difference, and then the adder 244 adds the difference to the output 217 of the accumulator 202. The value is used to produce a result 215 which is the cumulative result in accumulator 202. Thus, during a plurality of time-frequency periods, the neural processing unit 126 can perform the summation of the differences. In a preferred embodiment, although the weight text 203 and the data text 209 are the same size (in terms of bits), they may have different binary decimal point positions, as will be described later. In a preferred embodiment, multiplier 242 and adder 244 are integer multipliers and adders. This arithmetic logic unit 204 has low complexity, small size, and fast compared to arithmetic logic units that use floating point operations. With the advantages of low energy consumption. However, in other embodiments of the invention, arithmetic logic unit 204 may also perform floating point operations.

Although only one multiplier 242 and adder 244 are shown in the arithmetic logic unit 204 of the second figure, in a preferred embodiment, the arithmetic logic unit 204 includes other components to perform the other different operations described above. For example, the arithmetic logic unit 204 can include a comparator (not shown) to compare the accumulator 202 with a data/weight text, and a multiplexer (not shown) to select between two values specified by the comparator. The larger (maximum) is stored to accumulator 202. In another example, arithmetic logic unit 204 includes selection logic (not shown) that utilizes a data/weight text to skip multiplier 242, causing adder 224 to add this data/weight text to value 217 of accumulator 202. To generate a total amount to store Adder 202. These additional operations are described in more detail in subsequent sections, such as Figures 18 through 29A, which also contribute to the implementation of convolution operations and common source operations.

The start function unit 212 receives the output 217 of the accumulator 202. The start function unit 212 performs a start function on the output of the accumulator 202 to produce the result 133 of the first map. In general, the start function in the neurons of the intervening layer of the artificial neural network can be used to normalize the total number of multiply accumulates, especially in a non-linear manner. To "normalize" the accumulated total, the current neuron's start function will produce a result value within the range of values that other neurons connected to the current neuron are expected to receive as input. (The result of normalization is sometimes referred to as "startup". In this paper, the start is the output of the current node, and the receiving node multiplies this output by a weight associated with the link between the output node and the receiving node to produce a product. And this product will be multiplied by the other input links associated with the receiving node.) For example, if the received/linked neurons are expected to receive as input values between 0 and 1, the output neurons It may be necessary to non-linearly squeeze and/or adjust (eg, shift up to convert a negative value to a positive value) beyond the cumulative total of the range of 0 and 1 to fall within the expected range. Thus, the operation performed by the start function unit 212 on the accumulator 202 value 217 will bring the result 133 to a known range. The results 133 of the N neural execution units 126 can all be written back to the data random access memory 122 or the weighted random access memory 124. In a preferred embodiment, the boot function unit 212 is operative to execute a plurality of boot functions, and for example, an input from the control register 127 selects one of the start functions to execute on the output of the accumulator 202. . These start functions may include, but are not limited to, step functions, correction functions, sigmoid functions, hyperbolic tangent functions, and soft addition functions (also referred to as smoothing correction functions). The analytical formula of the soft addition function is f(x)=ln(1+e ^x ), which is the total natural logarithm of the sum of 1 and e ^x , where “e” is the Euler's number and x is this Input 217 for the function. In a preferred embodiment, the start function may also include a pass-through function that directly passes the value 217 of the accumulator 202 or a portion thereof, as will be described later. In an embodiment, the circuitry of the startup function unit 212 performs the startup function in a single time-frequency cycle. In an embodiment, the start function unit 212 includes a plurality of forms that receive the accumulated value and output a value. For some start functions, such as a sigmoid function, a double take tangent function, a soft add function, etc., the value is approximated. The value provided by the real startup function.

In a preferred embodiment, the width (in terms of bits) of accumulator 202 is greater than the width of output 133 of start function function 212. For example, in one embodiment, the accumulator has a width of 41 bits to avoid accumulation in the product of up to 512 32-bits (this portion corresponds to the thirty-th figure in subsequent sections). There will be a more detailed description of the loss accuracy, and the result 133 has a width of 16 bits. In one embodiment, in a subsequent time-frequency period, the start function unit 212 passes the other unprocessed portions of the output 217 of the accumulator 202 and writes the portions back to the data random access memory 122 or the weights are stored in random. The memory 124 is taken, and this section will be described in more detail in the subsequent sections corresponding to the eighth figure. Thus, the unprocessed accumulator 202 value can be loaded back to the media register 118 via the MFNN instruction, whereby the instructions executed by the other execution units 112 of the processor 100 can perform complex startups that the boot function unit 212 cannot perform. Functions, such as the common softmax function, are also called normalized exponential functions. In one embodiment, the instruction set architecture of processor 100 includes instructions to perform one of the exponential functions, generally denoted as e ^x or exp(x), which may be used by other execution units 112 of processor 100 to promote soft maximal startup. The execution speed of the function.

In an embodiment, the neural processing unit 126 is a pipeline design. For example, the neural processing unit 126 can include a temporary register of the arithmetic logic unit 204, such as a register between the multiplier and the adder and/or other circuits of the arithmetic logic unit 204. The neural processing unit 126 can also include A register that loads the output of the boot function function 212. Other embodiments of this neural processing unit 126 are described in subsequent sections.

The third diagram is a block diagram showing N multiplex registers 208 of the N neural processing units 126 of the neural network unit 121 of the first figure, obtained for the data random access memory 122 of the first figure. One of the column data words 207 performs the operation of a rotator or a circular shifter such as the same N characters. In the embodiment of the third figure, N is 512. Thus, neural network unit 121 has 512 multiplex registers 208, labeled 0 through 511, corresponding to 512 neural processing units 126, respectively. Each multiplex register 208 receives the corresponding data text 207 on one of the D columns of the data random access memory 122. That is to say, the multiplex register 0 will receive the data text 0 from the data random access memory 122 column, and the multiplex register 1 will receive the data text 1 from the data random access memory 122 column, and the multiplex temporary storage 2 will receive the data text 2 from the data random access memory 122 column, and so on, the multiplex register The 511 receives the data 511 from the data random access memory 122 column. In addition, the multiplex register 1 receives the output 209 of the multiplex register 0 as another input 211, and the multiplex register 2 receives the output 209 of the multiplex register 1 as another input 211, multiplexing. The register 3 receives the output 209 of the multiplex register 2 as another input 211, and so on, the multiplex register 511 receives the output 209 of the multiplex register 510 as another input 211, and more The scratchpad 0 receives the output 209 of the multiplex register 511 as the other input 211. Each multiplex register 208 receives a control input 213 to control its selection profile 207 or loop input 211. In one of the modes of operation, the control input 213 controls each of the multiplex registers 208 to select the data text 207 for storage in the scratchpad and to provide to the arithmetic logic unit 204 in subsequent steps during a first time-frequency period. And in subsequent time-frequency periods (such as the aforementioned M-1 time-frequency periods), control input 213 controls each multiplex register 208 to select loop input 211 for storage to the scratchpad and provide it to subsequent steps. Arithmetic logic unit 204.

Although in the embodiment depicted in the third figure (and subsequent seventh and nineteenth figures), a plurality of neural processing units 126 may be used to rotate the values of these multiplex registers 208/705 to the right, ie by the nerves The processing unit J moves toward the neural processing unit J+1, although the invention is not limited thereto, and in other embodiments (e.g., corresponding to the embodiments of the twenty-fourth to twenty-sixth embodiments), the plurality of neural processing units 126 It can be used to rotate the value of the multiplex register 208/705 to the left, that is, to move from the neural processing unit J toward the neural processing unit J-1. Moreover, in other embodiments of the invention, the neural processing unit 126 can selectively rotate the value of the multiplex register 208/705 to the left or to the right, for example, the selection can be Specified by the neural network unit directive.

The fourth diagram is a table showing a program stored in the program memory 129 of the neural network unit 121 of the first figure and executed by the neural network unit 121. As mentioned earlier, this sample program performs calculations related to one layer of the artificial neural network. The table in Figure 4 shows four columns and three rows. Each column corresponds to an address in the program memory 129 labeled in the first row. The second line specifies the corresponding instruction, and the third line indicates the number of time-frequency cycles associated with this instruction. In the preferred embodiment, the number of time-frequency cycles is indicative of the number of effective time-frequency cycles per instruction time-frequency period value in the embodiment of the pipeline execution, rather than the instruction delay. As shown in the figure, because the neural network unit 121 has the nature of pipeline execution, each instruction has an associated time-frequency period. The instruction at address 2 is an exception, and the instruction actually repeats 511 times. Therefore, 511 time-frequency cycles are required, as will be described later.

All neural processing units 126 will process each instruction in the program in parallel. That is to say, all N neural processing units 126 execute the instructions of the first column in the same time-frequency cycle, and all N neural processing units 126 execute the instructions of the second column in the same time-frequency cycle. analogy. However, the invention is not limited thereto, and in other embodiments of the subsequent sections, some instructions are executed in the form of a sequence of partially parallel portions, for example, as described in the embodiment of the eleventh embodiment, in multiple neural processing In an embodiment where unit 126 shares a start function unit, the start function and the output instructions at addresses 3 and 4 are executed in this manner. The example in the fourth figure assumes that one layer has 512 neurons (neural processing unit 126) and each neuron has 512 from the previous layer. 512 neuron link inputs, for a total of 256K links. Each neuron receives a 16-bit data value from each of the link inputs and multiplies the 16-bit data value by an appropriate 16-bit weight value.

The first column of address 0 (which can also be assigned to another address) specifies an initialization neuroprocessing unit instruction. This initialization instruction clears the value of accumulator 202 to zero. In an embodiment, the initialization command may also be loaded into the column of the data random access memory 122 or the weight random access memory 124 in the accumulator 202, thereby instructing the corresponding text specified by the instruction. This initialization command also loads the configuration values into the control register 127, which is described in more detail in subsequent twenty-ninth and twenty-ninth B-pictures. For example, the width of the data text 207 and the weight text 209 can be loaded for use by the arithmetic logic unit 204 to confirm the size of the operation performed by the circuit, which width also affects the result 215 stored in the accumulator 202. In one embodiment, the neural processing unit 126 includes a circuit that fills the output 215 before the output 215 of the arithmetic logic unit 204 is stored in the accumulator 202, and the initialization command loads a configuration value into the circuit. Affects the aforementioned filling operation. In an embodiment, it may also be specified in an arithmetic logic unit function instruction (such as a multiplication accumulating instruction of address 1) or an output instruction (such as a write start function unit output instruction of address 4) to accumulate the accumulator. 202 clears to zero value.

The second column located in address 1 specifies a multiply accumulate instruction to instruct the 512 neural processing units 126 to load a corresponding data word from the data random access memory 122 and from the weight random access memory 124. A column loads a corresponding weight text, and a first multiplication is performed on the data text input 207 and the weight text input 206. The addition operation, that is, the initialization accumulator 202 zero value is added. Further, this command will instruct sequencer 128 to generate a value at control input 213 to select data entry 207. In the example of the fourth figure, the designated column of the data random access memory 122 is column 17, and the designated column of the weight random access memory 124 is column 0, so the sequencer is instructed to output the value 17 as a data random. The memory address address 123 is accessed and the value 0 is output as a weighted random access memory address 125. Thus, 512 data words from column 17 of data random access memory 122 are provided as corresponding data inputs 207 for 512 neural processing units 126, and 512 weights from column 0 of weight random access memory 124. The text is provided as a corresponding weight input 206 for 512 neural processing units 126.

The third column located in address 2 specifies a multiply-accumulate rotation instruction having a count of 511 to indicate that the 512 neural processing units 126 perform 511 multiply-accumulate operations. This instruction instructs the 512 neural processing units 126 to input the data word 209 of the arithmetic logic unit 204 into the rotation value 211 from the adjacent neural processing unit 126 in each of the 511 multiply-accumulate operations. That is, this command will instruct sequencer 128 to generate a value at control input 213 to select rotation value 211. In addition, the instruction will instruct the 512 neural processing units 126 to load one of the 511 multiply-accumulate operations into the "next" column of the weighted random access memory 124. That is, the instruction will instruct sequencer 128 to increment the weighted random access memory address 125 from the value of the previous time-frequency period by one, in this example, the first time-frequency period of the instruction is column 1, the next time The frequency period is column 2, the next time-frequency period is column 3, and so on, the 511th time-frequency period is Is column 511. In each of the 511 multiply-accumulate operations, the product of the rotary input 211 and the weighted text input 206 is added to the previous value of the accumulator 202. The 512 neural processing units 126 perform the 511 multiply-accumulate operations in 511 time-frequency cycles, each of the neural processing units 126 for different data words from the column 17 of the data random access memory 122 - that is, The adjacent neural processing unit 126 performs the arithmetic data of the operation in the previous time-frequency cycle, and performs a multiplication and accumulation operation on the different weight words associated with the data characters, which are conceptually different input inputs of the neurons. This example assumes that each neural processing unit 126 (neuron) has 512 linked inputs, thus involving the processing of 512 data words and 512 weighted words. After the last iteration of the multiply-accumulate rotation instruction of column 2, the sum of the products of the 512 connection inputs is stored in the accumulator 202. In one embodiment, the instruction set of the neural processing unit 126 includes an "execute" instruction to instruct the arithmetic logic unit 204 to perform an arithmetic logic unit operation specified by the initialization neural processing unit instruction, such as the arithmetic of the twenty-ninth A-picture. The logic unit function 2926 specifies, rather than having separate instructions for each of the different types of arithmetic logic operations (e.g., the aforementioned multiply accumulate, accumulator and weight max, etc.).

The fourth column located in address 3 specifies a start function instruction. This start function instruction instructs the start function unit 212 to execute the specified start function for the accumulator 202 value to produce the result 133. The embodiment of the startup function will be described in more detail in subsequent chapters.

The fifth column located in address 4 specifies a write start function unit output instruction to indicate that the 512 neural processing units 216 will The boot function unit 212 output is written back as a result 133 to one of the columns of data random access memory 122, in this example column 16. That is, this instruction will instruct sequencer 128 to output a value of 16 as data random access memory address 123 and a write command (corresponding to the read command specified by the multiply accumulate instruction of address 1). In a preferred embodiment, the write start function unit output instructions can be executed concurrently with other instructions because of the nature of the pipeline execution, so the write start function unit output instructions can actually be executed in a single time-frequency cycle.

In a preferred embodiment, each neural processing unit 126 acts as a pipeline having various functional components, such as multiplex register 208 (and multiplex register 705 of the seventh diagram), arithmetic. Logic unit 204, accumulator 202, start function unit 212, multiplexer 802 (please refer to FIG. 8), column buffer 1104 and start function unit 1112 (please refer to FIG. 11), etc., some of which are themselves Can be executed in pipeline. In addition to the data text 207 and the weight text 206, this pipeline also receives instructions from the program memory 129. These instructions flow along the pipeline and control multiple functional units. In another embodiment, the program does not include a start function instruction, but the initialization neural processing unit instruction specifies a start function that executes the value 217 of the accumulator 202, indicating that the value of one of the specified start functions is stored in a configuration. The register, the portion of the start function unit 212 for the pipeline, is utilized after the last accumulator 202 value 217 is generated, that is, after the last execution of the multiply-accumulate rotation instruction of address 2 is repeated. In a preferred embodiment, in order to save energy, the boot function unit 212 portion of the pipeline will be in an inactive state before the write start function unit output instruction arrives. When the instruction arrives, the function unit 212 is started. The start function is executed and the accumulator 202 output 217 specified by the initialization instruction is executed.

The fifth diagram shows a timing diagram of the execution of the program of the fourth diagram by the neural network unit 121. Each column of this timing diagram corresponds to the continuous time-frequency period indicated by the first row. The other lines correspond to the different neural processing units 126 of the 512 neural processing units 126, respectively, and indicate their operations. Only the operation of the neural processing unit 0, 1, 511 is shown in the figure to simplify the explanation.

At time-frequency period 0, each of the 512 neural processing units 126 performs an initialization instruction of the fourth map, and in the fifth diagram, a zero value is assigned to the accumulator 202.

In time-frequency period 1, each of the 512 neural processing units 126 performs a multiply-accumulate instruction of address 1 in the fourth figure. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 (i.e., zero) to the text 0 of the column 17 of the data random access memory 122 and the text 0 of the column 0 of the weight random access memory 124. Product; the neural processing unit 1 adds the value of the accumulator 202 (i.e., zero) to the product of the text 1 of the column 17 of the data random access memory 122 and the text 1 of the column 0 of the weight random access memory 124; Similarly, the neural processing unit 511 adds the value of the accumulator 202 (i.e., zero) to the product of the text 511 of the column 17 of the data random access memory 122 and the text 511 of the column 0 of the weight random access memory 124.

In time-frequency period 2, each of the 512 neural processing units 126 performs the first iteration of the multiply-accumulate rotation instruction of address 2 in the fourth figure. As shown in the figure, the nerve The processing unit 0 adds the value of the accumulator 202 to the rotated data text 211 (i.e., the data text 511 received by the data random access memory 122) received by the multiplexer 208 output 209 of the neural processing unit 511, and the weights are random. The product of the text 0 of the column 1 of the memory 124 is accessed; the neural processing unit 1 adds the value of the accumulator 202 to the rotated data 211 received by the output 209 of the multiplex register 208 of the neural processing unit 0 (ie, by data) The product text 0) received by the random access memory 122 and the text 1 of the column 1 of the weight random access memory 124; and so on, the neural processing unit 511 adds the value of the accumulator 202 to the neural processing unit 510. The multiplexer 208 outputs 209 the product of the rotated data text 211 (ie, the data text 510 received by the data random access memory 122) and the text 511 of the column 1 of the weight random access memory 124.

In time-frequency period 3, each of the 512 neural processing units 126 performs a second iteration of the multiply-accumulate rotation instruction of address 2 in the fourth figure. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 to the rotated data text 211 received by the multiplexer 208 output 209 of the neural processing unit 511 (i.e., received by the data random access memory 122). The data text 510) is multiplied by the text 0 of column 2 of the weight random access memory 124; the neural processing unit 1 adds the value of the accumulator 202 to the rotation received by the output 209 of the multiplex register 208 of the neural processing unit 0. The product text 211 (i.e., the data text 511 received by the data random access memory 122) and the text 1 of the column 2 of the weight random access memory 124; and so on, the neural processing unit 511 will increment the value of the accumulator 202. The rotated data text 211 received by the output 209 of the multiplex register 208 of the neural processing unit 510 is added (ie, the data is stored randomly). The product of the data text 509 received by the memory 122 and the text 511 of the column 2 of the weight random access memory 124 is taken. As the ellipsis of the fifth figure shows, the next 509 time-frequency periods will continue until the time-frequency period 512.

At time-frequency period 512, each of the 512 neural processing units 126 performs the 511th iteration of the multiply-accumulate rotation instruction of address 2 in the fourth diagram. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 to the rotated data text 211 received by the multiplexer 208 output 209 of the neural processing unit 511 (i.e., received by the data random access memory 122). The data text 1) is multiplied by the text 0 of the column 511 of the weight random access memory 124; the neural processing unit 1 adds the value of the accumulator 202 to the rotation received by the output 209 of the multiplex register 208 of the neural processing unit 0. The product text 211 (i.e., the data text 2 received by the data random access memory 122) and the text 1 of the column 511 of the weight random access memory 124; and so on, the neural processing unit 511 will increment the value of the accumulator 202. The rotated data text 211 (ie, the data text 0 received by the data random access memory 122) received by the multiplexer 208 output 209 of the neural processing unit 510 and the column 511 of the weight random access memory 124 are added. The product of the text 511. In one embodiment, multiple time-frequency periods are required to read the data text and weight text from the data random access memory 122 and the weight random access memory 124 to perform the multiplication accumulation instruction of the address 1 in the fourth figure; however, The data random access memory 122, the weighted random access memory 124, and the neural processing unit 126 are pipelined, such that after the first multiply-accumulate operation begins (as shown in the time-frequency period 1 of the fifth figure), subsequent Multiply-accumulate operation (such as the time-frequency of the fifth picture) Cycles 2-512) will begin to execute during the connected time-frequency period. In a preferred embodiment, the data random access memory 122 and/or the weight random access memory are utilized in response to architectural instructions, such as MTNN or MFNN instructions (described in subsequent fourteenth and fifteenth figures). The access to the body 124, or the micro-instructions translated by the architectural instructions, will temporarily rest on these neural processing units 126.

At time-frequency period 513, the start function unit 212 of each of the 512 neural processing units 126 performs the start function of address 3 in the fourth diagram. Finally, at time-frequency period 514, each of the 512 neural processing units 126 will perform a fourth by writing the result 133 back to the corresponding text in column 16 of the data random access memory 122. The write start function unit output instruction of the address 4 in the figure, that is, the result 133 of the neural processing unit 0 is written to the text 0 of the data random access memory 122, and the result 133 of the neural processing unit 1 is written. The text 1 of the data random access memory 122 is entered, and so on, the result 133 of the neural processing unit 511 is written to the text 511 of the data random access memory 122. Corresponding block diagrams corresponding to the operations of the aforementioned fifth figure are shown in Figure 6A.

Figure 6A is a block diagram showing the execution of the fourth diagram of the neural network unit 121 of the first figure. The neural network unit 121 includes 512 neural processing units 126, a data random access memory 122 that receives the address input 123, and a weighted random access memory 124 that receives the address input 125. At time interval 0, the 512 neural processing units 126 execute initialization instructions. This operation is not shown in the figure. As shown in the figure, at time-frequency period 1, 512 16-bit elements of column 17 The data text is read from the data random access memory 122 and provided to the 512 neural processing units 126. During the time-frequency period 1 to 512, 512 16-bit weighted texts of columns 0 through 511 are read from weight random access memory 122 and provided to the 512 neural processing units 126, respectively. During the time-frequency period 1, the 512 neural processing units 126 perform their corresponding multiply-accumulate operations on the loaded data text and weight text. This operation is not shown in the figure. During the time-frequency period 2 to 512, the multiplexer 208 of the 512 neural processing unit 126 operates as a rotator with 512 16-bit characters, and will be previously accessed by the data random access memory. The data characters loaded in column 122 are rotated to the adjacent neural processing unit 126, and the neural processing units 126 perform the corresponding data characters after the rotation and the corresponding weight characters loaded by the weight random access memory 124. Multiply accumulate operation. At time FIFO period 513, the 512 start function units 212 execute a start command. This operation is not shown in the figure. During time-frequency period 514, the 512 neural processing units 126 write their corresponding 512 16-bit results 133 back to column 16 of data random access memory 122.

As shown in the figure, the number of time-frequency cycles required to generate the resulting text (neuron output) and write back to the data random access memory 122 or the weighted random access memory 124 is approximately the data received by the current layer of the neural network. Enter the square root of the number of (links). For example, if the current layer has 512 neurons and each neuron has 512 links from the previous layer, the total number of these links is 256K, and the number of time-frequency cycles required to produce the current layer result is slightly greater than 512. Therefore, the neural network unit 121 can provide extremely high performance in neural network calculation.

FIG. 6B is a flow chart showing that the processor 100 of the first figure executes an architectural program to perform a typical multiply-accumulate start function operation of a neuron associated with a hidden layer of an artificial neural network using the neural network unit 121. The operation is as if it were performed by the program of the fourth figure. The example in Figure 6B assumes that there are four hidden layers (variables labeled NUM_LAYERS in initialization step 602), each hidden layer has 512 neurons, and each neuron is connected to all 512 neurons in the previous layer (through the fourth Figure program). However, it is to be understood that the selection of the number of these layers and neurons is illustrative of the present invention. The neural network unit 121 can apply similar calculations to different numbers of hidden layers, with different numbers of nerves in each layer. Embodiments of the element, or embodiments in which the neurons are not all connected. In one embodiment, the weight value for a neuron that does not exist in this layer or a neuron that does not exist will be set to zero. In a preferred embodiment, the architectural program writes the first set of weights to the weighted random access memory 124 and activates the neural network unit 121 when the neural network unit 121 is performing the calculation associated with the first layer. The architecture program writes the second set of weights to the weighted random access memory 124. Thus, once the neural network unit 121 completes the calculation of the first hidden layer, the neural network unit 121 can begin the calculation of the second layer. Thus, the architecture program will travel to and from the two regions of the weighted random access memory 124 to ensure that the neural network unit 121 can be fully utilized. This process begins in step 602.

In step 602, as described in the relevant section of FIG. A, the processor 100 executing the architecture program writes the input value to the current neuron hidden layer of the data random access memory 122, that is, the data is written. The machine accesses column 17 of memory 122. These values may also already be in column 17 of data random access memory 122 as the result 133 (e.g., convolution, common source, or input layer) of neural network unit 121 for the previous layer. Second, the architecture program initializes the variable N to a value of 1. The variable N represents the current layer in the hidden layer that is to be processed by the neural network unit 121. In addition, the architecture program initializes the variable NUM_LAYERS to a value of 4 because there are four hidden layers in this example. The flow then proceeds to step 604.

At step 604, processor 100 writes the weight text of layer 1 into weight random access memory 124, such as columns 0 through 511 shown in Figure 6A. The flow then proceeds to step 606.

In step 606, the processor 100 writes a multiply-accumulate start function program (as shown in the fourth figure) to the neural network unit 121 program memory by specifying a function 1432 to write the MTNN instruction 1400 of the program memory 129. 129. The processor 100 then utilizes an MTNN instruction 1400 to initiate a neural network unit program that specifies a function 1432 to begin execution of the program. The flow then proceeds to step 608.

In decision step 608, the architecture program determines if the value of the variable N is less than NUM_LAYERS. If so, the flow proceeds to step 612; otherwise, it proceeds to step 614.

In step 612, processor 100 writes the weight text of layer N+1 into weighted random access memory 124, such as columns 512 through 1023. Therefore, the architecture program can write the weight text of the next layer to the weight random access memory 124 when the neural network unit 121 performs the hidden layer calculation of the current layer, thereby completing the calculation of the current layer, that is, writing. After the data random access memory 122, the neural network unit 121 can be opened immediately Start the hidden layer calculation of the next layer. Next, proceed to step 614.

In step 614, the processor 100 confirms that the neural network unit program being executed (in the case of layer 1, starting at step 606, and in the case of layers 2 through 4, starting at step 618) has been completed. . In a preferred embodiment, processor 100 reads neural network unit 121 status register 127 by executing an MFNN instruction 1500 to confirm whether execution has been completed. In another embodiment, the neural network unit 121 generates an interrupt indicating that the multiply-accumulate start function layer program has been completed. The flow then proceeds to decision step 616.

In decision step 616, the architecture program determines if the value of the variable N is less than NUM_LAYERS. If so, the flow will proceed to step 618; otherwise, proceed to step 622.

In step 618, the processor 100 updates the multiply-accumulate start function program to enable the hidden layer calculation of layer N+1. Further, the processor 100 updates the column value of the data random access memory 122 of the multiply-accumulate instruction of the address 1 in the fourth figure to the column in which the calculation result of the previous layer in the data random access memory 122 is written ( For example, update to column 16) and update the output column (for example, update to column 15). The processor 100 then begins updating the neural network unit program. In another embodiment, the program of the fourth figure is the same column of the output instruction of the specified address 4 as the column specified by the multiply-accumulate instruction of the address 1 (that is, the column read by the data random access memory 122). ). In this embodiment, the current column of the input data text is overwritten (because the column data has been read into the multiplex register 208 and rotated between the neural processing units 126 via the N-text rotator, as long as this column The text does not need to be used for other purposes. It can be allowed). In this case, there is no need to update the neural network unit program in step 618, but only need to restart it. The flow then proceeds to step 622.

In step 622, processor 100 reads the result of the neural network unit program of layer N from data random access memory 122. However, if these results are only used in the next layer, the architecture program does not need to read the results from the data random access memory 122, but can retain it in the data random access memory 122 for the next hidden layer calculation. Use. The flow then proceeds to step 624.

In decision step 624, the architecture program determines if the value of the variable N is less than NUM_LAYERS. If so, the flow proceeds to step 626; otherwise, the process is terminated.

In step 626, the architecture program increments the value of N by one. The flow then returns to decision step 608.

As shown in the example of FIG. B, the neuroprocessing unit 126 performs a read and a write to the data random access memory 122 substantially every 512 time-frequency cycles (through the neural network of the fourth figure). The effect of the operation of the road unit program). In addition, the neural processing unit 126 reads the weighted random access memory 124 for each time-frequency period to read a list of weighted words. Therefore, the full bandwidth of the weighted random access memory 124 is consumed because the neural network unit 121 performs the hidden layer operation in a mixed manner. Furthermore, assuming that in one embodiment there is a write and read buffer, such as buffer 1704 of FIG. 17, the neural processing unit 126 performs the reading while the processor 100 performs the weighted random access memory 124. Write, so buffer 1704 roughly A write to weight random access memory 124 is performed every 16 time-frequency cycles to write weight text. Thus, in an embodiment where the weighted random access memory 124 is a single unit (as described in the corresponding section of Figure 17), the neuroprocessing units 126 are temporarily placed on the weights for every 16 time-frequency periods. The random access memory 124 reads, and the buffer 1704 can write to the weight random access memory 124. However, in embodiments of the dual weight random access memory 124, the neural processing units 126 need not be placed on hold.

The seventh figure is a block diagram showing another embodiment of the neural processing unit 126 of the first figure. The neural processing unit 126 of the seventh diagram is similar to the neural processing unit 126 of the second figure. However, the neural processing unit 126 of the seventh diagram additionally has a dual input multiplex register 705. The multiplex register 705 selects one of the inputs 206 or 711 to be stored in its register and provides its output 203 for subsequent time-frequency periods. Input 206 receives the weight text from weight random access memory 124. Another input 711 is the output 203 of the second multiplex register 705 that receives the adjacent neural processing unit 126. In a preferred embodiment, the input 711 of the neural processing unit J receives the output 203 of the multiplex register 705 of the neural processing unit 126 arranged at J-1, and the output 203 of the neural processing unit J is provided. The input 711 to the multiplex register 705 of the neural processing unit 126 arranged at J+1. Thus, the multiplex registers 705 of the N neural processing units 126 can operate together, such as the same N text rotator, and the operation is similar to that shown in the third figure above, but for the weight text. Non-data text. The multiplex register 705 uses a control input 213 to control which of the two inputs is selected by the multiplex register 705 for storage in its register. And provided to output 203 in the subsequent.

Utilizing the multiplex register 208 and/or the multiplex register 705 (and the multiplex registers in other embodiments as shown in the eighteenth and twenty-third figures) actually constitutes a large rotator The data/weights from one of the data random access memory 122 and/or the weighted random access memory 124 are rotated, and the neural network unit 121 does not need to be stored in the data random access memory 122 and/or the weights. A very large multiplexer is used between memory 124 to provide the required data/weight text to the appropriate neural network unit.

Write back the accumulator value in addition to the start function result

For some applications, processor 100 is received (eg, received through MFNN instruction of FIG. 15 to media register 118) unprocessed accumulator 202 value 217 for execution by other execution units 112. The instruction execution calculations do have their usefulness. For example, in an embodiment, the startup function unit 212 is not configured for execution of the soft maximal startup function to reduce the complexity of the startup function unit 212. Therefore, the neural network unit 121 outputs the unprocessed accumulator 202 value 217 or one of the subsets to the data random access memory 122 or the weighted random access memory 124, and the architecture program can be randomly randomized in subsequent steps. The access memory 122 or the weight random access memory 124 reads and calculates the unprocessed value. However, the application of the value 217 for the unprocessed accumulator 202 is not limited to performing soft maximal operations, and other applications are also encompassed by the present invention.

The eighth diagram shows the neural processing unit 126 of the first figure. A block diagram of yet another embodiment. The nerve processing unit 126 of the eighth diagram is similar to the neural processing unit 126 of the second figure. However, the neural processing unit 126 of the eighth diagram includes a multiplexer 802 within the startup function unit 212, and the activation function unit 212 has a control input 803. The width (in bits) of the accumulator 202 is greater than the width of the data text. Multiplexer 802 has a plurality of inputs to receive the data text width portion of accumulator 202 output 217. In one embodiment, the accumulator 202 has a width of 41 bits, and the neural processing unit 216 can be used to output a 16-bit result text 133; thus, for example, the multiplexer 802 (or the thirty-th diagram) The multiplexer 3032 and/or multiplexer 3037) has three inputs that receive the bits [15:0], the bits [31:16], and the bits [47:32] of the output 217 of the accumulator 202, respectively. In a preferred embodiment, output bits (e.g., bits [47:41]) that are not provided by accumulator 202 are forced to be zero-valued bits.

The sequencer 128 will generate a value at the control input 803, and the control multiplexer 802 selects one of the words (e.g., 16 bits) of the accumulator 202 to respond to an accumulator instruction, such as in the subsequent ninth diagram. Write accumulator instruction at address 3 through 5. In a preferred embodiment, multiplexer 802 has one or more inputs to receive the output of a start function circuit (e.g., elements 3022, 3024, 3026, 3018, 3014, and 3016 in FIG. 30). The width of the output produced by these start function circuits is equal to one data word. The sequencer 128 will generate a value at the control input 803 to control the multiplexer 802 to select one of these start function circuit outputs instead of selecting one of the words of the accumulator 202 in response to the bit in the fourth figure. The start function unit of address 4 outputs an instruction.

The ninth picture is a table showing one stored in the first picture The program memory 129 of the neural network unit 121 is executed by the neural network unit 121. The example program in the ninth figure is similar to the program in the fourth figure. In particular, the two instructions are exactly the same at addresses 0 through 2. However, the instructions for addresses 3 and 4 in the fourth figure are replaced by a write accumulator instruction in the ninth figure, which instructs 512 neural processing units 126 to write back their accumulator 202 output 217 as a result 133. The three columns of data random access memory 122, in this example, columns 16 through 18. That is, the write accumulator instruction instructs the sequencer 128 to output a data random access memory address 123 of a value of 16 and a write command during the first time-frequency period, and output a second time-frequency period. The data random access memory address 123 and a write command of value 17 output a data random access memory address 123 and a write command of a value of 18 in the third time-frequency cycle. In a preferred embodiment, the execution time of the write accumulator instruction can overlap with other instructions, so that the write accumulator instruction can actually be executed during the three time-frequency periods, each of which is time-frequency. The cycle is written to one of the columns of data random access memory 122. In one embodiment, the user specifies the value of the start function 2934 and the output command 2956 of the control register 127 (FIG. 29A) to write the desired portion of the accumulator 202 to the data random access. Memory 122 or weight random access memory 124. Additionally, the write accumulator instruction can selectively write back a subset of the accumulators 202 instead of writing back all of the accumulator 202. In an embodiment, the standard type accumulator 202 can be written back. This section will be described in more detail in the subsequent sections corresponding to the twenty-ninth to thirty-first.

The tenth diagram shows a timing diagram of the execution of the program of the ninth diagram by the neural network unit 121. The timing chart of the tenth figure is similar to the time of the fifth figure. The sequence diagram, in which the time-frequency periods 0 to 512 are the same. However, in time-frequency periods 513-515, the start function unit 212 of each of the 512 neural processing units 126 performs one of the write accumulator instructions of addresses 3 through 5 in the ninth figure. . In particular, in the time-frequency period 513, each of the 512 neural processing units 126 will write back the bit random access memory 122 to the data random access memory 122 as the result 133 of the accumulator 202 output 217 bits [15:0]. The corresponding text in column 16; in the time-frequency period 514, each of the 512 neural processing units 126 will write back the data as the result 133 of the 217 bits [31:16] of the accumulator 202 output. Corresponding characters in column 17 of random access memory 122; and in time-frequency period 515, each of 512 neural processing units 126 will output accumulator 202 to bits 217 [40:32] As a result 133, the corresponding text in the column 18 of the data random access memory 122 is written back. In a preferred embodiment, bit [47:41] is forced to a zero value.

Shared boot function unit

The eleventh diagram is a block diagram showing an embodiment of a neural network unit 121 of the first figure. In the embodiment of the eleventh diagram, a neuron is divided into two parts, that is, a start function unit part and an arithmetic logic unit part (this part includes a shift register part), and each start function unit part is composed of The arithmetic logic units are partially shared. In the eleventh diagram, the arithmetic logic unit portion refers to the neural processing unit 126, and the shared startup function unit portion refers to the startup function unit 1112. Relative to the embodiment of the second figure, each neuron contains its own The function unit 212 is started. Accordingly, in an example of the eleventh embodiment, the neural processing unit 126 (arithmetic logic unit portion) may include the accumulator 202 of the second graph, the arithmetic logic unit 204, the multiplex register 208, and the register. 205, but does not include the start function unit 212. In the embodiment of the eleventh diagram, the neural network unit 121 includes 512 neural processing units 126, but the present invention is not limited thereto. In the example of the eleventh figure, the 512 neural processing units 126 are divided into 64 groups, labeled as groups 0 to 63 in the eleventh figure, and each group has eight neural processing units 126.

The neural network unit 121 includes a column of buffers 1104 and a plurality of shared start function units 1112. The start function units 1112 are coupled between the neural processing unit 126 and the column buffer 1104. The width of the column buffer 1104 (in bits) is the same as one of the data random access memory 122 or the weight random access memory 124, such as 512 words. Each of the neural processing units 126 has a start function unit 1112, that is, each start function unit 1112 corresponds to a group of neural processing units 126; thus, there are 64 in the embodiment of the eleventh embodiment. The start function unit 1112 corresponds to a group of 64 neural processing units 126. The eight neural processing units 126 of the same group share the start function unit 1112 corresponding to this group. The invention is also applicable to embodiments having a different number of activation function units and a different number of neural processing units in each group. For example, the present invention is also applicable to embodiments in which two, four, or sixteen neural processing units 126 share the same start function unit 1112 in each group.

The shared start function unit 1112 helps to reduce the size of the neural network unit 121. Size reduction will sacrifice performance. That is, Depending on the sharing rate, an additional time-frequency period may be required to generate the result 133 of the entire neural processing unit 126 array, for example, as shown in Figure 12 below, at a sharing ratio of 8:1. Seven additional time-frequency cycles are required. However, in general, compared to the number of time-frequency cycles required to generate the cumulative total (for example, for each neuron with 512 connections, 512 time-frequency periods are required), the aforementioned additional time The number of frequency cycles (for example, 7) is quite small. Therefore, the shared startup function unit has a very small impact on performance (e.g., an increase of approximately one percent of the computation time), which can be a costly reduction for the size of the neural network unit 121 that can be reduced.

In one embodiment, each neural processing unit 126 includes a start function unit 212 for performing a relatively simple start function, these simple start function units 212 having a smaller size to be included in each of the neural processing units 126 Conversely, the shared complex startup function unit 1112 performs a relatively complex startup function that is significantly larger in size than the simple startup function unit 212. In this embodiment, an additional time-frequency period is required only if a complex startup function is specified and needs to be performed by the shared complex startup function unit 1112, in the event that the specified startup function can be executed by the simple startup function unit 212, This additional time-frequency period is not required.

The twelfth and thirteenth drawings are timing charts showing the execution of the program of the fourth diagram by the neural network unit 121 of the eleventh diagram. The timing chart of the twelfth figure is similar to the timing chart of the fifth figure, and the time-frequency periods 0 and 512 of both are the same. However, the operation in the time-frequency period 513 is not the same because the neural processing unit 126 of the eleventh figure shares the startup function unit. 1112; That is, the same group of neural processing units 126 will share the start function unit 1112 associated with the group, and the eleventh figure shows the shared architecture.

Each column of the timing diagram of the thirteenth map corresponds to a continuous time-frequency period indicated in the first row. The other lines correspond to different start function units 1112 of the 64 start function units 1112, respectively, and indicate their operations. Only the operations of the neural processing units 0, 1, 63 are shown in the figure to simplify the explanation. The time-frequency period of the thirteenth map corresponds to the time-frequency period of the twelfth figure, but the operation of the neural processing unit 126 sharing the start-up function unit 1112 is displayed in a different manner. As shown in the thirteenth diagram, in the time-frequency period 0 to 512, the 64 start function units 1112 are all in the inactive state, and the neural processing unit 126 performs the initialization of the neural processing unit instruction, the multiply accumulate instruction, and the multiply accumulate rotation instruction. .

As shown in the twelfth and thirteenth diagrams, at time-frequency period 513, the start function unit 0 (the start function unit 1112 associated with group 0) begins the execution of the specified value for the accumulator 202 value 217 of the neural processing unit 0. The function, neural processing unit 0 is the first neural processing unit 216 in group 0, and the output of the start function unit 1112 will be stored in text 0 of column register 1104. Also in time-frequency period 513, each start function unit 1112 begins the execution of the specified start function for the accumulator 202 value 217 of the first neural processing unit 126 in the corresponding neural processing unit 216 group. Thus, as shown in FIG. 13, at time-frequency period 513, the start function unit 0 begins executing the specified start function on the accumulator 202 of the neural processing unit 0 to generate the text 0 to be stored in the column register 1104. The result; the start function unit 1 begins to accumulate the neural processing unit 8 The controller 202 executes the specified start function to generate the result of the text 8 to be stored in the column register 1104; and so on, the start function unit 63 begins executing the specified start function on the accumulator 202 of the neural processing unit 504. The result of the text 504 to be stored in the column register 1104 is generated.

In the time-frequency period 514, the start function unit 0 (the start function unit 1112 associated with the group 0) begins executing the specified start function for the accumulator 202 value 217 of the neural processing unit 1, the neural processing unit 1 being in group 0. The second neural processing unit 216, and the output of the start function unit 1112 will be stored in the text 1 of the column register 1104. Also at time-frequency period 514, each start function unit 1112 begins to perform the specified start function for the accumulator 202 value 217 of the second neural processing unit 126 in the corresponding neural processing unit 216 group. Thus, as shown in FIG. 13, at time-frequency period 514, the start function unit 0 begins executing the specified start function on the accumulator 202 of the neural processing unit 1 to generate the text 1 to be stored in the column register 1104. As a result, the start function unit 1 begins executing the specified start function on the accumulator 202 of the neural processing unit 9 to produce the result of the literal 9 to be stored in the column register 1104; and so on, the start function unit 63 begins to The accumulator 202 of the neural processing unit 505 executes the specified start function to produce the result of the text 505 to be stored in the column register 1104. Such processing continues until the time-frequency period 520, and the start function unit 0 (the start function unit 1112 associated with the group 0) begins executing the specified start function for the accumulator 202 value 217 of the neural processing unit 7, the neural processing unit 7 That is, the eighth (last) neural processing unit 216 in group 0, and the output of the start function unit 1112 will be stored in the text 7 of the column register 1104. Also in the time-frequency period 520, Each start function unit 1112 begins the execution of the specified start function for the accumulator 202 value 217 of the eighth neural processing unit 126 in the group of corresponding neural processing units 216. Thus, as shown in FIG. 13, at time-frequency period 520, the start function unit 0 begins executing the specified start function on the accumulator 202 of the neural processing unit 7 to generate the text 7 to be stored in the column register 1104. As a result, the start function unit 1 begins executing the specified start function on the accumulator 202 of the neural processing unit 15 to produce the result of the literal 15 to be stored in the column register 1104; and so on, the start function unit 63 begins to The accumulator 202 of the neural processing unit 511 executes the specified start function to produce the result of the text 511 to be stored in the column register 1104.

In the time-frequency period 521, once all 512 results of the 512 neural processing units 126 have been generated and written to the column register 1104, the column register 1104 begins to write its contents to the data random access memory. 122 or weight random access memory 124. Thus, the start function unit 1112 of each of the neural processing units 126 group performs a portion of the start function instruction of address 3 in the fourth figure.

The embodiment of sharing the start function unit 1112 in the arithmetic logic unit 204 group as shown in FIG. 11 is particularly helpful in conjunction with the use of the integer arithmetic logic unit 204. This section will be described in subsequent sections as shown in Figures 29 to 33.

MTNN and MFNN architecture instructions

Figure 14 is a block diagram showing a Move to Neural Network (MTNN) architecture instruction 1400 and its corresponding to the first figure The operation of the portion of the neural network unit 121. The MTNN instruction 1400 includes an execution code field 1402, a src1 field 1404, a src2 field, a gpr field 1408, and an immediate field 1412. The MTNN instruction is an architectural instruction, that is, the instruction is included in the instruction set architecture of the processor 100. In a preferred embodiment, the instruction set architecture utilizes one of the execution code field bits 1402 to distinguish between the MTNN instruction 1400 and other instructions within the instruction set architecture. The execution code 1402 of the MTNN instruction 1400 may or may not be included in a prefix such as the x86 architecture.

Immediate field 1412 provides a value to specify a function 1432 to control logic 1434 of neural network unit 121. In a preferred embodiment, this function 1432 is an immediate operand as one of the microinstructions 105 of the first figure. These functions 1432, which can be executed by the neural network unit 121, include the write data random access memory 122, the write weight random access memory 124, the write program memory 129, the write control register 127, and start execution. The program in the program memory 129, the execution of the program in the program memory 129, the completion of the notification request (for example, interruption) after executing the program in the program memory 129, and the reset of the neural network unit 121 are not limited thereto. In a preferred embodiment, the neural network unit instruction set includes an instruction, the result of which indicates that the neural network unit program has completed. In addition, the neural network unit instruction set includes an explicitly generated interrupt instruction. In a preferred embodiment, the operation of resetting the neural network unit 121 includes, in addition to the data random access memory 122, the weight random access memory 124, and the program memory 129 in the neural network unit 121. The data will remain intact and other parts, effectively forcing a return to a reset state (for example, emptying the internal state machine and setting it up) Set to idle state). In addition, internal registers, such as accumulator 202, are not affected by the reset function and must be explicitly cleared, such as using the initialization neural processing unit instruction of address 0 in the fourth figure. In an embodiment, the function 1432 can include a direct execution function, the first source register of which includes a micro-operation (for example, reference to the micro-operation 3418 of the thirty-fourth figure). This direct execution function instructs the neural network unit 121 to directly perform the specified micro-operation. In this way, the architecture program can directly control the neural network unit 121 to perform operations instead of writing the instructions to the program memory 129 and subsequently instructing the neural network unit 121 to execute the instructions in the program memory 129 or through the MTNN instructions. Execution of 1400 (or MFNN instruction 1500 of Figure 15). The fourteenth figure shows an example of the function of this write data random access memory 122.

This gpr field specifies a general purpose register in the general register file 116. In one embodiment, each general purpose register is 64 bits. The universal scratchpad file 116 provides the value of the selected universal register to the neural network unit 121, as shown in the figure, and the neural network unit 121 uses this value as the address 1422. This address 1422 selects one of the columns of memory specified in function 1432. In the case of data random access memory 122 or weighted random access memory 124, this address 1422 additionally selects a data block that is twice the size of the media register in the selected column (eg, 512). Bit). In a preferred embodiment, this location is at a 512 bit boundary. In one embodiment, the multiplexer selects the address 1422 (either address 1422 in the case of the MFNN instruction 1400 described below) or the address from the sequencer 128 123/125/131 to the data. Random access memory 124 / weight random access memory 124 / Program memory 129. In one embodiment, the data random access memory 122 has a double port, enabling the neural processing unit 126 to read/write to the data random access memory 122 using the media register 118, while reading/writing. The data is randomly accessed into the memory 122. In an embodiment, the weighted random access memory 124 also has double turns for similar purposes.

Both the src1 field 1404 and the src2 field 1406 in the figure specify a media register for the media register file 118. In one embodiment, each media register 118 is 256 bits. The media register file 118 provides the connected data (eg, 512 bits) from the selected media register to the data random access memory 122 (or the weighted random access memory 124 or the program memory). 129) The selected column 1428 specified by the write address 1422 and the location specified by the address 1422 in the selected column 1428, as shown in the figure. Through the execution of a series of MTNN instructions 1400 (and the MFNN instructions 1500 described below), the architecture program executing on the processor 100 can fill the data random access memory 122 columns and the weight random access memory 124 columns and A program write program memory 129, such as the one described herein (such as the programs shown in Figures 4 and 9), allows neural network unit 121 to perform operations on data and weights at very fast speeds to complete the manual. Neural network. In one embodiment, the architecture program directly controls the neural network unit 121 rather than writing the program to the program memory 129.

In one embodiment, the MTNN instruction 1400 specifies a starting source register and the number of source registers, Q, rather than specifying two source registers (as specified by fields 1404 and 1406). This form of MTNN instruction 1400 will instruct the processor 100 to designate as the start The media register 118 of the source register and the next Q-1 consecutive media register 118 are written to the neural network unit 121, that is, the specified data random access memory 122 or the weight random memory is written. Take the memory 124. In a preferred embodiment, the instruction translator 104 translates the MTNN instruction 1400 into the number of microinstructions required to write all of the Q designated media registers 118. For example, in one embodiment, when the MTNN instruction 1400 designates the scratchpad MR4 as the starting source register and Q is 8, the instruction translator 104 translates the MTNN instruction 1400 into four microinstructions, where The first microinstruction is written to the scratchpads MR4 and MR5, the second microinstruction is written to the scratchpads MR6 and MR7, the third microinstruction is written to the scratchpads MR8 and MR9, and the fourth micro The instructions are written to the scratchpads MR10 and MR11. In another embodiment, the data path from media register 118 to neural network unit 121 is 1024 bits instead of 512 bits, in which case instruction translator 104 translates MTNN instruction 1400 into two A microinstruction in which the first microinstruction is written to the scratchpads MR4 to MR7, and the second microinstruction is written to the scratchpads MR8 to MR11. The present invention is also applicable to embodiments in which the MFNN instruction 1500 specifies a number of start destination registers and destination registers, such that each MFNN instruction 1500 can be accessed from the data random access memory 122 or the weighted random access memory. One of the columns 124 reads a block of data larger than the single media register 118.

The fifteenth diagram is a block diagram showing the operation of a mobile-to-neural network (MTNN) architecture command 1500 and its portion corresponding to the neural network unit 121 of the first diagram. The MFNN instruction 1500 includes an execution code field 1502, a dst field 1504, and a gpr field 1508. And an immediate field 1512. The MFNN instruction is an architectural instruction, that is, the instruction is included in the instruction set architecture of the processor 100. In a preferred embodiment, the instruction set architecture utilizes one of the execution code fields 1502 to distinguish between the MFNN instruction 1500 and other instructions within the instruction set architecture. The execution code 1502 of the MFNN instruction 1500 may or may not be included in a prefix such as the x86 architecture.

Immediate field 1512 provides a value to specify a function 1532 to control logic 1434 of neural network unit 121. In a preferred embodiment, this function 1532 is an immediate operand as one of the microinstructions 105 of the first figure. The functions 1532 that the neural network unit 121 can execute include a read data random access memory 122, a read weight random access memory 124, a read program memory 129, and a read status register 127, but not Limited to this. The example of the fifteenth figure shows a function 1532 of the read data random access memory 122.

This gpr field 1508 specifies a general purpose register within the general register file 116. The universal scratchpad file 116 provides the value of the selected universal register to the neural network unit 121, as shown in the figure, and the neural network unit 121 uses this value as the address 1522 and is similar to the tenth The operation of the address 1422 of the four graphs is performed to select one of the memories specified in the function 1532. In the case of the data random access memory 122 or the weighted random access memory 124, the address 1522 additionally selects a data block whose size is the media register (eg, 256 bits) in the selected column. position. In a preferred embodiment, this location is at a 256 bit boundary.

This dst field 1504 is attached to a media register file. A media register is specified in 118. As shown in the figure, the media register file 118 receives data (eg, 256 bits) from the data random access memory 122 (or the weighted random access memory 124 or the program memory 129) to the selected media. The scratchpad, this data is read from the selected column 1528 specified by the data receiving address 1522 and the location specified by the address 1522 in the selected column 1528.

Neural network unit internal random access memory configuration

Figure 16 is a block diagram showing an embodiment of the data random access memory 122 of the first figure. The data random access memory 122 includes a memory array 1606, a read buffer 1602 and a write buffer 1604. The memory array 1606 is loaded with data text. In a preferred embodiment, the data is arranged in an array of N words of D columns as previously described. In one embodiment, the memory array 1606 includes an array of 64 horizontally arranged static random access memory cells, each of which has a width of 128 bits and a height of 64 bits. A 64 KB data random access memory 122 is provided having a width of 8192 bits and having 64 columns, and the data random access memory 122 uses a die area of approximately 0.2 square millimeters. However, the invention is not limited thereto.

In a preferred embodiment, write 埠 1602 is coupled to neural processing unit 126 and media register 118 in a multiplexed manner. Further, the media registers 118 can be coupled to the read buffer through the result bus, and the resulting bus is also used to provide data to the rearrangement buffer and/or the result transfer bus to be provided to the other execution units 112. . These ones The neural processing unit 126 and the media register 118 share the read buffer 1602 to read the data random access memory 122. Moreover, in a preferred embodiment, write 埠 1604 is also multiplexed to neural processing unit 126 and media register 118. The neural processing unit 126 shares the write buffer 1604 with the media register 118 to write the data random access memory 122. In this manner, the media register 118 can write the data random access memory 122 while the neural processing unit 126 reads the data random access memory 122, and the neural processing unit 126 can temporarily store the data in the media. The device 118 is writing to the data random access memory 122 while reading the data random access memory 122. This way of doing things can improve performance. For example, the neural processing unit 126 can read the data random access memory 122 (eg, continuously perform calculations) while the media register 118 can write more data to the data random access memory 122. . In another example, the neural processing unit 126 can write the calculation result to the data random access memory 122, and at the same time, the media register 118 can read the calculation result from the data random access memory 122. In one embodiment, the neural processing unit 126 can write a list of calculation results to the data random access memory 122 while also reading a list of data characters from the data random access memory 122. In one embodiment, memory array 1606 is configured as a memory bank. When the neural processing unit 126 accesses the data random access memory 122, all memory blocks are activated to access a complete column of the memory array 1606; however, the data is randomly accessed in the media register 118. When the memory 122 is accessed, only the specified memory block will be activated. In one embodiment, each memory block has a width of 128 The width of the media register 118 is 256 bits. Thus, for example, each time the media register 118 is accessed, two memory blocks need to be activated. In one embodiment, one of the 埠 1602/1604 is a read/write 埠. In an embodiment, these 埠 1602/1604 are both read/write 埠.

An advantage of having these neural processing units 126 with the capabilities of a rotator as described herein is that the architectural program (via the media register 118) is continuously provided with information to ensure that the neural processing unit 126 is fully utilized. The data random access memory 122 and the memory array required to retrieve the results from the data random access memory 122 while the neural processing unit 126 is performing the calculations, this ability helps to reduce the data random access memory 122 The number of columns of the memory array 1606 can thus be reduced in size.

Internal random access memory buffer

Figure 17 is a block diagram showing an embodiment of a weighted random access memory 124 and a buffer 1704 of the first figure. The weight random access memory 124 includes a memory array 1706 and a stack 1702. The memory array 1706 is loaded with weighting text. For a preferred embodiment, the weighting texts are arranged in an array of N words of W columns as previously described. In one embodiment, the memory array 1706 includes an array of 128 horizontally arranged static random access memory cells, each of which has a width of 64 bits and a height of 2048 bits. A 2MB weight random access memory 124 is provided having a width of 8192 bits and having 2048 columns, and the weight random access memory 124 uses a die area of approximately 2.4 square millimeters. However, the invention is not limited thereto.

In a preferred embodiment, the 埠 1702 is multiplexed to the neural processing unit 126 and the buffer 1704. The neural processing unit 126 and the buffer 1704 are read through the buffer 1702 and written to the weight random access memory 124. The buffer 1704 is coupled to the media register 118 of the first figure. Thus, the media register 118 can read and write the weight random access memory 124 through the buffer 1704. An advantage of this approach is that the media register 118 can also write to or read from the buffer 118 when the neural processing unit 126 is reading or writing to the weighted random access memory 124 (but if The neural processing unit 126 is executing, and preferably, the neural processing unit 126 is placed to avoid accessing the weighted random access memory 124 when the buffer 1704 accesses the weighted random access memory 124. This approach can improve performance, particularly since the read and write of the media scratchpad 118 for the weighted random access memory 124 is significantly less than the read and write by the neural processing unit 126 for the weighted random access memory 124. . For example, in one embodiment, the neural processing unit 126 reads/writes 8192 bits (one column) at a time, however, the width of the media register 118 is only 256 bits, and each MTNN instruction 1400 is only Two media registers 118 are written, ie 512 bits. Therefore, in the case where the architecture program executes sixteen MTNN instructions 1400 to fill the buffer 1704, the collision time between the neural processing unit 126 and the architectural program accessing the weight random access memory 124 may be less than substantially all of the time. Six percent. In another embodiment, the instruction translator 104 translates one MTNN instruction 1400 into two microinstructions 105, and each microinstruction writes a single data register 118 to the buffer. 1704, as such, the frequency at which the neural processing unit 126 and the architectural program collide when accessing the weighted random access memory 124 is further reduced.

In an embodiment that includes buffer 1704, the use of an architectural program to write weighted random access memory 124 requires a plurality of MTNN instructions 1400. One or more MTNN instructions 1400 specify a function 1432 to write to the data block specified in buffer 1704, and then an MTNN instruction 1400 specifies a function 1432 to instruct neural network unit 121 to write the contents of buffer 1704 to weight random access. One of the memories 124 is selected. The size of a single data block is twice the number of bits in the media register 118, and these data blocks are naturally aligned in the buffer 1704. In one embodiment, each of the specified functions 1432 in the write buffer 1704 specifies that the MTNN instruction 1400 of the data block includes a bitmask having bits corresponding to the respective data blocks of the buffer 1704. The data from the two designated source registers 118 are written into the data block of the buffer 1704, and the corresponding bits in the bit mask are the respective data blocks that are set. This embodiment facilitates the case where there is a duplicate data value in one of the columns of the weighted random access memory 124. For example, to zero the buffer 1704 (and the next column of weighted random access memory 124), the programmer can load zero values into the source register and set all bits of the bit mask. . In addition, the bit mask can also allow the programmer to write only the selected data blocks in buffer 1704, leaving other data blocks to maintain their previous data state.

In an embodiment that includes buffer 1704, reading the weight random access memory 124 with the architectural program requires a plurality of MFNN instructions 1500. The initial MFNN instruction 1500 specifies a function 1532 to load a specified column of one of the weighted random access units 124 into the buffer 1704, followed by an OR A plurality of MFNN instructions 1500 specify a function 1532 to read a designated data block of one of the buffers 1704 to the destination register. The size of a single data block is the number of bits in the media register 118, and these data blocks are naturally aligned in the buffer 1704. The technical features of the present invention are also applicable to other embodiments. For example, the weight random access memory 124 has a plurality of buffers 1704, which further reduces the number of neural processing units by increasing the number of accesses of the architecture program when the neural processing unit 126 executes. The conflict between the 126 and the architecture program due to the access weight random memory 124 is increased by the buffer 1704 during the time-frequency period in which the neural processing unit 126 does not need to access the weight random access memory 124. Take the possibility.

The sixteenth figure depicts a pair of data random access memory 122, however, the present invention is not limited thereto. The technical features of the present invention are also applicable to other embodiments in which the weighted random access memory 124 is also a dual-turn design. Further, a buffer matching weight random access memory 124 is used in the seventeenth figure, but the present invention is not limited thereto. The technical features of the present invention are also applicable to embodiments in which the data random access memory 122 has a corresponding buffer similar to the buffer 1704.

Dynamically configurable neural processing unit

Figure 18 is a block diagram showing a dynamically configurable neural processing unit 126 of the first figure. The neural processing unit 126 of the eighteenth diagram is similar to the neural processing unit 126 of the second figure. However, the neural processing unit 126 of Fig. 18 is dynamically configurable to operate in one of two different configurations. In the first configuration, the operation of the neural processing unit 126 of Fig. 18 is similar to the neural processing unit 126 of the second figure. That is, in the first configuration, it is labeled "wide" here. In a configuration or "single one" configuration, the arithmetic logic unit 204 of the neural processing unit 126 performs operations on a single wide data text and a single wide weight text (e.g., 16 bits) to produce a single wide result. In contrast, in the second configuration, which is labeled herein as a "narrow" configuration or a "double" configuration, the neural processing unit 126 will have two narrow data words and two narrow weight texts (eg, eight). Bits) perform operations that produce two narrow results, respectively. In one embodiment, the configuration (wide or narrow) of the neural processing unit 126 is achieved by initializing a neural processing unit instruction (e.g., an instruction located at address 0 in the aforementioned twentieth diagram). Alternatively, this configuration can be accomplished by an MTNN command having a configuration (wide or narrow) that the function 1432 specifies to set the neural processing unit settings. In a preferred embodiment, the program memory 129 instructs or determines that the configuration (wide or narrow) MTNN instruction fills up the configuration register. For example, the output of the configuration register is provided to the logic logic unit 204, the startup function unit 212, and the logic that generates the multiplex register control signal 213. Basically, the elements of the neural processing unit 126 of Fig. 18 and the elements of the same number in the second figure perform similar functions, from which reference can be made to understand the embodiment of Fig. 18. The following description of the embodiment of the eighteenth embodiment includes the differences from the second figure.

The neural processing unit 126 of Fig. 18 includes two registers 205A and 205B, two three-input multiplex registers 208A and 208B, one arithmetic logic unit 204, two accumulators 202A and 202B, and two start-ups. Function units 212A and 212B. The registers 205A/205B have one-half (e.g., 8 bits) of the width of the register 205 of the second figure, respectively. The registers 205A/205B are respectively from the weight random access memory 124 A corresponding narrow weight text 206A/B 206 (e.g., 8 bits) is received and its output 203A/203B is provided to the operand selection logic 1898 of the arithmetic logic unit 204 at a subsequent time-frequency period. When the neural processing unit 126 is in a wide configuration, the registers 205A/205B operate together to receive a wide weight text 206A/206B (e.g., 16 bits) from the weighted random access memory 124, similar to the second In the embodiment of the embodiment, when the neural processing unit 126 is in a narrow configuration, the registers 205A/205B are actually operated independently, each receiving a narrow weight text from the weighted random access memory 124. 206A/206B (e.g., 8 bits), as such, the neural processing unit 126 is essentially equivalent to two narrow neural processing units operating independently of each other. However, regardless of the configuration of the neural processing unit 126, the same output bits of the weighted random access memory 124 are coupled and provided to the registers 205A/205B. For example, the register 205A of the neural processing unit 0 receives the byte 0, the register 205B of the neural processing unit 0 receives the byte 1, the register 205A of the neural processing unit 1 receives the byte 2, and the nerve The register 205B of the processing unit 1 receives the byte 3, and so on, and the register 205B of the neural processing unit 511 receives the byte 1023.

The multiplex registers 208A/208B have one-half (e.g., 8 bits) of the width of the register 208 of the second figure, respectively. Multiplex register 208A selects one of inputs 207A, 211A, and 1811A to store to its scratchpad and is provided by output 209A for subsequent time-frequency cycles. Multiplexed register 208B selects among inputs 207B, 211B, and 1811B. One is stored to its register and is provided by the output 209B to the operand selection logic 1898 at subsequent time-frequency cycles. Input 207A receives a narrow from data random access memory 122 Data text (e.g., 8 bits), input 207B receives a narrow data text from data random access memory 122. When the neural processing unit 126 is in a wide configuration, the multiplex registers 208A/208B will actually operate together to receive a wide data text 207A/207B from the data random access memory 122 (e.g., 16 bits). ), similar to the multiplex register 208 in the embodiment of the second figure; when the neural processing unit 126 is in a narrow configuration, the multiplex registers 208A/208B will actually operate independently, each receiving random data from One of the narrow data words 207A/207B (e.g., 8 bits) is accessed by the memory 122. Thus, the neural processing unit 126 is essentially equivalent to the operation of the two narrow neural processing units. However, regardless of the configuration of the neural processing unit 126, the same output bits of the data random access memory 122 are coupled and provided to the multiplex registers 208A/208B. For example, the multiplexer 208A of the neural processing unit 0 receives the byte 0, the multiplexer 208B of the neural processing unit 0 receives the byte 1, and the multiplexer 208A of the neural processing unit 1 receives The octet 2, the multiplexer 208B of the neural processing unit 1 receives the byte 3, and so on, and the multiplexer 208B of the neural processing unit 511 receives the byte 1023.

Input 211A receives output 209A of multiplex register 208A of neighboring neural processing unit 126, and input 211B receives output 209B of multiplex register 208B of neighboring neural processing unit 126. Input 1811A receives output 209B of multiplex register 208B of proximity neural processing unit 126, and input 1811B receives output 209A of multiplex register 208A of adjacent neural processing unit 126. The nerve processing unit 126 shown in FIG. 18 belongs to one of the N nerve processing units 126 shown in the first figure. Also labeled as nerve processing unit J. That is to say, the neural processing unit J is a representative example of one of the N neural processing units. In a preferred embodiment, the multiplexer 208A input 211A of the neural processing unit J receives the output 209A of the multiplexer 208A of the neural processing unit 126 of the example J-1, and the number of the neural processing unit J is The physical register 208A input 1811A will receive the multiplex buffer 208B output 209B of the neural processing unit 126 of the example J-1, and the multiplex register 208A output 209A of the neural processing unit J will be provided to the sample J+1 at the same time. The multiplexer register 208A of the neural processing unit 126 inputs 211A and the multiplexer 208B of the neural processing unit 126 of the example J inputs 211B; the input 211B of the multiplexer 208B of the neural processing unit J receives the sample J The multiplex register 208B of the neural processing unit 126 of the -1 outputs 209B, and the input 1811B of the multiplex register 208B of the neural processing unit J receives the output 209A of the multiplexer 208A of the neural processing unit 126 of the example J. And, the output 209B of the multiplexer 208B of the neural processing unit J is simultaneously provided to the multiplexer 208A input 1811A of the neural processing unit 126 of the example J+1 and the neural processing unit 126 of the example J+1. The multiplex register 208B inputs 211B.

Control input 213 controls each of multiplex registers 208A/208B, selects one of the three inputs to store to its corresponding register, and provides it to the corresponding output 209A/209B in a subsequent step. When the neural processing unit 126 is instructed to load a column from the data random access memory 122 (for example, the multiply-accumulate instruction of address 1 in the twentieth map, as will be described later), regardless of whether the neural processing unit 126 is in a wide configuration Or a narrow configuration, the control input 213 controls each of the multiplex registers 208A/208B, from the data random access record A corresponding narrow text file 207A/207B (eg, 8-bit) is selected from the corresponding narrow text of the selected column of the memory 122.

When the neural processing unit 126 receives the indication that the previously received data column value needs to be rotated (for example, the multiplication accumulation rotation instruction of the address 2 in the twentieth figure, as will be described later), if the neural processing unit 126 is in a narrow configuration, the control Input 213 controls each of the multiplex registers 208A/208B to select the corresponding input 1811A/1811B. In this case, the multiplex registers 208A/208B will actually operate independently and the neural processing unit 126 will behave as two separate narrow neural processing units. Thus, the multiplexers 208A and 208B of the N neural processing units 126 operate together as the same 2N narrow text rotators, which is described in more detail later in the corresponding FIG.

When the neural processing unit 126 receives an indication that the previously received data column value needs to be rotated, if the neural processing unit 126 is in a wide configuration, the control input 213 controls each of the multiplex registers 208A/208B. The device selection corresponds to input 211A/211B. In this case, the multiplex registers 208A/208B will operate together as if the neural processing unit 126 were a single wide neural processing unit 126. Thus, the multiplex registers 208A and 208B of the N neural processing units 126 operate together as the same N wide-text rotators, similarly to the manner described in the third figure.

Arithmetic logic unit 204 includes operand selection logic 1898, a wide multiplier 242A, a narrow multiplier 242B, a wide dual input multiplexer 1896A, a narrow dual input multiplexer 1896B, a Wide adder 244A and a narrow adder 244B. In fact, this arithmetic logic unit 204 can be understood to include operand selection logic, a wide arithmetic logic unit 204A (including the aforementioned wide multiplier 242A, the aforementioned wide multiplexer 1896A and the aforementioned wide adder 244A) and a narrow arithmetic logic unit. 204B (including the aforementioned narrow multiplier 242B, the aforementioned narrow multiplexer 1896B and the aforementioned narrow adder 244B). In a preferred embodiment, wide multiplier 242A can multiply two wide words, similar to multiplier 242 of the second figure, such as a 16-bit by 16-bit multiplier. Narrow multiplier 242B can multiply two narrow words, such as an 8-bit by 8-bit multiplier to produce a 16-bit result. When the neural processing unit 126 is in a narrow configuration, the wide multiplier 242A can be fully utilized as a narrow multiplier to multiply the two narrow words by the assistance of the operand selection logic 1898, so that the neural processing unit 126 Two narrowly operated nerve processing units. In a preferred embodiment, wide adder 244A adds the output of wide multiplexer 1896A to output 217A of wide accumulator 202A to produce a total number 215A for use by wide accumulator 202A, which operates similarly to Adder 244 of the second figure. Narrow adder 244B adds the output of narrow multiplexer 1896B to narrow accumulator 202B output 217B to produce a total number 215B for use by narrow accumulator 202B. In one embodiment, the narrow accumulator 202B has a width of 28 bits to avoid loss of accuracy when performing an accumulation operation of up to 1024 16-bit products. When the neural processing unit 126 is in a wide configuration, the narrow multiplier 244B, the narrow accumulator 202B, and the narrow start function unit 212B are preferably in an inactive state to reduce energy consumption.

The operand selection logic 1898 will select the operands from 209A, 209B, 203A, and 203B to provide to the arithmetic logic unit 204. His components are described in detail later. In a preferred embodiment, operand selection logic 1898 also has other functions, such as performing symbolic extensions of signed numeric data and weighted text. For example, if the neural processing unit 126 is in a narrow configuration, the operand selection logic 1898 extends the sign of the narrow data text and the weight text to the width of the wide text and then provides it to the wide multiplier 242A. Similarly, if the arithmetic logic unit 204 accepts an indication that a narrow data/weight text is to be passed (using the wide multiplexer 1896A to skip the wide multiplier 242A), the operand selection logic 1898 extends the sign of the narrow data text and the weight text to The width of the wide text is then provided to the wide adder 244A. For a preferred embodiment, the logic for performing the symbol stretching function is also present within the arithmetic logic operation 204 of the neural processing unit 126 of the second figure.

The wide multiplexer 1896A receives the output of the wide multiplier 242A and one of the operands from the operand selection logic 1898, and selects one of these inputs for the wide adder 244A, and the narrow multiplexer 1896B receives the narrow multiplier 242B. The output is derived from one of the operand selection logic 1898 and one of these inputs is provided to the narrow adder 244B.

The operand selection logic 1898 provides operands in accordance with the configuration of the neural processing unit 126 and the arithmetic and/or logical operations to be performed by the arithmetic logic unit 204, which is based on the function specified by the instructions executed by the neural processing unit 126. Decide. For example, if the instruction instructs the arithmetic logic unit 204 to perform a multiply-accumulate operation and the neural processing unit 126 is in a wide configuration, the operand selection logic 1898 concatenates the outputs 209A and 209B to form a wide text to provide a wide multiplication. One of the inputs 242A is input, and the outputs 203A and 203B are connected in series to form one wide text to be supplied to the other input, and the narrow multiplier 242B is not activated. Thus, the operation of the neural processing unit 126 is similar to that of a single one. The wide nerve processing unit 126 of the neural processing unit 126 of the second figure. However, if the instruction instructs the arithmetic logic unit to perform a multiply-accumulate operation and the neural processing unit 126 is in a narrow configuration, the operand selection logic 1898 provides an extended or expanded version of the narrow data word 209A to one of the wide multipliers 242A. Input, while the extended version of the narrow weight text 203A is provided to another input; in addition, the operand selection logic 1898 provides the narrow data text 209B to one of the narrow multipliers 242B and the narrow weight text 203B to the other An input. In order to achieve the operation of extending or expanding the narrow text as described above, if the narrow text has a symbol, the operation element selection logic 1898 will perform symbol extension on the narrow text; if the narrow text has no symbol, the operation element selection logic 1898 The upper zero value is added to the narrow text.

In another example, if the neural processing unit 126 is in a wide configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation operation of a weighted text, the wide multiplier 242A is skipped and the operand selection logic 1898 will output 203A. Serial multiplexer 1896A is provided in series with 203B for supply to wide adder 244A. However, if the neural processing unit 126 is in a narrow configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation operation of a weighted text, the wide multiplier 242A is skipped and the operand selection logic 1898 will have an extended version output. 203A is provided to wide multiplexer 1896A for supply to wide adder 244A; in addition, narrow multiplier 242B is skipped and operand selection logic 1898 will be extended The output 203B of the latter version is provided to the narrow multiplexer 1896B for supply to the narrow adder 244B.

In another example, if the neural processing unit 126 is in a wide configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation of data, the wide multiplier 242A is skipped and the operand selection logic 1898 will output 209A. Serial multiplexer 1896A is provided in series with 209B for supply to wide adder 244A. However, if the neural processing unit 126 is in a narrow configuration and the instruction instructs the arithmetic logic unit 204 to perform a data word accumulation operation, the wide multiplier 242A is skipped and the operand selection logic 1898 will have an extended version output. 209A is provided to wide multiplexer 1896A for supply to wide adder 244A; in addition, narrow multiplier 242B is skipped, and operand selection logic 1898 provides extended version 209B to narrow multiplexer 1896B for provision to Narrow adder 244B. The cumulative calculation of weights/data texts contributes to the averaging operation, which can be used in the pooling layer of some artificial neural network applications, such as image processing.

In a preferred embodiment, the neural processing unit 126 further includes a second wide multiplexer (not shown) for skipping the wide adder 244A to facilitate wide data/weight text in a wide configuration. Or a narrow data/weight text loaded into the wide accumulator 202A in a narrow configuration, and a second narrow multiplexer (not shown) to skip the narrow adder 244B to facilitate narrow configuration The next narrow data/weight text is loaded into the narrow accumulator 202B. In a preferred embodiment, the arithmetic logic unit 204 further includes a wide and narrow comparator/multiplexer combination (not shown) that receives the corresponding accumulator value 217A. /217B And the corresponding multiplexer 1896A/1896B output, so that the maximum value is selected between the accumulator value 217A/217B and a data/weight text 209A/209B/203A/203B, and the common source of some artificial neural network applications (pooling) The layer system uses this operation, which is described in more detail in subsequent sections, for example, corresponding to the twenty-seventh and twenty-eighth diagrams. In addition, the operand selection logic 1898 is used to provide an arithmetic element of zero value (for addition of zeros or to clear the accumulator) and to provide an arithmetic element of value one (for multiplication by one).

Narrow start function unit 212B receives output 217B of narrow accumulator 202B and performs a start function thereon to produce a narrow result 133B. Wide start function unit 212A receives output 217A of wide accumulator 202A and performs a start function thereon to generate a The wide result is 133A. When the neural processing unit 126 is in a narrow configuration, the wide start function unit 212A will understand the output 217A of the accumulator 202A and perform a start function on it to generate a narrow result, such as an 8-bit, which is corresponding in subsequent sections. A more detailed description is given in Figures 29 to 30.

As previously mentioned, a single neural processing unit 126 can actually operate as two narrow neural processing units when in a narrow configuration, so that for smaller text, it is generally comparable to a wide configuration. Up to twice the processing power. For example, suppose the neural network layer has 1024 neurons, and each neuron receives 1024 narrow inputs (and has narrow weighted text) from the previous layer, thus producing one million links. For a neural network unit 121 having 512 neural processing units 126, in a narrow configuration (equivalent to 1024 narrow neural processing units), although it is a narrow text rather than a wide text, it is capable of The number of links can be up to four times the wide configuration (256K connections on one million links), and the required time is roughly half (about 1026 time-frequency periods versus 514 time-frequency periods).

In one embodiment, the dynamic configuration neural processing unit 126 of FIG. 18 includes a three-input multiplexer similar to the multiplex registers 208A and 208B in place of the registers 205A and 205B to form a rotator. The weighted character string received by the weighted random access memory 124 is processed in a manner similar to that described in the embodiment of the seventh embodiment but applied to the dynamic configuration described in the eighteenth figure.

Figure 19 is a block diagram showing the 2N multiplex registers 208A/208B of the N neural processing units 126 of the neural network unit 121 of the first figure according to the embodiment of the eighteenth figure. The data random access memory 122 of the first figure obtains a list of data words 207 to perform operations as the same rotator. In the embodiment of the nineteenth embodiment, N is 512, and the neural processing unit 121 has 1024 multiplex registers 208A/208B, labeled 0 to 511, corresponding to 512 neural processing units 126 and 1024 actually. Narrow nerve processing unit. The two narrow neural processing units within the neural processing unit 126 are labeled A and B, respectively, and in each multiplex register 208, the corresponding narrow neural processing unit is also labeled. Further, the multiplex register 208A of the neural processing unit 126 labeled 0 is labeled 0-A, and the multiplex register 208B of the neural processing unit 126 labeled 0 is labeled 0-B, labeled as The multiplex register 208A of the neural processing unit 126 of 1 is labeled 1-A, and the multiplex register 208B of the neural processing unit 126 labeled 1 is labeled 1-B, and the neural processing unit 126 is labeled 511. The multiplex register 208A is labeled as 511-A, and the multiplex register 208B of the neural processing unit 126, labeled 511, is labeled 511-B, the value of which also corresponds to the narrow neural processing unit described in the subsequent twenty-first figure.

Each multiplex register 208A receives its corresponding narrow data 207A in one of the D columns of the data random access memory 122, and each multiplex buffer 208B is in the data random access memory. One of the D columns of 122 receives its corresponding narrow data text 207B. That is, the multiplex register 0-A receives the narrow data text 0 of the data random access memory 122 column, and the multiplex register 0-B receives the narrow data text 1 of the data random access memory 122 column. The multiplexed register 1-A receives the narrow data text 2 of the data random access memory 122, the multiplex register 1-B receives the narrow data text of the data random access memory 122, and so on. The multiplexed register 511-A receives the narrow data text 1022 of the data random access memory 122, and the multiplex register 511-B receives the narrow data text 1023 of the data random access memory 122. In addition, the multiplex register 1-A receives the output 209A of the multiplex register 0-A as its input 211A, and the multiplex register 1-B receives the output 209B of the multiplex register 0-B as its input. 211B, and so on, the multiplex register 511-A receives the output 209A of the multiplex register 510-A as its input 211A, and the multiplex register 511-B receives the output of the multiplex register 510-B. 209B as its input 211B, and the multiplex register 0-A receives the output 209A of the multiplex register 511-A as its input 211A, and the multiplex register 0-B receives the multiplex register 511-B. Output 209B is its input 211B. Each multiplex register 208A/208B receives control input 213 to control its selection data 207A/207B or rotates input 211A/211B or rotates and then loses Into 1811A/1811B. Finally, the multiplex register 1-A receives the output 209B of the multiplex register 0-B as its input 1811A, and the multiplex register 1-B receives the output 209A of the multiplex register 1-A as its input. 1811B, and so on, the multiplex register 511-A receives the output 209B of the multiplex register 510-B as its input 1811A, and the multiplex register 511-B receives the output of the multiplex register 511-A. 209A as its input 1811B, and the multiplex register 0-A receives the output 209B of the multiplex register 511-B as its input 1811A, and the multiplex register 0-B receives the multiplex register 0-A Output 209A is its input 1811B. Each multiplex register 208A/208B receives a control input 213 to control its selection data 207A/207B or a post-rotation input 211A/211B or a post-rotation input 1811A/1811B. In an operational mode, during the first time-frequency period, control input 213 controls each multiplex register 208A/208B to select data words 207A/207B for storage to the scratchpad for subsequent supply to arithmetic logic unit 204; Subsequent time-frequency periods (eg, the M-1 time-frequency period described above), control input 213 controls each multiplex register 208A/208B to select the rotated input 1811A/1811B to store to the scratchpad for subsequent supply to the arithmetic logic unit 204, this section will be described in more detail in subsequent chapters.

Figure 20 is a table showing a program stored in the program memory 129 of the neural network unit 121 of the first figure and executed by the neural network unit 121, and the neural network unit 121 has the eighteenth The nerve processing unit 126 shown in the embodiment of the figure. The example program in Figure 20 is similar to the program in Figure 4. The differences are explained below. The initialization neural processing unit instruction at address 0 specifies that the neural processing unit 126 will enter a narrow configuration. In addition, as shown in the figure, in position The multiply-accumulate rotation command of address 2 specifies a count value of 1023 and requires 1023 time-frequency periods. This is because the example in the twentieth diagram assumes that there are actually 1024 narrow (eg, 8-bit) neurons (ie, neural processing units) in one layer, each with 1024 neurons from the previous layer. The link of the yuan is entered, so there are a total of 1024K links. Each neuron receives an 8-bit data value from each of the link inputs and multiplies the 8-bit data value by an appropriate 8-bit weight value.

The twenty-first figure shows a timing chart in which a neural network unit 121 executes the program of the twentieth diagram, and the neural network unit 121 has the neural processing unit 126 as shown in the eighteenth diagram executed in a narrow configuration. The timing chart of the twenty-first figure is similar to the timing chart of the fifth figure. The differences are explained below.

In the timing diagram of the twenty-first figure, these neural processing units 126 will be in a narrow configuration because the initialization neural processing unit instruction at address 0 initializes it to a narrow configuration. Therefore, the 512 neural processing units 126 actually function as 1024 narrow neural processing units (or neurons), which are treated with neuroprocessing units 0-A and neural processing units in the field. 0-B (two narrow neuroprocessing units of neuroprocessing unit 126 labeled 0), neuroprocessing unit 1-A and neuroprocessing unit 1-B (two narrow neural processing units of neuroprocessing unit 126 labeled 1) ), and so on until the neural processing unit 511-A and the neural processing unit 511-B (the two narrow neural processing units of the neural processing unit 126 labeled 511) are indicated. To simplify the description, only the operations of the narrow neural processing units 0-A, 0-B, and 511-B are shown. Because the count value specified by the multiply-accumulate rotation instruction at address 2 is 1023, and requires 1023 time-frequency cycles to operate. Therefore, the number of timing charts in the twenty-first figure includes up to 1026 time-frequency periods.

In a time-frequency period of 0, each of the 1024 neural processing units performs an initialization instruction of the fourth map, i.e., assigns a zero value to the operation of the accumulator 202 as shown in the fifth figure.

In time-frequency period 1, each of the 1024 narrow neural processing units performs a multiply-accumulate instruction at address 1 in the twentieth diagram. As shown in the figure, the narrow neural processing unit 0-A adds the value of the accumulator 202A (i.e., zero) to the column of the data random access unit 122, the narrow text 0, and the weight random access unit 124, the narrow text 0. Product; the narrow neural processing unit 0-B adds the value of the accumulator 202B (i.e., zero) to the product of the narrow text 1 of the data random access unit 122 and the narrow text 1 of the weight random access unit 124; Analogy until the narrow neural processing unit 511-B adds the value of the accumulator 202B (i.e., zero) to the product of the narrow text 1023 of the column 17 of the data random access unit 122 and the narrow text 1023 of the weight random access unit 124.

In time-frequency period 2, each of the 1024 narrow neural processing units performs the first iteration of the multiply-accumulate rotation instruction at address 2 in the twentieth diagram. As shown in the figure, the narrow neural processing unit 0-A adds the value 217A of the accumulator 202A to the rotated narrow data text 1811A received by the multiplexer 208B output 209B of the narrow neural processing unit 511-B (ie, The narrow data word 1023) received by the data random access memory 122 and the narrow text 0 of the column 1 of the weight random access unit 124; the narrow neural processing unit 0-B adds the value 217B of the accumulator 202B to the narrow nerve Processing unit 0-A multiplex register 208A receives 209A received After rotation, the narrow data text 1811B (that is, the narrow data text 0 received by the data random access memory 122) is multiplied by the narrow text 1 of the weight random access unit 124; and so on, until narrow nerve processing Unit 511-B adds the accumulator 202B value 217B to the rotated narrow data text 1811B received by the multiplexer 208A output 209A of the narrow neural processing unit 511-A (i.e., by the data random access memory 122). The product of the narrow data text 1022 received and the narrow text 1023 of the column 1 of the weight random access unit 124.

In time-frequency period 3, each of the 1024 narrow neural processing units performs the second iteration of the multiply-accumulate rotation instruction at address 2 in the twentieth diagram. As shown in the figure, the narrow neural processing unit 0-A adds the value 217A of the accumulator 202A to the rotated narrow data text 1811A received by the multiplexer 208B output 209B of the narrow neural processing unit 511-B (ie, The product of the narrow data text 1022 received by the data random access memory 122 and the narrow text 0 of the column 2 of the weight random access unit 124; the narrow neural processing unit 0-B adds the value 217B of the accumulator 202B by the narrow nerve The multiplexer 208A of the processing unit 0-A outputs the rotated narrow data text 1811B received by the 209A (that is, the narrow data text 1023 received by the data random access memory 122) and the weight random access unit 124. Column 2 is the product of narrow text 1; and so on, until the narrow neural processing unit 511-B adds the accumulator 202B value 217B to the rotation received by the multiplexer 208A output 209A of the narrow neural processing unit 511-A. The product of the narrow data text 1811B (i.e., the narrow data text 1021 received by the data random access memory 122) and the narrow text 1023 of the weight random access unit 124. As shown in Figure 21, this operation will follow in the next 1021. The time-frequency period continues until the time-frequency period 1024 described below.

In the time-frequency period 1024, each of the 1024 narrow neural processing units performs the 1023th iteration of the multiply-accumulate rotation instruction at address 2 in the twentieth diagram. As shown in the figure, the narrow neural processing unit 0-A adds the value 217A of the accumulator 202A to the rotated narrow data text 1811A received by the multiplexer 208B output 209B of the narrow neural processing unit 511-B (ie, The product of the narrow data text 1) received by the data random access memory 122 and the narrow text 0 of the column 1023 of the weight random access unit 124; the narrow neural processing unit 0-B adds the value 217B of the accumulator 202B to the narrow nerve The multiplexer 208A of the processing unit 0-A outputs the rotated narrow data text 1811B received by the 209A (that is, the narrow data character 2 received by the data random access memory 122) and the weight random access unit 124. Column 1023 is the product of narrow text 1; and so on, until narrow neuroprocessing unit 511-B adds accumulator 202B value 217B to the rotation received by multiplexer 208A output 209A of narrow neural processing unit 511-A. The product of the narrow data text 1811B (that is, the narrow data text 0 received by the data random access memory 122) and the narrow text 1023 of the column 1023 of the weight random access unit 124.

At time-frequency period 1025, the start function unit 212A/212B of each of the 1024 narrow neural processing units performs the start function instruction at address 3 in the twentieth diagram. Finally, in a time-frequency period 1026, each of the 1024 narrow neural processing units writes its narrow result 133A/133B back to the corresponding narrow text in column 16 of the data random access memory 122 to perform the second The write start function unit instruction at address 4 in the ten figure. That is, the narrow result of the neuroprocessing unit 0-A 133A The narrow text 0 of the data random access memory 122 will be written, the narrow result 133B of the neural processing unit 0-B will be written to the narrow text 1 of the data random access memory 122, and so on, until the neural processing unit The narrow result 133B of 511-B is written to the narrow text 1023 of the data random access memory 122. The twenty-second figure shows the aforementioned operation corresponding to the twenty-first figure in a block diagram.

The twenty-second diagram shows a block diagram of the neural network unit 121 of the first diagram, the neural network unit 121 having the neural processing unit 126 as shown in Fig. 18 to execute the program of the twentieth diagram. The neural network unit 121 includes 512 neural processing units 126, namely 1024 narrow neural processing units, data random access memory 122, and weight random access memory 124. The data random access memory 122 receives its bits. Address input 123, weighted random access memory 124 receives its address input 125. Although not shown in the figure, in the time-frequency period 0, the 1024 narrow neural processing units will execute the initialization instructions of the twentieth map. As shown in the figure, at time-frequency period 1, 1024 octet data words of column 17 are read from data random access memory 122 and provided to the 1024 narrow neural processing units. In the time-frequency period 1 to 1024, 1024 8-bit weight texts of columns 0 to 1023 are read out from the weight random access memory 124 and supplied to the 1024 narrow neural processing units, respectively. Although not shown in the figure, in the time-frequency period 1, the 1024 narrow neural processing units perform the corresponding multiply-accumulate operation on the loaded data text and the weight text. In the time-frequency period of 2 to 1024, the 1024 narrow neural processing unit multiplexer 208A/208B operates as a 1024 8-bit text rotator that will load the previously accessed data random access memory. 122 The data text of 17 is rotated to the adjacent narrow nerve processing unit, and the narrow neural processing unit performs a multiplication and accumulation operation on the corresponding rotated data text and the corresponding narrow weight text loaded by the weight random access memory 124. Although not shown in the figure, at time-frequency period 1025, the 1024 narrow start function units 212A/212B execute a start command. At time-frequency period 1026, the 1024 narrow neural processing units write their corresponding 1024 octet results 133A/133B back to column 16 of data random access memory 122.

It can be seen that, compared to the embodiment of the second figure, the embodiment of the eighteenth embodiment gives the programmer the flexibility to choose to use wide data and weight text (such as 16 bits) and narrow data and weight text (such as 8-bit) Perform calculations to meet the accuracy requirements for a particular application. From a single perspective, the embodiment of the eighteenth embodiment provides twice the performance compared to the embodiment of the second figure for narrow data applications, but additional narrow components must be added (eg, multiplexed temporary storage) At the expense of 208B, register 205B, narrow arithmetic logic unit 204B, narrow accumulator 202B, narrow start function unit 212B), these additional narrow elements will increase the neural processing unit 126 by about 50%.

Three-mode neural processing unit

A twenty-third figure is a block diagram showing another embodiment of a dynamically configurable neural processing unit 126 of the first figure. The neural processing unit 126 of the twenty-third diagram can be used not only for wide and narrow configurations, but also for a third configuration, referred to herein as a "funnel" configuration. The neural processing unit 126 of the twenty-third figure is similar to the eighteenth The nerve processing unit 126. However, the wide adder 244A in Fig. 18 is replaced by a three-input wide adder 2344A in the neural processing unit 126 of the twenty-third figure, and the three-input wide adder 2344A receives a third addend 2399. It is an extended version of the output of the narrow multiplexer 1896B. The program executed by the neural network unit having the neural processing unit of the twenty-third figure is similar to the program of the twenty-first figure. However, the initialization neural processing unit instructions located at address 0 will initialize these neural processing units 126 into a funnel configuration rather than a narrow configuration. In addition, the multiply-accumulate rotation instruction at address 2 has a count value of 511 instead of 1023.

When in the funnel configuration, the operation of the neural processing unit 126 is similar to being in a narrow configuration. When performing the multiply accumulate instruction of address 1 in the twentieth diagram, the neural processing unit 126 receives two narrow data words 207A/207B and Two narrow weight texts 206A/206B; wide multiplier 242A multiplies data text 209A by weight text 203A to produce product 246A selected by wide multiplexer 1896A; narrow multiplier 242B compares data text 209B with weight text 203B Multiply by the product 246B that produces the narrow multiplexer 1896B selection. However, the wide adder 2344A adds the product 246A (selected by the wide multiplexer 1896A) and the product 246B/2399 (selected by the wide multiplexer 1896B) to the wide accumulator 202A output 217A, and the narrow adder 244B The narrow accumulator 202B is not activated. In addition, when in the funnel configuration to perform the multiply-accumulate rotation instruction of address 2 as in the twentieth diagram, the control input 213 causes the multiplex register 208A/208B to rotate two narrow words (eg, 16 bits), that is, It is said that the multiplex register 208A/208B will select its corresponding input 211A/211B as if it were in a wide configuration. However, the wide multiplier 242A will present the text 209A with the weight text. 203A is multiplied to produce a product 246A selected by the wide multiplexer 1896A; the narrow multiplier 242B multiplies the data word 209B by the weight text 203B to produce a product 246B selected by the narrow multiplexer 1896B; and the wide adder 2344A will Product 246A (selected by wide multiplexer 1896A) and product 246B/2399 (selected by wide multiplexer 1896B) are all added to wide accumulator 202A output 217A, while narrow adder 244B and narrow accumulator 202B are as described above. Do not start. Finally, when in the funnel configuration to execute the start function instruction of address 3 as in the twentieth diagram, the wide start function unit 212A performs a start function on the total number of results 215A to produce a narrow result 133A, while the narrow start function unit 212B is Do not start. Thus, only the narrow nerve processing unit labeled A will produce a narrow result 133A, and the narrow result 133B produced by the narrow nerve processing unit labeled B is ineffective. Therefore, writing back the result column (as indicated by the instruction of address 4 in Figure 20) will contain holes because only the narrow result 133A is valid and the narrow result 133B is invalid. Therefore, conceptually, in each time-frequency period, each neuron (the neural processing unit of the twenty-third figure) performs two linked data inputs, ie multiplies two narrow data words by their corresponding weights and The two products are added. In contrast, the second and eighteenth embodiments perform only one link data input per time-frequency period.

In the embodiment of the twenty-third figure, it can be found that the number of result words (neuron output) generated and written back to the data random access memory 122 or the weight random access memory 124 is the received data input (link). Half of the square root of the number, and the result of the write back column has a hole, that is, every other narrow text result is invalid, more precisely, the result of the narrow neural processing unit labeled B is meaningless. Therefore, the twentieth The three-figure embodiment is particularly efficient for a neural network with two consecutive layers. For example, the first layer has twice as many neurons as the second layer (eg, the first layer has 1024 neurons fully connected to 512 neurons in the second layer). In addition, other execution units 122 (eg, media units, such as x86 advanced vector expansion units) may perform a packing operation on a scatter result column (ie, having holes) to make it compact (ie, not void), if necessary. . Subsequently, when the neural processing unit 121 performs other calculations associated with the data random access memory 122 and/or the other columns of the weight random access memory 124, the processed data column can be used for calculation.

Hybrid neural network unit operation: convolution and common source computing

An advantage of the neural network unit 121 described in the embodiments of the present invention is that the neural network unit 121 can simultaneously operate in a manner similar to a coprocessor executing its own internal program and executing in a processing unit similar to a processor. The published architecture directive (or the microinstruction translated by the architectural directive). The architectural instructions are contained within an architectural program executed by a processor having a neural network unit 121. As such, the neural network unit 121 can operate in a mixed manner while maintaining the high utilization of the neural processing unit 121. For example, the twenty-fourth to twenty-sixth diagrams show the operation of the neural network unit 121 to perform a convolution operation in which the neural network unit is fully utilized, and the twenty-seventh to twenty-eighth diagrams show the nerve The network unit 121 performs the operation of the common source operation. Applications such as convolutional layers, common source layers, and other digital data calculations, such as image processing (such as edge detection, sharpening, blurring, recognition/classification), require these operations. Do not The hybrid operation of the neural processing unit 121 is not limited to performing convolution or common source operations, and the hybrid feature can also be used to perform other operations, such as the conventional neural network multiply accumulate operation and the start function described in the fourth to thirteenth figures. Operation. That is, the processor 100 (more precisely, the reservation station 108) will issue the MTNN command 1400 and the MFNN command 1500 to the neural network unit 121, and in response to this issued command, the neural network unit 121 will write the data to the memory. The body 122/124/129 reads the result from the memory 122/124 written by the neural network unit 121, and at the same time, executes the program of the program memory 129 for executing the processor 100 (via the MTNN1400 instruction). The neural network unit 121 reads and writes to the memory 122/124/129.

The twenty-fourth diagram is a block diagram showing an example of a data structure used by the neural network unit 121 of the first figure to perform a convolution operation. The block diagram includes a convolution kernel 2402, a data array 2404, and the data random access memory 122 and the weight random access memory 124 of the first figure. In a preferred embodiment, data array 2404 (e.g., corresponding to image pixels) is loaded into system memory (not shown) coupled to processor 100 and loaded by processor 100 by executing MTNN instructions 1400. The network unit 121 weights the random access memory 124. The convolution operation convolves a first array with a second array, which is the convolution kernel described herein. As described herein, a convolution kernel is a matrix of coefficients, which may also be referred to as weights, parameters, elements, or values. For a preferred embodiment, the convolution kernel 2042 is static data of the architecture program executed by the processor 100.

This data array 2404 is a two-dimensional array of data values, and the size of each data value (such as image pixel value) is The machine accesses the size of the text of the memory 122 or the weight random access memory 124 (for example, 16 bits or 8 bits). In this example, the data value is 16-bit characters, and the neural network unit 121 is configured with 512 wide-configured neural processing units 126. Moreover, in this embodiment, the neural processing unit 126 includes a multiplex register to receive the weight text 206 from the weighted random access memory 124, such as the multiplex register 705 of the seventh figure, whereby the weights are random. The access memory 124 receives a list of data values to perform a collective rotator operation, which is described in more detail in subsequent sections. In this example, data array 2404 is a 2560 row x 1600 column pixel array. As shown in the figure, when the architecture program convolves the data array 2404 with the convolution kernel 2402, the data array 2402 is divided into 20 data blocks, and each data block is a 512x400 data array 2406.

In this example, convolution kernel 2402 is a 3x3 array of coefficients, weights, parameters, or elements. The first column of these coefficients is labeled C0,0; C0,1; and C0,2; the second column of these coefficients is labeled C1,0; C1,1; and C1,2; the third column of these coefficients is labeled C2,0; C2,1; and C2,2. For example, a convolution kernel with the following coefficients can be used to perform edge detection: 0, 1, 0, 1, 4, 1, 0, 1, 0. In another embodiment, a convolution kernel having the following coefficients can be used to perform Gaussian blur operations: 1, 2, 1, 2, 4, 2, 1, 2, 1. In this example, a division is typically performed on the final accumulated value, where the sum of the absolute values of the elements of the divisor-convolution kernel 2042 is 16, in this example. In another example, the divisor may be the number of elements of the convolution kernel 2042. In yet another example, the divisor may be a value used to compress the convolution operation to a target value range, the divisor being the element value of the convolution kernel 2042, The range of the target and the range of input value arrays that perform the convolution operation are determined.

Referring to the twenty-fourth diagram and the twenty-fifth diagram detailing the details therein, the architecture program writes the coefficients of the convolution kernel 2042 into the data random access memory 122. In a preferred embodiment, all of the text on each of the nine consecutive columns of data random access memory 122 (the number of elements in convolution kernel 2402) will utilize different elements of convolution kernel 2402. Columns are written in their primary order. That is to say, as shown in the figure, each character in the same column is written with the first coefficient C0,0; the next column is written with the second coefficient C0,1; the next column is the third coefficient. C0,2 is written; the next column is written with the fourth coefficient C1,0; and so on, until each character of the ninth column is written with the ninth coefficient C2,2. In order to perform a convolution operation on the data matrix 2406 of the data block segmented by the data array 2404, the neural processing unit 126 repeatedly reads the nine columns of the coefficients of the convolution kernel 2042 loaded in the data random access memory 122 in this order. In the subsequent sections, particularly the sections corresponding to the twenty-sixth A diagram, there will be a more detailed description.

Referring to the twenty-fourth diagram and the twenty-fifth diagram detailing the details therein, the architecture program writes the value of the data matrix 2406 into the weight random access memory 124. When the neural network unit program performs the convolution operation, the result array is written back to the weighted random access memory 124. In a preferred embodiment, the architecture program writes a first data matrix 2406 to the weighted random access memory 124 and causes the neural network unit 121 to operate, when the neural network unit 121 is in the first data matrix. When the convolution operation is performed by the convolution kernel 2402, the architecture program writes a second data matrix 2406 into the weight random access memory 124. Thus, the neural network unit After the convolution operation of the first data matrix 2406 is completed, the convolution operation of the second data matrix 2406 can be started, which is described in more detail later in the corresponding figure. In this manner, the architecture program will travel to and from the two regions of the weighted random access memory 124 to ensure that the neural network unit 121 is fully utilized. Therefore, the example of the twenty-fourth figure shows a first data matrix 2406A and a second data matrix 2406B, and the first data matrix 2406A corresponds to the first data block occupying the columns 0 to 399 of the weight random access memory 124. And the second data matrix 2406B corresponds to the second data block occupying the columns 500 to 899 in the weight random access memory 124. In addition, as shown in the figure, the neural network unit 121 writes the result of the convolution operation back to the columns 900-1299 of the weighted random access memory 124 and the columns 1300-1699, and then the architecture program will use the weight random access memory. Body 124 reads these results. The data value of the data matrix 2406 loaded in the weighted random access memory 124 is labeled as "Dx, y", where "x" is the number of columns of the weighted random access memory 124, and "y" is the weighted random access memory. The text, or the number of lines. For example, the data text 511 located in column 399 is labeled D399, 511 in the twenty-fourth figure, and is received by the multiplex register 705 of the neural processing unit 511.

The twenty-fifth diagram is a flowchart showing that the processor 100 of the first figure executes an architectural program to perform a convolution operation on the convolution kernel 2042 on the data array 2404 of the twenty-fourth map using the neural network unit 121. This process begins in step 2502.

In step 2502, the processor 100, that is, the processor 100 executing the architecture program, writes the convolution kernel 2402 of the twenty-fourth graph into the data random access memory in the manner described in the twenty-fourth diagram. 122. In addition, the architecture program initializes a variable N to a value of 1. The variable N is a data block that the neural network unit 121 is processing in the data array 2404. In addition, the architecture program initializes a variable NUM_CHUNKS to a value of 20. The flow then proceeds to step 2504.

In step 2504, as shown in FIG. 24, the processor 100 writes the data matrix 2406 of the data block 1 into the weight random access memory 124 (eg, the data matrix 2406A of the data block 1). The flow then proceeds to step 2506.

In step 2506, the processor 100 writes a convolution program to the neural network unit 121 program memory 129 using a MTNN instruction 1400 that specifies a function 1432 to write to the program memory 129. The processor 100 then uses a MTNN instruction 1400 that specifies a function 1432 to begin execution of the program to initiate the neural network unit convolution program. An example of a neural network unit convolution program is described in more detail at correspondence to Figure 26A. The flow then proceeds to step 2508.

At decision step 2508, the architecture program determines if the value of the variable N is less than NUM_CHUNKS. If so, the flow will proceed to step 2512; otherwise, proceed to step 2514.

At step 2512, as shown in FIG. 24, processor 100 writes data matrix 2406 of data block N+1 into weighted random access memory 124 (e.g., data matrix 2406B of data block 2). Therefore, when the neural network unit 121 is performing a convolution operation on the current data block, the architecture program can write the data matrix 2406 of the next data block into the weight random access memory 124, thus completing the current data block. Convolution operation After writing to the weighted random access memory 124, the neural network unit 121 can immediately begin performing a convolution operation on the next data block.

At step 2514, the processor 100 confirms whether the neural network unit program being executed (for data block 1 but from step 2506, and for block 2-20, execution from step 2518) has completed execution. In a preferred embodiment, processor 100 reads neural network unit 121 status register 127 by executing an MFNN instruction 1500 to confirm whether execution has been completed. In another embodiment, the neural network unit 121 generates an interrupt indicating that the convolution procedure has been completed. The flow then proceeds to decision step 2516.

In decision step 2516, the architecture program determines if the value of the variable N is less than NUM_CHUNKS. If so, the flow proceeds to step 2518; otherwise, the process proceeds to step 2522.

In step 2518, processor 100 updates the convolution program to execute on data block N+1. More precisely, the processor 100 updates the column value of the initialized neural processing unit instruction corresponding to address 0 in the weighted random access memory 124 to the first column of the data matrix 2406 (eg, updated to the data matrix 2406A). Column 0 is either column 500 of the material matrix 2406B and the output column is updated (eg, updated to column 900 or 1300). The processor 100 then begins executing the updated neural network unit convolution program. The flow then proceeds to step 2522.

In step 2522, the processor 100 reads the execution result of the neural network unit convolution program of the data block N from the weight random access memory 124. The flow then proceeds to decision step 2524.

In decision step 2524, the architecture program confirms the variable N Is the value less than NUM_CHUNKS? If so, the flow proceeds to step 2526; otherwise, it terminates.

In step 2526, the architecture program increments the value of N by one. The flow then returns to decision step 2508.

The twenty-sixth A diagram is a list of programs of a neural network unit program that performs a convolution operation on a data matrix 2406 and writes it back using the convolution kernel 2402 of the twenty-fourth graph. Weight random access memory 124. This program loops the instruction loop formed by the instructions of addresses 1 through 9 a certain number of times. The initialization neural processing unit instruction at address 0 specifies the number of times each neural processing unit 126 performs this instruction loop. The example of the twenty-sixth A diagram has a loop count value of 400, corresponding to the twenty-fourth. The number of columns in the data matrix 2406 of the graph, and the loop command at the loop terminal (at address 10) decrements the current loop count value, and if the result is non-zero, it returns to the command loop. The top (ie, the instruction to return to address 1). Initializing the neural processing unit instructions also clears the accumulator 202 to zero. In a preferred embodiment, the loop command at address 10 also clears accumulator 202 to zero. Alternatively, the multiply accumulate instruction at address 1 as described above can also clear accumulator 202 to zero.

For each execution of the in-program instruction loop, the 512 neural processing units 126 simultaneously perform a convolution operation of 512 3x3 convolution kernels and 512 corresponding 3x3 sub-matrices of the data matrix 2406. The convolution operation is the sum of the nine products calculated from the elements of the convolution kernel 2042 and the corresponding elements in the corresponding sub-matrix. In the embodiment of Figure 26A, the origin of each of the 512 corresponding 3x3 sub-matrices (Central element) is the data character Dx+1, y+1 in the twenty-fourth figure, where y (row number) is the number of the neural processing unit 126, and x (column number) is the current weight random access memory 124 The column number read by the multiply-accumulate instruction of address 1 in the program of Figure 26 (this column number will also be initialized by the initialization neural processing unit instruction of address 0, and will also be in the execution position. The multiplication of addresses 3 and 5 is incremented by the instruction and is also updated by the decrement instruction at address 9. Thus, in each cycle of the program, the 512 neural processing units 126 compute 512 convolution operations and write the results of the 512 convolution operations back to the instruction sequence of the weighted random access memory 124. Edge handling is omitted herein to simplify the description, but it should be noted that utilizing the collective rotation features of these neural processing units 126 results in a data matrix 2406 (for the image processor, the data matrix of the image) Two rows of multi-line data are wrapped from the vertical edge of one side to the other (eg, from the left edge to the right edge, and vice versa). Now explain the instruction loop.

Address 1 is a multiply-accumulate instruction that specifies column 0 of data random access memory 122 and implicitly utilizes the current weight of random access memory 124, which column is preferably loaded in sequencer 128 (and It is initialized to zero by the instruction at address 0 to perform the operation of the first instruction loop transfer). That is to say, the instruction at address 1 causes each neural processing unit 126 to read its corresponding text from the column 0 of the data random memory 122, and reads the corresponding data from the current weight random access memory 124 column. Text, and perform a multiply accumulate operation on the two words. Thus, for example, the neural processing unit 5 multiplies C0,0 by Dx,5 (where "x" is the current weight random access memory 124 column), the result is added to the accumulator 202 value 217, and the total is written back to the accumulator 202.

Address 2 is a multiply-accumulate instruction that specifies that the column of data random access memory 122 is incremented (i.e., increased to one) and then read from the incremented address of data random access memory 122. This instruction will also specify that the value in the multiplex register 705 of each neural processing unit 126 is rotated to the adjacent neural processing unit 126, in this example from the weighted random access memory in response to the instruction of address 1. The body 124 reads the data matrix 2406 values. In the embodiment of the twenty-fourth to twenty-sixth embodiments, the neural processing unit 126 is configured to rotate the value of the multiplex register 705 to the left, that is, from the neural processing unit J to the neural processing unit J- 1, instead of rotating from the neural processing unit J to the neural processing unit J+1 as in the aforementioned third, seventh and nineteenth views. It should be noted that in the embodiment in which the neural processing unit 126 rotates to the right, the architecture program writes the convolution kernel 2042 coefficient values into the data random access memory 122 in different orders (eg, rotating around its center row) to achieve The purpose of similar convolution results. In addition, the architecture program can perform additional convolution kernel preprocessing (such as transposition) when needed. In addition, the instruction specifies a count value of 2. Thus, the instruction at address 2 causes each neural processing unit 126 to read its corresponding text from column 1 of data random access memory 122, and receive the rotated text into multiplex register 705, and The text performs a multiply accumulate operation. Since the count value is 2, this command also causes each neural processing unit 126 to repeat the foregoing operations. That is, the sequencer 128 increments the data random access memory 122 column address 123 (i.e., increases to 2), and each neural processing unit 126 fetches from the data random access memory 122. 2 Reading its corresponding text and receiving the rotated text to the multiplex register 705, and performing a multiply-accumulate operation on the two characters. Thus, for example, assuming that the current weight random access memory 124 is listed as 27, after executing the instruction of address 2, the neural processing unit 5 will multiply the product of C0,1 and D27,6 with C0,2 and D27. The multiplication of 7 is added to its accumulator 202. Thus, after the instruction of address 1 and address 2 is completed, the product of C0, 0 and D27, 5, the product of C0, 1 and D27, 6 and C0, 2 and D27, 7 are accumulated to accumulator 202, and added. All other accumulated values from the previously passed instruction loop.

The operations performed by the instructions of addresses 3 and 4 are similar to the instructions of addresses 1 and 2, using the weighted random access memory 124 column to increment the performance of the indicator, these instructions will be below the weight random access memory 124 The operations are performed and these instructions operate on the next three columns of data random access memory 122, columns 3 through 5. That is to say, taking the neural processing unit 5 as an example, after completing the instructions of the address 1 to 4, the product of C0, 0 and D27, 5, the product of C0, 1 and D27, 6, C0, 2 and D27, 7 The product of product, C1,0 and D28,5, the product of C1,1 and D28,6, and the product of C1,2 and D28,7 are accumulated to accumulator 202, adding all other orders from the previously passed instruction loop. Accumulated value.

The operations performed by the instructions of addresses 5 and 6 are similar to the instructions of addresses 3 and 4, which will be a column below the weighted random access memory 124 and the next three columns of the data random access memory 122, That is, columns 6 to 8 perform calculations. That is to say, taking the neural processing unit 5 as an example, after completing the instructions of addresses 1 to 6, the product of C0, 0 and D27, 5, the product of C0, 1 and D27, 6, C0, 2 and D27, 7 Product, product of C1,0 and D28,5, product of C1,1 and D28,6, C1,2 and D28,7,C2,0 and D29,5 The product, the product of C2,1 and D29,6, and the product of C2,2 and D29,7 are accumulated to accumulator 202, adding all other accumulated values from the previously passed instruction loop. That is to say, after the instruction of address 1 to 6 is completed, it is assumed that the weight random access memory 124 is listed as 27 when the instruction loop starts, and the neuroprocessing unit 5 is taken as an example, and the convolution kernel 2042 is used for the following 3x3. Submatrix for convolution: D27,5 D27,6 D27,7 D28,5 D28,6 D28,7 D29,5 D29,6 D29,7 In general, after completing the instructions of addresses 1 to 6, this 512 Each of the neural processing units 126 has used the convolution kernel 2042 to convolute the following 3x3 submatrices: Dr, n Dr, n+1 Dr, n+2 Dr+1, n Dr+1, n+1 Dr+1 , n+2 Dr+2, n Dr+2, n+1 Dr+2, n+2, where r is the address of the address of the random access memory 124 when the instruction loop starts, and n is the neural processing The number of unit 126.

The instruction of address 7 transmits the accumulator 202 value 217 through the start function unit 121. This transfer function passes a text whose size (in bits) is equivalent to the text read by the data random access memory 122 and the weighted random access memory 124 (in this example, 16 bits). . In a preferred embodiment, the user can specify an output format, such as how many bits in the output bit are fractional bits, as will be explained in more detail in subsequent sections. In addition, this specification specifies a division start function instead of specifying a pass start letter. The division start function divides the accumulator 202 value 217 by a divisor, as described herein in relation to the twenty-ninth and thirty-first diagrams, for example, using the "divider" 3014/3016 of FIG. one of them. For example, in the case of a convolution kernel 2042 with coefficients, as described above with a Gaussian blur kernel with a factor of one-sixteenth, the instruction of address 7 specifies a division start function (eg, divided by 16), and Do not specify a transfer function. In addition, the architecture program may perform the division by 16 on the coefficients of the convolution kernel 2042 before writing the convolution kernel coefficients into the data random access memory 122, and adjust the binary decimal point of the convolution kernel 2042 value accordingly. The position, for example, uses the data binary point 2922 of the twenty-ninth figure as described below.

The instruction of address 8 writes the output of the start function unit 212 to the column specified in the weight random access memory 124 by the current value of the output column register. This current value is initialized by the instruction of address 0, and this value is incremented by the increment indicator within the instruction after each pass.

As described in the twenty-fourth to twenty-sixth diagrams having an example of a 3x3 convolution kernel 2402, the neural processing unit 126 reads the weighted random access memory 124 approximately every three time-frequency cycles to read the data matrix 2406. One column, and the convolution kernel result matrix is written to the weighted random access memory 124 approximately every twelve time-frequency cycles. Furthermore, it is assumed that in one embodiment, having one of the buffers 1704 of the seventeenth embodiment, the write and read buffers, while the neural processing unit 126 is reading and writing, the processor 100 can have random weights. The memory 124 is read and written, and the buffer 1704 performs a read and write operation on the weight random access memory every sixteen time-frequency cycles to read the data matrix and Write the convolution kernel result matrix. Thus, approximately half of the bandwidth of the weighted random access memory 124 is consumed by the convolution kernel operation performed by the neural network unit 121 in a mixed manner. This example includes a 3x3 convolution kernel 2042. However, the present invention is not limited thereto, and other sizes of convolution kernels, such as 2x2, 4x4, 5x5, 6x6, 7x7, 8x8, etc., may also be applicable to different neural networks. Unit program. In the case of larger convolution kernels, because of the rotated version of the multiply-accumulate instruction (such as the instructions of addresses 2, 4, and 6 in Figure 26A, larger convolution kernels will need to use these instructions) With a larger count value, the time ratio of the neural processing unit 126 reading the weight random access memory 124 is reduced, and therefore, the bandwidth usage ratio of the weight random access memory 124 is also lowered.

In addition, the architectural program can cause the neural network unit program to overwrite the columns that are no longer needed in the input data matrix 2406, rather than writing the convolution operation results back to different columns of the weighted random access memory 124 (eg, column 900). -1299 and 1300-1699). For example, in the case of a 3x3 convolution kernel, the architecture program can write the data matrix 2406 into columns 2-401 of the weighted random access memory 124 instead of writing columns 0-399, and the neural processing unit program The convolution operation result is written from the column 0 of the weight random access memory 124, and the number of columns is incremented every time the instruction loop is passed. In this way, the neural network unit program will only overwrite the columns that are no longer needed. For example, after the first pass through the instruction loop (or more precisely, after loading the instruction of address 1 it loads the column 0 of the weight random access memory 124), the data of column 0 can be overwritten. Write, however, the data in columns 1-3 needs to be left for the second pass through the operation of the instruction loop and cannot be overwritten; likewise, the second pass is passed After the loop is made, the data of column 1 can be overwritten. However, the data of column 2-4 needs to be left for the third pass through the operation of the instruction loop and cannot be overwritten; and so on. In this embodiment, the height of each data matrix 2406 (data block) can be increased (e.g., 800 columns) so that fewer data blocks can be used.

In addition, the architecture program can cause the neural network unit program to write the result of the convolution operation back to the data random access memory 122 column above the convolution kernel 2402 (eg, above column 8) instead of writing the convolution operation result back. The weight random access memory 124, when the neural network unit 121 writes the result, the architecture program can read the result from the data random access memory 122 (for example, using the data random access memory 122 in the twenty-sixth figure) Recently written to column 2606 address). This configuration is applicable to embodiments having a weighted random access memory 124 and a dual data random access memory.

According to the operation of the neural network unit 121 in the embodiment of the twenty-fourth to twenty-sixth embodiment, it can be found that each execution of the program of the twenty-sixth A diagram requires about 5000 time-frequency periods, and thus, the second The convolution operation of the entire 2560x1600 data array 2404 in Figure 14 requires approximately 100,000 time-frequency cycles, significantly less than the number of time-frequency cycles required to perform the same task in the conventional manner.

The twenty-sixth B diagram is a block diagram showing one embodiment of certain fields of the control register 127 of the neural network unit 121 of the first figure. The status register 127 includes a field 2602 indicating the address of the weighted random access memory 124 that was recently written by the neural processing unit 126; a field 2606 indicating the most recent of the data random access memory 122. The address written by the neural processing unit 126; a field 2604, An address of the column of the weighted random access memory 124 that was recently read by the neural processing unit 126 is indicated; and a field 2608 indicating the bit of the data random access memory 122 that was most recently read by the neural processing unit 126. site. In this way, the processing program executed by the processor 100 can confirm the processing progress of the neural network unit 121, and read and/or write data to the data random access memory 122 and/or the weight random access memory 124. Time to enter. Using this capability, plus the overwrite of the input data matrix as previously described (or writing the results to the data random access memory 122 as previously described), as described in the following example, the data array 2404 of the twenty-fourth graph It can be viewed as five 512x1600 data blocks instead of 20 512x400 data blocks. The processor 100 writes the first 512x1600 data block from the column 2 of the weight random access memory 124 and causes the neural network unit program to start (the program has a loop count of 1600 values, and the weights are randomized). The output column of the access memory 124 is initialized to 0). When the neural network unit 121 executes the neural network unit program, the processor 100 monitors the output position/address of the weight random access memory 124, thereby (1) (using the MFNN instruction 1500) reading the weight random access memory. Body 124 has a list of valid convolution operations written by neural network unit 121 (starting with column 0); and (2) overwrites the second 512x1600 data matrix 2406 (starting with column 2) The result of the effective convolution operation is read, so that when the neural network unit 121 completes the neural network unit program for the first 512x1600 data block, the processor 100 can immediately update the neural network unit program and start the neural network again if necessary. The way unit program is executed on the second 512x1600 data block. This program will repeat the execution of three remaining 512x1600 data blocks three times so that the neural network unit 121 can be charged. Use separately.

In an embodiment, the start function unit 212 has the ability to effectively perform an effective division operation on the accumulator 202 value 217, which in the subsequent sections corresponds in particular to the twenty-ninth, twenty-nine, and twenty-sixth and thirty-th. There will be more detailed instructions. For example, a start function neural network unit instruction that divides the value of accumulator 202 by a divide by 16 can be used for the Gaussian blur matrix described below.

The convolution kernel 2402 used in the example of the twenty-fourth figure is a small static convolution kernel applied to the entire data matrix 2404. However, the present invention is not limited thereto, and the convolution kernel may also be a large matrix. The particular weight corresponds to a different data value of the data array 2404, such as a convolution kernel commonly found in convolutional neural networks. When the neural network unit 121 is used in this manner, the architecture program interchanges the data matrix with the location of the convolution kernel, that is, the data matrix is placed in the data random access memory 122 and the convolution kernel is placed in the weight random. Within memory 124, the number of columns required to execute a neural network unit program is relatively small.

Figure 27 is a block diagram showing an example of a weighted random access memory 124 filled with input data in the first figure. The input data is subjected to a common source operation by the neural network unit 121 of the first figure ( Pooling operation). The common source operation is performed by one common source layer of the artificial neural network, and the input data matrix is reduced by taking the sub-region or sub-matrix of the input matrix and calculating the maximum or average value of the sub-matrix as a result matrix, ie, a common source matrix ( The size of an image or a convolved image. In the examples of the twenty-seventh and twenty-eighth, The common source operation calculates the maximum value of each submatrix. Common source operations are particularly useful for artificial neural networks that perform object classification or detection. In general, the common source operation can actually reduce the input matrix by the number of elements of the detected sub-matrix, and in particular, the direction of each dimension of the input matrix can be reduced by the number of elements in the corresponding dimension direction of the sub-matrix. In the example of the twenty-seventh figure, the input data is a 512 x 1600 matrix of wide text (e.g., 16 bits) stored in columns 0 through 1599 of the weighted random access memory 124. In the twenty-seventh figure, these characters are indicated by their row position. For example, the text in column 0 row 0 is marked as D0, 0; the text in column 0 row 1 is marked as D0, 1; The text of column 0 and line 2 is labeled as D0, 2; and so on, the text in column 0 line 511 is labeled D0, 511. Similarly, the text in column 1 row 0 is labeled D1, 0; the text in column 1 row 1 is labeled D1, 1; the text in column 1 row 2 is labeled D1, 2; and so on, in the column The text of line 1 511 is marked as D1, 511; and so on, the text in column 1599 line 0 is marked as D1599, 0; the text in column 1599 line 1 is marked as D1599, 1 is in column 1599 line 2 The text is labeled D1599, 2; and so on, the text in column 1599, line 511 is labeled D1599, 511.

The twenty-eighth figure is a list of programs of a neural network unit program that performs the common source operation of the input data matrix of the twenty-seventh figure and writes it back to the weight random access memory 124. . In the example of the twenty-eighth figure, the common source operation calculates the maximum of each 4x4 submatrix in the input data matrix. This program executes the instruction loop consisting of instructions 1 through 10 multiple times. The initialization neural processing unit instruction at address 0 specifies that each neural processing unit 126 executes the instruction loop. The number of times, in the example of the twenty-eighth figure, the loop count value is 400, and the loop command at the end of the loop (at address 11) decrements the current loop count value, and if the result is A non-zero value returns it to the top of the instruction loop (ie, the instruction back to address 1). The input data matrix in the weighted random access memory 124 is essentially treated by the neural network unit program as 400 mutually exclusive groups of four adjacent columns, namely columns 0-3, columns 4-7, columns. 8-11, and so on, until column 1596-1599. Each group consisting of four adjacent columns includes 128 4x4 sub-matrices, which are 4x4 sub-matrices formed by the elements at the intersection of four columns and four adjacent rows of the group. The adjacent row is line 0-3, line 4-7, line 8-11, and so on until line 508-511. Each of the 512 neural processing units 126, a fourth set of four neural processing units 126 (a total of 128) performs a common source operation on a corresponding 4x4 submatrix, while the other three neural processing units 126 is not used. More precisely, the neural processing units 0, 4, 8, and so on until the neural processing unit 508 performs a common source operation on the corresponding 4x4 submatrix, and the leftmost row number of the 4x4 submatrix corresponds to The neural processing unit number, and the lower column corresponds to the column value of the current weight random access memory 124, this value will be initialized to zero by the initialization instruction of address 0 and will increase by 4 after repeating each instruction loop. Some will be explained in more detail in subsequent chapters. The repetition of the 400 instruction loops corresponds to the number of 4x4 submatrix groups in the input data matrix of the twenty-seventh graph (ie, the input data matrix has 1600 columns divided by 4). Initializing the neural processing unit instructions also clears the accumulator 202 to zero. In a preferred embodiment, the loop instruction of address 11 also clears accumulator 202 to zero. In addition, address 1 The maxwacc instruction will specify that the accumulator 202 is cleared to zero.

Each time the instruction loop of the program is executed, the 128 used neural processing units 126 perform 128 common source operations simultaneously on 128 individual 4x4 sub-matrices in the current four-column group of the input data matrix. Further, this common source operation will identify the largest of the 16 elements of the 4x4 submatrix. In the embodiment of the twenty-eighth figure, for each of the 128 used neural processing units 126, the lower left element of the 4x4 submatrix is the element Dx in the twenty-seventh graph. , y, where x is the number of columns of the current weight random access memory 124 at the beginning of the instruction loop, and the column data is read by the maxwacc instruction of the address 1 in the program of the twenty-eighth figure (this column number is also It will be initialized by the initialization neural processing unit instruction of address 0 and incremented each time the maxwacc instruction of addresses 3, 5 and 7 is executed). Thus, for each loop of the program, the 128 used neural processing units 126 will write back the maximum elements of the corresponding 128 4x4 sub-matrices of the current column group back to the weighted random access memory 124. The specified column. The following is a description of this instruction loop.

The maxwacc instruction of address 1 implicitly uses the current weight random access memory 124 column, which is preferably loaded in sequencer 128 (and initialized to zero by the instruction at address 0 to execute the first time) Pass the operation of the instruction loop.) The instruction of address 1 causes each neural processing unit 126 to read its corresponding text from the current column of the weighted random access memory 124, compare this text with the value 217 of the accumulator 202, and compare the two values. The largest is stored in accumulator 202. Thus, for example, the neural processing unit 8 will confirm the accumulator 202 value 217 and the message text. The word Dx,8 (where "x" is the current weight random access memory 124 column) has the maximum value and is written back to the accumulator 202.

Address 2 is a maxwacc instruction that specifies that the value in the multiplex register 705 of each neural processing unit 126 is rotated adjacent to the neural processing unit 126, where the instruction for address 1 is just The weighted random access memory 124 reads one of the column input data array values. In the embodiment of the twenty-seventh to twenty-eighth embodiment, the neural processing unit 126 is configured to rotate the multiplexer 705 value to the left, that is, from the neural processing unit J to the neural processing unit J-1, as in the foregoing Corresponds to the chapters of the twenty-fourth to twenty-sixth figures. In addition, this instruction will specify a count value of 3. Thus, the instruction of address 2 causes each neural processing unit 126 to receive the rotated text into the multiplex register 705 and confirm the maximum value of the rotated text and the accumulator 202, and then repeat the operation twice. . That is, each neural processing unit 126 performs three operations of receiving the rotated text to the multiplex register 705 and confirming the maximum value of the rotated text and accumulator 202 values. Thus, for example, assuming that the instruction loop is started, the current weight random access memory 124 is listed as 36. Taking the neural processing unit 8 as an example, after executing the instructions of addresses 1 and 2, the neural processing unit 8 will The maximum value of the accumulator 202 and the four weight random access memory 124 characters D36, 8, D36, 9, D36, 10 and D36, 11 will be stored in its accumulator 202.

The operations performed by the maxwacc instructions of addresses 3 and 4 are similar to the instructions of address 1, using the weighted random access memory 124 column increment indicator has the effect, the address 3 and 4 instructions will be weighted random access memory A column below 124 is executed. In other words, assume that the instruction loop begins The current weight random access memory 124 column is 36. Taking the neural processing unit 8 as an example, after completing the instructions of addresses 1 to 4, the neural processing unit 8 will accumulate the storage loops in its accumulator 202 at the beginning of the loop. 202 and eight weight random access memories 124 have the maximum values of characters D36, 8, D36, 9, D36, 10, D36, 11, D37, 8, D37, 9, D37, 10 and D37, 11.

The operations performed by the maxwacc instructions of addresses 5 through 8 are similar to the instructions of addresses 1 through 4, and the instructions of addresses 5 through 8 are executed for the two columns below the weighted random access memory 124. That is to say, it is assumed that the current weight random access memory 124 column is 36 at the beginning of the instruction loop, and the neural processing unit 8 is taken as an example. After the instructions of addresses 1 to 8 are completed, the neural processing unit 8 will accumulate in it. The accumulator 202 and the sixteen weight random access memory 124 characters D36, 8, D36, 9, D36, 10, D36, 11, D37, 8, D37, 9, D37, 10 are stored in the buffer 202. The maximum of D37, 11, D38, 8, D38, 9, D38, 10, D38, 11, D39, 8, D39, 9, D39, 10 and D39, 11. That is to say, assuming that the current weight random access memory 124 column is 36 at the beginning of the instruction loop, the neural processing unit 8 is taken as an example. After completing the instructions of addresses 1 to 8, the neural processing unit 8 will complete the confirmation of the following. Maximum value of 4x4 submatrix: D36, 8 D36, 9 D36, 10 D36, 11 D37, 8 D37, 9 D37, 10 D37, 11 D38, 8 D38, 9 D38, 10 D38, 11 D39, 8 D39, 9 D39 10 D39, 11 Basically, after completing the instructions of addresses 1 through 8, each of the 128 used neural processing units 126 will complete the confirmation of the following maximum values of the 4x4 submatrix: Dr,n Dr,n+1 Dr,n+2 Dr,n+3 Dr+1,n Dr+1,n+1 Dr+1,n+2 Dr+1,n+3 Dr+2,n Dr +2,n+1 Dr+2,n+2 Dr+2,n+3 Dr+3,n Dr+3,n+1 Dr+3,n+2 Dr+3,n+3 where r is the command The column address value of the random access memory 124 is currently weighted at the beginning of the loop, and n is the number of the neural processing unit 126.

The instruction of address 9 transmits the accumulator 202 value 217 through the start function unit 212. This transfer function passes a text whose size (in bits) is equivalent to the text read by the weighted random access memory 124 (in this example, 16 bits). In a preferred embodiment, the user can specify an output format, such as how many bits in the output bit are fractional bits, as will be explained in more detail in subsequent sections.

The instruction of address 10 writes the accumulator 202 value 217 into the column of the weighted random access memory 124 specified by the current value of the output column register, which is initialized by the instruction of address 0, and Use the increment indicator in the instruction to increment this value after each pass through the loop. Further, the instruction of address 10 writes a wide text (e.g., 16 bits) of accumulator 202 into weighted random access memory 124. In a preferred embodiment, the instruction will write the 16 bits in accordance with the output binary point 2916, which will be in the following corresponding to the twenty-ninth and twenty-ninth B-pictures. More detailed instructions.

As described above, iterating once the instruction loop write weights random access memory 124 column will contain holes with invalid values. That is, the result 133 is wide text 1 to 3, 5 to 7, 9 to 11, and so on. Until the wide text 509 to 511 are invalid or unused. In one embodiment, the start function unit 212 includes a multiplexer that enables the result of merging the adjacent words into the column buffer, such as the column buffer 1104 of the eleventh figure, to write back the output weight random access memory. 124 columns. In a preferred embodiment, the start function instruction specifies the number of words in each hole, and the number of words in the hole controls the multiplexer result. In an embodiment, the number of holes may be specified as a value of 2 to 6 to combine the outputs of the 3x3, 4x4, 5x5, 6x6, or 7x7 sub-matrices of the common source. In addition, the architecture program executing on the processor 100 reads the sparse (ie, having holes) result columns generated from the weighted random access memory 124 and utilizes other execution units 112, such as media units that use the schema merge instructions, such as The x86 single instruction multiple data flow extension (SSE) instruction performs the merge function. In a manner similar to that described above and utilizing the hybrid nature of neural network unit 121, the architectural program executing on processor 100 can read state register 127 to monitor the most recently written column of weighted random access memory 124 ( For example, the field 2602 of FIG. 26B is used to read one of the generated sparse result columns, merge them and write them back to the same column of the weighted random access memory 124, so that preparation is completed and can be used as an input data. The matrix is provided for use under the neural network, such as a roll of layers or a traditional neural network layer (ie, a multiply-accumulate layer). In addition, the embodiments described herein perform a common source operation in a 4x4 submatrix, but the present invention is not limited thereto, and the neural network unit program of the twenty-eighth figure can be adjusted to be a sub-matrix of other sizes, such as 3x3. , 5x5, 6x6 or 7x7, perform a common source operation.

As can be seen from the foregoing, the number of result columns of the write weight random access memory 124 is four quarters of the number of columns of the input data matrix. One. Finally, data random access memory 122 is not used in this example. However, the data random access memory 122, rather than the weighted random access memory 124, can also be used to perform the common source operation.

In the twenty-seventh and twenty-eighth embodiments, the common source operation calculates the maximum value of the sub-region. However, the program of Figure 28 can be adjusted to calculate the average of the sub-regions by replacing the maxwacc instruction with the sumwacc command (adding the weight text to the accumulator 202 value 217) and starting the address 9. The function instruction is modified to divide the accumulated result by the number of elements in each sub-region (preferably by reciprocal multiplication as described below), which is sixteen in this example.

According to the operations of the neural network unit 121 according to the twenty-seventh and twenty-eighth diagrams, each time the program of the twenty-eighth figure is executed, it takes about 6000 time-frequency periods to be used for the twenty-seventh figure. The entire 512x1600 data matrix performs a common source operation, and the number of time-frequency cycles used in this operation is significantly less than the number of time-frequency cycles required to perform similar tasks in the conventional manner.

In addition, the architecture program can cause the neural network unit program to write the result of the common source operation back to the data random access memory 122 column instead of writing the result back to the weight random access memory 124, when the neural network unit 121 will result When the data random access memory 122 is written (for example, using the data of the twenty-sixth B-picture data random access memory 122 recently written to the address of the column 2606), the architecture program reads from the data random access memory 122. result. This configuration is applicable to embodiments having a weighted random access memory 124 and a dual data random access memory 122.

Fixed-point arithmetic operation with user-supplied decimal point, full-precision fixed-point accumulation, user-specified reciprocal value, random rounding of accumulator values, and selectable start/output functions

Generally, a hardware unit that performs an arithmetic operation in a digital computing device is an integer or a floating point number according to an object on which an arithmetic operation is performed, and is generally classified into an "integer" unit and a "floating point" unit. A floating point number has a magnitude (or mantissa) and an exponent, usually with a sign. An index is an indicator of the position of a radix point (usually a binary point) relative to a value. In contrast, an integer does not have an exponent, but only has a value, usually with a sign. Floating point units allow programmers to get the numbers they need to work from a very wide range of different values, while hardware is responsible for adjusting the index value of this number when needed, without the programmer having to deal with it. For example, suppose two floating point numbers 0.111 x 10 ^{29 are} multiplied by 0.81 x 10 ³¹ . (Although floating-point units typically work on 2-based floating-point numbers, decimal digits, or 10-based floating-point numbers, are used in this example.) Floating-point units are automatically responsible for multiplying the mantissa, exponential phase Add, then normalize the result to the value .8911 x 10 ⁵⁹ . In another example, assume that the same two floating point numbers are added. The floating point unit is automatically responsible for aligning the decimal point of the mantissa before the addition to produce a total number of .81111 x 10 ³¹ .

However, it is well known that such complicated operations result in an increase in the size of the floating point unit, an increase in energy consumption, an increase in the number of time-frequency cycles required per instruction, and/or an increase in cycle time. For this reason, many devices, such as embedded processors, microcontrollers, and relatively low cost and/or low power microprocessors, do not have floating point units. From the aforementioned examples It has been found that the complex structure of a floating-point unit includes logic that performs exponential calculations associated with floating-point addition and multiplication/division (ie, an adder that performs an addition/subtraction operation on an index of an operand to produce an index value of floating-point multiplication/division, a subtractor that subtracts the operand index to confirm the binary point of the floating point addition offset offset), including the offset of the binary point alignment of the mantissa in the floating point addition, including the pair of floating points The result is an offset that is standardized. In addition, the process usually needs to perform the logic of rounding the floating point result, perform conversion between integer format and floating point format, and between different floating point formats (such as amplification precision, double precision, single precision, half precision). Logic, leading zero and preamble detectors, and logic for handling special floating point numbers, such as anomalous, non-numeric, and infinite.

In addition, the verification of the correctness of the floating-point unit will greatly increase the complexity of the design due to the increased numerical space required to be verified, which will extend the product development cycle and time to market. In addition, as mentioned above, floating-point arithmetic operations need to store and use the mantissa field and the exponent field for each floating-point number used for calculation, respectively, to increase the required storage space and/or in a given storage space. Reduce accuracy when storing integers. Many of these shortcomings can be avoided by performing arithmetic operations on integer units.

Programmers usually need to write a program that processes decimals, which is a non-complete number. Such a program may need to be executed on a processor that does not have a floating point unit, or if the processor has a floating point unit, it is faster to execute an integer instruction by an integer unit of the processor. In order to take advantage of the performance of integer processors, programmers use well-known fixed-point calculations for fixed-point numbers. Operational calculations. Such programs will include instructions that execute on integer units to process integer or integer data. The software knows that the data is a decimal, and the software contains instructions to perform operations on the integer data to handle the fact that the data is actually a decimal, such as an alignment offset. Basically, the pointing software can manually perform functions that some or all of the floating point units can perform.

In this paper, a "fixed point" number (or value or operand or input or output) is a number whose stored bit is understood to contain a bit to represent a fractional part of the fixed point number, which is referred to herein. It is a "decimal bit". A fixed-point storage location is included in a memory or scratchpad, such as an 8-bit or 16-bit literal in a memory or scratchpad. In addition, the storage bits of the fixed-point number are all used to express a value, and in some cases, one of the bits will be used to express the symbol, but a fixed-bit storage element will be used to express the number. index. In addition, the number of decimal places or the number of decimal places of the fixed point number is specified in a storage space different from the fixed point storage bit, and the number of decimal places or the number of the decimal places is indicated in a shared or common manner. The decimal point position is shared with a set of fixed-point numbers containing the fixed-point number, such as input operands, accumulated values, or a set of output results of the processing unit array.

In the embodiment described herein, the arithmetic logic unit is an integer unit, however, the start function unit is either a floating point arithmetic hardware assist or acceleration. This can make the arithmetic logic unit portion smaller and faster to facilitate the use of more arithmetic logic units on a given wafer space. This also means that more neurons can be placed on the unit wafer space, which is particularly advantageous for neural network units.

In addition, exponential storage bits are required compared to each floating point number. The fixed point number in the embodiment described herein expresses the number of storage bits belonging to the decimal place in all the digital sets by one index, however, This indicator is located in a single, shared storage space and broadly identifies all numbers of the entire collection, such as the input set of a series of operations, the set of accumulated numbers of a series of operations, the set of outputs, and the number of decimal places. In a preferred embodiment, a user of the neural network unit can specify the number of decimal storage bits for the set of numbers. Therefore, it can be understood that although in many cases (such as general mathematics), the term "integer" refers to a signed complete number, that is, a number that does not have a fractional part, but in the context of this article, " The term "integer" can mean a number with a fractional part. In addition, in the context of this article, the term "integer" is used to distinguish from floating-point numbers. For floating-point numbers, some of the bits in their respective storage spaces are used to express the exponent of floating-point numbers. Similarly, integer arithmetic operations, such as integer multiplication or addition or comparison operations performed by integer units, assume that there are no indices in the operands, therefore, integer components of integer units, such as integer multipliers, integer adders, integer comparators, There is no need to include logic to process the index. For example, there is no need to move the mantissa to align the binary point for addition or comparison operations, and there is no need to add the indices for multiplication.

In addition, the embodiments described herein include a large hardware integer accumulator to accumulate a large series of integer operations (eg, 1000 multiply-accumulate operations) without loss of precision. This prevents the neural network unit from processing the floating point number while maintaining the full precision of the accumulated number without saturating or generating an inaccurate junction due to the overflow. fruit. Once the series of integer operations adds a result to the full-precision accumulator, the fixed-point hardware assist performs the necessary scaling and saturation operations to take advantage of the user-specified cumulative value of the fractional number of bits and the output value required. The number of decimal places, this full precision accumulated value is converted into an output value, which will be explained in more detail in subsequent chapters.

When it is desired to compress the accumulated value from full precision form for use in one of the input functions of the start function or for transfer, in a preferred embodiment, the start function unit can selectively perform a random round operation on the accumulated value. This section will be explained in more detail in subsequent chapters. Finally, depending on the different needs of a given layer of one of the neural networks, the neural processing unit can selectively accept indications to use different start functions and/or output many different forms of accumulated values.

The twenty-ninth A diagram is a block diagram showing an embodiment of the control register 127 of the first figure. The control register 127 can include a plurality of control registers 127. As shown in the figure, the control register 127 includes the following fields: configuration 2902, signed data 2912, signed weight 2914, data binary decimal point 2922, weight binary decimal point 2924, arithmetic logic unit function 2926, Rounding control 2932, start function 2934, reciprocal 2942, offset 2944, output random access memory 2952, output binary point 2954, and output command 2956. Controlling the register 127 value can utilize the MTNN instruction 1400 and the NNU program instructions, such as the start command, to perform the write operation.

The configuration 2902 value specifies that the neural network unit 121 is of a narrow configuration, a wide configuration, or a funnel configuration, as previously described. Configuration 2902 is also set by data random access memory 122 and weight random access memory The size of the input text received by the body 124. In a narrow configuration and funnel configuration, the size of the input text is narrow (for example, 8-bit or 9-bit), but in a wide configuration, the size of the input text is wide (for example, 12-bit or 16-bit) ). In addition, the configuration 2902 also sets the size of the output result 133 that is the same size as the input text.

When the signed data value 2912 is true, it means that the data characters received by the data random access memory 122 are signed values, and if they are false, the data characters are unsigned values. When the signed weight value 2914 is true, it means that the weighted text received by the weight random access memory 122 is a signed value, and if it is false, it means that the weighted characters are unsigned values.

The data binary decimal point value of 2922 indicates the binary decimal point position of the data text received by the data random access memory 122. In a preferred embodiment, for the position of the binary point, the data binary point 2922 value represents the number of bit positions calculated from the right side of the binary point. In other words, the data binary decimal point 2922 indicates the number of decimal places in the least significant bit of the data text, that is, the number of bits located to the right of the decimal point. Similarly, the weighted binary point 2924 value represents the binary point position of the weighted text received by the weighted random access memory 124. In a preferred embodiment, when the arithmetic logic unit function 2926 is a multiply and accumulate or output accumulation, the neural processing unit 126 determines the number of bits to the right of the binary digit of the value of the accumulator 202 as the data two. The sum of the carry decimal point 2922 and the weight binary decimal point 2924. So, for example, if the data binary decimal point 2922 has a value of 5 and the weight binary decimal point The value of 2924 is 3, and the value in accumulator 202 will have 8 bits to the right of the decimal point. When the arithmetic logic unit function 2926 is a total/maximum accumulator and data/weight text or a transfer data/weight text, the neural processing unit 126 will load the number of bits to the right of the decimal point of the value of the accumulator 202. It is determined as the data/weight binary decimal point 2922/2924. In another embodiment, a single accumulator binary decimal point 2923 is specified without specifying an individual data binary decimal point 2922 and a weight binary decimal point 2924. This section will be described in more detail later in the corresponding map of Figure 29.

Arithmetic logic unit function 2926 specifies a function that is executed by arithmetic logic unit 204 of neural processing unit 126. As previously described, the arithmetic logic unit function 2926 can include the following operations, but is not limited to: multiplying the data text 209 by the weight text 203 and adding the product to the accumulator 202; adding the accumulator 202 to the weight text 203; The loader 202 is added to the data text 209; the maximum value of the accumulator 202 and the data text 209; the maximum value of the accumulator 202 and the weight text 209; the output accumulator 202; the transfer data 209; the transfer weight text 209; value. In one embodiment, the arithmetic logic unit function 2926 is specified by a neural network unit initialization instruction and is used by the arithmetic logic unit 204 to respond to an execution instruction (not shown). In one embodiment, the arithmetic logic unit function 2926 is specified by individual neural network unit instructions, such as the aforementioned multiply accumulate and maxwacc instructions.

The rounding control 2932 specifies (in the thirty-fifth figure) the form of the rounding operation used by the rounder 3004. In an embodiment, the rounding modes that can be specified include, but are not limited to, not rounding, rounding to the nearest value, And random rounding. For a preferred embodiment, processor 100 includes a random bit source 3003 (see FIG. 30) to generate random bits 3005 that are sampled to perform random rounding to reduce The possibility of rounding deviations. In one embodiment, when the rounding bit 3005 is one and the sticky bit is zero, if the sampled random bit 3005 is true, the neural processing unit 126 rounds up if the random bit is sampled. If the 3005 is false, the neural processing unit 126 will not round up. In one embodiment, the random bit source 3003 is sampled based on the random electronic characteristics of the processor 100 to generate random bits 3005, such random semiconductor characteristics or thermal noise of the semiconductor diode or resistor, although the present invention Not limited to this.

The start function 2934 specifies a function for the accumulator 202 value 217 to generate the output 133 of the neural processing unit 126. As described herein, the start function 2934 includes, but is not limited to, a sigmoid function; a hyperbolic tangent function; a soft addition function; a correction function; divide by a specified power of two; multiply a user-specified reciprocal value to achieve, etc. Dividing the effect; passing the entire accumulator; and passing the accumulator in standard dimensions, as described in more detail in the following sections. In an embodiment, the startup function is specified by a neural network unit startup function instruction. In addition, the startup function may also be specified by the initialization instruction and used in response to an output instruction, such as the start function unit output instruction of address 4 in the fourth figure. In this embodiment, the address 3 is activated in the fourth figure. Function instructions are included in the output instructions.

The reciprocal 2942 value specifies a number multiplied by the accumulator 202 value 217 to achieve the division of the accumulator 202 value 217 value. That is, the reciprocal 2942 value specified by the user will be the reciprocal of the divisor that you actually want to execute. This facilitates collocation with convolution or common source operations as described herein. In a preferred embodiment, the user will specify the reciprocal 2942 value as two portions, which will be described in more detail later in the corresponding twenty-ninth C-picture. In one embodiment, the control register 127 includes a field (not shown) that allows the user to specify a division among a plurality of built-in divisor values that are equivalent to commonly used convolutions. The size of the core, such as 9, 25, 36 or 49. In this embodiment, the start function unit 212 stores the reciprocal of these built-in divisors for multiplying the accumulator 202 value 217.

Offset 2944 is the number of bits that specify one of the start function unit 212 shifters to shift the value of the accumulator 202 217 to the right to divide it by the power of two. This facilitates the operation of a convolution kernel with a power of two.

The output random access memory 2952 value specifies one of the data random access memory 122 and the weight random access memory 124 to receive the output result 133.

The output binary point 2954 value represents the position of the decimal point of the output result 133. For a preferred embodiment, for the position of the binary point of the output 133, the output binary point 2954 value represents the number of bit positions calculated from the right side. In other words, the output binary decimal point 2954 represents the number of decimal places in the least significant bit of the output result 133, that is, the number of bits located to the right of the binary decimal point. The start function unit 212 will be based on the value of the output binary point 2954 (in most cases, it will also be based on the data binary The operation of rounding, compression, saturation, and size conversion is performed on the decimal point 2922, the weighted binary decimal point 2924, the start function 2934, and/or the value of the configuration 2902.

Output command 2956 will output 133 from many control outputs. In an embodiment, the startup function unit 121 utilizes the concept of a standard size that is twice the width (in bits) specified by the configuration 2902. Thus, for example, if the configuration 2902 sets the size of the input text received by the data random access memory 122 and the weight random access memory 124 to be 8-bit, the standard size will be 16 bits; in another example If the configuration 2902 sets the size of the input text received by the data random access memory 122 and the weight random access memory 124 to be 16 bits, the standard size will be 32 bits. As described herein, the accumulator 202 is relatively large in size (for example, the narrow accumulator 202B is 28 bits and the wide accumulator 202A is 41 bits) to maintain intermediate calculations, such as 1024 and 512 nerves. Network unit multiply accumulates instructions with full precision. Thus, the accumulator 202 value 217 will be greater than (in terms of the bit) standard size, and for most of the value of the start function 2934 (in addition to passing the entire accumulator), the function unit 212 is activated (eg, the following corresponds to the thirty-th figure) The standard size compressor 3008) described in the paragraph compresses the accumulator 202 value 217 to the size of the standard size. The first predetermined value of the output command 2956 will instruct the startup function unit 212 to execute the specified startup function 2934 to generate an internal result and output the internal result as the output result 133, the size of the internal result being equal to the size of the original input text, ie Half the standard size. The second preset value of the output command 2956 will instruct the boot function unit 212 to execute the specified start function 2934 to generate an internal knot. And the lower half of the internal result is output as an output result 133, the size of the internal result is equal to twice the size of the original input text, that is, the standard size; and the third preset value of the output command 2956 indicates the start function unit 212 outputs the upper half of the internal result of the standard size as the output result 133. The fourth preset value of the output command 2956 will instruct the start function unit 212 to output the unprocessed least significant text of the accumulator 202 as the output result 133; and the fifth preset value of the output command 2956 will indicate that the start function unit 212 will The unprocessed intermediate valid text of the accumulator 202 is output as an output 133; the sixth preset value of the output command 2956 instructs the startup function unit 212 to treat the unprocessed most significant text of the accumulator 202 (its width is configured by The output specified by 2902) is output as output 133, which is described in more detail in the section corresponding to the eighth to tenth figures. As before, the internal result of outputting the entire accumulator 202 size or standard size helps the other execution units 112 of the processor 100 to execute a boot function, such as a soft maximal start function.

The fields described in the twenty-ninth A diagram (and the twenty-ninth and twenty-ninth C diagrams) are located inside the control register 127, however, the invention is not limited thereto, and one or more of the fields It can also be located in other parts of the neural network unit 121. In a preferred embodiment, a plurality of fields may be included within the neural network unit instructions and decoded by the sequencer 128 to generate a microinstruction 3416 (see Figure 34) to control the arithmetic logic unit. 204 and/or start function unit 212. In addition, these fields may also be included in the micro-operations 3414 stored in the media register 118 (see FIG. 34) to control the arithmetic logic unit 204 and/or the start function unit 212. This embodiment can reduce the initialization god Used by network unit instructions, in other embodiments the initialization neural network unit instructions can be removed.

As described above, the neural network unit instructions can specify a memory operand (such as text from the data random access memory 122 and/or the weight random access memory 123) or a post-rotation operand (eg, from a multiplex The memory 208/705) performs an arithmetic logic instruction operation. In one embodiment, the neural network unit instructions may also designate an operand as a scratchpad output of a boot function (such as the output of register 3038 in FIG. 30). Moreover, as previously described, the neural network unit instructions can be specified to increment the current column address of one of the data random access memory 122 or the weight random access memory 124. In an embodiment, the neural network unit instruction may specify an immediate signed integer difference to join the current column for the purpose of incrementing or decrementing a value other than one.

The twenty-ninth B diagram is a block diagram showing another embodiment of the control register 127 of the first figure. The control register 127 of FIG. 29B is similar to the control register 127 of FIG. 29A. However, the control register 127 of FIG. 29B includes an accumulator binary decimal point 2923. . The accumulator binary decimal point 2923 represents the binary position of the accumulator 202. In a preferred embodiment, the accumulator binary point 2923 value indicates the number of bit positions from the right side of the binary point position. In other words, the accumulator binary decimal point 2923 represents the number of decimal places in the least significant bit of accumulator 202, ie, the bit located to the right of the binary decimal point. In this embodiment, the accumulator binary point 2923 is explicitly indicated, rather than being implicitly confirmed as in the twenty-ninth embodiment.

The twenty-ninth C-picture shows a block diagram of an embodiment in which the inverse of the twenty-ninth A picture is stored in two parts. The first portion 2962 is an offset value indicating the number 2962 of zeros that were previously suppressed by the user in the real reciprocal value of the value 217 of the accumulator 202. The number of leading zeros is the number of zeros consecutively arranged to the right of the binary decimal point. The second portion 2694 is the leading zero suppression reciprocal value, which is the true reciprocal value after all leading zeros have been removed. In one embodiment, the suppressed leading zeros number 2962 is stored in 4 bits, and the leading zero suppression inverted value 2964 is stored in 8-bit unsigned values.

For example, assume that the user wants to multiply the accumulator 202 value 217 by the reciprocal value of the value 49. The reciprocal value of the value 49 is presented in two dimensions and the 13 decimal places are set to be 0.0000010100111 with five leading zeros. Thus, the user will fill the suppressed leading zero number 2962 into the value 5 and the leading zero suppression reversal value 2964 into the value 10100111. After the reciprocal multiplier "divider A" 3014 (see Fig. 30) multiplies the accumulator 202 value 217 by the leading zero suppression reciprocal value 2964, the resulting product is shifted to the right according to the suppressed leading zero number 2962. Such an embodiment facilitates the use of relatively few bits to express the reciprocal 2942 value to achieve high accuracy requirements.

The thirtieth diagram is a block diagram showing one embodiment of the start function unit 212 of the second figure. The start function unit 212 includes control logic 127 of the first diagram, a positive type converter (PFC) and an output binary point aligner (OBPA) 3002 to receive the accumulator 202 value 217, a rounder 3004 to receive The accumulator 202 has a value 217 and an output index of the number of bits removed by the decimal point aligner 3002. The aforementioned random bit source 3003 is used to generate a random bit 3005, a first multiplexer 3006 to receive the output of the positive type converter and the output binary point aligner 3002, and the output of the rounder 3004, a standard size. A compressor (CCS) and saturator 3008 to receive the output of the first multiplexer 3006, a bit selector and saturator 3012 to receive the output of the standard size compressor and saturator 3008, and a corrector 3018 to receive the standard size The output of the compressor and saturator 3008, a reciprocal multiplier 3014 to receive the output of the standard size compressor and saturator 3008, a rightward shifter 3016 to receive the output of the standard size compressor and saturator 3008, a double take A tanh module 3022 receives the output of the bit selector and saturator 3012, an S-type module 3024 to receive the output of the bit selector and saturator 3012, and a soft add block 3026 to receive the bit selection. And the output of the saturator 3012, a second multiplexer 3032 to receive the dual take-out tangent module 3022, the S-type module 3024, the soft add-on module 3026, the corrector 3018, the reciprocal multiplier 3014, and the right shifter 301 The output of 6 and the standard size output 3028 passed by the standard size compressor and saturator 3008, a symbol restorer 3034 to receive the output of the second multiplexer 3032, a size converter and saturator 3036 to receive the symbol restorer 3034 The output, a third multiplexer 3037 to receive the output of the size converter and saturator 3036 and the accumulator output 217, and an output register 3038 to receive the output of the multiplexer 3037, the output of which is the first Results 133 in the figure.

The positive type converter and the output binary point aligner 3002 receive the accumulator 202 value 217. In a preferred embodiment, accumulator 202 value 217 is a full precision value as previously described. That is, accumulator 202 has a sufficient number of storage bits to load an accumulated number, which is the total number of products produced by integer adder 244 that are summed by a series of integer multipliers 242, and this operation is performed. Any one of the individual products of the multiplier 242 or the respective totals of the adders is not discarded to maintain accuracy. In a preferred embodiment, accumulator 202 has at least a sufficient number of bits to load the maximum number of multiply-accumulates that a neural network unit 121 can be programmed to perform. For example, please refer to the program of the fourth figure. In the wide configuration, the maximum number of multiplications and accumulations that can be generated by the neural network unit 121 can be 512, and the cumulative number of 202 bits is 41. In another example, referring to the program of the twentieth diagram, in a narrow configuration, the maximum number of multiply-accumulates that can be generated by the neural network unit 121 can be 1024, and the cumulative number of 202 bits is 28. Basically, full precision accumulator 202 has at least Q bits, where Q is the sum of M and log ₂ P, where M is the bit width of the integer product of multiplier 242 (for example, for narrow multiplier 242) It is a 16-bit element, which is 32 bits for the wide multiplier 242), and P is the maximum allowable number of products that the accumulator 202 can accumulate. In a preferred embodiment, the maximum number of multipliers is specified in accordance with the program specification of the programmer of neural network unit 121. In one embodiment, assume that a previous multiply-accumulate instruction is used to load the data/weight text 206/207 column from the data/weight random access memory 122/124 (as in the instruction of address 1 in the fourth figure). The maximum value of the count of the sequencer 128 executing the multiply-accumulate neural network unit instruction (such as the instruction of address 2 in the fourth figure) is, for example, 511.

Accumulating one of the accumulating operations with a full-precision value that has a sufficient bit width and a maximum number of allowed accumulations The design of the arithmetic logic unit 204 of the neural processing unit 126 can be simplified. In particular, such processing can alleviate the need to use logic to perform saturation operations on the total number of integer adder 244 generations, since integer adder 244 causes an overflow accumulator to generate an overflow, while continuing to track the accumulator's binary fraction The point position is used to confirm whether an overflow is generated to confirm whether a saturation operation needs to be performed. For example, for a design with a non-full precision accumulator but with saturation logic to handle the overflow of a non-full accumulator, the following is assumed.

(1) The range of data values is between 0 and 1 and all storage bits are used to store decimal places. The weight text value ranges from -8 to +8 and all but three of the storage bits are used to store the decimal places. The cumulative value of the input as a hyperbolic tangent start function is between -8 and 8, and all but three of the storage bits are used to store the decimal places.

(2) The bit width of the accumulator is not full precision (such as the bit width of only the product).

(3) Assuming that the accumulator is full precision, the final accumulated value will also be between -8 and 8 (eg +4.2); however, the product before "point A" in this sequence will produce positive values more frequently. , and the product after point A will produce negative values more frequently.

In this case, it is possible to obtain incorrect results (such as results other than +4.2). This is because at some point in front of point A, when the accumulator needs to reach a value greater than its saturation maximum of +8, such as +8.2, it will lose an extra 0.2. The accumulator will even keep the remaining multiply-accumulated result at a saturated value, and will lose more positive values. Therefore, the final value of the accumulator It may be less than the value calculated using an accumulator with a full-precision bit width (ie less than +4.2).

The positive type converter 3004 converts the positive value to 217 when the value 217 of the accumulator 202 is negative, and generates an extra bit to indicate the positive or negative value of the original value. This bit is passed down to the start function unit 212 along with this value. Pipeline. Converting a negative number to a positive type simplifies the operation of the subsequent start function unit 121. For example, after this processing, only the positive value will input the hyperbolic tangent module 3022 and the S-type module 3024, thereby simplifying the design of these modules. In addition, the rounder 3004 and the saturator 3008 can also be simplified.

The output binary point aligner 3002 will shift or scale the positive type value to the right to align with the output binary point 2954 specified in the control register 127. In a preferred embodiment, the output binary point aligner 3002 calculates the number of decimal places of the value 217 of the accumulator 202 (eg, as specified by the accumulator binary decimal point 2923 or the data binary decimal point 2922). The sum of the number of decimal places (as specified by the output binary point 2954) minus the number of decimal places of the output (as specified by the output binary point 2954) is used as the offset. Thus, for example, if the accumulator 202 binary decimal point 2923 is 8 (ie, the above embodiment) and the output binary decimal point 2954 is 3, the output binary decimal point aligner 3002 will have this positive type value right. Five bits are shifted to produce the result provided to multiplexer 3006 and rounder 3004.

The rounder 3004 performs a rounding operation on the accumulator 202 value 217. In a preferred embodiment, the rounder 3004 generates a positive class for the positive type converter and the output binary point aligner 3002. The type value produces a rounded version and provides this rounded version to multiplexer 3006. The rounder 3004 performs a rounding operation in accordance with the rounding control 2932 described above. As described herein, the rounding control described above may include random rounding using random bits 3005. The multiplexer 3006 will select one of its plurality of inputs, that is, from the positive type converter and the output binary point aligner 3002, depending on the rounding control 2932 (which may include random rounding as described herein). The positive type value is either from the rounded version of the rounder 3004 and the selected value is provided to the standard size compressor and saturator 3008. In a preferred embodiment, if the rounding control specifies that rounding is not performed, the multiplexer 3006 selects the output of the positive type converter and the output binary point aligner 3002, otherwise the rounder is selected. The output of 3004. In other embodiments, additional rounding operations may also be performed by the start function unit 212. For example, in one embodiment, when the bit selector 3012 compresses the output of the standard size compressor and saturator 3008 (described below), the bit selector 3012 is based on the missing low order bit. Perform rounding operations. In another example, the product of the reciprocal multiplier 3014 (as described below) is subjected to a rounding operation. In yet another example, the size converter 3036 needs to convert the appropriate output size (as described below), which may involve dropping some of the low order bits used to determine the rounding, and performing a rounding operation.

The standard size compressor 3008 compresses the multiplexer 3006 output value to a standard size. Thus, for example, if the neural processing unit 126 is in a narrow configuration or funnel configuration 2902, the standard size compressor 3008 can compress the 28-bit multiplexer 3006 output value to 16 bits; if the neural processing unit 126 is In wide configuration 2902, standard size compression The device 3008 can compress the 41-bit multiplexer 3006 output value to 32 bits. However, before compressing to the standard size, if the pre-compression value is greater than the maximum value that can be expressed by the standard version, the saturator 3008 fills the pre-compression value to the maximum value that can be expressed by the standard pattern. For example, if any bit in the pre-compression value that is to the left of the most significant pre-compression value bit is a value of 1, the saturator 3008 fills up to the maximum value (if filled to all 1s).

In a preferred embodiment, the hyperbolic tangent module 3022, the S-type module 3024, and the soft-add module 3026 all include a lookup table, such as a programmable logic array (PLA), a read-only memory (ROM). , combination logic gates, and so on. In an embodiment, in order to simplify and reduce the size of the modules 3022/3024/3026, the input values provided to the modules are 3.4, that is, three integer bits and four decimal places, that is, The input value has four bits to the right of the binary decimal point and three bits to the left of the binary decimal point. Since the output value is asymptotically close to its min/max value at the extremes of the input value range (-8, +8) of the 3.4 type, these values are selected. However, the present invention is not limited thereto, and the present invention is also applicable to other embodiments in which a binary decimal point is placed at a different position, such as a 4.3 type or a 2.5 type. The bit selector 3012 selects and selects a bit that satisfies the 3.4 type specification among the bits output by the standard size compressor and the saturator 3008. This involves compression processing, that is, some bits are lost because the standard type has a comparison. More bits. However, before selecting/compressing the output value of the standard size compressor and the saturator 3008, if the pre-compression value is greater than the maximum value that can be expressed by the 3.4 type, the saturator 3012 will fill the pre-compression value to the 3.4 type. maximum value. For example, if any bit in the pre-compression value that is to the left of the most significant 3.4 type bit is a value of 1, the saturator 3012 fills up to the maximum value (eg, fills up to all 1s).

The hyperbolic tangent module 3022, the S-type module 3024, and the soft-add module 3026 perform a corresponding start function (as described above) on the 3.4-type value output by the standard size compressor and the saturator 3008 to produce a result. In a preferred embodiment, the hyperbolic tangent module 3022 and the S-type module 3024 produce a 0.7-bit 7-bit result, ie, zero integer bits and seven decimal places, that is, input. The value has seven bits to the right of the binary decimal point. In a preferred embodiment, the soft add module 3026 produces a 3.4-bit 7-bit result, i.e., the pattern is the same as the input pattern of the module 3026. In a preferred embodiment, the outputs of the hyperbolic tangent module 3022, the S-type module 3024, and the soft-add module 3026 are extended to a standard version (eg, with leading zeros if necessary) and aligned The binary decimal point is specified by the output binary digit 2954.

The corrector 3018 produces a corrected version of the output values of the standard size compressor and saturator 3008. That is, if the output value of the standard size compressor and the saturator 3008 (as described above, the symbol is shifted down the pipeline), the corrector 3018 will output a zero value; otherwise, the corrector 3018 will output its input value. . In a preferred embodiment, the output of the corrector 3018 is of a standard version and has a binary point specified by the output binary point 2954 value.

The reciprocal multiplier 3014 multiplies the output of the standard size compressor and saturator 3008 by a user specified reciprocal value assigned to the reciprocal value 2942 to produce a product of the standard size, which is actually The output value of the standard size compressor and saturator 3008 is the quotient calculated by the inverse of the inverse value 2942 as the divisor. In a preferred embodiment, the output of the reciprocal multiplier 3014 is a standard version and has a binary point specified by the output binary point 2954.

The right shifter 3016 moves the output of the standard size compressor and saturator 3008 by the number of user-specified bits specified by the offset value 2944 to produce a quotient of the standard size. In a preferred embodiment, the output to the right shifter 3016 is a standard version and has a binary point specified by the output binary point 2954.

The multiplexer 3032 selects the appropriate input specified by the value of the start function 2934 and provides its selection to the symbol restorer 3034. If the original accumulator 202 has a negative value of 217, the symbol restorer 3034 outputs the multiplexer 3032. The positive type value is converted to a negative type, for example to a two-complement type.

The size converter 3036 converts the output of the symbol restorer 3034 to the appropriate size in accordance with the value of the output command 2956 as described in FIG. 29A. In a preferred embodiment, the output of symbol restorer 3034 has a binary point specified by the output binary point 2954 value. In a preferred embodiment, the size converter 3036 discards the upper half of the output of the symbol restorer 3034 for the first predetermined value of the output command. In addition, if the output of the symbol restorer 3034 is positive and exceeds the maximum value that can be expressed by the text size specified by the configuration 2902, or the output is negative and smaller than the minimum value that can be expressed by the text size, the saturator 3036 outputs it. Fill up to the text size to express the maximum/minimum value. For the second and third preset values, the size is turned The converter 3036 passes the output of the symbol restorer 3034.

The multiplexer 3037 selects one of the data converter and saturator 3036 output and accumulator 202 output 217 for supply to the output register 3038 in accordance with the output command 2956. Further, for the first and second preset values of the output command 2956, the multiplexer 3037 selects the lower text of the output of the size converter and saturator 3036 (the size is specified by configuration 2902). For the third preset value, the multiplexer 3037 selects the upper text of the output of the size converter and saturator 3036. For the fourth preset value, the multiplexer 3037 selects the lower text of the unprocessed accumulator 202 value 217; for the fifth preset value, the multiplexer 3037 selects the middle of the unprocessed accumulator 202 value 217 The text; and for the sixth preset value, the multiplexer 3037 selects the upper text of the unprocessed accumulator 202 value 217. As previously mentioned, in a preferred embodiment, the start function unit 212 adds a zero value upper orientation element to the upper text of the unprocessed accumulator 202 value 217.

The thirty-first figure shows an example of the operation of the start function unit 212 of the thirty-th figure. As shown in the figure, the configuration 2902 of the neural processing unit 126 is set to a narrow configuration. In addition, the signed material 2912 and the signed weight 2914 are true. In addition, the data binary decimal point value of 2922 indicates that for the data random access memory 122 text, there are 7 bits to the right of the binary decimal point position, and one of the first data characters received by the neural processing unit 126 is an example value. The line is presented as 0.1001110. In addition, the weight binary decimal point 2924 value indicates that for the weight random access memory 124 text, there are 3 bits to the right of the binary decimal point position, and one of the first weight texts received by the neural processing unit 126 is an example value. Department It is now 00001.010.

The 16-bit product of the first data and the weight text (this product is added to the initial zero value of the accumulator 202) is presented as 000000.1100001100. Since the data binary decimal point 2912 is 7 and the weighted binary decimal point 2914 is 3, for the implicit accumulator 202 binary decimal point, there will be 10 bits to the right. In the case of a narrow configuration, as shown in this embodiment, the accumulator 202 has 28 bit widths. For example, after all arithmetic logic operations have been completed (eg, all 1024 multiply-accumulate operations in the twentieth diagram), the value 217 of the accumulator 202 would be 000000000000000001.1101010100.

The output binary digit 2954 value indicates that the output binary digit has 7 digits to the right of the decimal point. Thus, after passing the output decimal point aligner 3002 and the standard size compressor 3008, the accumulator 202 value 217 is scaled, rounded, and compressed to the value of the standard version, 000000001.1101011. In this example, the output binary decimal point address represents 7 decimal places, and the accumulator 202 binary decimal point position represents 10 decimal places. Thus, the output binary point aligner 3002 calculates the difference 3 and scales it by shifting the accumulator 202 value 217 to the right by 3 bits. In the thirty-first figure, it is shown that the accumulator 202 value 217 will lose 3 least significant bits (two digits of 100). Moreover, in this example, the rounding control 2932 value indicates that random rounding is used, and in this example it is assumed that the sampling random bit 3005 is true. Thus, as before, the least significant bit is rounded up because of the rounding bit of the value 217 of the accumulator 202 (the three are removed from the bit due to the scaling operation of the value 217 of the accumulator 202). The most significant bit) is one, The sticky bits (the three Boolean or arithmetic results of the two least significant bits due to the scaling operation of the value 217 of the accumulator 202) are zero.

In this example, the start function 2934 indicates that an sigmoid function is used. Thus, bit selector 3012 selects the bit of the standard pattern value such that the input of S-type module 3024 has three integer bits and four decimal places, as previously described, the value 001.1101 shown. The output value of the S-type module 3024 will be placed in the standard version, which is the value shown as 000000000.1101110.

The output command 2956 of this example specifies the first preset value, i.e., the text size represented by output configuration 2902, in this case narrow text (8 bits). Thus, the size converter 3036 converts the standard S-type output value into an 8-bit quantity having an implicit binary point, ie, 7 bits to the right of the binary point, producing an output The value is 01101110, as shown in the figure.

The thirty-second figure shows a second example of the operation of the start function unit 212 of the thirty-th figure. The example of the thirty-second diagram describes the operation of the start function unit 212 when the start function 2934 indicates that the accumulator 202 value 217 is passed in a standard size. As shown in the figure, this configuration 2902 is set to a narrow configuration of the neural processing unit 216.

In this example, the width of the accumulator 202 is 28 bits, and the accumulator 202 has 10 bits to the right of the decimal point position (this is because in the embodiment the data binary decimal point 2912 and the weight binary carry The sum of the decimal points 2914 is always 10, or in another embodiment the accumulator binary decimal point 2923 is explicitly designated to have a value of 10). For example It is said that after performing all the arithmetic logic operations, the value 217 of the accumulator 202 shown in the thirty-second figure is 000001100000011011.1101111010.

In this example, the output binary point 2954 value indicates that for the output, there are 4 bits to the right of the binary point. Thus, after passing the output binary point aligner 3002 and the standard size compressor 3008, the accumulator 202 value 217 is saturated and compressed to the standard pattern value 111111111111.1111 shown, which is received by the multiplexer 3032. The value is passed as a standard size 3028.

Two output commands 2956 are shown in this example. The first output command 2956 specifies a second preset value, which is the lower text of the output standard type size. Since the size indicated by configuration 2902 is a narrow text (8-bit), the standard size will be 16-bit, and the size converter 3036 will select the lower 8 bits below the standard size transfer value 3028 to produce as shown in the figure. The 8-bit value is 11111111. The second output command 2956 specifies a third preset value, that is, the upper text of the output standard type size. As such, the size converter 3036 selects the upper 8 bits of the standard size transfer value 3028 to produce the 8-bit value 11111111 as shown.

The thirty-third figure shows a third example of the operation of the start function unit 212 of the thirty-th figure. The example of the thirty-third diagram reveals the operation of the start function unit 212 when the start function 2934 indicates that the entire unprocessed accumulator 202 value 217 is to be passed. As shown in the figure, this configuration 2902 is set to a wide configuration of the neural processing unit 126 (e.g., 16-bit input text).

In this example, the width of the accumulator 202 is 41 bits, and the accumulator 202 has 8 bits to the right of the decimal point position (this This is because in one embodiment the data binary decimal point 2912 and the weighted binary decimal point 2914 add up to 8, or in another embodiment the accumulator binary decimal point 2923 is explicitly designated to have a value of 8). For example, after performing all of the arithmetic logic operations, the value 217 of the accumulator 202 shown in the thirty-third figure is 001000000000000000001100000011011.11011110.

Three output commands 2956 are shown in this example. The first output command specifies a fourth preset value, that is, outputs the text below the value of the unprocessed accumulator 202; the second output command specifies a fifth preset value, that is, the middle of the output of the unprocessed accumulator 202. The text is output; the third output command specifies a sixth preset value, that is, the text above the value of the unprocessed accumulator 202 is output. Because the size indicated by configuration 2902 is a wide text (16-bit), as shown in the thirty-third figure, in response to the first output command 2956, the multiplexer 3037 selects the 16-bit value 0001101111011110; in response to the second output command 2956 The multiplexer 3037 selects the 16-bit value 0000000000011000; and in response to the third output command 2956, the multiplexer 3037 selects the 16-bit value 0000000001000000.

As mentioned above, the neural network unit 121 can execute on integer data instead of floating point data. As such, it is helpful to simplify the individual neural processing units 126, or at least the arithmetic logic unit 204 portions thereof. For example, the arithmetic logic unit 204 does not need to include, for the multiplier 242, an adder that is used in floating point operations to add the exponents of the multipliers. Similarly, this arithmetic logic unit 204 does not need to include a shifter that is used in the floating point operation to align the binary digits of the addend for the adder 234. A person skilled in the art can understand, floating point The elements are often very complex; therefore, the examples described herein are only simplified for the arithmetic logic unit 204, and the integer embodiment with the hardware fixed point assistance allowing the user to specify the associated binary point can also be used for other parts. Simplify. In contrast to floating point embodiments, the use of integer units as arithmetic logic unit 204 can produce a smaller (and faster) neural processing unit 126 that facilitates the integration of a large array of neural processing units 126 into the neural network. Within unit 121. The portion of the start function unit 212 may process the scaling and saturation operations of the value 217 of the accumulator 202 based on the user specified, the number of decimal places required for the accumulated number, and the number of decimal bits required for the output value, preferably based on use. Designated. Any additional complexity and accompanying increase in size, as well as energy and/or time loss within the fixed-point hardware assistance of the start function unit 212, can be shared by sharing the start function unit 212 between the arithmetic logic units 204. This is because, as shown in the embodiment of the eleventh figure, the embodiment using the sharing mode can reduce the number of startup function units 1112.

The embodiments described herein can enjoy many advantages of using integer arithmetic units to reduce hardware complexity (compared to using floating point arithmetic units), while also being used for arithmetic operations of decimals, that is, having a binary decimal point. digital. The advantage of floating-point arithmetic is that it can provide data arithmetic operations to the individual values of the data falling within a very wide range of values (actually limited only by the size of the exponent range, so it will be a very large range). That is, each floating point number has its own unique index value. However, the embodiments described herein understand and utilize applications in which the input data is highly parallel and falls within a relatively narrow Within the scope, all parallel data have the same "index" characteristics. As such, these embodiments allow the user to assign a binary decimal point position to all input values and/or accumulated values at a time. Similarly, by understanding and utilizing the parallel output having similar range characteristics, these embodiments allow the user to assign the binary point position to all of the output values at a time. Artificial neural networks are an example of such an application, although embodiments of the invention may also be applied to perform calculations for other applications. By assigning a binary decimal point position to multiple inputs instead of giving individual input numbers, embodiments of the present invention can utilize memory space more efficiently than using floating point operations (eg, requiring less memory) Accuracy is improved in the case of using a similar amount of memory, because the bits of the exponent used for floating-point operations can be used to improve numerical precision.

Moreover, embodiments of the present invention understand that accuracy may be lost when performing an accumulation of a large series of integer operations (such as overflow or loss of less important fractional bits), thus providing a solution, primarily using a sufficiently large Accumulator to avoid loss of precision.

Direct execution of neural network unit micro-operation

The thirty-fourth diagram is a block diagram showing a portion of the details of the processor 100 of the first diagram and the neural network unit 121. The neural network unit 121 includes a pipeline stage 3401 of the neural processing unit 126. Each pipeline stage 3401 is distinguished by a stage register and includes combinatorial logic to accomplish the operations of the neural processing unit 126 herein, such as a Boolean logic gate, a multiplexer, an adder, a multiplier, a comparator, and the like. Pipeline stage 3401 receives a micro-operation 3418 from multiplexer 3402. Micro-operation 3418 will flow down to the pipeline level 3401 and control its combinatorial logic. Micro-operation 3418 is a set of bits. For a preferred embodiment, the micro-operation 3418 includes a bit of the data random access memory 122 memory address 123, a bit of the weight random access memory 124 memory address 125, and a program memory 129 memory address. The bit of 131, the multiplex register 208/705 control signal 213/713, and a number of fields that control the register 217 (e.g., the control registers of the twenty-ninth to twenty-ninth C-pictures). In an embodiment, the micro-operation 3418 includes approximately 120 bits. Multiplexer 3402 receives micro-operations from three different sources and selects one of them as a micro-operation 3418 that is provided to pipeline stage 3401.

One micro-operation source of multiplexer 3402 is sequencer 128 of the first figure. The sequencer 128 decodes the neural network unit instructions received by the program memory 129 and accordingly provides a micro-operation 3416 to the first input of the multiplexer 3402.

The second source of micro-computations of multiplexer 3402 is a decoder 10404 that receives microinstructions 105 from reservation station 108 of the first diagram and operands from general register 116 and media register 118. In a preferred embodiment, as previously described, the microinstruction 105 is generated by the instruction translator 104 in response to the translation of the MTNN instruction 1400 and the MFNN instruction 1500. Microinstruction 105 may include an immediate column to specify a particular function (specified by an MTNN instruction 1400 or an MFNN instruction 1500), such as the start and stop execution of a program in program memory 129, executing directly from media register 118. Micro-operation, or memory of one of the neural network units read/written as described above. The decoder 3404 decodes the microinstructions 105 and accordingly generates a micro-operation 3412 to provide a second input to the multiplexer. For a preferred embodiment, some functions of the 1400/MFNN instruction 1500 for the MTNN instruction For 1432/1532, the decoder 3404 does not need to generate a micro-operation 3412 to be transferred down to the pipeline 3401, for example, to the write control register 127, to start executing the program in the program memory 129, and to suspend execution of the program memory 129. The program, the program in the wait program memory 129 completes execution, reads from the status register 127, and resets the neural network unit 121.

The third source of micro-computation for multiplexer 3402 is media register 118 itself. In a preferred embodiment, as previously described in relation to FIG. 14, the MTNN instruction 1400 can specify a function to instruct the neural network unit 121 to directly perform a provision from the media register 118 to the multiplexer 3402. The three-input micro-operation 3414. Direct execution of the micro-operations 3414 provided by the architectural media register 118 facilitates testing of the neural network unit 121, such as built-in self-test (BIST), or debugging.

In a preferred embodiment, decoder 3404 generates a mode indicator 3422 that controls the selection of multiplexer 3402. When the MTNN instruction 1400 specifies a function to begin execution of a program from the program memory 129, the decoder 3404 generates a mode indicator 3422 value causing the multiplexer 3402 to select the micro-operation 3416 from the sequencer 128 until an error occurs or until decoding occurs. The device 3404 encounters an MTNN instruction 1400 that specifies a function to stop execution of the program from the program memory 129. When the MTNN instruction 1400 specifies a function to instruct the neural network unit 121 to directly perform a micro-operation 3414 provided by the media register 118, the decoder 3404 generates a mode indicator 3422 value to cause the multiplexer 3402 to select from the specified media. The micro-operation 3414 of the register 118. Otherwise, decoder 3404 will generate a mode indicator 3422 value to cause multiplexer 3402 to select micro-operation 3412 from decoder 3404.

Variable rate neural network unit

In many cases, the neural network unit 121 will enter the standby state after executing the program, waiting for the processor 100 to process some things that need to be processed before executing the next program. For example, assuming that it is in a situation similar to that described in Figures 3-6A, the neural network unit 121 will accumulate a start function program (also known as a pre-nego network layer program). Neural network layer program)) Performs two or more times in succession. Compared to the time it takes for the neural network unit 121 to execute the program, the processor 100 obviously takes a long time to write a weight value of 512 KB into the weight random access memory 124 for use by the next neural network unit program. . In other words, the neural network unit 121 executes the program in a short time and then enters the standby state until the processor 100 writes the next weight value to the weight random access memory 124 for use in the next program execution. For the case, refer to the thirty-sixth A diagram, as will be described later. In this case, the neural network unit 121 can operate at a lower time frequency to extend the execution time of the program, so that the energy consumption required to execute the program is dispersed over a longer time range, and the neural network unit 121, and even the neural network unit 121 Throughout the processor 100, it is maintained at a lower temperature. This case is called a mitigation mode, and can be referred to the thirty-sixth B-picture, as will be described later.

The thirty-fifth diagram is a block diagram showing a processor 100 having a variable rate neural network unit 121. This processor 100 is similar to processor 100 of the first figure, and elements having the same reference numerals are similar in the figures. The processor 100 of the thirty-fifth diagram has a time-frequency generating logic 3502 coupled to the functional units of the processor 100, namely, the instruction fetching unit 101, the instruction cache 102, the instruction translator 104, and the renaming unit. 106, reservation station 108, neural network unit 121, other execution units 112, memory subsystem 114, general purpose register 116 and media register 118. The time-frequency generating logic 3502 includes a time-frequency generator, such as a phase-locked loop (PLL), to generate a time-frequency signal having a dominant time frequency or time-frequency. For example, the primary time frequency can be 1 GHz, 1.5 GHz, 2 GHz, and the like. The time frequency means the number of cycles per second, such as the number of oscillations of the time-frequency signal between high and low states. Preferably, the frequency signal has a duty cycle, that is, one half of the cycle is a high state and the other half is a low state; in addition, the frequency signal can also have an unbalanced period, that is, a time frequency. The signal is in a high state for longer than it is at a low state, and vice versa. Preferably, the phase locked loop is used to generate a primary time-frequency signal of a plurality of time frequencies. Preferably, the processor 100 includes a power management module that automatically adjusts the primary time frequency according to various factors, including the dynamic detection operating temperature of the processor 100, the utilization, and the system software (such as the operating system). , Basic Input Output System (BIOS)) A command that indicates the required performance and/or energy efficiency indicators. In an embodiment, the power management module includes microcode of the processor 100.

The time-frequency generation logic 3502 includes a time-frequency spreading network, or a clock tree. The time-frequency tree will spread the main time-frequency signal to the functional unit of the processor 100. As shown in the thirty-fifth figure, the spreading action is to transmit the time-frequency signal 3506-1 to the instruction capturing unit 101, and the time-frequency signal 3506. -2 is transmitted to the instruction cache 102, the time frequency signal 3506-10 is transmitted to the instruction translator 104, the time frequency signal 3506-9 is transmitted to the rename unit 106, and the time frequency signal 3506-8 is transmitted to the reservation station 108. Transmitting the time-frequency signal 3506-7 to the neural network unit 121, The frequency signal 3506-4 is transmitted to the other execution unit 112, the time-frequency signal 3506-3 is transmitted to the memory subsystem 114, the time-frequency signal 3506-5 is transmitted to the general-purpose register 116, and the time-frequency signal 3506-6 is transmitted. The signals are transmitted to the media register 118, which are collectively referred to as time-frequency signals 3506. The frequency tree now has a node or line to transmit the primary time-frequency signal 3506 to its corresponding functional unit. In addition, preferably, the time-frequency generating logic 3502 can include a time-frequency buffer, when it is required to provide a clean time-frequency signal and/or need to increase the voltage level of the main time-frequency signal, especially for a distant node. The time-frequency buffer can regenerate the main time-frequency signal. In addition, each functional unit has its own sub-time-frequency tree that regenerates and/or boosts the voltage level of the corresponding primary time-frequency signal 3506 received as needed.

The neural network unit 121 includes time-frequency reduction logic 3504, and the time-frequency reduction logic 3504 receives a mitigation indicator 3512 and a primary time-frequency signal 3506-7 to generate a second time-frequency signal. The second time-frequency signal has a one-time frequency. If the frequency is not the same as the main time frequency, it is in a mode of mitigation. The frequency is reduced by a value from the main time to reduce the heat generation. This value is programmed to the mitigation index 3512. The time-frequency reduction logic 3504 is similar to the time-frequency generation logic 3502, which has a time-frequency spreading network, or a time-frequency tree, to spread the second time-frequency signal to the various functional blocks of the neural network unit 121, and the spreading action is time-consuming. The frequency signal 3508-1 is transmitted to the neural processing unit array 126, and the time-frequency signal 3508-2 is transmitted to the sequencer 128 to transmit the time-frequency signal 3508-3 to the interface logic 3514, collectively referred to as the second time-frequency signal. 3508. Preferably, the neural processing unit 126 includes a plurality of pipeline stages 3401. As shown in the thirty-fourth diagram, the pipeline stage 3401 includes a pipeline grading register for time-frequency reduction logic. The 3504 receives the second time-frequency signal 3508-1.

The neural network unit 121 also has interface logic 3514 to receive the primary time-frequency signal 3506-7 and the second time-frequency signal 3508-3. The interface logic 3514 is coupled between the lower portion of the front end of the processor 100 (eg, the reservation station 108, the media register 118 and the universal register 116) and the various functional blocks of the neural network unit 121. These functional blocks are reduced in real time. Logic 3504, data random access memory 122, weighted random access memory 124, program memory 129 and sequencer 128. Interface logic 3514 includes a data random access memory buffer 3522, a weighted random access memory buffer 3524, a thirty-fourth graph decoder 3404, and a mitigation indicator 3512. The mitigation indicator 3512 is loaded with a value that specifies how slow the neural processing unit array 126 will execute the neural network unit program instructions. Preferably, the mitigation indicator 3512 specifies a division value N, and the time-frequency reduction logic 3504 divides the main time-frequency signal 3506-7 by the division value to generate the second time-frequency signal 3508. Thus, the second time-frequency signal is The frequency will be 1/N. Preferably, the value of N can be programmed into any one of a plurality of different preset values. The preset values can cause the time-frequency reduction logic 3504 to generate a plurality of second time-frequency signals 3508 having different time frequencies. The time frequency is less than the primary frequency.

In one embodiment, the time-frequency reduction logic 3504 includes a time-frequency divider circuit for dividing the primary time-frequency signal 3506-7 by the mitigation indicator 3512 value. In an embodiment, the time-frequency reduction logic 3504 includes a time-frequency gate (such as an AND gate), and the time-frequency gate can gate the main time-frequency signal 3506-7 through a start signal, and the enable signal is at each N of the main time-frequency signal. Only one true value is generated in one cycle. One containing a counter to generate For example, the circuit that starts the signal can count up to N. When the accompanying logic detects that the output of the counter matches N, the logic generates a true pulse at the second time-frequency signal 3508 and resets the counter. Preferably, the mitigation indicator 3512 value can be programmed by an architectural instruction, such as the MTNN instruction 1400 of FIG. Preferably, before the architecture program instructs the neural network unit 121 to start executing the neural network unit program, the architecture program operating on the processor 100 will program the mitigation value to the mitigation indicator 3512, which in the subsequent corresponds to the thirtieth There will be more detailed explanations at the seven maps.

The weighted random access memory buffer 3524 is coupled between the weighted random access memory 124 and the media register 118 as a buffer for data transmission therebetween. Preferably, weighted random access memory buffer 3524 is similar to one or more embodiments of buffer 1704 of FIG. Preferably, the weighted random access memory buffer 3524 receives the data from the media register 118 by using the primary time-frequency signal 3506-7 having the primary time frequency as the time frequency, and the weight random access memory buffer 3524 from The portion of the weighted random access memory 124 that receives the data is the second time-frequency signal 3508-3 having the second time frequency as the time frequency, and the second time frequency can be adjusted from the main time frequency according to the value programmed in the mitigation index 3512. Down or no, that is, whether the neural network unit 121 performs the mitigation or normal mode to perform the down or no. In one embodiment, the weight random access memory 124 is 單埠. As described in the foregoing FIG. 17, the weight random access memory 124 can be transmitted by the media register 118 through the weight random access memory buffer 3524. And accessed by the neural processing unit 126 or the eleventh column buffer 1104 in an arbitrated fashion. In another implementation In the example, the weight random access memory 124 is a double port. As described in the sixteenth figure above, each port can be transmitted by the media register 118 through the weight random access memory buffer 3524 and by the neural processing unit 126 or the column buffer. 1104 is accessed in parallel.

Similar to the weighted random access memory buffer 3524, the data random access memory buffer 3522 is coupled between the data random access memory 122 and the media register 118 as a buffer for data transfer therebetween. Preferably, data random access memory buffer 3522 is similar to one or more embodiments of buffer 1704 of FIG. Preferably, the data random access memory buffer 3522 receives the data from the media register 118 by using the main time-frequency signal 3506-7 having the main time frequency as the time frequency, and the data random access memory buffer 3522 The data random access memory 122 receives the data by using a second time-frequency signal 3508-3 having a second time frequency as a time frequency, and the second time frequency can be adjusted from the main time frequency according to the value programmed in the mitigation index 3512. Down or no, that is, whether the neural network unit 121 performs the mitigation or normal mode to perform the down or no. In one embodiment, the data random access memory 122 is 單埠. As described in FIG. 17 above, the data random access memory 122 can be buffered by the media buffer 118 through the data random access memory buffer 3522. And accessed by the neural processing unit 126 or the column buffer 1104 of the eleventh figure in an arbitration manner. In another embodiment, the data random access memory 122 is a double port. As described in the sixteenth figure above, each port can be accessed by the media register 118 through the data random access memory buffer 3522 and by the neural processing unit 126. Or column buffer 1104 is accessed in parallel.

Preferably, regardless of the data random access memory 122 and / Or the weight random access memory 124 is 單埠 or 埠, and the interface logic 3514 includes a data random access memory buffer 3522 and a weight random access memory buffer 3524 to synchronize the primary time frequency domain with the second time frequency domain. Preferably, the data random access memory 122, the weight random access memory 124 and the program memory 129 both have a static random access memory (SRAM), which includes individual read enable signals, and writes The signal and memory can be selected to enable the signal.

As before, the neural network unit 121 is one of the execution units of the processor 100. The execution unit is a micro-instruction in the processor that executes the schema instruction or a functional unit that executes the architecture instruction itself, such as executing the micro-instruction 105 or the architecture instruction 103 itself translated by the architecture instruction 103 in the first figure. The execution unit receives operands from the general purpose register of the processor, such as from the general purpose register 116 and the media register 118. The execution unit executes the micro or architectural instructions and produces a result that is written to the general purpose register. The MTNN instructions 1400 and MFNN instructions 1500 described in Figures 14 and 15 are examples of architectural instructions 103. Microinstructions are used to implement architectural instructions. More precisely, the collective execution of one or more microinstructions that are executed by the execution unit for the architectural instructions is the result of the input of the architectural instruction specified by the architectural instruction specified by the architectural instruction to produce the result of the architectural instruction definition.

The thirty-sixth A diagram is a timing diagram showing that the processor 100 has an operational example in which the neural network unit 121 operates in a general mode, which operates at a primary time frequency. In the timing diagram, the progress of time is from left to right. The processor 100 executes the architectural program at the primary time frequency. More precisely, the front end of the processor 100 (such as instruction capture Unit 101, instruction cache 102, instruction translator 104, rename unit 106 and reservation station 108) retrieve, decode and issue architectural instructions to neural network unit 121 and other execution units 112.

Initially, the architecture program executes an architectural instruction (e.g., MTNN instruction 1400), and the processor front end 100 issues the architectural instruction to the neural network unit 121 to instruct the neural network unit 121 to begin executing the neural network in the program memory 129. Unit program. Previously, the architecture program executed an architectural instruction to write the value of a specified primary time frequency to the mitigation indicator 3512, even if the neural network unit was in the normal mode. More precisely, the value programmed to the mitigation indicator 3512 causes the time-frequency reduction logic 3504 to generate the second time-frequency signal 3508 at the dominant time frequency of the primary time-frequency signal 3506. Preferably, in this example, the time-frequency buffer of the time-frequency reduction logic 3504 simply boosts the voltage level of the primary time-frequency signal 3506. In addition, the architecture program executes the architectural instructions to write the data random access memory 122, the weighted random access memory 124, and the neural network unit program to the program memory 129. In response to the neural network unit program MTNN command 1400, the neural network unit 121 will begin executing the neural network unit program at the primary time frequency because the mitigation indicator 3512 is programmed with the primary time frequency value. After the neural network unit 121 begins execution, the architecture program continues to execute the architectural instructions at the primary time frequency, including writing and/or reading the data random access memory 122 and the weight random access memory 124 primarily by the MTNN instruction 1400. To complete the next instance of the neural network unit program, or the preparation of the invocation or run.

In the example of Figure 36A, compared to the architecture The time it takes for the data random access memory 122 and the weight random access memory 124 to write/read is completed, and the neural network unit 121 can be completed in significantly less time (for example, one quarter of the time). The execution of the neural network unit program. For example, in the case of operating at the primary time frequency, the neural network unit 121 spends approximately 1000 time-frequency cycles to execute the neural network unit program, however, the architecture program would take approximately 4000 time-frequency cycles. As such, the neural network unit 121 will be in a standby state for the remainder of the time, which in this example is a relatively long period of time, such as approximately 3000 major time frequency periods. As shown in the example of Figure 36A, the foregoing mode is executed again depending on the size and configuration of the neural network, and may continue to be executed many times. Since the neural network unit 121 is a relatively large and transistor-intensive functional unit in the processor 100, the operation of the neural network unit 121 will generate a large amount of thermal energy, especially when operating at a primary time frequency.

The thirty-sixth B diagram is a timing diagram, and the display processor 100 has an operation example in which the neural network unit 121 operates in the mitigation mode, and the mitigation mode operates at a frequency lower than the main time frequency. The timing chart of the thirty-sixth B diagram is similar to the thirty-sixth A diagram. In the thirty-sixth A diagram, the processor 100 executes an architecture program at the main time frequency. This example assumes that the architecture program and the neural network unit program in Figure 36B are identical to the architecture program and neural network unit program in Figure 36A. However, prior to initiating the neural network unit program, the architecture program executes an MTNN instruction 1400 to program a mitigation indicator 3512, which causes the time-frequency reduction logic 3504 to generate a second frequency less than the primary frequency. Time-frequency signal 3508. In other words, the architecture program will make the neural network single The element 121 is in the mitigation mode of the thirty-sixth B-picture, instead of the general mode of the thirty-sixth A picture. Thus, the neural processing unit 126 executes the neural network unit program at the second time frequency, and in the mitigation mode, the second time frequency is less than the primary time frequency. In this example, it is assumed that the mitigation indicator 3512 is programmed with a value that specifies the second time frequency as a quarter of the primary frequency. Thus, the time taken by the neural network unit 121 to execute the neural network unit program in the mitigation mode is four times that of the normal mode, as shown in the thirty-sixth and thirty-sixth B-pictures. Comparing these two figures, it can be seen that the length of time during which the neural network unit 121 is in the standby state is significantly shortened. Thus, the duration of the neural network unit 121 performing the neural network unit program consuming energy in the thirty-sixth B diagram is approximately four times that of the neural network unit 121 executing the program in the normal mode in the thirty-sixth A picture. Therefore, the thermal energy generated by the neural network unit 121 in the unit of the neural network unit 121 in the unit time is approximately one quarter of the thirty-sixth A picture, and has the advantages described herein.

Figure 37 is a flow chart showing the operation of the processor 100 of the thirty-fifth figure. The operation described in this flowchart is similar to the previous operation corresponding to the thirty-fifth, thirty-sixth and thirty-sixth B diagrams. This process begins in step 3702.

In step 3702, processor 100 executes MTNN instruction 1400 to write weights to weight random access memory 124 and writes the data to data random access memory 122. The flow then proceeds to step 3704.

In step 3704, the processor 100 executes the MTNN instruction 1400 to programmatically mitigate the indicator 3512 with a numerical value. A frequency lower than the dominant frequency is also determined, even if the neural network unit 121 is in the mitigation mode. The flow then proceeds to step 3706.

In step 3706, the processor 100 executes the MTNN instruction 1400 to instruct the neural network unit 121 to begin executing the neural network unit program, i.e., in a manner similar to that presented in Figure 36B. The flow then proceeds to step 3708.

In step 3708, neural network unit 121 begins executing the neural network unit program. At the same time, the processor 100 executes the MTNN instruction 1400 to write new weights to the weight random access memory 124 (and possibly also write new data to the data random access memory 122), and/or execute the MFNN instruction 1500. The result is read from the data random access memory 122 (the result may also be read from the weighted random access memory 124). The flow then proceeds to step 3712.

In step 3712, the processor 100 executes the MFNN instruction 1500 (e.g., the read status register 127) to detect that the neural network unit 121 has finished executing the program. Assuming that the architecture program selects a good mitigation indicator 3512 value, the time taken by the neural network unit 121 to execute the neural network unit program is the same as the processor 100 executing the partial architecture program to access the weighted random access memory 124 and / Or the time it takes for the data to randomly access the memory 122, as shown in Figure 36B. The flow then proceeds to step 3714.

At step 3714, processor 100 executes MTNN instruction 1400 and utilizes a numerically programmed mitigation indicator 3512 that specifies the primary time frequency, even if neural network unit 121 is in the normal mode. Next, proceed to step 3716.

In step 3716, the processor 100 executes the MTNN instruction 1400 to instruct the neural network unit 121 to begin executing the neural network unit program, i.e., in a manner similar to that presented in the thirty-sixth A diagram. The flow then proceeds to step 3718.

In step 3718, neural network unit 121 begins executing the neural network unit program in a normal mode. This process ends at step 3718.

As described above, execution in the mitigation mode can disperse the execution time while avoiding high temperatures compared to executing the neural network unit program in the normal mode (i.e., at the main time frequency of the processor). Further, when the neural network unit executes the program in the mitigation mode, the neural network unit generates thermal energy at a lower time frequency, and the thermal energy can smoothly pass through the neural network unit (for example, a semiconductor device, a metal layer and a lower layer) The substrate) is discharged from the surrounding package and the cooling mechanism (such as the heat sink, fan). Therefore, devices in the neural network unit (such as transistors, capacitors, wires) are more likely to operate at lower temperatures. Overall, operating in a mitigation mode also helps to reduce device temperature in other parts of the processor die. Lower operating temperatures, especially for junction temperatures of these devices, can reduce leakage current generation. In addition, because the amount of current flowing in per unit time decreases, the inductance noise and IR drop noise are also reduced. In addition, the temperature reduction has a positive effect on the negative bias temperature instability (NBTI) and positive bias instability (PBSI) of the metal oxide half field effect transistor (MOSFET) in the processor, which improves reliability and / Or the life of the device and the processor portion. The temperature is reduced and the Joule heat and electromigration effects within the metal layer of the processor are mitigated.

Communication mechanism between architecture programs and non-architecture programs for sharing resources between neural network units

As described above, in the examples of the twenty-fourth to twenty-eighth and thirty-fifth to thirty-seventh figures, the data random access memory 122 and the weight random access memory 124 are shared. The neural processing unit 126 shares the data random access memory 122 and the weight random access memory 124 with the front end of the processor 100. More precisely, the neural processing unit 126 and the front end of the processor 100, such as the media register 118, both read and write the data random access memory 122 and the weight random access memory 124. In other words, the architecture program executed by the processor 100 and the neural network unit program executing in the neural network unit 121 share the data random access memory 122 and the weight random access memory 124, and in some cases As mentioned earlier, it is necessary to control the flow between the architecture program and the neural network unit program. The resources of program memory 129 are also shared to some extent because the architecture program will write to it and sequencer 128 will read it. The embodiments described herein provide a high performance solution to control the flow of shared resources between the architectural program and the neural network unit.

In the embodiments described herein, neural network unit programs are also referred to as non-architectural programs, neural network unit commands are also referred to as non-architectural instructions, and neural network unit instruction sets (also referred to as neural processing as previously described). The unit instruction set) is also known as the non-architectural instruction set. A non-architectural instruction set is different from a architectural instruction set. In embodiments where processor 100 includes instruction translator 104 to translate architectural instructions out of microinstructions, the non-architectural instruction set is also distinct from the microinstruction set.

The thirty-eighth diagram is a block diagram showing the sequencer 128 of the neural network unit 121 in detail. Sequencer 128 provides a memory address to program memory 129 to select the non-architected instructions provided to sequencer 128, as previously described. As shown in the thirty-eighth figure, the memory address is loaded in a program counter 3802 of the sequencer 128. The sequencer 128 will typically increment sequentially in the address sequence of the program memory 129 unless the sequencer 128 encounters a non-architectural instruction, such as a loop or branch instruction, and in this case, the sequencer 128 will The program counter 3802 is updated to the target address of the control instruction, i.e., updated to the address of the non-architected instruction located at the target of the control instruction. Thus, the address 131 loaded in the program counter 3802 specifies the address of the non-architected instruction in the program memory 129 that is currently being fetched for execution by the neural processing unit 126. The value of the program counter 3802 can be obtained by the architecture program through the neural network unit program counter field 3912 of the status register 127, as described in the subsequent thirty-first figure. In this way, the architecture program can determine the location of the data random access memory 122 and/or the weight random access memory 124 to read/write data according to the progress of the non-architecture program.

The sequencer 128 includes a loop counter 3804 which operates in conjunction with a non-architected loop command, such as a loop of address 10 to a command and a twenty-eighth in the twenty-sixth A diagram. In the figure, the loop of address 11 is up to 1 command. In the examples of the twenty-sixth and twenty-eighth diagrams, the loop counter 3804 is loaded with the value specified by the non-architectural initialization instruction of address 0, such as the value 400. Each sequencer 128 encounters a loop instruction and jumps to the target instruction (such as the multiply accumulate instruction at address 1 in Figure 26A or the address 1 in the twenty-eighth picture). The maxwacc command), the sequencer 128 will decrement the loop counter 3804. Once the loop counter 3804 is reduced to zero, the sequencer 128 turns to the next non-architected instruction. In another embodiment, the loop count value specified in a loop command is loaded in the loop counter for the first time the loop command is encountered, thereby eliminating the need to initialize the loop counter 3804 with the non-architectural initialization command. Therefore, the value of the loop counter 3804 will indicate the number of times the loop group of the non-architected program has yet to be executed. The value of the loop counter 3804 can be obtained by the architecture program through the loop count field 3914 of the status register 127, as shown in the subsequent thirty-ninth figure. In this way, the architecture program can determine the location of the data random access memory 122 and/or the weight random access memory 124 to read/write data according to the progress of the non-architecture program. In one embodiment, the sequencer includes three additional loop counters to match the nest loops in the non-architected program, and the values of the three loop counters are also readable by the status register 127. The loop instruction has a bit to indicate which of the four loop counters is available for use by the current loop instruction.

Sequencer 128 also includes an iteration count counter 3806. The iteration number counter 3806 is paired with non-architectural instructions, such as the multiply-accumulate instruction of address 2 in the fourth, nine, twenty- and twenty-sixth A pictures, and the maxwacc instruction of address 2 in the twenty-eighth figure. This will be referred to as the "execute" instruction. In the foregoing example, each execution instruction specifies an execution count 511, 511, 1023, 2, and 3, respectively. When sequencer 128 encounters an execution instruction that specifies a non-zero iteration count, sequencer 128 loads the iteration count counter 3806 with this specified value. In addition, sequencer 128 generates an appropriate micro-operation 3418 to control the thirty-fourth map. The logic within the pipeline stage 3401 is executed by the middle neural processing unit 126 and the iteration count counter 3806 is decremented. If the iteration count counter 3806 is greater than zero, the sequencer 128 will again generate an appropriate micro-operation 3418 to control the logic within the neural processing unit 126 and decrement the iteration count counter 3806. Sequencer 128 will continue to operate in this manner until the value of iteration count counter 3806 is zeroed. Therefore, the value of the iteration count counter 3806 is the number of operations to be executed in the non-architectural execution instruction (these operations are multiply and accumulate for the accumulated value and a data/weight text, take the maximum value, add the total operation, etc.). The value of the iteration count counter 3806 can be obtained by the architecture program through the iteration count field 3916 of the state register 127, as described in the subsequent thirty-ninth figure. In this way, the architecture program can determine the location of the data random access memory 122 and/or the weight random access memory 124 to read/write data according to the progress of the non-architecture program.

The thirty-ninth diagram is a block diagram showing the fields of the control and status register 127 of the neural network unit 121. These fields include an address 2602 including a weighted random access memory column that the neural processing unit 126 performs a non-architected program to write recently, and the neural processing unit 126 performs a weighted random access memory column that is recently read by the non-architected program. At address 2604, the neural processing unit 126 executes the address 2606 of the data random access memory column recently written by the non-architected program, and the neural processing unit 126 performs the address of the data random access memory column recently read by the non-architected program. 2608, as shown in the aforementioned twenty-sixth B. In addition, these fields include a neural network unit program counter 3912 field, a loop counter 3914 field, and an iteration count counter 3916 field. As described above, the architecture program can read the data in the state register 127 to the media temporary storage. The timer 118 and/or the general purpose register 116, for example, through the MFNN instruction 1500, reads the values including the neural network unit program counter 3912, the loop counter 3914 and the iteration count counter 3916. The value of the program counter field 3912 reflects the value of the program counter 3802 in the thirty-eighth figure. The value of the loop counter field 3914 reflects the value of the loop counter 3804. The value of the iteration count counter field 3916 reflects the value of the iteration count counter 3806. In one embodiment, the sequencer 128 updates the program counter field 3912, the loop counter field 3914 and the iteration count counter column each time the program counter 3802, loop counter 3804, or iteration number counter 3806 needs to be adjusted. The value of bit 3916, so that when the architecture program reads it, the value of these fields will be the current value. In another embodiment, when the neural network unit 121 executes the architectural instructions to read the status register 127, the neural network unit 121 only takes the value of the program counter 3802, the loop counter 3804 and the iteration count counter 3806 and It provides back architecture instructions (eg, to media register 118 or general purpose register 116).

It can be seen that the value of the field of the state register 127 of the thirty-ninth figure can be understood as the information of the progress of the execution of the non-architectural instruction by the neural network unit. Some specific aspects of the execution schedule of the non-architecture program, such as the program counter 3802 value, the loop counter 3804 value, the iteration count counter 3806 value, and the most recent read/write weight random access memory 124 address 125 field. 2602/2604, and the most recent read/write data random access memory 122 address 123 field 2606/2608, has been described in the previous section. The architecture program executing on the processor 100 can be slaved from the state register 127. Read the non-architected program progress values in Figure 39 and use this information to make decisions, such as through architectural instructions such as comparison and branch instructions. For example, the architecture program may determine the data/weight reading/writing queue for the data random access memory 122 and/or the weight random access memory 124 to control the data random access memory 122 or weight. The inflow and outflow of data from random access memory 124, especially for overlapping of large data sets and/or different non-architectural instructions. Examples of these decisions using the architecture program can be found in the previous and subsequent sections of this article.

For example, as described in the foregoing figure 26A, the architecture program sets the non-architecture program to write the result of the convolution operation back to the column above the convolution kernel 2402 in the data random access memory 122 (eg, above column 8). When the neural network unit 121 writes the result using the address of the recently written data random access memory 122 column 2606, the architecture program reads the result from the data random access memory 122.

In another example, as described in the foregoing section 26B, the architecture program uses the information from the field register 127 field of the thirty-eighth figure to confirm that the non-architecture program is the data array 2404 of the twenty-fourth figure. Divided into five 512 x 1600 data blocks to perform the progress of the convolution operation. The architecture program writes the first 512 x 1600 data block of the 2560 x 1600 data array into the weighted random access memory 124 and initiates the non-architected program with a loop count of 1600 and an output of the weighted random access memory 124 initialization. Listed as 0. When the neural network unit 121 executes the non-architecture program, the architecture program reads the status register 127 to confirm the most recent write column 2602 of the weighted random access memory 124, so that the architecture program can be read by the non-architect program. The result of the effective convolution operation, and use the next 512 after reading The x 1600 data block overwrites the result of this efficient convolution operation, so that after the neural network unit 121 completes execution of the non-architected program for the first 512 x 1600 data block, the processor 100 can update the non-architected immediately when necessary. The program starts the non-architected program again to execute the next 512 x 1600 data block.

In another example, assume that the architectural program causes the neural network unit 121 to perform a series of typical neural network multiply-accumulate start functions, wherein the weights are stored in the weighted random access memory 124 and the results are written back to the data randomly. The memory 122 is accessed. In this case, the architecture program reads the weight of one of the random access memories 124 and then does not read it. Thus, after the current weight has been read/used by the non-architect program, the architecture program can be used to begin to overwrite the weights on the weighted random access memory 124 to provide an example of the non-architectural program ( For example, the next neural network layer). In this case, the architecture program reads the state register 127 to obtain the address of the most recent read column 2604 of the weighted random access memory to determine its write of the new weight in the weighted random access memory 124. position.

In another example, assume that the architectural program knows that the non-architected program includes an execution instruction with a large number of iteration counts, such as the non-architectural multiply-accumulate instruction of address 2 in the twentieth diagram. In this case, the architecture program needs to know the iteration count of 3916 to know how many time-frequency cycles are needed to complete the non-architectural instruction to determine which of the two or more actions the architecture program will take next. . For example, if it takes a long time to complete execution, the architecture program will give up control to another architecture program, such as the operating system. similar It is assumed that the architecture program knows that the non-architecture program includes a loop group with a fairly large loop count, such as the non-architectural program of Figure 28. In this case, the architecture program will need to know the loop count 3914 to know how many time-frequency cycles are needed to complete the non-architectural instruction to determine which of the two or more actions to take next.

In another example, it is assumed that the architectural program causes the neural network unit 121 to perform a common source operation similar to that described in the twenty-seventh and twenty-eighthth embodiments, wherein the data to be co-sourced is stored in the weighted random access memory 124. The result is written back to the weighted random access memory 124. However, unlike the examples of the twenty-seventh and twenty-eighth diagrams, it is assumed that the results of this example are written back to the top 400 columns of the weighted random access memory 124, such as columns 1600 through 1999. In this case, after the non-architecture program finishes reading the data of the weighted random access memory 124 of the four common sources, the non-architected program will not read it again. Therefore, once the current four columns of data have been read/used by the non-architected program, the architecture program can be used to start the new data (such as the weight of the next paradigm of the non-architected program, for example, for example, the typical execution of the data. The multiplication accumulates the non-architectural program of the function operation to overwrite the data of the weighted random access memory 124. In this case, the architecture program reads the state register 127 to obtain the address of the most recent read column 2604 of the weighted random access memory to determine the location of the new weight rewrite write weight random access memory 124. .

Time recurrent neural network acceleration

Traditional feedforward neural networks do not have memory that was previously entered by the storage network. The feedforward neural network is usually used to perform multiple inputs that are input to the network over time in the task, and each of the inputs is independent. It is such a task. In contrast, time-recurrent neural networks often help to perform tasks that are important in the order in which tasks are input to the neural network over time. (The order here is often referred to as the time step.) Thus, the time recurrent neural network includes a conceptual memory or internal state that loads the information generated by the network in response to calculations performed by previous inputs in the sequence. The output of the time recurrent neural network is associated with the input of this internal state and the next time step. The following tasks, such as speech recognition, language modeling, text generation, language translation, image description generation, and some forms of handwriting recognition, are good examples of time recursive neural networks that can perform well.

Examples of three conventional time recurrent neural networks are the Elman time recurrent neural network, the Jordan time recurrent neural network and the long and short term memory (LSTM) neural network. The Elman time recurrent neural network contains content nodes to remember the hidden layer state of the time recurrent neural network in the current time step, which will be used as input to the hidden layer in the next time step. The Jordan time recurrent neural network is similar to the Elman time recurrent neural network except that the content nodes remember the output layer state of the time recurrent neural network rather than the hidden layer state. Long- and short-term memory neural networks include long-term and short-term memory layers composed of long- and short-term memory cells. Each long-term and short-term memory cell has one of the current time steps of the current state with a current output, and one of the new or subsequent time steps of the new state with a new output. The long-term and short-term memory cells include an input gate and an output gate, and a forgetting gate, which can cause the neuron to lose its state of memory. These three time recurrent neural networks are described in more detail in subsequent chapters.

As described herein, for a time recurrent neural network, such as the Elman or Jordan time recurrent neural network, the neural network unit uses a time step each time to perform a set of input layer node values and perform the necessary calculations. It propagates through a time recurrent neural network to produce output layer node values as well as hidden layer and content layer node values. Thus, the input layer node value is associated with the time step of computing the hidden, outputting the value with the content layer node; while the hidden, output and content layer node values are associated with the time step of generating these node values. The input layer node values are samples of systems simulated by the time recurrent neural network, such as images, speech samples, and snapshots of commercial market data. For long- and short-term memory neural networks, each execution of the neural network unit uses a time step to obtain a set of memory cell input values and perform the necessary calculations to generate memory cell output values (as well as memory cell states and input gates, The forgetting gate and the output gate value) can also be understood as the memory cell input value through the long-term and short-term memory layer memory. Therefore, the memory cell input value is associated with the time step of calculating the memory cell state and the input gate, the forgetting gate and the output gate value; and the memory cell state and the input gate, the forgetting gate and the output gate value are associated with the time at which these node values are generated. step.

The content layer node value, also known as the state node, is the state value of the neural network based on the input layer node values associated with the previous time step, and not only the input layer node values associated with the current time step. The calculations performed by the neural network unit for the time step (eg, for the hidden layer node value calculations of the Elman or Jordan time recurrent neural network) are a function of the content layer node values generated by the previous time step. Therefore, the network status value (content node value) at the beginning of the time step Will affect the output layer node value generated during this time step. In addition, the network status value at the end of the time step is affected by the input node value for this time step and the network status value at the beginning of the time step. Similarly, for long- and short-term memory cells, the memory cell state value is associated with the memory cell input value of the previous time step, rather than just the memory cell input value of the current time step. Since the calculation of the neural network unit for the time step execution (eg, the calculation of the next memory cell state) is a function of the memory cell state value generated by the previous time step, the network state value (memory cell state value) at the beginning of the time step will The memory cell output value generated in this time step is affected, and the network state value at the end of this time step is affected by the memory cell input value and the previous network state value of this time step.

The fortieth figure is a block diagram showing an example of an Elman time recurrent neural network. The Elman time recurrent neural network of the fortieth figure includes input layer nodes, or neurons, denoted as D0, D1 to Dn, collectively referred to as multiple input layer nodes D and individually referred to as input layer nodes D; hidden layer nodes/ Neurons, labeled Z0, Z1 to Zn, collectively referred to as multiple hidden layer nodes Z and individually referred to as hidden layer nodes Z; output layer nodes/neurons, labeled Y0, Y1 to Yn, collectively referred to as multiple outputs Layer nodes Y are collectively referred to as output layer nodes Y; and content layer nodes/neurons, designated C0, C1 through Cn, collectively referred to as multiple content layer nodes C and individually referred to as content layer nodes C. In the example of the Elman time recurrent neural network of the fortieth figure, each hidden layer node Z has an input connected to the output of each input layer node D, and has an input connected to the output of each content layer node C; each output Layer node Y has an input link The output to each hidden layer node Z; and each content layer node C has an input coupled to an output of a corresponding hidden layer node Z.

In many ways, the operation of the Elman time recurrent neural network is similar to the traditional feed-forward artificial neural network. That is to say, for a given node, each input link of this node will have an associated weight; the value received by the node in an input link will be multiplied by the associated weight to generate a product; this node will Adding the product associated with all input joins produces a total (this offset may also contain an offset); in general, a total is also executed for this total to generate the output value of the node. Sometimes referred to as the startup value of this node. For traditional feedforward networks, data always flows in the direction from the input layer to the output layer. That is, the input layer provides a value to the hidden layer (usually there are multiple hidden layers), while the hidden layer produces its output value to the output layer, and the output layer produces an output that can be fetched.

However, unlike the traditional feedforward network, the Elman time recurrent neural network also includes some feedback links, that is, the connection from the hidden layer node Z to the content layer node C in the fortieth figure. The operation of the Elman time recurrent neural network is as follows. When the input layer node D provides an input value to the hidden layer node Z in a new time step, the content node C provides a value to the hidden layer Z, which is the hidden layer node Z. The output value in response to the previous input, which is the current time step. In this sense, the content node C of the Elman time recursive neural network is a memory based on the input values of the previous time steps. The operational examples of the neural network unit 121 that performs the calculation associated with the Elman time recurrent neural network associated with the fortieth map will be described in the forty-first and forty-second figures.

To illustrate the present invention, an Elman time recurrent neural network is a time recurrent neural network comprising at least one input node layer, one hidden node layer, one output node layer and one content node layer. For a given time step, the content node layer stores the results of the hidden node layer generated in the previous time step and fed back to the content node layer. The result of this feedback to the content layer can be the result of the execution of the startup function or the result of the hidden node layer performing the accumulation operation without executing the startup function.

The forty-first diagram is a block diagram showing the data random access memory 122 and the weight of the neural network unit 121 when the neural network unit 121 performs the calculation associated with the Elman time recurrent neural network of the fortieth map. An example of data configuration within random access memory 124. In the example of the 41st graph, it is assumed that the Elman time recurrent neural network of the fortieth graph has 512 input nodes D, 512 hidden nodes Z, 512 content nodes C, and 512 output nodes Y. In addition, it is also assumed that the Elman time recurrent neural network is fully connected, that is, all 512 input nodes D are connected to each hidden node Z as an input, and all 512 content nodes C are connected to each hidden node Z as an input, and all 512 The hidden nodes Z are each connected to each output node Y as an input. Moreover, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, such as a widened configuration. Finally, this example assumes that the weights associated with the links from content node C to hidden node Z are all a value of 1, so that these weight values are not required to be stored.

As shown in the figure, the lower 512 columns (columns 0 to 511) of the weighted random access memory 124 are loaded with weight values associated with the connections between the input node D and the hidden node Z. More precisely, as shown in the figure, Column 0 loads the weight associated with the input link from input node D0 to hidden node Z, ie, text 0 loads the weight associated with the link between input node D0 and hidden node Z0, and text 1 is loaded associated with the input node. The weight of the connection between D0 and the hidden node Z1, the text 2 will load the weight associated with the connection between the input node D0 and the hidden node Z2, and so on, the text 511 will be loaded with the link between the input node D0 and the hidden node Z511. The weight of the column 1 is loaded with the weight associated with the input link from the input node D1 to the hidden node Z, that is, the text 0 loads the weight associated with the link between the input node D1 and the hidden node Z0, and the text 1 loads the association. For the weight of the connection between the input node D1 and the hidden node Z1, the text 2 will load the weight associated with the connection between the input node D1 and the hidden node Z2, and so on, the text 511 will be loaded with the input node D1 and the hidden node Z511. The weight of the link; until column 511, the column 511 is loaded with the weight associated with the input link from the input node D511 to the hidden node Z, that is, the text 0 is loaded associated with the input The weight of the connection between the node D511 and the hidden node Z0, the character 1 will load the weight associated with the connection between the input node D511 and the hidden node Z1, and the character 2 will load the weight associated with the connection between the input node D511 and the hidden node Z2. And so on, the text 511 will load the weight associated with the connection between the input node D511 and the hidden node Z511. This configuration and use is similar to the embodiment described above corresponding to Figures 4-6A.

As shown in the figure, the subsequent 512 columns of weight random memory 124 (columns 512 through 1023) load the weight associated with the association between hidden node Z and output node Y in a similar manner.

Data random access memory 122 is loaded with Elman The inter-recurrent neural network node values are used for a series of time steps. Further, the data random access memory 122 provides a node value for a given time step in three columns as a group load. As shown in the figure, taking a data random access memory 122 having 64 columns as an example, the data random access memory 122 can load node values for use in 20 different time steps. In the example of the 41st graph, columns 0 through 2 are loaded with node values for time step 0, columns 3 through 5 are loaded with node values for time step 1, and so on, columns 57 through 59 are loaded for time steps. 19 node value used. The first column in each group is loaded with the value of input node D for this time step. The second column in each group loads the value of the hidden node Z for this time step. The third column in each group loads the value of the output node Y for this time step. As shown in the figure, each row of data random access memory 122 loads the node value of its corresponding neuron or neural processing unit 126. That is to say, row 0 is loaded with node values associated with nodes D0, Z0 and Y0, and its calculation is performed by neural processing unit 0; row 1 is loaded with node values associated with nodes D1, Z1 and Y1, and the calculation is Executed by the neural processing unit 1; and so on, the row 511 is loaded with node values associated with nodes D511, Z511 and Y511, the calculation of which is performed by the neural processing unit 511, which in the subsequent corresponds to the forty-second map There will be more detailed instructions.

As indicated in the 41st figure, for a given time step, the value of the hidden node Z in the second column of each group of three columns of memory will be the value of the content node C of the next time step. That is, the value of the node Z calculated and written by the neural processing unit 126 in a time step becomes the value of the node C used by the neural processing unit 126 to calculate the value of the node Z in the next time step ( Together with this Enter the value of node D for the next time step). The initial value of the content node C (the value of the node C used to calculate the value of the node Z in the column 1 at time step 0) is assumed to be zero. This will be explained in more detail in the subsequent sections of the non-architected program corresponding to the 42nd figure.

Preferably, the value of the input node D (column 0, 3 in the example of the 41st graph, and so on to the value of column 57) is written by the architecture program executed by the processor 100 through the MTNN instruction 1400 / The data random access memory 122 is populated and is read/used by a non-architected program executed by the neural network unit 121, such as the non-architected program of the forty-second graph. Conversely, the value of the hidden/output node Z/Y (columns 1 and 2, 4 and 5 in the example of the 41st graph, and so on to the values of columns 58 and 59) are performed by the neural network. The non-architectural program of unit 121 writes/fills data random access memory 122 and is read/used by MFNN instruction 1500 by an architectural program executing on processor 100. The example in the 41st figure assumes that the architecture program performs the following steps: (1) For 20 different time steps, the value of the input node D is filled in the data random access memory 122 (column 0, 3, This type is pushed to column 57); (2) to start the non-architecture program of the 42nd chart; (3) to detect whether the non-architecture program is executed; (4) to read the output node Y from the data random access memory 122. The values (columns 2, 5, and so on to column 59); and (5) repeat steps (1) through (4) several times until the task is completed, such as the calculation required to identify the phone user's words.

In another implementation, the architecture program performs the following steps: (1) for a single time step, filling the data random access memory 122 (such as column 0) with the value of the input node D; (2) starting the non-architectural Cheng (A modified version of one of the non-architectural programs in Figure 42 does not require loops and only accesses a single set of three columns of data stored in memory 122); (3) detects whether the non-architected program has been executed. (4) reading the value of the output node Y from the data random access memory 122 (e.g., column 2); and (5) repeating steps (1) through (4) several times until the task is completed. Which of the two methods is preferred depends on the sampling method of the input value of the time recurrent neural network. For example, if this task allows sampling of inputs (eg, about 20 time steps) and performing calculations in multiple time steps, the first approach is ideal because it may result in more computational resource efficiency and / Or better performance, however, if this task only allows sampling in a single time step, the second method is required.

The third embodiment is similar to the second method described above. However, unlike the second method, a single group of three columns of data random access memory 122 is used. In this manner, the non-architectural program uses multiple sets of three columns of memory, that is, Each time step uses a different set of three columns of memory, which is similar to the first. In this third embodiment, preferably, the architecture program includes a step before step (2), in which the architecture program updates the non-architecture program before starting, for example, the instruction of the address 1 is The data random access memory 122 column is updated to point to the next set of three columns of memory.

The forty-second graph is a table showing a program stored in the program memory 129 of the neural network unit 121. The program is executed by the neural network unit 121, and the data and weights are used according to the configuration of the 41st graph. To achieve the Elman time recurrent neural network. Picture 42 (and forty-fifth, forty-eight, fifty-one, fifty-four and fifty-seven) Some of the instructions in the non-architecture program are as described above (for example, multiply accumulate (MULT-ACCUM), loopback (LOOP), initialize (INITIALIZE) instructions). The following paragraphs assume that these instructions are consistent with the above description unless otherwise stated. .

The sample program in Figure 42 contains 13 non-architected instructions located at addresses 0 through 12. The instruction of address 0 (INITIALIZE NPU, LOOPCNT = 20) clears the accumulator 202 and initializes the loop counter 3804 to a value of 20 to perform 20 loop groups (instructions of addresses 4 through 11). Preferably, the initialization command also causes the neural network unit 121 to be in a wide configuration. Thus, the neural network unit 121 is configured as 512 neural processing units 126. As described in the subsequent sections, during the execution of the instructions of addresses 1 through 3 and addresses 7 through 11, the 512 neural processing units 126 operate as 512 corresponding hidden layer nodes Z, and at address 4 During the execution of the instructions up to 6, the 512 neural processing units 126 operate as 512 corresponding output layer nodes Y.

The instructions of addresses 1 through 3 are not part of the program's loop group and will only be executed once. These instructions calculate the initial value of the hidden layer node Z and write it to column 1 of the data random access memory 122 for the first execution of the instructions of addresses 4 through 6 to calculate the first time step (time step) 0) Output layer node Y. In addition, the values of the hidden layer nodes Z calculated by the instructions of the addresses 1 to 3 and written into the column 1 of the data random access memory 122 become the value of the content layer node C for the addresses of the addresses 7 and 8. The use is performed once to calculate the value of the hidden layer node Z for use in the second time step (time step 1).

In the execution of the instructions of addresses 1 and 2, these 512 Each of the neural processing units 126 performs 512 multiplication operations, multiplying the 512 input node D values of the data random access memory 122 column 0 by the weight 0 of the weighted random access memory 124 to The weight of the row of the neural processing unit 126 is corresponding to 511 to generate 512 multiply accumulators for the accumulator 202 of the corresponding neural processing unit 126. During the execution of the instruction of address 3, the values of the 512 accumulators 202 of the 512 neural processing units are passed and written to column 1 of the data random access memory 122. That is to say, the output instruction of address 3 writes the value of the accumulator 202 of each of the 512 neural processing units into the data random access memory 122, which is the initial hidden layer. The Z value, then, this instruction clears the accumulator 202.

The operations performed by the instructions of addresses 1 through 2 of the non-architected program of the 42nd graph are similar to those performed by the instructions of addresses 1 through 2 of the non-architectural instructions of the fourth figure. Further, the instruction of address 1 (MULT_ACCUM DR ROW 0) indicates that each of the 512 neural processing units 126 reads the corresponding text of column 0 of the data random access memory 122 into it. The scratchpad 208 reads the corresponding text of the column 0 of the weight random access memory 124 into its multiplex register 705, multiplies the data text by the weight text to generate a product, and adds the product to the accumulator 202. The instruction of address 2 (MULT-ACCUM ROTATE, WR ROW+1, COUNT=511) indicates that each of the 512 neural processing units 126 transfers the text from the adjacent neural processing unit 126 to its multiplex The buffer 208 (using a 512-word rotator formed by the collective operation of the 512 multiplex registers 208 of the neural network unit 121, these registers are the instructions of the address 1 to indicate the random access memory of the data. The register of the volume 122 is read into the register, and the corresponding text in the lower column of the weight random access memory 124 is read into the multiplex register 705, and the data text is multiplied by the weight text to generate a product. The product is added to the accumulator 202, and the aforementioned operation is performed 511 times.

In addition, the single non-architectural output instruction (OUTPUT PASSTHRU, DR OUT ROW 1, CLR ACC) of address 3 in Figure 42 will write the operation of the start function instruction and the write output of addresses 3 and 4 in the fourth figure. The instructions are merged (although the program of the 42nd graph transfers the value of the accumulator 202, and the program of the fourth graph executes a start function for the accumulator 202 value). That is, in the program of the 42nd graph, the start function of the value of the accumulator 202, if any, is specified in the output instruction (also specified in the output instructions of addresses 6 and 11), and Not shown in the program in Figure 4 is specified in a different non-architectural start function instruction. Another embodiment of the non-architectural program of the fourth figure (and the twentieth, twenty-sixth and twenty-eighth) is also about to start the operation of the function instruction and the write output instruction (such as the address of the fourth figure 3 It is also within the scope of the present invention to incorporate 4) into a single non-architectural output instruction as shown in FIG. The example in Figure 42 assumes that the node of the hidden layer (Z) does not perform a start function on the accumulator value. However, the embodiment in which the hidden layer (Z) performs the start function on the accumulator value is also within the scope of the present invention. These embodiments can perform operations using the instructions of addresses 3 and 11, such as S-type, hyperbolic tangent, correction function, etc. .

Instructions that are compared to addresses 1 through 3 are only executed once, and instructions with addresses 4 through 11 are placed in the program loop and are executed a number of times, as specified by the loop count (for example, 20). Address 7 to 11 The first nine executions of the instruction are used to calculate the value of the hidden layer node Z and write it to the data random access memory 122 for the second to twenty executions of the instructions of addresses 4 through 6 to calculate the remaining time steps. Output layer node Y (time steps 1 to 19). (The last/twentieth execution of the instructions of addresses 7 through 11 calculates the value of the hidden layer node Z and writes it to column 61 of the data random access memory 122, however, these values are not used.)

In the first execution of the instructions of addresses 4 and 5 (MULT-ACCUM DR ROW+1, WR ROW 512 and MULT-ACCUM ROTATE, WR ROW+1, COUNT=511) (corresponding to time step 0), this Each of the 512 neural processing units 126 performs 512 multiplication operations to randomly store the values of the 512 hidden nodes Z of the column 1 of the memory 122 (these values are from addresses 1 to 3). The single execution of the instruction is generated and written. The weights of the rows corresponding to the neural processing unit 126 in the columns 512 to 1023 of the weight random access memory 124 are generated to generate 512 multiplication accumulations for the corresponding neural processing unit 126. Accumulator 202. In the first execution of the instruction of address 6 (OUTPUT ACTIVATION FUNCTION, DR OUT ROW+1, CLR ACC), a start function (eg S-type, hyperbolic tangent, correction function) is executed for the 512 accumulated values. To calculate the value of the output layer node Y, the execution result is written to column 2 of the data random access memory 122.

In the second execution of the instructions of addresses 4 and 5 (corresponding to time step 1), each of the 512 neural processing units 126 performs 512 multiplication operations to access the data random access memory. The value of 512 hidden nodes Z of column 4 of 122 (these values are generated and written by the first execution of the instructions of addresses 7 to 11) multiplied by the weight The ranks of the rows 512 through 1023 of the random access memory 124 correspond to the rows of the neural processing unit 126 to generate 512 multiply accumulates for the accumulator 202 of the corresponding neural processing unit 126, and the instruction at address 6 In the second execution, a start function is executed for the 512 accumulated values to calculate the value of the output layer node Y, and the result is written in the data random access memory 122 column 5; in the address 4 and 5 instructions In the third execution (corresponding to time step 2), each of the 512 neural processing units 126 performs 512 multiplication operations, and 512 hidden nodes of the data random access memory 122. The value of Z (these values are generated and written by the second execution of the instructions of addresses 7 through 11) is multiplied by the row 512 to 1023 of the weighted random access memory 124 corresponding to the neural processing unit 126. The weights are generated to generate 512 multiply accumulators added to the accumulator 202 of the corresponding neural processing unit 126, and in the third execution of the instruction of the address 6, a start function is executed for the 512 accumulated values to calculate the output layer node Y value, this result is Write data column 8 of data random access memory 122; and so on, in the twentieth execution of instructions of addresses 4 and 5 (corresponding to time step 19), each of the 512 neural processing units 126 The neural processing unit 126 performs 512 multiplication operations to store the values of the 512 hidden nodes Z of the column 58 of the data random access memory 122 (these values are generated by the nineteenth execution of the instructions of addresses 7 through 11). Multiplied by the weight of the row corresponding to the neural processing unit 126 in the columns 512 to 1023 of the weighted random access memory 124 to generate 512 multiply accumulates to the accumulator 202 of the corresponding neural processing unit 126, In the twentieth execution of the instruction of address 6, a start function is executed for the 512 accumulated values to calculate the value of the output layer node Y, and the execution result is written. A column 59 of random access memory 122.

In the first execution of the instructions of addresses 7 and 8, each of the 512 neural processing units 126 accumulates the values of the 512 content nodes C of the data array 1 of the memory 122. Accumulator 202, these values are generated by a single execution of the instructions of addresses 1 through 3. Further, the instruction of address 7 (ADD_D_ACC DR ROW+0) will instruct each of the 512 neural processing units 126 to store the data random access memory 122 in the current column (in the first execution process) The corresponding text in column 0) is read into its multiplex register 208 and the text is added to accumulator 202. The instruction of address 8 (ADD_D_ACC ROTATE, COUNT=511) indicates that each of the 512 neural processing units 126 transfers the text from the adjacent neural processing unit 126 to its multiplex register 208 (using The 512 multiplex registers 208 of the neural network unit 121 collectively operate to form a 512-word rotator, and the multiplex registers are the instructions of the address 7 to read the data random access memory 122. The register is added to the accumulator 202 and the aforementioned operation is performed 511 times.

In the second execution of the instructions of addresses 7 and 8, each of the 512 neural processing units 126 will accumulate the values of the 512 content nodes C of the data 4 of the memory 122. To its accumulator 202, these values are generated and written by the first execution of the instructions of addresses 9 through 11; in the third execution of the instructions of addresses 7 and 8, the 512 neural processing units 126 Each of the neural processing units 126 will accumulate the values of the 512 content nodes C of the column 7 of the data random access memory 122 to its accumulator 202, which are represented by the address 9 The second execution of the instruction to 11 is generated and written; and so on, in the twentieth execution of the instructions of addresses 7 and 8, each of the 512 neural processing units 126 will The data is randomly summed to the value of 512 content nodes C of column 58 of memory 122 to its accumulator 202, which are generated and written by the nineteenth execution of the instructions of addresses 9 through 11.

As described above, the example of the forty-second diagram assumes that the weight associated with the link of the content node C to the hidden layer node Z has a value of one. However, in another embodiment, the links within the Elman time recurrent neural network have non-zero weight values that are placed in the weighted random access memory 124 prior to execution of the program in the forty-second graph. (For example, columns 1024 to 1535), the program instructions for address 7 are MULT-ACCUM DR ROW+0, WR ROW 1024, and the program instructions for address 8 are MULT-ACCUM ROTATE, WR ROW+1, COUNT=511. Preferably, the instruction of address 8 does not access the weighted random access memory 124, but the instruction of the rotated address 7 reads the value of the multiplexed register 705 from the weighted random access memory 124. The bandwidth is not accessed during the time-frequency period of the 511 execution address 8 instructions to reserve more bandwidth for use by the architectural program access weight random access memory 124.

In the first execution of the instructions of addresses 9 and 10 (MULT-ACCUM DR ROW+2, WR ROW 0 and MULT-ACCUM ROTATE, WR ROW+1, COUNT=511) (corresponding to time step 1), this Each of the 512 neural processing units 126 performs 512 multiplication operations to randomize the memory 122. The value of 512 input nodes D of column 3 is multiplied by the weight of the row corresponding to the neural processing unit 126 in columns 0 to 511 of the weighted random access memory 124 to generate 512 products, together with the instructions of addresses 7 and 8. The accumulation operation performed on the 512 content node C values is accumulated in the accumulator 202 of the corresponding neural processing unit 126 to calculate the value of the hidden layer node Z, and the instruction at address 11 (OUTPUT PASSTHRU, DR OUT ROW+2, In the first execution of CLR ACC), the 512 accumulators 202 of the 512 neural processing units 126 are passed and written to column 4 of the data random access memory 122, and the accumulator 202 is cleared; In the second execution of the instructions of addresses 9 and 10 (corresponding to time step 2), each of the 512 neural processing units 126 performs 512 multiplication operations to access the data random access memory 122. The value of 512 input nodes D of column 6 is multiplied by the weight of the row corresponding to the neural processing unit 126 in columns 0 to 511 of the weighted random access memory 124 to generate 512 products, together with the instructions of addresses 7 and 8. Cumulative load performed on 512 content node C values The accumulator 202 is added to the corresponding neural processing unit 126 to calculate the value of the hidden layer node Z. In the second execution of the instruction of the address 11, the 512 accumulators 202 of the 512 neural processing units 126 are numerically Passing and writing column 7 of data random access memory 122, and accumulator 202 is cleared; and so on, in the nineteenth execution of instructions of addresses 9 and 10 (corresponding to time step 19) Each of the 512 neural processing units 126 performs 512 multiplication operations, multiplying the value of the 512 input nodes D of the column 57 of the data random access memory 122 by the weight random access memory 124. The weights of the rows from 0 to 511 corresponding to the neural processing unit 126 are generated to generate 512 multiplications. The accumulation, together with the instructions of addresses 7 and 8, for the 512 content node C values, is accumulated in the accumulator 202 of the corresponding neural processing unit 126 to calculate the value of the hidden layer node Z, and at address 11 In the nineteenth execution of the instruction, the 512 accumulators 202 of the 512 neural processing units 126 are passed and written to the column 58 of the data random access memory 122, and the accumulator 202 is cleared. As previously mentioned, the value of the hidden layer node Z generated and written in the twentieth execution of the instructions of addresses 9 and 10 is not used.

The instruction of address 12 (LOOP 4) decrements the loop counter 3804 and returns to the instruction of address 4 if the new loop counter 3804 has a value greater than zero.

A forty-third figure is a block diagram showing an example of a Jordan time recurrent neural network. The 43rd chart of the Jordan time recurrent neural network is similar to the 40th Elman time recurrent neural network with input layer nodes/neuron D, hidden layer nodes/neurons Z, output layer nodes/neurons Y , with content layer nodes / neurons C. However, in the Jordan time recurrent neural network of the 43rd graph, the content layer node C uses the output feedback from its corresponding output layer node Y as its input link instead of the Elman time recursion as in the fortieth chart. The neural network is the output from the hidden layer node Z as its input link.

To illustrate the present invention, the Jordan Time Recurrent Neural Network is a time recurrent neural network comprising at least one input node layer, one hidden node layer, one output node layer and one content node layer. At the beginning of a given time step, the content node layer stores the node generated by the output node layer in the previous time step and fed back to the content node layer. fruit. The result of this feedback to the content layer can be the result of the startup function or the result of the output node performing the accumulation operation without executing the startup function.

The forty-fourth figure is a block diagram showing the data random access memory 122 of the neural network unit 121 when the neural network unit 121 performs the calculation of the Jordan time recurrent neural network associated with the forty-third figure. An example of data configuration within weight random access memory 124. In the example of the forty-fourth figure, it is assumed that the Jordan time recurrent neural network of the forty-third figure has 512 input nodes D, 512 hidden nodes Z, 512 content nodes C, and 512 output nodes Y. In addition, it is also assumed that the Jordan time recurrent neural network is fully connected, that is, all 512 input nodes D are connected to each hidden node Z as an input, and all 512 content nodes C are connected to each hidden node Z as an input, and all 512 The hidden nodes Z are each connected to each output node Y as an input. The example of the Jordan time recurrent neural network of the forty-fourth figure applies a start function to the value of the accumulator 202 to generate the value of the output layer node Y. However, this example assumes that the accumulation function will be applied before the start function. The value of the 202 is passed to the content layer node C instead of the true output layer node Y value. In addition, the neural network unit 121 is provided with 512 neural processing units 126, or neurons, for example, in a wide configuration. Finally, this example assumes that the weights associated with the links from content node C to hidden node Z all have a value of one; thus, it is not necessary to store these as a weight value.

As in the example of the 41st graph, as shown in the figure, the lower 512 columns (columns 0 to 511) of the weight random access memory 124 load the weight value associated with the link between the input node D and the hidden node Z. And weighting the subsequent 512 columns of random access memory 124 (columns 512 through 1023) The weight value associated with the link between the hidden node Z and the output node Y is loaded.

The data random access memory 122 is loaded with Jordan time recursive neural network node values for use in a series of time steps similar to the example of the 41st graph; however, the example of the forty-fourth graph is a set of four The column's memory load provides the node value for a given time step. As shown in the figure, in an embodiment having 64 columns of data random access memory 122, data random access memory 122 can carry node values required for 15 different time steps. In the example of the forty-fourth figure, columns 0 to 3 are loaded with node values for time step 0, columns 4 to 7 are loaded with node values for time step 1, and so on, columns 60 to 63 are loaded. The node value used for time step 15. The first column of the four columns of memory is loaded with the value of input node D for this time step. The second column of the four columns of memory loads the value of the hidden node Z for this time step. The third column of the four columns of memory is loaded with the value of the content node C of this time step. The fourth column of the four columns of memory is the value of the output node Y that is loaded for this time step. As shown in the figure, each row of data random access memory 122 loads the node value of its corresponding neuron or neural processing unit 126. That is to say, row 0 is loaded with node values associated with nodes D0, Z0, C0 and Y0, and its calculation is performed by neural processing unit 0; row 1 is loaded with node values associated with nodes D1, Z1, C1 and Y1, The calculation is performed by the neural processing unit 1; and so on, the row 511 is loaded with node values associated with the nodes D511, Z511, C511 and Y511, the calculation of which is performed by the neural processing unit 511. This section will be described in more detail later in the corresponding figure 44.

The value of the content node C for a given time step in Figure 44 is generated during this time step and is entered as the next time step. That is, the value of the node C calculated and written by the neural processing unit 126 during this time step becomes the value of the node C used by the neural processing unit 126 to calculate the value of the node Z in the next time step ( Together with the value of input node D for this next time step). The initial value of the content node C (i.e., the value of the node C used to calculate the value of the node 1 node Z at time step 0) is assumed to be zero. This section will be described in more detail in the subsequent sections of the non-architected program corresponding to the 45th.

As described in the foregoing 41st, preferably, the value of the input node D (columns 0, 4 in the example of the forty-fourth graph, and so on, to the value of the column 60) is executed by the processor 100. The architecture program writes/fills the data random access memory 122 through the MTNN command 1400 and is read/used by a non-architected program executed by the neural network unit 121, such as the non-architected program of the forty-fifth figure. Conversely, the value of the hidden node Z/content node C/output node Y (in the example of the forty-fourth figure is column 1/2/3, 5/6/7, and so on, to column 61/62/63) The value is written/filled into the data random access memory 122 by the non-architectural program executed by the neural network unit 121, and is read/used by the architecture program executed by the processor 100 through the MFNN instruction 1500. The example in the forty-fourth figure assumes that the architecture program performs the following steps: (1) For 15 different time steps, the value of the input node D is filled in the data random access memory 122 (column 0, 4, This type is pushed to column 60); (2) to start the non-architectural program of the 45th figure; (3) to detect whether the non-architected program is executed (4) reading the value of the output node Y from the data random access memory 122 (column 3, 7, and so on to column 63); and (5) repeating steps (1) through (4) several times until Complete tasks, such as the calculations needed to identify the phone user's words.

In another implementation, the architecture program performs the following steps: (1) for a single time step, filling the data random access memory 122 (such as column 0) with the value of the input node D; (2) starting the non-architectural The program (the revised version of the non-architected program in the forty-fifth figure does not require loops, and only accesses a single group of four columns of data storage memory 122); (3) detects whether the non-architected program is executed. (4) reading the value of the output node Y from the data random access memory 122 (e.g., column 3); and (5) repeating steps (1) through (4) several times until the task is completed. Which of the two methods is preferred depends on the sampling method of the input value of the time recurrent neural network. For example, if this task allows sampling of inputs (for example, about 15 time steps) and performing calculations in multiple time steps, the first method is ideal because it can bring more computing resource efficiency and / or better performance, however, if this task only allows sampling in a single time step, you need to use the second method.

The third embodiment is similar to the second method described above. However, unlike the second method, a single group of four data random access memory 122 columns is used. The non-architectural program in this manner uses multiple sets of four columns of memory, that is, Different sets of four columns of memory are used at various time steps, which is similar to the first way. In this third embodiment, preferably, the architecture program includes a step before step (2), in which the architecture program updates the non-architecture program before starting, for example, the instruction of address 1 The data random access memory 122 column is updated to point to the next set of four columns of memory.

The forty-fifth figure is a table showing a program stored in the program memory 129 of the neural network unit 121, which is executed by the neural network unit 121, and uses data and weights according to the configuration of the forty-fourth figure. To achieve the Jordan time recurrent neural network. The non-architectural program of the 45th figure is similar to the non-architectural program of the 42nd figure. The difference between the two can be referred to the description of the relevant section of this article.

The sample program of Figure 45 includes 14 non-architected instructions located at addresses 0 through 13. The instruction of address 0 is an initialization instruction to clear accumulator 202 and initialize loop counter 3804 to a value of 15 to perform 15 loop groups (instructions of addresses 4 through 12). Preferably, this initialization command causes the neural network unit 121 to be in a wide configuration and configured as 512 neural processing units 126. As described herein, during the execution of instructions for addresses 1 through 3 and addresses 8 through 12, the 512 neural processing units 126 correspond to and operate as 512 hidden layer nodes Z, at addresses 4, 5 During execution of the instruction with 7, these 512 neural processing units 126 correspond and operate as 512 output layer nodes Y.

The instructions of addresses 1 through 5 and address 7 are identical to the instructions of addresses 1 through 6 in the forty-second figure and have the same function. The instructions of address 1 to 3 calculate the initial value of the hidden layer node Z and write it to the column 1 of the data random access memory 122 for the first execution of the instructions of the addresses 4, 5 and 7 to calculate The output layer node Y of the first time step (time step 0).

During the first execution of the output instruction of address 6, the number of accumulators 202 generated by the accumulation of the instructions of addresses 4 and 5 The values (these values will be used by the output instruction of address 7 to calculate and write to the output layer node Y) will be passed and written to column 2 of the data random access memory 122. These values are the first. The content layer node C value generated in the time step (time step 0) is used in the second time step (time step 1); during the second execution of the output instruction of address 6, the 512 bits are used The instructions of addresses 4 and 5 accumulate the value of the accumulator 202 generated (these values will be used by the output instruction of address 7 to calculate and write the value of the output layer node Y) and will be passed to the data random access. Columns 6 of memory 122, these values are the content layer node C values generated in the second time step (time step 1) and used in the third time step (time step 2); and so on, at address 6 During the fifteenth execution of the output instruction, the 512 accumulators 202 are incremented by the instructions of addresses 4 and 5 (these values are then used by the output instruction of address 7 to calculate and write The value of the output layer node Y) will be passed and the data will be stored randomly. Taking the column 58 of the memory 122, these values are the value of the content layer node C generated in the fifteenth time step (time step 14) (and read by the instruction of address 8, but will not be used).

The instructions of addresses 8 through 12 are substantially identical to the instructions of addresses 7 through 11 in the forty-second figure and have the same function, with only one difference point. The difference is that the instruction of address 8 (ADD_D_ACC DR ROW+1) in the forty-fifth figure increases the number of columns of data random access memory 122 by one, and the instruction of address 7 in the forty-second figure. (ADD_D_ACC DR ROW+0) increments the number of columns of data random access memory 122 by zero. Data random access memory The configuration of the data in 122 is different. In particular, the configuration of the four columns in the forty-fourth figure includes a separate column for the value of the content layer node C (such as columns 2, 6, 10, etc.), and the forty-first The configuration of the three columns in the figure does not have this independent column, but the value of the content layer node C and the value of the hidden layer node Z share the same column (such as columns 1, 4, 7, etc.). Fifteen executions of the instructions of addresses 8 through 12 calculate the value of the hidden layer node Z and write it to the data random access memory 122 (write columns 5, 9, 13 and so on until column 57). The second to sixteenth execution of the instructions of addresses 4, 5 and 7 is used to calculate the output layer node Y of the second to fifteenth time steps (time steps 1 to 14). (The last/fifth execution of the instructions of addresses 8 through 12 calculates the value of the hidden layer node Z and writes it to column 61 of the data random access memory 122, but these values are not used.)

The loop command of address 13 causes the loop counter 3804 to decrement and return to the instruction of address 4 if the new loop counter 3804 has a value greater than zero.

In another embodiment, the design of the Jordan time recurrent neural network utilizes the content node C to load the start function value of the output node Y, which is the accumulated value after the execution of the function. In this embodiment, since the value of the output node Y is the same as the value of the content node C, the non-architectural instruction of the address 6 is not included in the non-architectural program. Therefore, the number of columns used in the data random access memory 122 can be reduced. More precisely, the columns of the values of the loaded content nodes C (e.g., columns 2, 6, 59) in the forty-fourth graph are not present in this embodiment. In addition, the time steps of this embodiment only require three columns of data random access memory 122, and will be combined with 20 time steps instead of 15, in the forty-fifth figure. The address of the instruction of the non-architected program will also be adjusted appropriately.

Long-term and short-term memory

The use of long and short term memory cells for time recurrent neural networks is a concept well known in the art. For example, Long Short-Term Memory, Sepp Hochreiter and Jürgen Schmidhuber, Neural Computation, November 15, 1997, Vol. 9, No. 8, Pages 1735-1780; Learning to Forget: Continual Prediction with LSTM, Felix A. Gers Jürgen Schmidhuber, and Fred Cummins, Neural Computation, October 2000, Vol. 12, No. 10, Pages 2451-2471; these documents are available from the MIT Press Journals. Long- and short-term memory cells can be constructed in many different types. The long-short-term memory cell 4600 of the 46th diagram described below is based on the website http://deeplearning.net/tutorial/lstm.html titled LSTM Networks for Sentiment Analysis. The long and short-term memory cells described in the tutorial are models. A copy of this tutorial was downloaded on October 19, 2015 (hereafter referred to as the “long and short-term memory tutorial”) and is provided in the US application data disclosure case in this case. This long and short term memory cell 4600 can be used to generally describe the ability of the neural network unit 121 embodiments described herein to be effective in performing calculations associated with long and short term memory. It should be noted that the embodiments of the neural network unit 121, including the embodiment described in the forty-ninth figure, can effectively perform other long-term and short-term relationships other than the long-term and short-term memory cells described in the forty-sixth figure. Calculation of memory cells.

Preferably, the neural network unit 121 can be used to target one A time recurrent neural network with long and short-term memory cell layers connected to other levels performs computations. For example, in this long-term and short-term memory tutorial, the network includes a mean common source layer to receive the long-short-term memory cell output (H) of the long- and short-term memory layer, and a logistic regression layer to receive the output of the mean common source layer. .

The forty-sixth diagram is a block diagram showing one embodiment of long and short term memory cells 4600.

As shown in the figure, the long-and short-term memory cell 4600 includes a memory cell input (X), a memory cell output (H), an input gate (I), an output gate (O), and a forget gate (F). A memory state (C) and a candidate memory state (C'). The input gate (I) can gate the signal input from the memory cell input (X) to the memory cell state (C), while the output gate (O) can gate the signal state of the memory cell state (C) to the memory cell output (H). . This memory cell state (C) is fed back as a candidate memory cell state (C') for a time step. The forgetting gate (F) can gate this candidate memory cell state (C'), and the candidate memory cell state will feed back and become the memory cell state (C) of the next time step.

The embodiment of the forty-sixth embodiment uses the following equations to calculate the various different values described above:

(1) I=SIGMOID(Wi * X+Ui * H+Bi)

(2) F=SIGMOID(Wf * X+Uf * H+Bf)

(3) C'=TANH(Wc * X+Uc * H+Bc)

(4) C=I * C'+F * C

(5) O=SIGMOID(Wo * X+Uo * H+ Bo)

(6) H=O * TANH(C)

Wi and Ui are weight values associated with the input gate (I), and Bi is the offset value associated with the input gate (I). Wf and Uf are weight values associated with the forgotten gate (F), and Bf is the offset value associated with the forgotten gate (F). Wo and Uo are weight values associated with the output gate (O), and Bo is the offset value associated with the output gate (O). As described above, equations (1), (2), and (5) calculate the input gate (I), the forget gate (F), and the output gate (O), respectively. Equation (3) calculates the candidate memory cell state (C'), and Equation (4) calculates the candidate memory cell state (C') with the current memory cell state (C) as input, and the current memory cell state (C) Memory cell state (C) of the current time step. Equation (6) calculates the memory cell output (H). However, the invention is not limited thereto. Embodiments for calculating input gates, forgetting gates, output gates, candidate memory cell states, memory cell states, and memory cell output long-term memory cells using other methods are also encompassed by the present invention.

To illustrate the invention, the long and short term memory cells include a memory cell input, a memory cell output, a memory cell state, a candidate memory cell state, an input gate, an output gate, and a forget gate. For each time step, the input gate, output gate, forget gate, and candidate memory cell state are functions of the memory cell input of the current time step and the memory cell output and associated weight of the previous time step. The memory cell state of this time step is a function of the memory cell state of the previous time step, the candidate memory cell state, the input gate and the output gate. In this sense, the state of the memory cell feeds back the state of the memory cell used to calculate the next time step. The memory cell output of this time step is a function of the state of the memory cell and the output gate calculated at this time step. Long-term and short-term memory neural networks are one with a length The neural network of the memory cell layer.

The forty-seventh figure is a block diagram showing the data of the neural network unit 121 when the neural network unit 121 performs the calculation of the long-short-term memory cell 4600 layer associated with the long-short-term memory neural network of the forty-sixth figure. An example of data configuration in random access memory 122 and weighted random access memory 124. In the example of the forty-seventh diagram, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, such as a widened configuration, however, there are only 128 neural processing units 126 (eg, neural processing units 0 to 127) The resulting value will be used because the long and short term memory layer in this example has only 128 long and short term memory cells 4600.

As shown in the figure, the weight random access memory 124 loads the weight values, offset values and intermediate values of the corresponding neural processing units 0 to 127 of the neural network unit 121. The rows 0 to 127 of the weight random access memory 124 load the weight values, offset values and intermediate values of the corresponding neural processing units 0 to 127 of the neural network unit 121. Each of the columns 0 to 14 is loaded with 128 values corresponding to the above equations (1) to (6) to be supplied to the neural processing units 0 to 127, which are: Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, C', TANH(C), C, Wo, Uo, Bo. Preferably, the weight value and the offset value -Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, Wo, Uo, Bo (located in columns 0 to 8 and columns 12 to 14) are executed The architecture program of the processor 100 writes/fills the weight random access memory 124 through the MTNN instruction 1400, and is read/used by the non-architecture program executed by the neural network unit 121, such as the 48th figure. Architecture program. Preferably, the intervening values -C', TANH(C), C (in columns 9 to 11) are written/filled by the non-architected program executed by the neural network unit 121. The body 124 is read and used, as will be described later.

As shown in the figure, data random access memory 122 loads input (X), output (H), input gate (I), forget gate (F) and output gate (O) values for a series of time steps. Further, the five columns of this memory are loaded with the values of X, H, I, F and O for a given time step. Taking a random access memory 122 having 64 columns as an example, as shown in the figure, the data random access memory 122 can carry the memory cell values for use in 12 different time steps. In the example of the 47th figure, columns 0 to 4 are loaded with the memory cell values used for time step 0, columns 5 to 9 are loaded with the memory cell values used for time step 1, and so on, columns 55 to 59 The memory cell value used for time step 11 is loaded. The first of the five columns of memory is loaded with the X value of this time step. The second of the five columns of memory is loaded with the H value for this time step. The third of the five columns of memory is loaded with the I value for this time step. The fourth of the five columns of memory is loaded with the F value for this time step. The fifth of the five columns of memory is loaded with the O value for this time step. As shown in the figure, each line within the data random access memory 122 is loaded with values for use by the corresponding neuron or neural processing unit 126. That is to say, the row 0 is loaded with the value associated with the long and short term memory cell 0, and its calculation is performed by the neural processing unit 0; the row 1 is loaded with the value associated with the long and short term memory cell 1, and its calculation is performed by the nerve The processing unit 1 performs; and so on, the row 127 is loaded with the value associated with the long- and short-term memory cells 127, and its calculation is performed by the neural processing unit 127, as described in the subsequent forty-eighth diagram.

Preferably, the X value (located in columns 0, 5, 9, according to this The analogy to column 55) is written/filled into the data random access memory 122 by the architecture program executed by the processor 100 through the MTNN instruction 1400, and is read/used by the non-architecture program executed by the neural network unit 121. , such as the non-architected program shown in Figure 48. Preferably, the I value, the F value and the O value (located in columns 2/3/4, 7/8/9, 12/13/14, and so on to column 57/58/59) are performed by neural processing. The non-architectural program of the unit 121 writes/fills the data random access memory 122 as will be described later. Preferably, the H values (in columns 1, 6, 10, and so on to column 56) are written/filled into the random access memory 122 by the non-architectural program executed by the neural processing unit 121 and read. / used, and read by the MFNN instruction 1500 by an architectural program executing on the processor 100.

The example in the 47th figure assumes that the architecture program performs the following steps: (1) For 12 different time steps, the value of the input X is filled in the data random access memory 122 (column 0, 5, according to this) Analogy to column 55); (2) start the non-architectural program of the 48th figure; (3) detect whether the non-architected program is executed; (4) read the value of the output H from the data random access memory 122 ( Columns 1, 6, and so on to column 59); and (5) repeat steps (1) through (4) several times until the task is completed, such as the calculation required to identify the phone user's words.

In another implementation, the architecture program performs the following steps: (1) for a single time step, filling in the data random access memory 122 (such as column 0) with the value of input X; (2) starting the non-architect program (The forty-eighth version of the non-architected program is a modified version that does not require loops and only accesses a single set of five columns of data storage memory 122); (3) detects whether the non-architected program has been executed; (4) Random access memory from data Body 122 reads the value of output H (e.g., column 1); and (5) repeats steps (1) through (4) several times until the task is completed. Which of the two methods is superior depends on the sampling method of the input X value of the long-term and short-term memory layers. For example, if this task allows sampling of inputs (eg, about 12 time steps) and performing calculations in multiple time steps, the first approach is ideal because it may result in more computational resource efficiency and / Or better performance, however, if this task only allows sampling in a single time step, the second method is required.

The third embodiment is similar to the second method described above. However, unlike the second method, a single group of five columns of data random access memory 122 is used. In this manner, the non-architectural program uses multiple sets of five columns of memory, that is, Each time step uses a different five-column set of memory, which is similar to the first. In this third embodiment, preferably, the architecture program includes a step before the step (2), in which the architecture program updates the non-architecture program before starting, for example, the instruction of the address 0 is The data random access memory 122 column is updated to point to the next set of five columns of memory.

The forty-eighth figure is a table showing a program stored in the program memory 129 of the neural network unit 121. The program is executed by the neural network unit 121 and uses data and weights according to the configuration of the 47th figure. To achieve the calculation associated with the long- and short-term memory cell layer. The sample program of the forty-eighth figure includes 24 non-architectural instructions located at addresses 0 through 23, respectively. The instruction of address 0 (INITIALIZE NPU, CLR ACC, LOOPCNT=12, DR IN ROW=-1, DR OUT ROW=2) will clear accumulator 202 and initialize loop counter 3804 to a value of 12 to Execute 12 loop groups (addresses 1 to 22). This initialization command initializes the column to be read of the data random access memory 122 to a value of -1, and the value is incremented to zero after the first execution of the instruction of the address 1. This initialization instruction initializes the columns to be written to the data random access memory 122 (e.g., the registers 2660 of the twenty-sixth and thirty-ninth figures) into column 2. Preferably, the initialization command causes the neural network unit 121 to be in a wide configuration. Thus, the neural network unit 121 is configured with 512 neural processing units 126. As described in subsequent sections, 128 of the 512 neural processing units 126 correspond to and operate as 128 long-term memory cells 4600 during the execution of instructions at addresses 0 through 23.

In the first execution of the instructions of addresses 1 through 4, each of the 128 neural processing units 126 (i.e., neural processing units 0 through 127) will be directed to the first of the corresponding long and short term memory cells 4600. The time step (time step 0) calculates the input gate (I) value and writes the I value to the corresponding text of column 2 of the data random access memory 122; in the second execution of the instructions of addresses 1 to 4, Each of the 128 neural processing units 126 calculates an I value for the second time step (time step 1) of the corresponding long and short term memory cell 4600 and writes the I value to the data random access memory 122. Corresponding text of column 7; and so on, in the twelfth execution of the instructions of addresses 1 to 4, each of the 128 neural processing units 126 will target the corresponding long-term and short-term memory cells 4600 The twelfth time step (time step 11) calculates the I value and writes the I value to the corresponding text of column 57 of the data random access memory 122, as shown in FIG.

Further, the multiply accumulate instruction of address 1 reads a column below the current column of the data random access memory 122 (column 0 in the first execution, column 5 in the second execution, and so on, In the twelfth execution, column 55), the column contains the memory cell input (X) value associated with the current time step, and the instruction reads the column 0 of the Wi value contained in the weight random access memory 124. And the aforementioned read values are multiplied to produce an accumulator 202 that adds the first multiply accumulation to the instruction that was just cleared by the initialization instruction of address 0 or the address of address 22. Subsequently, the multiply accumulate instruction of address 2 reads the next data random access memory 122 column (column 1 in the first execution, column 6 in the second execution, and so on, in the twelfth execution) That is, column 56), this column contains the memory cell output (H) value associated with the current time step, and this instruction will read the column 1 containing the Ui value in the weight random access memory 124, and the aforementioned value Multiply by to generate a second multiply accumulation and add to accumulator 202. The H value associated with the current time step is read by the data random access memory 122 by the instruction of address 2 (and the instructions of addresses 6, 10 and 18), generated at the previous time step, and by address 22 The output instruction is written to the data random access memory 122; however, in the first execution, the instruction of the address 2 writes the column 1 of the data random access memory as an H value with an initial value. Preferably, the architecture program writes the initial H value to column 1 of the data random access memory 122 (eg, using the MTNN instruction 1400) prior to initiating the non-architecture of the twenty-eighth figure; however, the invention is not limited Thus, other embodiments in which the non-architectural program includes an initialization command to write the initial H value to column 1 of the data random access memory 122 are also within the scope of the present invention. In an embodiment, this initial H value is zero. Next, the instruction of the address 3 to add the weight text to the accumulator (ADD_W_ACC WR ROW 2) reads the column 2 containing the Bi value in the weighted random access memory 124 and adds it to the accumulator 202. Finally, the output instruction of address 4 (OUTPUT SIGMOID, DR OUT ROW+0, CLR ACC) performs an S-type start function on the value of the accumulator 202 and writes the execution result to the current output column of the data random access memory 122. (Column 2 in the first execution, column 7 in the second execution, and column 57 in the twelfth execution) and the accumulator 202 is cleared.

In the first execution of the instructions of addresses 5 through 8, each of the 128 neural processing units 126 will calculate its first time step (time step 0) for the corresponding long and short term memory cells 4600. Forgotten gate (F) values and write F values to the corresponding text of column 3 of data random access memory 122; in the second execution of instructions of addresses 5 through 8, the 128 neural processing units 126 Each of the neural processing units 126 calculates its forgetting gate (F) value for the second time step (time step 1) of the corresponding long-term and short-term memory cell 4600 and writes the F value to the column 8 of the data random access memory 122. Corresponding text; and so on, in the twelfth execution of the instructions of addresses 5 to 8, each of the 128 neural processing units 126 will be directed to the tenth of the corresponding long-term and short-term memory cells 4600. The second time step (time step 11) calculates its forgetting gate (F) value and writes the F value to the corresponding text of column 58 of data random access memory 122, as shown in FIG. The instructions of addresses 5 through 8 calculate the F value in a similar manner to the above instructions of addresses 1 through 4, however, the instructions of addresses 5 through 7 will be from column 3, column 4 and column of weight random access memory 124, respectively. 5 Read the Wf, Uf and Bf values to perform multiplication and/or addition operations.

In the twelve executions of the instructions of addresses 9 through 12, each of the 128 neural processing units 126 calculates its candidate memory cell state for the corresponding time step of the corresponding long-term and short-term memory cells 4600 ( The C') value and the C' value are written to the corresponding text of column 9 of the weighted random access memory 124. The instructions of addresses 9 through 12 calculate the C' value in a manner similar to the above-mentioned instructions of addresses 1 through 4, however, the instructions of addresses 9 through 11 will be respectively from the weighted random access memory 124, column 6, column 7 and Column 8 reads the Wc, Uc and Bc values to perform multiplication and/or addition operations. In addition, the output instruction of address 12 performs a hyperbolic tangent start function instead of (as the output instruction execution of address 4) an S-type start function.

Further, the multiply accumulate instruction of address 9 reads the current column of the data random access memory 122 (column 0 in the first execution, column 5 in the second execution, and so on, The twelfth execution is column 55). This current column contains the memory cell input (X) value associated with the current time step. This instruction also reads the column 6 of the weighted random access memory 124 that contains the Wc value. And multiplying the aforementioned values to produce a first multiply accumulation plus an accumulator 202 that has just been cleared by the instruction of address 8. Next, the multiply-accumulate instruction of address 10 reads the next column of data random access memory 122 (column 1 in the first execution, column 6 in the second execution, and so on, in the The eleventh execution is column 56), which includes the memory cell output (H) value associated with the current time step, and the instruction reads the column 7 containing the Uc value in the weighted random access memory 124, and The aforementioned values are multiplied to produce a second multiplication accumulation applied to accumulator 202. Next, the instruction of the address 11 to add the weight text to the accumulator will read the weighted random access memory 124 containing the column 8 of the Bc value and It is added to the accumulator 202. Finally, the output instruction of address 12 (OUTPUT TANH, WR OUT ROW 9, CLR ACC) performs a hyperbolic tangent start function on the accumulator 202 value and writes the execution result to column 9 of the weighted random access memory 124. And the accumulator 202 is cleared.

In the twelve executions of the instructions of addresses 13 through 16, each of the 128 neural processing units 126 calculates a new memory cell state for the corresponding time step of the corresponding long-term and short-term memory cells 4600 ( C) the value and writing the new C value to the corresponding text of column 11 of the weighted random access memory 122, each neural processing unit 126 also calculates tanh(C) and writes it into the weighted random access memory. Corresponding text of column 10 of 124. Further, the multiply-accumulate instruction of address 13 reads a column below the current column of the data random access memory 122 (column 2 in the first execution and column 7 in the second execution, By analogy, in the twelfth execution, column 57), this column contains the input gate (I) value associated with the current time step, and the instruction reads the weighted random access memory 124 containing the candidate memory cell state (C ') The value of column 9 (just written by the instruction of address 12), and the aforementioned value is multiplied to produce a first multiply accumulated plus accumulator 202 that has just been cleared by the instruction of address 12. Next, the multiply accumulate instruction of address 14 reads a column below the data random access memory 122 (column 3 in the first execution, column 8 in the second execution, and so on, in the The eleventh execution is column 58), this column contains the forgetting gate (F) value associated with the current time step, and the instruction reads the current memory cell included in the weighted random access memory 124 and calculated in the previous time step. Column 11 of the state (C) value (written by the last execution of the instruction of address 15), and multiplying the aforementioned values to produce the first The diploid product is added to the accumulator 202. Next, the output instruction of the address 15 (OUTPUT PASSTHRU, WR OUT ROW 11) passes the value of the accumulator 202 and writes it to the column 11 of the weighted random access memory 124. It should be understood that the C value read by the address of the address 14 by the column 11 of the data random access memory 122 is the C value generated and written by the instruction of the address 13 to 15 in the most recent execution. The output instruction of address 15 does not clear accumulator 202, so its value can be used by the instruction of address 16. Finally, the output command of address 16 (OUTPUT TANH, WR OUT ROW 10, CLR ACC) performs a hyperbolic tangent start function on the value of accumulator 202 and writes its execution result to column 10 of weight random access memory 124. The instructions for address 21 are used to calculate the memory cell output (H) value. The instruction of address 16 clears accumulator 202.

In the first execution of the instructions of addresses 17 through 20, each of the 128 neural processing units 126 will calculate its first time step (time step 0) for the corresponding long- and short-term memory cells 4600. The gate (O) value is output and the O value is written to the corresponding text of column 4 of the data random access memory 122; in the second execution of the instructions of addresses 17 through 20, the 128 neural processing units 126 Each of the neural processing units 126 calculates its output gate (O) value for the second time step (time step 1) of the corresponding long-term memory cell 4600 and writes the O value to the column 9 of the data random access memory 122. Corresponding text; and so on, in the twelfth execution of the instructions of addresses 17 to 20, each of the 128 neural processing units 126 will be directed to the tenth of the corresponding long-term and short-term memory cells 4600. The second time step (time step 11) calculates its output gate (O) value and writes the O value into the data random access record. The corresponding text of column 58 of body 122 is shown in Figure 47. The instructions of addresses 17 through 20 calculate the value of O similarly to the instructions of addresses 1 through 4 described above, however, the instructions of addresses 17 through 19 will be from column 12, column 13 and column of weight random access memory 124, respectively. 14 Read the Wo, Uo and Bo values to perform multiplication and/or addition operations.

In the first execution of the instructions of addresses 21 through 22, each of the 128 neural processing units 126 will calculate its first time step (time step 0) for the corresponding long-term and short-term memory cells 4600. The memory cell outputs (H) values and writes the H values to the corresponding text of column 6 of the data random access memory 122; in the second execution of the instructions of addresses 21 through 22, the 128 neural processing units 126 Each of the neural processing units 126 calculates its memory cell output (H) value for the second time step (time step 1) of the corresponding long-term and short-term memory cell 4600 and writes the H value into the data random access memory 122. 11 corresponding text; and so on, in the twelfth execution of the instructions of addresses 21 to 22, each of the 128 neural processing units 126 will target the corresponding long-term and short-term memory cells 4600 The twelfth time step (time step 11) calculates its memory cell output (H) value and writes the H value to the corresponding text of column 60 of the data random access memory 122, as shown in FIG.

Further, the multiply accumulate instruction of the address 21 reads the third column behind the current column of the data random access memory 122 (column 4 in the first execution and column 9 in the second execution, This type of push, in the twelfth execution, is column 59), this column contains the output gate (O) value associated with the current time step, and this instruction reads the weighted random access memory. 124 includes a column 10 of tanh (C) values (written by the instruction of address 16) and multiplies the aforementioned values to produce a multiply accumulator plus the accumulator 202 that was just cleared by the instruction of address 20. Subsequently, the output instruction of address 22 transfers the value of accumulator 202 and writes it to the next second output column 11 of data random access memory 122 (on the first execution, column 6, in the second time) Execution is column 11, and so on, at the twelfth execution, column 61), and the accumulator 202 is cleared. It should be understood that the H value of the data random access memory 122 column is written by the instruction of the address 22 (column 6 in the first execution, column 11 in the second execution, and so on, in The twelfth execution is the value of H consumed/read in the subsequent execution of the instructions of the instructions of addresses 2, 6, 10 and 18. However, the H value written in column 61 during the twelfth execution is not consumed/read by the execution of the instructions of addresses 2, 6, 10 and 18; for a preferred embodiment, this value will It is consumed/read by the architecture program.

The instruction of bit 23 (LOOP 1) decrements the loop counter 3804 and returns to the instruction of address 1 if the value of the new loop counter 3804 is greater than zero.

A forty-ninth diagram is a block diagram showing an embodiment of a neural network unit 121 having an output buffer masking and feedback capability within the neural processing unit group of this embodiment. The forty-ninth diagram shows a single neural processing unit group 4901 consisting of four neural processing units 126. Although the forty-ninth diagram shows only a single neural processing unit group 4901, it should be understood that each of the neural processing units 126 in the neural network unit 121 is included in a neural processing unit group 4901, and therefore, a total of There will be N/J neural processing unit groups 4901, where N is nerve processing The number of units 126 (for example, 512 for a wide configuration, 1024 for a narrow configuration) and J is the number of neural processing units 126 within a single group 4901 (for example, forty In the case of the nine figure, it is four). In the forty-ninth diagram, the four neural processing units 126 in the neural processing unit group 4901 are referred to as a neural processing unit 0, a neural processing unit 1, a neural processing unit 2, and a neural processing unit 3.

Each of the neural processing units in the embodiment of the forty-ninth embodiment is similar to the neural processing unit 126 of the seventh embodiment described above, and elements having the same reference numerals are similar in the drawings. However, multiplex register 208 is tuned to include four additional inputs 4905 that are tuned to include four additional inputs 4907 that are adjusted to be input from the original Selections 211 and 207 and additional inputs 4905 are provided to output 209, and selection input 713 is adjusted to provide selection from the original inputs 711 and 206 and additional inputs 4907 to output 203.

As shown in the figure, the column buffer 1104 of the eleventh diagram is the output buffer 1104 in the forty-ninth figure. Further, the characters 0, 1, 2, and 3 of the output buffer 1104 shown in the figure receive the corresponding outputs associated with the four start function units 212 of the neural processing units 0, 1, 2, and 3. The output buffer 1104 of this section contains N words corresponding to a neural processing unit group 4901, which is referred to as an output buffer text group. In the embodiment of the forty-ninth embodiment, N is four. The four texts of output buffer 1104 are fed back to multiplex registers 208 and 705, and are received by multiplex register 208 as four additional inputs 4905 and by multiplex register 705 as four additional inputs 4907. Received. lose The buffered text group is fed back to the feedback action of its corresponding neural processing unit group 4901, so that the arithmetic instructions of the non-architectural program can be from the text associated with the output buffer 1104 of the neural processing unit group 4901 (ie, the output buffer text group) Select one or two words as the input in the group). For an example, please refer to the non-architectural program in the following 51st figure, as shown in the figure 4, 8, 11, 12 and 15. That is, the output buffer 1104 text specified in the non-architected instruction will confirm the value generated by the selection input 213/713. This capability actually allows the output buffer 1104 to act as a scratch pad memory that allows non-architected programs to reduce the write data random access memory 122 and/or the weighted random access memory 124 and subsequent The number of reads, such as the value of the intervening generation and use during the process. Preferably, the output buffer 1104, or column buffer 1104, includes a one-dimensional array of registers for storing 1024 narrow words or 512 wide words. Preferably, the reading of the output buffer 1104 can be performed in a single time-frequency period, while the writing to the output buffer 1104 can be performed in a single time-frequency period. Different from the data random access memory 122 and the weighted random access memory 124, the data can be accessed by the architecture program and the non-architecture program. The output buffer 1104 cannot be accessed by the architecture program, but can only be stored by the non-architecture program. take.

Output buffer 1104 will be adjusted to receive a mask input 4903. Preferably, the mask input 4903 includes four bits corresponding to the four characters of the output buffer 1104 associated with the four neural processing units 126 of the neural processing unit group 4901. Preferably, if this corresponds to the cover of the output buffer 1104 The cover input 4903 bits are true, and the text of the output buffer 1104 maintains its current value; otherwise, the text of the output buffer 1104 is updated by the output of the start function unit 212. That is, if the mask input 4903 of the text corresponding to the output buffer 1104 is false, the output of the start function unit 212 is written to the text of the output buffer 1104. Thus, the output command of the non-architected program can selectively write the output of the start function unit 212 to some characters of the output buffer 1104 and maintain the current value of the other characters of the output buffer 1104. For an example, please refer to the example. The instructions of the non-architecture program in the subsequent fifty-first figure are as shown in the instructions of addresses 6, 10, 13 and 14. That is, the text of the output buffer 1104 specified in the non-architected program is generated from the value of the mask input 4903.

To simplify the description, the input 1811 of the multiplex register 208/705 is not shown in the forty-ninth figure (as shown in Figures 18, 19 and 23). However, embodiments that simultaneously support feedback/masking that can dynamically configure the neural processing unit 126 and the output buffer 1104 are also within the scope of the present invention. Preferably, in these embodiments, the output buffer text group is dynamically configurable accordingly.

It should be understood that although the number of neural processing units 126 in the neural processing unit group 4901 of this embodiment is four, the present invention is not limited thereto, and the number of intra-group neural processing units 126 is large or small. The examples are all within the scope of the invention. Moreover, in the case of an embodiment having a shared activation function unit 1112, as shown in the fifty-second diagram, the number of neural processing units 126 within a group of neural processing units 4901 and the nerves within a group of activation function units 212 Processing unit 126 The quantity will have a synergistic effect. The occlusion and feedback capabilities of the output buffer 1104 within the neural processing unit group are particularly useful for improving the computational efficiency associated with the long and short term memory cells 4600, as described in subsequent fifty and fifty-first figures.

Figure 50 is a block diagram showing the neural network of the forty-ninth figure when the neural network unit 121 performs the calculation associated with a hierarchy of 128 long- and short-term memory cells 4600 in the forty-sixth figure. An example of data configuration in the data random access memory 122 of the unit 121, the weighted random access memory 124 and the output buffer 1104. In the example of Fig. 50, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, for example, in a wide configuration. As in the forty-seventh and forty-eighth examples, there are only 128 long- and short-term memory cells 4600 in the long- and short-term memory layers of the fifty and fifty-first examples. However, in the example of Fig. 50, all of the values generated by 512 neural processing units 126 (e.g., neural processing units 0 through 127) will be used. When performing the non-architectural program of Figure 51, each neural processing unit group 4901 will collectively operate as a long-term and short-term memory cell 4600.

As shown in the figure, the data random memory 122 loads the memory cell input (X) and output (H) values for use in a series of time steps. Further, for a given time step, a pair of two columns of memory are loaded with X values and H values, respectively. Taking a random access memory 122 having 64 columns as an example, as shown in the figure, the memory cell value loaded by the data random access memory 122 can be used in 31 different time steps. In the example of Fig. 50, columns 2 and 3 are loaded with values for time step 0, columns 4 and 5 are loaded with values for time step 1, and so on, columns 62 and 63 are loaded for time step 30. Value. The pair of two columns of memory The first column loads the X value for this time step, and the second column loads the H value for this time step. As shown in the figure, each of the four rows in the data random access memory 122 corresponds to a value stored in the memory system of the neural processing unit group 4901 for use by its corresponding long-term and short-term memory cells 4600. That is, rows 0 to 3 are loaded with values associated with long- and short-term memory cells 0, which are calculated by neuroprocessing unit 0-3, ie, neuroprocessing unit group 0 is executed; rows 4 to 7 are loaded associated with long The value of the short-term memory cell 1 is calculated by the neural processing unit 4-7, that is, the neural processing unit group 1 is executed; and so on, the lines 508 to 511 are loaded with the values associated with the long- and short-term memory cells 127, and the calculation is performed. It is performed by the neural processing unit 508-511, that is, the neural processing unit group 127, as shown in the subsequent fifty-first figure. As shown in the figure, column 1 is not used, column 0 is loaded with the initial memory cell output (H) value. In a preferred embodiment, the value can be filled in by the architecture program, however, the invention is not limited Thus, it is also within the scope of the present invention to fill in the initial memory cell output (H) values of column 0 using non-architectural program instructions.

Preferably, the X value (in columns 2, 4, 6 and so on to column 62) is written/filled into the data random access memory 122 by the architecture program executed by the processor 100 through the MTNN instruction 1400, and The non-architecture program executed by the neural network unit 121 performs reading/using, such as the non-architected program shown in FIG. Preferably, the H value (in columns 3, 5, 7 and so on to column 63) is written/filled into the data random access memory 122 by the non-architectural program executed by the neural network unit 121 and read. / Use, as described later. Preferably, the H value is read by the MFNN instruction 1500 by an architectural program executing on the processor 100. Need to pay attention Yes, the non-architectural scheme of the fifty-first graph assumes that each group of four rows of memory corresponding to the neural processing unit group 4901 (eg, rows 0-3, rows 4-7, rows 5-8, and so on) In 508-511), the four X values in a given column are filled in with the same value (for example, filled in by the architecture program). Similarly, the non-architectural program of the fifty-first graph calculates and writes the same value for the four H values of a given column in each of the four rows of memory corresponding to the neural processing unit group 4901.

As shown in the figure, the weighted random access memory 124 is loaded with the weights, offsets, and memory cell state (C) values required by the neural processing unit of the neural network unit 121. In each group of four rows of memory corresponding to the neural processing unit group 121 (eg, rows 0-3, rows 4-7, rows 5-8, and so on to rows 508-511): (1) the row number divided by The remainder of 4 is equal to 3, and the values of Wc, Uc, Bc, and C are loaded in columns 0, 1, 2, and 6, respectively; (2) the line number divided by 4 is equal to 2, and will be in Columns 3, 4, and 5 respectively load the values of Wo, Uo, and Bo; (3) the row number divided by the remainder of 4 is equal to 1, and the values of Wf, Uf, and Bf are loaded in columns 3, 4, and 5, respectively; And (4) the row number divided by the remainder of 4 is equal to 0, and the values of Wi, Ui and Bi are loaded in columns 3, 4 and 5, respectively. Preferably, these weights and offset values - Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, Wo, Uo, Bo (in columns 0 to 5) - are executed by the processor 100. The architecture program writes/fills the weighted random access memory 124 through the MTNN instruction 1400 and is read/used by a non-architected program executed by the neural network unit 121, such as the non-architected program of FIG. Preferably, the intermediate C value is written/filled into the weight random access memory 124 by the non-architectural program executed by the neural network unit 121, and is read/used as described later.

The example in Figure 50 assumes that the architecture program performs the following steps: (1) For 31 different time steps, the value of the input X is filled in the data random access memory 122 (column 2, 4, and so on. Column 62); (2) start the non-architecture program of the 51st figure; (3) detect whether the non-architecture program is executed; (4) read the value of the output H from the data random access memory 122 (column 3) , 5, and so on to column 63); and (5) repeat steps (1) through (4) several times until the task is completed, such as the calculation required to identify the phone user's words.

In another implementation, the architecture program performs the following steps: (1) for a single time step, filling in the data random access memory 122 (such as column 2) by inputting the value of X; (2) starting the non-architectural program (The 51st version of the non-architected program is a modified version that does not require loops and only accesses a single pair of two columns of data storage memory 122); (3) detects whether the non-architected program has been executed; (4) Reading the value of the output H from the data random access memory 122 (e.g., column 3); and (5) repeating steps (1) through (4) several times until the task is completed. Which of the two methods is superior depends on the sampling method of the input X value of the long-term and short-term memory layers. For example, if this task allows sampling of inputs (eg, approximately 31 time steps) and performing calculations in multiple time steps, the first approach is desirable because it may result in more computational resource efficiency and / Or better performance, however, if this task only allows sampling in a single time step, the second method is required.

The third embodiment is similar to the second method described above. However, unlike the second method, a single pair of two columns of data random access memory 122 is used. In this way, the non-architectural program uses multiple pairs of memory columns, that is, It is to use different pairs of memory columns at various time steps, this part is similar to the first way. Preferably, the architecture program of the third embodiment includes a step before the step (2), in which the architecture program updates the non-architecture program before starting the non-architecture program, for example, in the instruction of the address 1 The data random access memory 122 column is updated to point to the next pair of two columns of memory.

As shown in the figure, for the neural processing units 0 to 511 of the neural network unit 121, after the execution of the instructions of the different addresses in the non-architectural program of the fifty-first figure, the output buffer 1104 is loaded with the memory cell output (H). ), candidate memory cell state (C'), input gate (I), forget gate (F), output gate (O), memory cell state (C) and tanh (C), each output buffer text group In the group (for example, the output buffer 1104 corresponds to the group of four words of the neural processing unit group 4901, such as words 0-3, 4-7, 5-8 and so on to 508-511), the text number is divided by The text with a remainder of 4 is represented as OUTBUF[3], the text with the remainder of the text number divided by 4 is represented as OUTBUF[2], and the text with the remainder of the text number divided by 4 is represented as OUTBUF [ 1], and the text with the remainder of the text number divided by 4 is represented as OUTBUF[0].

As shown in the figure, after the instruction of address 2 in the non-architecture program of the fifty-first figure is executed, for each neural processing unit group 4901, all four characters of the output buffer 1104 are written correspondingly. The initial memory cell output (H) value of the long-term and short-term memory cells 4600. After the instruction of address 6 is executed, for each neural processing unit group 4901, the OUTBUF[3] text of the output buffer 1104 is written to the candidate memory cell state (C') value of the corresponding long-term memory cell 4600. The other three characters of the output buffer 1104 maintain their previous values. Instruction at address 10 After execution, for each neural processing unit group 4901, the OUTBUF[0] text of the output buffer 1104 is written to the input gate (I) value of the corresponding long-term memory cell 4600, and the OUTBUF[1] text is written. Corresponding to the forgotten gate (F) value of the long-term and short-term memory cell 4600, the OUTBUF[2] text is written to the output gate (O) value of the corresponding long-term memory cell 4600, while the OUTBUF[3] text is maintained at its previous value. . After the instruction of address 13 is executed, for each neural processing unit group 4901, the OUTBUF[3] text of the output buffer 1104 is written to the new memory cell state (C) value of the corresponding long-term memory cell 4600. (For the output buffer 1104, the C value containing the slot 3 is written to the column 6 of the weight random access memory 124, as described in the subsequent fifty-first figure), and the output buffer 1104 The other three words are to maintain their previous values. After the instruction of address 14 is executed, for each neural processing unit group 4901, the OUTBUF[3] text of the output buffer 1104 is written to the tanh(C) value of the corresponding long-term memory cell 4600, and the output buffer is output buffer. The other three words of the device 1104 are to maintain their previous values. After the instruction of address 16 is executed, for each neural processing unit group 4901, all four words of output buffer 1104 are written to the new memory cell output (H) value of the corresponding long-term memory cell 4600. The execution flow of the above address 6 to 16 (that is, the execution of the address 2 is excluded, because the address 2 is not part of the program loop) is repeated 30 times, and the program returns to the address 3 as the address 17. Loop.

The fifty-first figure is a table showing a program stored in the program memory 129 of the neural network unit 121. The program is executed by the neural network unit 121 of the forty-ninth figure and configured according to the fifty-fifth figure. Use data and weights to achieve calculations associated with long- and short-term memory cell layers. The sample program of Figure 51 contains 18 non-architected instructions located at addresses 0 through 17. The instruction of address 0 is an initialization instruction to clear accumulator 202 and initialize loop counter 3804 to a value of 31 to perform 31 loop groups (instructions of addresses 1 through 17). The initialization command initializes the column to be written of the data random access memory 122 (for example, the register 2606 of the twenty-sixth/39th figure) to a value of 1, and the first instruction of the address 16 After the second execution, this value will increase to 3. Preferably, the initialization command causes the neural network unit 121 to be in a wide configuration. Thus, the neural network unit 121 is configured with 512 neural processing units 126. As described in the subsequent sections, during the execution of the instructions of addresses 0 through 17, the 128 neural processing unit groups 490 of the 512 neural processing units 126 operate as 128 corresponding long-term and short-term memory cells 4600.

The instructions for addresses 1 and 2 are not part of the program loop group and will only be executed once. These instructions will generate an initial memory cell output (H) value (eg, 0) and write it to all of the text in output buffer 1104. The instruction of address 1 reads the initial H value from column 0 of data random access memory 122 and places it in accumulator 202 cleared by instruction of address 0. The address 2 instruction (OUTPUT PASSTHRU, NOP, CLR ACC) passes the accumulator 202 value to the output buffer 1104 as shown in FIG. The "NOP" flag in the output instruction of address 2 (and the other output instructions in Figure 51) indicates that the output value will only be written to the output buffer 1104 and will not be written to the memory, ie it will not It is written to the data random access memory 122 or the weighted random access memory 124. The instruction of address 2 will clear accumulator 202.

The instructions of addresses 3 through 17 are located in the loop group, and the number of executions is the value of the loop count (such as 31).

Each execution of the instructions of addresses 3 through 6 calculates the tanh(C') value of the current time step and writes it to the text OUTBUF[3], which will be used by the instruction of address 11. More precisely, the multiply-accumulate instruction of address 3 reads the memory associated with this time step from the current read column of data random access memory 122 (e.g., columns 2, 4, 6 and so on to column 62). The cell inputs (X) value, reads the Wc value from column 0 of the weighted random access memory 124, and multiplies the aforementioned values to produce a product that is added to the accumulator 202 that is cleared by the instruction of address 2.

The multiply accumulate instruction of address 4 (MULT-ACCUM OUTBUF[0], WR ROW 1) reads the H value from the word OUTBUF[0] (ie all four neural processing units 126 of the neural processing unit group 4901), from Column 1 of weight random access memory 124 reads the Uc value and multiplies the aforementioned values to produce a second product to add accumulator 202.

Adding the weight text to the accumulator instruction (ADD_W_ACC WR ROW 2) of address 5 reads the Bc value from column 2 of the weight random memory 124 and adds it to accumulator 202.

The output instruction of address 6 (OUTPUT TANH, NOP, MASK[0:2], CLR ACC) performs a hyperbolic tangent start function on the accumulator 202 value, and only writes the execution result to the text OUTBUF[3] (also That is, only the neural processing unit 126 whose number is 4 in the neural processing unit group 4901 will write this result), and the accumulator 202 will be cleared. In other words, the output instruction of address 6 will obscure the characters OUTBUF[0], OUTBUF[1] and OUTBUF[2] (such as instruction terms). Maintain the current value as indicated by MASK[0:2], as shown in Figure 50. In addition, the output instruction of address 6 is not written to the memory (as indicated by the instruction term NOP).

Each execution of the instruction of address 7 to 10 calculates the input gate (I) value of the current time step, forgets the gate (F) value and the output gate (O) value and writes them to the text OUTBUF[0], OUTBUF respectively. [1], and OUTBUF[2], these values will be used by the instructions of addresses 11, 12 and 15. More precisely, the multiply-accumulate instruction of address 7 reads the memory associated with this time step from the current read column of data random access memory 122 (e.g., columns 2, 4, 6 and so on to column 62). The cell inputs (X) values, the Wi, Wf and Wo values are read from column 3 of the weighted random access memory 124, and the aforementioned values are multiplied to produce a product that is added to the accumulator 202 cleared by the instruction of address 6. More precisely, in the neural processing unit group 4901, the neural processing unit 126 with the remainder of the number divided by 4 calculates the product of X and Wi, and the neural processing unit 126 with the remainder of the number divided by 1 calculates X and The product of Wf, and the remainder of the number divided by 4, the neural processing unit 126 calculates the product of X and Wo.

The multiply accumulate instruction of address 8 reads the H value from the word OUTBUF[0] (i.e., all four neural processing units 126 of the neural processing unit group 4901), and reads Ui from column 4 of the weighted random access memory 124. The Uf and Uo values are multiplied by the aforementioned values to produce a second product that is added to the accumulator 202. More precisely, in the neural processing unit group 4901, the neural processing unit 126 with the remainder of the number divided by 4 calculates the product of H and Ui, and the neural processing unit 126 with the remainder of the number divided by 1 calculates H and The product of Uf, and the number except the remainder of 4 is 2 Processing unit 126 calculates the product of H and Uo.

Adding the weight text to the accumulator instruction (ADD_W_ACC WR ROW 2) of address 9 reads the Bi, Bf and Bo values from column 5 of the weighted random memory 124 and adds it to accumulator 202. More precisely, in the neural processing unit group 4901, the neural processing unit 126 with the remainder of the number divided by 4 performs the addition calculation of the Bi value, and the neural processing unit 126 with the remainder of the number divided by 1 performs the Bf. The addition of the values is calculated, and the neural processing unit 126, which has a remainder of 4 except the number 4, performs the addition calculation of the Bo value.

The output instruction of address 10 (OUTPUT SIGMOID, NOP, MASK[3], CLR ACC) performs an S-type start function on the value of the accumulator 202 and writes the calculated I, F and O values to the text OUTBUF[0, respectively. ], OUTBUF[1] and OUTBUF[2], this instruction will clear the accumulator 202 without writing to the memory. That is, the output instruction of address 10 masks the text OUTBUF[3] (as indicated by the instruction term MASK[3]) and maintains the current value of the text (ie, C'), as shown in Figure 50.

Each execution of the instructions of addresses 11 through 13 computes a new memory cell state (C) value generated by the current time step and writes it to column 6 of the weighted random access memory 124 for use in the next time step (also That is, the instruction for the address 12 is used when the next loop is executed.) More precisely, this value is written in the column 6 corresponding to the four lines of the neural processing unit group 4901. . In addition, each execution of the instruction at address 14 writes the tanh(C) value to OUTBUF[3] for instruction of address 15.

More precisely, the multiply accumulate instruction of address 11 (MULT-ACCUM OUTBUF[0], OUTBUF[3]) reads the input gate (I) value from the word OUTBUF[0] and reads from the text OUTBUF[3]. The candidate memory cell state (C') value is multiplied by the aforementioned value to produce a first product added to the accumulator 202 cleared by the instruction of address 10. More precisely, each of the four neural processing units 126 of the neural processing unit group 4901 calculates the first product of the I value and the C' value.

The multiply accumulate instruction of address 12 (MULT-ACCUM OUTBUF[1], WR ROW 6) instructs the neural processing unit 126 to read the forgetting gate (F) value from the text OUTBUF[1] from the weighted random access memory 124. Column 6 reads its corresponding text and multiplies it to produce a second product that is summed with the address of address 11 resulting from the first product in accumulator 202. More precisely, for the neural processing unit 126 with the remainder of the labeling 4 in the neural processing unit group 4901, the text read from column 6 is the current memory cell state (C) value calculated in the previous time step. The sum of the first product and the second product is the new memory cell state (C). However, for the other three neural processing units 126 of the neural processing unit group 4901, the text read from column 6 is an unneeded value because the accumulated values produced by these values will not be used, That is, the instructions of addresses 13 and 14 are not placed in the output buffer 1104 and are cleared by the instruction of the address 14. That is to say, only the new memory cell state (C) value generated by the neural processing unit 126 with the remainder of the label of 4 in the neural processing unit group 4901 will be used, that is, the instructions of the addresses 13 and 14. use. For the second to thirty-one execution of the instruction at address 12, the right to The C value read by column 6 of the re-random access memory 124 is the value written by the instruction of address 13 in the previous execution of the loop group. However, for the first execution of the instruction at address 12, the value of C in column 6 is written by the architecture program before the non-architecture program of the fifty-first diagram is started or the version is modified by one of the non-architecture programs. Enter the initial value.

The output instruction of address 13 (OUTPUT PASSTHRU, WR ROW 6, MASK[0:2]) will only pass the value of accumulator 202, ie the calculated C value, to the text OUTBUF[3] (that is, only neural processing) The neural processing unit 126 having the remainder of the label 4 in the unit group 4901 divides the calculated C value into the output buffer 1104), and the column 6 of the weighted random access memory 124 is updated. The output buffer 1104 is written as shown in Fig. 50. That is to say, the output instruction of address 13 will obscure the characters OUTBUF[0], OUTBUF[1] and OUTBUF[2] while maintaining their current values (ie, I, F and O values). As described above, only the column 6 corresponds to the C value in the four-line text of the neural processing unit group 4901 in which the remainder of the label except 4 is 3, that is, the instruction of the address 12 is used; therefore, the non-architecture The program does not care about the values of row 0-2, row 4-6, and so on to row 508-510 in column 6 of the random access memory 124, as shown in Fig. 50 (i.e., I, F and O value).

The output instruction of address 14 (OUTPUT TANH, NOP, MASK[0:2], CLR ACC) performs a hyperbolic tangent start function on the accumulator 202 value, and writes the calculated tanh(C) value to the text OUTBUF. [3], this instruction will also clear the accumulator 202 without writing to the memory. The output instruction of address 14 is the same as the output instruction of address 13. The text OUTBUF[0], OUTBUF[1] and OUTBUF[2] are maintained to maintain their original values, as shown in Figure 50.

Each execution of the instructions of addresses 15 through 16 calculates the memory cell output (H) value generated by the current time step and writes it to the second column after the current output column of the data random access memory 122, the value of which will be Read by the architecture program and used for the next time step (ie, used by the instructions of addresses 3 and 7 in the next loop execution). More precisely, the multiply accumulate instruction at address 15 reads the output gate (O) value from the word OUTBUF[2], reads the tanh(C) value from the text OUTBUF[3], and multiplies it to produce a The product is added to accumulator 202 which is cleared by the instruction of address 14. More precisely, each of the four neural processing units 126 of the neural processing unit group 4901 calculates the product of the value O and tanh (C).

The output instruction of address 16 will pass the value of accumulator 202 and write the calculated H value to column 3 in the first execution, the calculated H value to column 5 in the second execution, and so on. In the thirty-first execution, the calculated H value is written to column 63. As shown in Fig. 50, these values are used by the instructions of addresses 4 and 8. In addition, as shown in Fig. 50, these calculated H values are placed in the output buffer 1104 for subsequent use by the instructions of addresses 4 and 8. The output instruction of address 16 will clear accumulator 202. In one embodiment, the long- and short-term memory cell 4600 is designed such that the output instruction of address 16 (and/or the output instruction of address 22 in the forty-eighth picture) has a start function, such as an S-type or a hyperbolic tangent. The function, not the value of the accumulator 202.

The loop instruction of address 17 will cause the loop counter 3804 Decrement and return to the instruction of address 3 if the new loop counter 3804 has a value greater than zero.

It can be seen that, because of the feedback and shielding capability of the output buffer 1104 in the embodiment of the neural network unit 121 of the forty-ninth figure, the number of instructions in the loop group of the non-architectural program of the fifty-first graph is compared. The non-architectural instructions in Figure 48 are roughly reduced by 34%. In addition, because of the feedback and shielding capability of the output buffer 1104 in the embodiment of the neural network unit 121 of the forty-ninth figure, the memory configuration in the random access memory 122 of the non-architectural program of the fifty-first embodiment is shown in FIG. The number of time steps for collocation is roughly three times that of the forty-eighth figure. The foregoing improvements facilitate some architectural applications that utilize neural network unit 121 to perform long- and short-term memory cell layer calculations, particularly for applications where the number of long- and short-term memory cells 4600 in the long- and short-term memory cell layer is less than or equal to 128.

The embodiments of the forty-seventh to fifty-first embodiments assume that the weights and offset values in the various time steps remain unchanged. However, the present invention is not limited thereto, and embodiments in which other weights and offset values are changed over time are also within the scope of the present invention, wherein the weight random access memory 124 is not as shown in the forty-seventh to fifty-th Fill in a single set of weights and offset values, but fill in different sets of weights and offset values at each time step and the weights of the non-architectural programs of the 48th to 51st maps. Adjustment.

Basically, in the foregoing embodiments of the forty-seventh to fifty-firstth embodiments, weights, offsets, and intermediate values (e.g., C, C' values) are stored in the weighted random access memory 124, and input and output values. (such as X, H values) is stored in the data random access memory 122. This feature is advantageous In the case where the data random access memory 122 is a double port and the weighted random access memory 124 is an embodiment, there is more traffic from the non-architecture program and the architecture program to the data random access memory 122. . However, because the weighted random access memory 124 is large, in another embodiment of the present invention, the memory of the non-architectural and architectural program write values is exchanged (ie, the interchangeable data random access memory 122 and the random weights are randomized). Access memory 124). That is, the W, U, B, C', tanh (C) and C values are stored in the data random access memory 122 and the X, H, I, F and O values are stored in the weighted random access memory. Body 124 (adjusted embodiment of the forty-seventh figure); and W, U, B, and C values are stored in the data random access memory 122 and the X and H values are stored in the weighted random access memory 124 (the adjusted example of the fifty-fifth figure). Because the weight random access memory 124 is large, these embodiments can handle more time steps in one batch. For applications that use the neural network unit 121 to perform computational architectural programs, this feature facilitates certain applications that can benefit from more time steps and can be designed for memory (such as weighted random access memory). 124) Provide sufficient bandwidth.

The fifty-second diagram is a block diagram showing an embodiment of a neural network unit 121 having an output buffer masking and feedback capability within the group of neural processing units of this embodiment, and sharing a start function unit 1112. The neural network unit 121 of the fifty-second diagram is similar to the neural network unit 121 of the forty-seventh diagram, and elements having the same reference numerals are similar in the drawings. However, the four start function units 212 of the forty-ninth figure are replaced by a single shared start function unit 1112 in this embodiment, and the single start function unit receives four from four accumulators 202. Output 217 produces four outputs to the words OUTBUF[0], OUTBUF[1], OUTBUF[2] and OUTBUF[3]. The neural network unit 212 of the fifty-second diagram operates in a manner similar to the embodiment described in the forty-ninth to fifty-first figures above, and the manner in which it operates to share the start function unit 1112 is similar to that of the foregoing eleventh The embodiment described in the thirteenth figure.

The fifty-third figure is a block diagram showing the neural network of the forty-ninth figure when the neural network unit 121 performs the calculation associated with a hierarchy of 128 long- and short-term memory cells 4600 in the forty-sixth figure. Another embodiment of the data configuration in the data random access memory 122 of the unit 121, the weighted random access memory 124 and the output buffer 1104. The example of the fifty-third figure is similar to the example of the fifty-fifth figure. However, in the fifty-third figure, the Wi, Wf, and Wo values are in column 0 (instead of column 3 in Figure 50); Ui, Uf and Uo values are in column 1 (rather than fifth) Ten graphs are in column 4); Bi, Bf and Bo values are in column 2 (instead of column 5 in Fig. 50); C values are in column 3 (instead of column 50 in column 50) ). In addition, the content of the output buffer 1104 of the fifty-third figure is similar to the fifty-fifth figure, but because of the difference between the non-architectural programs of the fifty-fourth and fifty-first figures, the content of the third column (ie, The I, F, O and C' values are present in the output buffer 1104 after the instruction of address 7 is executed (instead of the instruction of address 10 as in Fig. 50); the content of the fourth column (i.e., I, The F, O and C values are present in the output buffer 1104 after the instruction of address 10 is executed (instead of the instruction of address 13 as in Fig. 50); the contents of the fifth column (i.e., I, F, O) And the tanh(C) value) appears in the output buffer 1104 after the instruction of address 11 is executed (rather than the address as in the fifty-fifth figure) The instruction of the sixth column); and the content of the sixth column (i.e., the H value) is present in the output buffer 1104 after the execution of the instruction of the address 13 (instead of the instruction of the address 16 as in the fifty-fifth figure), as described later. .

The fifty-fourth figure is a table showing a program stored in the program memory 129 of the neural network unit 121, which is executed by the neural network unit 121 of the forty-ninth figure and according to the fifty-third figure. Configure usage data and weights to achieve calculations associated with long- and short-term memory cell layers. The example program of the fifty-fourth figure is similar to the program of the fifty-first figure. More precisely, in the fifty-fourth and fifty-first figures, the instructions of the addresses 0 to 5 are the same; the instructions of the addresses 7 and 8 in the fifty-fourth figure are the same as the addresses in the fifty-first figure. The instructions of 10 and 11; and the instructions of addresses 10 through 14 in the fifty-fourth figure are identical to the instructions of addresses 13 to 17 in the fifty-first figure.

However, the instruction of address 6 in the fifty-fourth figure does not clear the accumulator 202 (in contrast, the instruction of address 6 in the fifty-first figure clears the accumulator 202). In addition, the instructions of addresses 7 through 9 in the fifty-first figure do not appear in the non-architectural program of the fifty-fourth figure. Finally, with respect to the instruction of address 9 in the fifty-fourth figure and the instruction of address 12 in the fifty-first figure, except that the instruction of address 9 in the fifty-fourth figure reads the weight random access memory The order of 124 is the same as that of the address 12 of the fifty-first figure except that the index of the weighted random access memory is 6 and the other parts are the same.

Because of the difference between the non-architecture program of the fifty-fourth figure and the non-architecture program of the fifty-first figure, the number of columns of the weighted random access memory 124 used in the configuration of the fifty-third figure is reduced by three, and the program is returned. The number of instructions in the circle will also be reduced by three. The loop group size in the non-architectural program of the fifty-fourth figure is essentially only the back of the non-architectural program of the forty-eighth figure. It is half the size of the circle group and is roughly only 80% of the size of the circle group in the non-architectural program of the 51st chart.

Fifty-fifth is a block diagram showing a portion of a neural processing unit 126 in accordance with another embodiment of the present invention. More precisely, for a single one of the plurality of neural processing units 126 of the forty-ninth diagram, the multiplex register 208 is shown with its associated inputs 207, 211 and 4905, and more The scratchpad 705 is associated with its inputs 206, 711 and 4907. In addition to the input of the forty-ninth figure, the multiplexer 208 and the multiplexer 705 of the neural processing unit 126 individually receive an in-group number (index_within_group) input 5599. The intra-group numbering input 5599 indicates the numbering of the particular neural processing unit 126 within its neural processing unit group 4901. Thus, for example, taking an embodiment in which each neural processing unit group 4901 has four neural processing units 126, within each neural processing unit group 4901, one of the neural processing units 126 is numbered in its group. A value of zero is received in 5599, wherein one of the neural processing units 126 receives a value of one in its group number entry 5599, wherein one of the neural processing units 126 receives the value two in its group number input 5599, and one of the neural processing units 126 receives the value three in the numbering entry 5599 in its group. In other words, the intra-group number input 5599 value received by the neural processing unit 126 is the remainder of the number of the neural processing unit 126 in the neural network unit 121 divided by J, where J is within the neural processing unit group 4901. The number of neural processing units 126. Thus, for example, the neural processing unit 73 receives a value of one in its group number 5599, the neural processing unit 353 receives a value three in its group number 5599, and the neural processing unit 6 is in its group The internal number input 5599 receives the value two.

In addition, when control input 213 specifies a predetermined value, denoted herein as "SELF," multiplex register 208 selects output buffer 1104 output 4905 corresponding to the number of group input 5599 in the group. Therefore, when a non-architectural instruction specifies the receipt of data from the output buffer 1104 by the value of SELF (indicated as OUTBUF[SELF] in the instructions of addresses 57 and 7 in Figure 57), each neural processing unit 126 The multiplex register 208 receives its corresponding text from the output buffer 1104. Thus, for example, when the neural network unit 121 performs the non-architectural instructions of addresses 2 and 7 in the fifty-fifth diagram, the multiplexer 208 of the neural processing unit 73 selects the second of the four inputs 4905. The number (number 1) is input to receive the text 73 from the output buffer 1104, and the multiplex register 208 of the neural processing unit 353 selects the fourth (number 3) input among the four inputs 4905 to receive the output buffer. The text 353 of 1104, and the multiplex register 208 of the neural processing unit 6 selects the third (number 2) input among the four inputs 4905 to receive the text 6 from the output buffer 1104. Although not used in the non-architecture of Figure 57, the non-architected instructions may also use the SELF value (OUTBUF[SELF]) to specify that the data from the output buffer 1104 is received and the control input 713 to specify a preset value. The multiplex register 705 of each neural processing unit 126 receives its corresponding text from the output buffer 1104.

Figure 56 is a block diagram showing the neural network unit 121 when the neural network unit performs the calculation associated with the Jordan time recurrent neural network of the 43rd diagram and utilizes the embodiment of the fifty-fifth diagram. Data random access memory 122 and weight random access memory 124 An example of data configuration. The weighting in the weight random access memory 124 in the figure is the same as the example in the forty-fourth figure. The configuration of the values in the data random access memory 122 is similar to the example of the forty-fourth figure, except that in this example, each time step has a corresponding one pair of two columns of memory to load the input layer node D. The value is compared to the output layer node Y value, rather than the set of four columns of memory as in the example of the forty-fourth figure. That is, in this example, the hidden layer Z value and the content layer C value are not written to the data random access memory 122. Rather, the output buffer 1104 is used as a hidden layer Z value and a content layer C value as a category of draft memory, as described in the non-architectural program of FIG. The feedback feature of the OUTBUF[SELF] output buffer 1104 can make the operation of the non-architected program faster (this is the two write and read operations to be performed on the data random access memory 122 for The output buffer 1104 performs two writes and two read operations instead of) and reduces the space of the data random access memory 122 used in each time step, so that the data random access memory 122 of the present embodiment The loaded data can be used for approximately twice the time steps of the fourty-fourth and forty-fifth embodiments, as shown in the figure, i.e., 32 time steps.

The fifty-seventh diagram is a table showing a program stored in the program memory 129 of the neural network unit 121. The program is executed by the neural network unit 121 and uses data and weights according to the configuration of the fifty-sixth figure. To achieve the Jordan time recurrent neural network. The non-architectural program of Figure 57 is similar to the non-architectural program of Figure 45. The differences are as follows.

The sample program of Figure 57 has 12 non-architectures The instructions are located at addresses 0 through 11, respectively. The initialization instruction of address 0 clears the accumulator 202 and initializes the value of the loop counter 3804 to 32, causing the loop group (instructions of addresses 2 through 11) to execute 32 times. The output instruction of address 1 places the zero value of accumulator 202 (cleared by the instruction of address 0) into output buffer 1104. It can thus be observed that during the execution of the instructions of addresses 2 to 6, the 512 neural processing units 126 correspond to and operate as 512 hidden layer nodes Z, and the execution of the instructions at addresses 7 to 10. In the process, it corresponds to and operates as 512 output layer nodes Y. That is, 32 executions of the instructions of addresses 2 through 6 calculate the hidden layer node Z values for the 32 corresponding time steps and place them in the output buffer 1104 for the corresponding address of the instructions 7 through 9. 32 executions are used to calculate the output layer node Y of the 32 corresponding time steps and write it to the data random access memory 122, and provide the corresponding 32 executions of the instruction of the address 10 to use this The content layer node C of the 32 corresponding time steps is placed in the output buffer 1104. (The content layer node C placed in the 32nd time step in the output buffer 1104 is not used.)

In the first execution of the instructions of addresses 2 and 3 (ADD_D_ACC OUTBUF[SELF] and ADD_D_ACC ROTATE, COUNT=511), each of the 512 neural processing units 126 will have 512 of the output buffer 1104. The content node C values are accumulated to their accumulators 202, which are generated and written by instruction execution of addresses 0 through 1. In the second execution of the instructions of addresses 2 and 3, each of the 512 neural processing units 126 will accumulate 512 content node C values of the output buffer 1104 to its accumulator 202. Content node C value is an instruction from address 7 to 8 and 10. Execution is generated and written. More precisely, the instruction of address 2 will instruct the multiplexer 208 of each neural processing unit 126 to select its corresponding output buffer 1104 text, as previously described, and add it to accumulator 202; address 3 The instructions instruct the neural processing unit 126 to rotate the content node C value within a 512-character rotator that is collectively operated by the multiplexed registers 208 of the 512 neural processing units. It is constructed such that each neural processing unit 126 can accumulate the 512 content node C values to its accumulator 202. The instruction of address 3 does not clear the accumulator 202, so the instruction of the address 4 and 5 can add the input layer node D value (multiplied by its corresponding weight) by the instruction of the address 2 and 3. Content layer node C value.

In each execution of the instructions of addresses 4 and 5 (MULT-ACCUM DR ROW+2, WR ROW 0 and MULT-ACCUM ROTATE, WR ROW+1, COUNT=511), the 512 neural processing units 126 Each neural processing unit 126 performs 512 multiplication operations to associate the data random access memory 122 with the current time step (eg, column 0 for time step 0 and column 0 for time step 1) Column 2, and so on, for the 512 input node D values for column 62) for time step 31, multiplied by columns 0 through 511 of weight random access memory 124 corresponding to the neural processing unit 126 The weights are weighted to produce 512 products, and the accumulated results of the 512 content node C values, together with the instructions of the addresses 2 and 3, are added to the accumulator 202 of the corresponding neural processing unit 126 to calculate the concealment. Node Z layer value.

In each execution of the instruction of address 6 (OUTPUT PASSTHRU, NOP, CLR ACC), these 512 neural processing units The 512 accumulator 202 values of 126 are passed to and written to the corresponding text of the output buffer 1104, and the accumulator 202 is cleared.

During the execution of the instructions of addresses 7 and 8 (MULT-ACCUM OUTBUF[SELF], WR ROW 512 and MULT-ACCUM ROTATE, WR ROW+1, COUNT=511), each of the 512 neural processing units 126 The neural processing unit 126 performs 512 multiplication operations, and the 512 hidden node Z values in the output buffer 1104 (generated and written by the corresponding execution of the instructions of the addresses 2 to 6) are multiplied by the weights. The weights corresponding to the rows of the neural processing unit 126 in the columns 512 through 1023 of the memory 124 are taken to generate 512 multiply accumulates to the accumulator 202 of the corresponding neural processing unit 126.

In each execution of the instruction of the address 9 (OUTPUT ACTIVATION FUNCTION, DR OUT ROW+2), a start function (such as a hyperbolic tangent function, a sigmoid function, a correction function) is performed on the 512 accumulated values to calculate Output node Y value, this output node Y value will be written into the data random access memory 122 corresponding to the current time step column (for example: for time step 0 is column 1, for time step 1 For column 3, and so on, for time step 31 it is column 63). The instruction of address 9 does not clear accumulator 202.

In each execution of the instruction of address 10 (OUTPUT PASSTHRU, NOP, CLR ACC), the 512 values accumulated by the instructions of addresses 7 and 8 are placed in the output buffer 1104 for the instructions of addresses 2 and 3. The next execution is used and the accumulator 202 is cleared.

The loop command of address 11 will decrement the value of the loop counter 3804, and if the value of the new loop counter 3804 is still greater than Zero indicates the instruction to return to address 2.

As described in the section corresponding to the forty-fourth figure, in the example of performing the Jordan time recurrent neural network using the non-architectural program of the fifty-seventh figure, a start function is applied to the accumulator 202 value to generate an output. The layer node Y value, however, this example assumes that the accumulator 202 value is passed to the content layer node C before the start function is applied, rather than passing the true output layer node Y value. However, for a Jordan time recurrent neural network that applies a start function to the accumulator 202 value to produce a content layer node C, the instruction of address 10 will be removed from the non-architectural program of the fifty-seventh figure. In the embodiments described herein, the Elman or Jordan time recurrent neural network has a single hidden node layer (such as the fortieth and forty-two maps), however, it is to be understood that these processors 100 and neural networks Embodiments of unit 121 can efficiently perform computations associated with temporal recurrent neural networks having multiple hidden layers, in a manner similar to that described herein.

As described above in the section corresponding to the second figure, each neural processing unit 126 operates as a neuron within an artificial neural network, and all of the neural processing units 126 in the neural network unit 121 are processed in large scale parallel. The way to effectively calculate the neuron output value at one level of this network. The parallel processing of this neural network unit, especially the rotator collectively constructed using the neural processing unit multiplex register, is not intuitively thought of as a conventional way of calculating the output of the neuron layer. Further, the traditional approach usually involves the calculation of a single neuron or a very small subset of neurons (for example, using parallel arithmetic units to perform multiplication and addition calculations), and then continue to perform associations with the same The calculation of a neuron below a level, and so on, continues in a sequential manner until the calculation of all neurons in this hierarchy is completed. In contrast, the present invention, in each time-frequency period, all of the neural processing units 126 (neurons) of the neural network unit 121 perform parallel execution of a small set associated with the generation of all neuron outputs (eg, a single A multiplication and accumulation calculation). At the end of approximately M time-frequency periods -M is the number of nodes connected in the current level - the neural network unit 121 calculates the output of all neurons. In many artificial neural network configurations, because of the large number of neural processing units 126, the neural network unit 121 can calculate its neuron output values for all neurons of the entire hierarchy at the end of the M time-frequency periods. As described herein, this calculation is efficient for all types of artificial neural network computations, including but not limited to feedforward and time recurrent neural networks such as Elman, Jordan, and long- and short-term memory networks. . Finally, although in the embodiments herein, the neural network unit 121 is configured as 512 neural processing units 126 (eg, taking a wide text configuration) to perform the calculation of the time recurrent neural network, the invention is not limited thereto, The neural network unit 121 is configured as 1024 neural processing units 126 (e.g., employing a narrow text configuration) to perform the calculation of the time recurrent neural network unit, and other numbers of neural processing units 126 having 512 and 1024 as previously described. The neural network unit 121 is also within the scope of the present invention.

The above is only the preferred embodiment of the present invention, and the scope of the invention is not limited thereto, that is, the simple equivalent changes and modifications made by the scope of the invention and the description of the invention are All remain within the scope of the invention patent. for example, The software can perform the functions, manufacture, shaping, simulation, description, and/or testing of the devices and methods described herein. This can be achieved by a general programming language (such as C, C++), a hardware description language (HDL) containing Verilog HDL, VHDL, etc., or other existing programs. The software can be placed on any known computer usable medium such as magnetic tape, semiconductor, magnetic disk, optical disk (such as CD-ROM, DVD-ROM, etc.), network cable, wireless or other communication medium. Embodiments of the apparatus and methods described herein can be included in a semiconductor intellectual property core, such as a micro processing core (such as an embodiment in a hardware description language) and converted to hardware by fabrication of integrated circuits. In addition, the devices and methods described herein may also comprise a combination of hardware and software. Therefore, any embodiments described herein are not intended to limit the scope of the invention. Furthermore, the present invention is applicable to a microprocessor device of a general-purpose computer. Finally, it is common for those skilled in the art to make use of the concepts and embodiments disclosed herein to form and adapt various structures to achieve the same objectives without departing from the scope of the invention.

100‧‧‧ processor

101‧‧‧Command capture unit

102‧‧‧ instruction cache

103‧‧‧Architecture Instructions

104‧‧‧Instruction Translator

105‧‧‧Microinstructions

106‧‧‧Renaming unit

108‧‧‧Reservation station

112‧‧‧Other execution units

114‧‧‧ memory subsystem

116‧‧‧Universal register

118‧‧‧Media register

121‧‧‧Neural Network Unit

122‧‧‧ Data Random Access Memory

124‧‧‧ weighted random access memory

126‧‧‧Neural Processing Unit

127‧‧‧Control and Status Register

128‧‧‧Sequencer

129‧‧‧Program memory

123,125,131‧‧‧ memory address

133‧‧‧ Results

Claims (23)

An apparatus comprising: a plurality of arithmetic logic units, each of the arithmetic logic units having: an accumulator; and an integer arithmetic unit that receives an integer input and performs an integer arithmetic operation thereon, and an integer of a series of integer arithmetic operations The result is accumulated to the accumulator as an integer accumulated value; a temporary register can be programmed by one of the index of the number of fractional bits of the integer accumulated value and the number of fractional bits of the integer output; wherein One of the accumulators has a first bit width that is greater than twice the width of the second bit of the integer output; and a plurality of adjustment units that are based on the number of decimal places of the integer accumulated value programmed in the register The indicator and the indicator of the number of decimal places output by the integer, scale and saturate the first bit width accumulated value to produce the second bit width integer output. The apparatus of claim 1, wherein the adjusting unit calculates a difference between the number of the decimal places of the output and the number of decimal places of the integer accumulated value, and the adjusting unit adds the integer accumulated value The offset of the difference is shifted to the right to scale the integer accumulated value. The device of claim 1, wherein the integer accumulated value after shifting to the right is greater than/less than the second bit width The degree represents the maximum/minimum value, and the adjustment unit fills the integer accumulated value to a maximum/minimum value that can be represented by the second bit width to fill the integer accumulated value. The apparatus of claim 1, wherein the accumulator has at least Q storage bits to store the integer accumulated value, wherein Q is a number of bits sufficient to accumulate a series of P integer results without loss of precision . The apparatus of claim 4, wherein Q is equal to M plus log2P, wherein M is one of the bit widths of the integer result. The device of claim 5, wherein P is one of the series of integer results that produces the integer accumulated value is a preset maximum allowable number. The apparatus of claim 5, wherein when the integer input has a bit width of 8, P is 1023 and Q is 28. The apparatus of claim 5, wherein when the integer input has a bit width of 16, P is 511 and Q is 41. The device of claim 1, further comprising: a memory to load the integer input and provide the plurality of arithmetic logic units. The device of claim 9, wherein the plurality of adjustment units write the integer input into the memory. The apparatus of claim 1, wherein each of the arithmetic logic units further comprises: an integer multiplier performing one integer multiplication of the integer input to generate an integer product; and an integer adder executing The integer product is summed with one of the integer accumulated values to produce an integer sum total stored back to the accumulator. The device of claim 1, wherein each of the adjusting units further comprises: a rounder, rounding the integer accumulated value according to the least significant J bits of the integer accumulated value, wherein Is the difference between the number of decimal places of the output and the number of decimal places of the integer accumulated value. The device of claim 1, wherein the register is further programmed by a first indicator and a second indicator, the first indicator indicating that the data received from the first memory is signed or not. A symbol indicating that the weight text received from the second memory is signed or unsigned. Such as the device of claim 1 of the patent scope, wherein the whole The one-bit width of the number input is the same as the second bit width of the integer output. The device of claim 1, wherein the device comprises a neural network unit; the adjusting unit comprises a start function unit for performing a start function on the scaled or saturated integer accumulated value, It is normalized in a non-linear manner to produce a result that falls within a predetermined range of values. A method comprising: programstizing a temporary register by using one of an indicator of the number of fractional bits of the integer accumulated value and the number of fractional bits of the integer output; using a plurality of arithmetic logic units, each having one An accumulator and an arithmetic logic unit of an integer arithmetic unit: using the integer arithmetic unit, performing an integer arithmetic operation on the integer input; and accumulating a series of integer result of the integer arithmetic operations to the accumulator as an integer accumulated value; Wherein the first bit width of one of the accumulators is greater than twice the width of the second bit of the integer output; and the adjusting unit of the plurality of adjusting units is programmed according to the temporary register The indicator of the number of decimal places of the integer accumulated value and the number of decimal places of the integer output, scaling and filling the first bit width accumulated value to generate the second bit width integer output . The method of claim 16, further comprising: calculating, by using the adjusting unit, a difference between the number of the decimal places of the output and the number of decimals of the integer accumulated value; wherein the adjusting unit is used to scale the The step of incrementing the integer includes moving the integer accumulated value to the right by the offset of the difference. The method of claim 16, wherein the step of filling the integer accumulated value comprises: when the rightward shifting, the integer accumulated value is greater than/less than a maximum/minimum value that can be represented by the second bit width, The adjustment unit fills the integer accumulated value to a maximum/minimum value that can be represented by the second bit width. The method of claim 16, wherein the accumulator has at least Q storage bits to store the integer accumulated value, wherein Q is a number of bits sufficient to accumulate a series of P integer results without loss of precision . The method of claim 19, wherein Q is equal to M plus log2P, wherein M is one of the bit widths of the integer result. For example, the method of claim 16 of the patent scope further includes: using each of the rounding units, the minimum effective J according to the integer accumulated value a bit, the rounding operation is performed on the integer accumulated value, where J is the difference between the number of the decimal places of the output and the number of the decimal places of the integer accumulated value. For example, the method of claim 16 further includes: using the adjusting unit, performing a starting function on the scaled or saturated integer accumulated value, and performing a normalization operation in a nonlinear manner to generate A result that falls within a predetermined range of values. A computer program product encoded in at least one non-transitory computer usable medium for use by a computer device, comprising: a computer usable code embodied in the medium for describing a neural network unit, the computer The use code includes: a first code to describe a plurality of arithmetic logic units, each of the arithmetic logic units having an accumulator and an integer arithmetic unit, the integer arithmetic unit receiving an integer input and performing an integer arithmetic operation thereon, and A series of integer arithmetic operations of the integer operations are accumulated to the accumulator as an integer accumulated value; a second code is used to describe a register, and the register can utilize the number of decimal places of the integer accumulated value One indicator and one of the number of decimal places of the integer output are programmed; wherein the first bit width of one of the accumulators is greater than twice the width of the second bit of the integer output; and the third code For describing a plurality of adjustment units, according to the number of decimal places of the integer accumulated value programmed in the register The indicator and the indicator of the number of decimal places output by the integer, scale and saturate the first bit width accumulated value to produce the second bit width integer output.