TW201714119A - Neural network unit with neural memory and array of neural processing units that collectively shift row of data received from neural memory - Google Patents

Abstract

An array of N processing units (PU) each has: an accumulator; an arithmetic unit performs an operation on first, second and third inputs to generate a result to store in the accumulator, the first input receives the accumulator output; a weight input is received by the second input to the arithmetic unit; a multiplexed register has first and second data inputs, an output received by the third input to the arithmetic unit, and a control input that controls the data input selection. The multiplexed register output is also received by an adjacent PU's multiplexed register second data input. The N PU's multiplexed registers collectively operate as an N-word rotater when the control input specifies the second data input. Respective first/second memories hold W/D rows of N weight/data words and provide the N weight/data words to the corresponding weight/ multiplexed register first data inputs of the N PUs.

Description

Neural network unit with neural memory and a neural processing unit array that collectively receives data from the neural memory

This application claims the international priority of the following U.S. Provisional Application. The entire contents of these priority cases are incorporated herein by reference.

This application is related to the following US provisional application filed concurrently. The entire contents of these related applications are incorporated herein by reference.

In recent years, artificial neural networks (ANN) have re-engaged people's attention. These studies are often referred to as deep learning, computer learning, and the like. The increased computing power of general-purpose processors has also boosted people's interest in artificial neural networks decades later. Recent applications of artificial neural networks include language and image recognition. The need to improve the computational efficiency and efficiency of artificial neural networks seems to be increasing.

In view of the above, the present invention provides an apparatus comprising an array of N processing units, a first memory and a second memory. Each of the processing units in the array of N processing units includes an accumulator, an arithmetic unit, a weight input, and a multiplex register. The accumulator has an output. The arithmetic unit has first, second and third inputs and performs an operation thereon to generate a result stored to the accumulator, the first input receiving the output of the accumulator. The weight input is received by the aforementioned second input to the arithmetic unit. The multiplex register has first and second data inputs, an output and a control input, and the output of the multiplex register is received by the third input to the arithmetic unit, and the control input controls the first and second data. Enter the choice. Wherein, the output of the multiplex register is received by the second data input of the multiplex register of an adjacent processing unit, and when the control input selects the second data input, the multiplex register of the N processing units The rotators that work together as the same N characters. The first memory system loads N weight texts of W columns and provides N weight texts of one of the W columns to the corresponding weight inputs of the N processing units of the processing unit array. The second memory system loads N data words of D columns, and provides N data words of one of the D columns to the corresponding first data of the multiplexed registers of the N processing units of the processing unit array. Input.

The invention also provides a processor. The processor includes an instruction set, an array of N processing units, a first memory and a second memory. The instruction set has architectural instructions to indicate that the processor is operational. Each of the processing units in the array of N processing units includes an accumulator, an arithmetic unit, a weight input, and a multiplex register. The accumulator has an output. The arithmetic unit has first, second and third inputs and performs an operation thereon to generate a result stored to the accumulator, the first input receiving the output of the accumulator. The weight input is received by the aforementioned second input to the arithmetic unit. The multiplex register has first and second data inputs, an output and a control input, and the output of the multiplex register is received by the third input to the arithmetic unit, and the control input controls the first and second data. Enter the choice. Among them, the multiplex register The output is received by the second data input of the multiplex register of an adjacent processing unit. When the control input selects the second data input, the multiplexed registers of the N processing units work together as the same N Text rotator. The first memory system loads N weight texts of W columns and provides N weight texts of one of the W columns to the corresponding weight inputs of the N processing units of the processing unit array. The second memory system loads N data words of D columns, and provides N data words of one of the D columns to the corresponding first data of the multiplexed registers of the N processing units of the processing unit array. Input.

The present invention also provides a computer program product encoded in at least one non-transitory computer usable medium for use by a computer device. The computer program product includes a computer usable code embodied in the medium for describing a device. The computer can use the code including the first code, the second code and the third code. The first code specifies an array of N processing units. Each processing unit in the array includes an accumulator, an arithmetic unit, a weight input, and a multiplex register. The accumulator has an output. The arithmetic unit has first, second and third inputs and performs an operation thereon to generate a result stored to the accumulator, the first input receiving the output of the accumulator. The weight input is received by the aforementioned second input to the arithmetic unit. The multiplex register has first and second data inputs, an output and a control input, and the output of the multiplex register is received by the third input to the arithmetic unit, and the control input controls the first and second data. Enter the choice. Wherein, the output of the multiplex register is received by the second data input of the multiplex register of an adjacent processing unit, and when the control input selects the second data input, N places The multiplexer registers of the management unit work together as a rotator of the same N characters. The second code specifies a first memory to load N weight words of the W columns, and provides N weight words of one of the W columns to the N processing units of the processing unit array. Weight input. The third code specifies a second memory to load N data words of the D columns, and provides N data characters of one of the D columns to the multiplex of the N processing units of the processing unit array. The corresponding data input of the register is the first data input.

The specific embodiments of the present invention will be further described by the following examples and drawings.

100‧‧‧ processor

101‧‧‧Command capture unit

102‧‧‧ instruction cache

103‧‧‧Architecture Instructions

104‧‧‧Instruction Translator

105‧‧‧Microinstructions

106‧‧‧Renaming unit

108‧‧‧Reservation station

112‧‧‧Other execution units

114‧‧‧ memory subsystem

116‧‧‧Universal register

118‧‧‧Media register

121‧‧‧Neural Network Unit

122‧‧‧ Data Random Access Memory

124‧‧‧ weighted random access memory

126‧‧‧Neural Processing Unit

127‧‧‧Control and Status Register

128‧‧‧Sequencer

129‧‧‧Program memory

123,125,131‧‧‧ memory address

133, 215, 133A, 133B, 215A, 215B‧‧‧ Results

202‧‧‧ accumulator

204‧‧‧Arithmetic Logic Unit

203,209,217,203A,203B,209A,209B,217A,217B‧‧‧ Output

205, 205A, 205B‧‧‧ register

206,207,211,1811,206A,206B,207A,207B,211A,211B,1811A,1811B,711‧‧‧ Input

208,705,208A,208B‧‧‧Multiplex register

213,713,803‧‧‧Control input

212‧‧‧Start function unit

242‧‧‧Multiplier

244‧‧‧Adder

246,246A, 246B‧‧‧ product

802‧‧‧Multiplexer

1104‧‧‧ column buffer

1112‧‧‧Start function unit

1400‧‧‧MTNN Directive

1500‧‧‧MFNN Directive

1432, 1532‧‧‧ function

1402, 1502‧‧‧ Code field

1404‧‧‧src1 field

1406‧‧‧src2 field

1408, 1508‧‧‧gpr field

1412, 1512‧‧‧ immediate field

1422, 1522‧‧‧ address

1424, 1426, 1524‧‧‧ data block

1428, 1528‧‧‧ Selected columns

1434‧‧‧Control logic

1504‧‧dst field

1602‧‧‧Reading

1604‧‧‧written 埠

1606‧‧‧Memory array

1702‧‧‧埠

1704‧‧‧buffer

1898‧‧‧Operator selection logic

1896A‧‧‧Wide multiplexer

1896B‧‧‧Narrow multiplexer

242A‧‧‧Wide Multiplier

242B‧‧‧Narrow multiplier

244A‧‧‧ wide adder

244B‧‧‧Narrow adder

204A‧‧‧wide arithmetic logic unit

204B‧‧‧Narrow arithmetic logic unit

202A‧‧‧Wide accumulator

202B‧‧‧Narrow Accumulator

212A‧‧‧wide start function unit

212B‧‧‧Narrow start function unit

2402‧‧‧Convolution Core

2404‧‧‧Data Array

2406A, 2406B‧‧‧ Data Matrix

2602, 2604, 2606, 2608, 2902, 2912, 2914, 2922, 2924, 2926, 2932, 2934, 2942, 2944, 2952, 2954, 2956, 2923, 2962, 2964‧‧‧

3003‧‧‧ Random bit source

3005‧‧‧ random bit

3002‧‧‧Positive type converter and output binary decimal point aligner

3004‧‧‧ rounder

3006‧‧‧Multiplexer

3008‧‧‧Standard size compressor and saturator

3012‧‧‧ bit selection and saturator

3018‧‧‧ Corrector

3014‧‧‧Reciprocal multiplier

3016‧‧‧ Right shifter

3028‧‧‧Standard size transfer value

3022‧‧‧Hyper tangent module

3024‧‧‧S type module

3026‧‧‧Soft Plus Module

3032‧‧‧Multiplexer

3034‧‧‧ Symbol Recovery

3036‧‧‧Dimensional Converter and Saturator

3037‧‧‧Multiplexer

3038‧‧‧Output register

3402‧‧‧Multiplexer

3401‧‧‧Neuro Processing Unit Pipeline Level

3404‧‧‧Decoder

3412, 3414, 3418‧‧‧ micro operations

3416‧‧‧ microinstructions

3422‧‧‧ model indicators

3502‧‧‧Time-frequency generation logic

3504‧‧‧Time-frequency reduction logic

3514‧‧‧Interface logic

3522‧‧‧ Data Random Access Memory Buffer

3524‧‧‧ weight random access memory buffer

3512‧‧‧Relief indicator

3802‧‧‧Program Counter

3804‧‧‧Circle counter

3806‧‧‧ Iteration number counter

3912, 3914, 3916‧‧‧ fields

4901‧‧‧Neuro Processing Unit Group

4903‧‧‧ mask

4905, 4907, 5599‧‧‧ Input

The first figure shows a block diagram of a processor including a neural network unit (NNU).

The second figure shows a block diagram of a neural processing unit (NPU) of the first figure.

The third figure is a block diagram showing N multiplex registers of N neural processing units of the neural network unit of the first figure, and one column of data is obtained for the data random access memory of the first figure. The operation of a rotator or a circular shifter such as the same N characters is performed.

The fourth diagram is a table showing a program stored in the program memory of the neural network unit of the first figure and executed by the neural network unit.

The fifth figure shows the timing of the execution of the program in the fourth diagram of the neural network unit. Figure.

Figure 6A is a block diagram showing the execution of the fourth diagram of the neural network unit of the first figure.

Figure 6B is a flow chart showing that the processor of the first figure executes an architectural program to perform a typical multiply-accumulate start function operation of a neuron associated with a hidden layer of an artificial neural network using a neural network unit As the operation performed by the program of the fourth figure.

Figure 7 is a block diagram showing another embodiment of the neural processing unit of the first figure.

The eighth figure is a block diagram showing still another embodiment of the neural processing unit of the first figure.

The ninth figure is a table showing a program stored in the program memory of the neural network unit of the first figure and executed by the neural network unit.

The tenth figure shows a timing diagram of the execution of the program of the ninth diagram by the neural network unit.

Figure 11 is a block diagram showing one embodiment of a neural network unit of the first figure. In the embodiment of the eleventh diagram, a neuron is divided into two parts, that is, a start function unit part and an arithmetic logic unit part (this part includes a shift register part), and each start function unit part is composed of The arithmetic logic units are partially shared.

Figure 12 is a timing diagram showing the execution of the fourth diagram of the neural network unit of the eleventh figure.

The thirteenth figure is a timing chart showing the execution of the program of the fourth figure by the neural network unit of the eleventh figure.

Figure 14 is a block diagram showing the movement to a neural network (MTNN) architecture instructions and their operation corresponding to portions of the neural network unit of the first figure.

The fifteenth diagram is a block diagram showing the operation of a mobile-to-neural network (MTNN) architecture instruction and its portion of the neural network unit corresponding to the first diagram.

Figure 16 is a block diagram showing an embodiment of the data random access memory of the first figure.

Figure 17 is a block diagram showing an embodiment of a weighted random access memory and a buffer of the first figure.

Figure 18 is a block diagram showing a dynamically configurable neural processing unit of the first figure.

Figure 19 is a block diagram showing the data of the first image by using the NN neural processing units of the neural network unit of the first figure according to the embodiment of the eighteenth embodiment. The random access memory obtains one of the data texts to perform the operation as the same rotator.

Figure 20 is a table showing a program stored in the program memory of the neural network unit of the first figure and executed by the neural network unit, and the neural network unit has the embodiment as shown in the eighteenth The neuroprocessing unit shown.

The twenty-first figure shows a timing diagram of a neural network unit executing the program of the twentieth diagram, the neural network unit having the neural processing unit as shown in the eighteenth diagram executed in a narrow configuration.

The twenty-second diagram shows a block diagram of the neural network unit of the first figure, the neural network unit having a neural processing list as shown in the eighteenth Yuan to execute the program of the twentieth chart.

A twenty-third figure is a block diagram showing another embodiment of a dynamically configurable neural processing unit of the first figure.

The twenty-fourth diagram is a block diagram showing an example of a data structure used by the neural network unit of the first figure to perform a convolution operation.

The twenty-fifth diagram is a flowchart showing that the processor of the first figure executes an architectural program to perform a convolution operation of the convolution kernel according to the data array of the twenty-fourth figure using the neural network unit.

The twenty-sixth A diagram is a list of programs of a neural network unit program that performs a convolution operation on a data matrix using the convolution kernel of the twenty-fourth graph and writes it back to the weight random Access memory.

Figure 26B is a block diagram showing one embodiment of certain fields of the control register of the neural network unit of the first figure.

Figure 27 is a block diagram showing an example of weighted random access memory filled with input data in the first figure. The input data is performed by the neural network unit of the first figure. ).

The twenty-eighth figure is a list of programs of a neural network unit program that performs the common source operation of the input data matrix of the twenty-seventh figure and writes it back to the weighted random access memory.

Figure 29A is a block diagram showing one embodiment of the control register of the first figure.

Figure 29B is a block diagram showing another embodiment of the control register of the first figure.

The twenty-ninth C-picture shows that the twenty-ninth A picture is stored in two parts. A block diagram of one embodiment of a reciprocal.

Figure 30 is a block diagram showing an embodiment of an activation function unit (AFU) of the second diagram.

The thirty-first figure shows an example of the operation of the start function unit of the thirtieth figure.

Figure 32 shows a second example of the operation of the start function unit of Figure 30.

The thirty-third figure shows a third example of the operation of the start function unit of the thirtieth figure.

The thirty-fourth diagram is a block diagram showing a portion of the details of the processor of the first diagram and the neural network unit.

The thirty-fifth diagram is a block diagram showing a processor having a variable rate neural network unit.

The thirty-sixth A diagram is a timing diagram showing an operational example in which a processor having a neural network unit operates in a general mode, which operates at a primary time frequency.

The thirty-sixth B diagram is a timing diagram showing an operation example in which a processor having a neural network unit operates in a mitigation mode, and the mitigation mode operates at a lower frequency than the primary time.

Figure 37 is a flow chart showing the operation of the processor of the thirty-fifth figure.

The thirty-eighth figure is a block diagram showing in detail the sequence of neural network elements.

The thirty-ninth figure is a block diagram showing certain fields of the control and status register of the neural network unit.

The fortieth is a block diagram showing the Elman time recurrent neural network An example of (recurrent neural network, RNN).

The forty-first figure is a block diagram showing the data random access memory and weight random access of the neural network unit when the neural network unit performs the calculation of the Elman time recurrent neural network associated with the fortieth map. An example of data configuration in memory.

Figure 42 shows a table showing a program memory stored in a neural network unit, which is executed by a neural network unit and uses data and weights according to the configuration of the 41st figure to achieve Elman time recurrent neural network

A forty-third figure is a block diagram showing an example of a Jordan time recurrent neural network.

The forty-fourth figure is a block diagram showing that when the neural network unit performs the calculation of the Jordan time recurrent neural network associated with the forty-third figure, the data random access memory and the weight of the neural network unit are randomly stored. Take an example of data configuration in memory.

The forty-fifth diagram is a table showing a program stored in a neural network unit of a program memory, the program is executed by the neural network unit, and the data and weights are used according to the configuration of the forty-fourth figure to achieve Jordan time recurrent neural network.

The forty-sixth diagram is a block diagram showing one embodiment of a long short term memory (LSTM) cell.

The forty-seventh diagram is a block diagram showing the data random access memory and weight random access of the neural network unit when the neural network unit performs the calculation associated with the long-short-term memory cell layer of the forty-sixth figure. An example of data configuration in memory.

The forty-eighth figure is a table showing a program memory stored in a neural network unit, which is executed by the neural network unit and uses data and weights according to the configuration of the 47th figure to achieve association. Calculation of the long and short term memory cell layer.

A forty-ninth diagram is a block diagram showing an embodiment of a neural network unit having an output buffer masking and feedback capability within the neural processing unit group of this embodiment.

Figure 50 is a block diagram showing the data random access memory of the neural network unit of the forty-ninth figure when the neural network unit performs the calculation of the long-short-term memory cell layer associated with the forty-sixth figure. An example of data configuration in weighted random access memory and output buffers.

The fifty-first figure is a table showing a program memory stored in the neural network unit, which is executed by the neural network unit of the forty-ninth figure and uses the data according to the configuration of the fifty-fifth figure. Weights are calculated to achieve a correlation with the long- and short-term memory cell layers.

A fifty-second diagram is a block diagram showing an embodiment of a neural network unit having an output buffer masking and feedback capability within the group of neural processing units of this embodiment, and sharing a start function unit.

The fifty-third figure is a block diagram showing the data random access memory of the neural network unit of the forty-ninth figure when the neural network unit performs the calculation of the long-short-term memory cell layer associated with the forty-sixth figure. Another embodiment of the configuration of the data in the weighted random access memory and the output buffer.

Figure 54 shows a table showing a program memory stored in a neural network unit. The program is executed by the neural network unit of the forty-ninth figure and is used according to the configuration of the fifty-third figure. With weights The calculation is related to the long- and short-term memory cell layer.

Fifty-fifth Figure is a block diagram showing a portion of a neural processing unit in accordance with another embodiment of the present invention.

Figure 56 is a block diagram showing the neural network unit when the neural network unit performs the calculation of the Jordan time recurrent neural network associated with the forty-third figure and utilizes the embodiment of the fifty-fifth figure. An example of data configuration in data random access memory and weight random access memory.

Figure 57 shows a table showing a program memory stored in a neural network unit. The program is executed by the neural network unit and uses data and weights according to the configuration of Figure 56 to achieve Jordan. Time recurrent neural network.

Processor with an architectural neural network unit

The first figure shows a block diagram of a processor 100 including a neural network unit (NNU) 121. As shown in the figure, the processor 100 includes an instruction capture unit 101, an instruction cache 102, an instruction translator 104, a rename unit 106, a plurality of reservation stations 108, a plurality of media registers 118, and a plurality of The universal register 116 is a plurality of execution units 112 and a memory subsystem 114 outside the neural network unit 121.

The processor 100 is an electronic device that acts as a central processing unit of the integrated circuit. The processor 100 receives the input digital data, processes the data according to instructions fetched by the memory, and generates a processing result of the operation indicated by the instruction as its output. This processor 100 can be used for a table A laptop, mobile device, or tablet, and used for computing, word processing, multimedia display, and web browsing. The processor 100 can be disposed in an embedded system to control a variety of devices including devices, mobile phones, smart phones, vehicles, and industrial controllers. The central processing unit executes electronic circuits (ie, hardware) that execute instructions of a computer program (or computer application or application) by performing operations including arithmetic, logic, and input/output on the data. An integrated circuit is a group of electronic circuits that are fabricated on a small semiconductor material, usually germanium. Integrated circuits are also commonly used to represent wafers, microchips or dies.

The instruction fetching unit 101 controls the operation of the system memory 103 (not shown) to retrieve the architecture instruction 103 to the instruction cache 102. The instruction fetch unit 101 provides a fetch address to the instruction cache 102 to specify that the processor 100 fetches the memory address of the cache line of the architectural instruction byte of the cache memory 102. The selection of the retrieved address is based on the current value of the instruction indicator (not shown) of the processor 100 or a program counter. In general, the program counters are incremented according to the instruction size until a control instruction such as a branch, call, or return occurs in the instruction stream, or an exception condition such as an interrupt, a trap, an exception, or an error occurs. The program counter is updated with non-sequential addresses such as branch target address, return address, or exception vector. In summary, the program counter is updated in response to execution of the execution unit 112/121. The program counter may also be updated upon detection of an exception condition, such as instruction instruction 104 encountering instruction 103 that is not defined in the instruction set architecture of processor 100.

The instruction cache 102 is stored from a fabric instruction 103 coupled to the system memory of the processor 100. These architectural instructions 103 This includes a Move to Neural Network (MTNN) command and a Neural Network Removal (MFNN) command, as described later. In one embodiment, the architectural instructions 103 are instructions of the x86 instruction set architecture and are appended with MTNN instructions and MFNN instructions. In the context of this disclosure, an x86 instruction set architecture processor is understood to be a processor that produces the same results as an Intel® 80386® processor at the instruction set architecture level while executing the same mechanical language instructions. However, other instruction set architectures, such as the Advanced Reduced Instruction Set Machine Architecture (ARM), Sun's Scalable Processor Architecture (SPARC), or the Enhanced Reduced Instruction Set Performance Computing Performance Architecture (PowerPC), It can also be used in other embodiments of the invention. Instruction cache 102 provides architectural instructions 103 to instruction translator 104 to translate architectural instructions 103 into microinstructions 105.

Microinstructions 105 are provided to rename unit 106 and ultimately executed by execution unit 112/121. These microinstructions 105 implement architectural instructions. In a preferred embodiment, the instruction translator 104 includes a first portion for translating frequently executed and/or relatively less complex architectural instructions 103 into microinstructions 105. The instruction translator 104 also includes a second portion having a microcode unit (not shown). The microcode unit has a microcode memory loaded microcode instruction to execute complex and/or less useful instructions in the architectural instruction set. The microcode unit includes a microsequencer that provides a non-architected microprogram counter (micro-PC) to the microcode memory. In a preferred embodiment, the microinstructions are translated into microinstructions 105 via a micro-translator (not shown). The selector selects the microinstruction 105 from the first portion or the second portion to provide the rename unit 106 depending on whether the microcode unit currently has control.

Renaming unit 106 will specify framework instruction 103 The architecture register is renamed to the physical scratchpad of processor 100. In a preferred embodiment, the processor 100 includes a rearrangement buffer (not shown). The rename unit 106 assigns the items of the rearrangement buffer to the respective microinstructions 105 in accordance with the program order. This allows the processor 100 to remove the microinstructions 105 and their corresponding architectural instructions 103 in accordance with the program sequence. In one embodiment, the media register 118 has a 256-bit width and the general-purpose register 116 has a 64-bit width. In one embodiment, media register 118 is an x86 media register, such as an Advanced Vector Expansion (AVX) register.

In one embodiment, each item of the rearrangement buffer has a storage space to store the results of the microinstructions 105. In addition, the processor 100 includes an architecture register file having a physical register corresponding to each architecture register, such as the media register 118, the general register 116, and other architecture temporary storage. Device. (For a preferred embodiment, for example, the media register 118 and the general register 116 are different in size, and a separate register file can be used to correspond to the two registers.) The instruction source 105 specifies a source operand of an architecture register, and the rename unit fills in the microinstruction 105 by using the reorder buffer directory of the latest microinstruction in the old microinstruction 105 written to the architecture register. The source operand field. When execution unit 112/121 completes execution of microinstruction 105, execution unit 112/121 writes its result to the reorder buffer entry of this microinstruction 105. When the microinstruction 105 is removed, the removal unit (not shown) writes the result from the reorder buffer field of the microinstruction to the scratchpad of the physical scratchpad file associated with the physical scratchpad file The schema destination register specified by the microinstruction 105 is thus removed.

In another embodiment, the processor 100 includes an entity The scratchpad file has more physical scratchpads than the number of architectural registers, however, the processor 100 does not include an architectural register file, and the result storage space is not included in the rearrangement buffer item. . (In a preferred embodiment, because the media register 118 and the general register 116 are different in size, a separate register file can be used to correspond to the two registers.) The processor 100 It includes an indicator table with corresponding indicators for each architecture register. For each operand specified in the microinstruction 105 with an architectural register, the rename unit fills the destination operand field in the microinstruction 105 with an indicator pointing to a free register in the physical scratchpad file. If there is no free register in the physical scratchpad file, the rename unit 106 will temporarily suspend the pipeline. For each source operand in the microinstruction 105 that specifies the architecture register, the rename unit will utilize a pointer to the entity register file and assign the latest microinstruction to the old microinstruction 105 of the write architecture register. The indicator of the register is filled in the source operand field in the microinstruction 105. When execution unit 112/121 completes execution of microinstruction 105, execution unit 112/121 writes the result to the destination operand field of microinstruction 105 in the physical scratchpad file. When the microinstruction 105 is removed, the removal unit copies the destination operand field value of the microinstruction 105 to the indicator associated with the indicator table of the schema destination register specified by the microinstruction 105.

The reservation station 108 will load the microinstructions 105 until they are ready to be issued to the execution unit 112/121 for execution. When all of the source operands of a microinstruction 105 are available and the execution unit 112/121 is also available for execution, the microinstruction 105 is ready for publication. Execution unit 112/121 is comprised of a rearrangement buffer or the aforementioned first implementation The architecture register file described in the example, or the physical register file of the foregoing second embodiment receives the scratchpad source operand. In addition, the execution unit 112/121 can receive the scratchpad source operand directly through the result transfer bus (not shown). Additionally, execution unit 112/121 can receive the immediate operand specified by microinstruction 105 from reservation station 108. The MTNN and MFNN architecture instructions 103 include an immediate operand to specify the functions to be performed by the neural network unit 121, and this functionality is provided by one or more microinstructions 105 generated by the MTNN and MFNN architecture instructions 103, as described below. .

Execution unit 112 includes one or more load/store units (not shown) that are loaded by memory subsystem 114 and store data to memory subsystem 114. In a preferred embodiment, the memory subsystem 114 includes a memory management unit (not shown), which may include, for example, a plurality of lookaside buffers, a table move ( A tablewalk unit, a hierarchy-data cache (with instruction cache 102), a hierarchy 2 unified cache, and a bus interface unit that serves as an interface between processor 100 and system memory. In one embodiment, the processor 100 of the first diagram is represented by one of a plurality of processing cores of a multi-core processor that shares a last-level cache memory. Execution unit 112 may include a plurality of integer units, a plurality of media units, a plurality of floating point units, and a branch unit.

The neural network unit 121 includes a weight random access memory (RAM) 124, a data random access memory 122, N neural processing units (NPU) 126, a program memory 129, and a sequencer 128 and more. Control and status register 127. These neural processing units 126 are conceptually like the functions of neurons in a neural network. Random weight The memory 124, the data random access memory 122, and the program memory 129 can be written and read through the MTNN and MFNN architecture instructions 103, respectively. The weight random access memory 124 is arranged in W columns, each column of N weight characters, and the data random access memory 122 is arranged in D columns, each column of N data characters. Each of the data texts and each weight text is a plurality of bits. For a preferred embodiment, it may be 8 bits, 9 bits, 12 bits, or 16 bits. Each data text is used as the output value of a neuron in the previous layer of the network (sometimes expressed as a startup value), and each weight text is used as a weight in the network associated with one of the neurons entering the current layer of the network. . Although in many applications of the neural network unit 121, the literal or operand loaded in the weighted random access memory 124 is actually associated with the weight of the link into a neuron, it should be noted that in the neural network In some applications of unit 121, the text loaded in weighted random access memory 124 is not weighted, but because the text is stored in weighted random access memory 124, it is still expressed in terms of "weighted text." For example, in some applications of the neural network unit 121, for example, an example of a convolution operation of the twenty-fourth to twenty-sixth A or an example of a common source operation of the twenty-seventh to twenty-eighth The weighted random access memory 124 loads objects other than weights, such as elements of a data matrix (such as image pixel data). Similarly, although in many applications of the neural network unit 121, the words or operands loaded in the data random access memory 122 are essentially the output values or activation values of the neurons, it should be noted that in the neural network In some applications of the way unit 121, the text loaded in the data random access memory 122 is not the case, but since the words are stored in the data random access memory 122, However, it is expressed in terms of “data”. For example, in some applications of neural network unit 121, such as the convolution operation of Figures 24 to 26A, data random access memory 122 will load non-neuron output, such as The element of the convolution kernel.

In an embodiment, the neural processing unit 126 and the sequencer 128 comprise combinational logic, sequencing logic, state machines, or a combination thereof. The architectural instructions (e.g., MFNN instruction 1500) load the contents of state register 127 into one of the general purpose registers 116 to confirm the state of neural network unit 121, such as neural network unit 121 having completed from program memory 129. The operation of a command or a program, or the neural network unit 121 is free to receive a new command or start a new neural network unit program.

The number of neural processing units 126 may be increased according to requirements, and the width and depth of the random storage memory 124 and the data random access memory 122 may be adjusted to expand. In a preferred embodiment, the weighted random access memory 124 is larger than the data random access memory 122 because there are many links in a typical neural network layer, and thus a larger storage space is required to be associated with The weight of each neuron. A number of embodiments are disclosed herein regarding the size of data and weight text, the size of the weighted random access memory 124 and the data random access memory 122, and the number of different neural processing units 126. In one embodiment, the neural network unit 121 has a data random access memory 122 of 64 KB (8192 bits x 64 columns) and a weighted random access memory of 2 MB (8192 bits x 2048 columns). 124, and 512 neural processing units 126. The neural network unit 121 is a Taiwan integrated circuit (TSMC) is manufactured in a 16 nm process with an area of approximately 3.3 mm square.

The sequencer 128 is fetched and executed by the program memory 129. The execution of the sequencer 128 further includes generating address and control signals for the data random access memory 122, the weight random access memory 124, and the neural processing unit 126. The sequencer 128 generates a memory address 123 and a read command to the data random access memory 122, whereby one of the N data words of the D columns is selected for supply to the N neural processing units 126. The sequencer 128 also generates a memory address 125 and a read command to the weighted random access memory 124, whereby one of the N weight words of the W columns is selected for supply to the N neural processing units 126. The sequence in which the sequencer 128 generates and provides the addresses 123, 125 to the neural processing unit 126 determines the "link" between the neurons. The sequencer 128 also generates a memory address 123 and a write command to the data random access memory 122, whereby one of the N data words of the D columns is selected for writing by the N neural processing units 126. In. The sequencer 128 also generates a memory address 125 and a write command to the weighted random access memory 124, whereby one of the N weight words of the W columns is selected for writing by the N neural processing units 126. In. The sequencer 128 also generates a memory address 131 to the program memory 129 to select one of the neural network unit instructions provided to the sequencer 128, as will be explained in subsequent sections. The memory address 131 corresponds to a program counter (not shown), and the sequencer 128 generally increments the program counter according to the position order of the program memory 129 unless the sequencer 128 encounters a control command, such as a loop command. (Please refer to Figure 26A). In this case, the sequencer 128 will program The counter is updated to the target address of this control instruction. The sequencer 128 also generates control signals to the neural processing unit 126, instructing the neural processing unit 126 to perform various different operations or functions, such as initialization, arithmetic/logic operations, rotation/shift operations, start functions, and write back. The calculations, related examples, are described in more detail in subsequent chapters (see micro-operations 3418 as shown in Figure 34).

The N neural processing units 126 generate N result words 133 which can be written back to one of the weighted random access memory 124 or the data random access memory 122. In a preferred embodiment, the weight random access memory 124 and the data random access memory 122 are directly coupled to the N neural processing units 126. Further, the weight random access memory 124 and the data random access memory 122 belong to the neural processing unit 126 and are not shared with other execution units 112 in the processor 100. The neural processing units 126 can continuously A column is obtained and completed from one or both of the weight random access memory 124 and the data random access memory 122 in each time-frequency period. In a preferred embodiment, the pipeline processing is performed. In one embodiment, each of the data random access memory 122 and the weighted random access memory 124 can provide 8192 bits to the neural processing unit 126 in each time-frequency period. These 8192 bits can be treated as 512 16-bits or 1024 8-bits, as will be described later.

The size of the data set processed by the neural network unit 121 is not limited by the weight of the random access memory 124 and the data random access memory 122, but is limited only by the size of the system memory, because the data System memory and weight random access memory 124 and data random access memory 122 are moved between the MTNN and MFNN commands (e.g., via media register 118). In one embodiment, the data random access memory 122 is assigned a double port, enabling reading of data characters or writing of data words to the data random access memory 122 by the data random access memory 122. The data text is written to the data random access memory 122. In addition, the large memory hierarchy of the memory subsystem 114, including the cache memory, provides a very large data bandwidth for data transfer between the system memory and the neural network unit 121. Moreover, in a preferred embodiment, the memory subsystem 114 includes a hardware data prefetcher that tracks memory access modes, such as neural data and weights loaded by system memory, and caches The hierarchical structure performs data prefetching to facilitate high frequency wide and low latency transmissions during transmission to the weighted random access memory 124 and the data random access memory 122.

Although in the embodiments herein, one of the operands provided by the weight memory to each of the neural processing units 126 is labeled as a weight, this term is commonly found in neural networks, but it should be understood that these operands may also be other Calculate the type of data associated with it, and its calculation speed can be improved by these devices.

The second figure shows a block diagram of a neural processing unit 126 of the first figure. As shown in the figure, the operation of the neural processing unit 126 can perform a number of functions or operations. In particular, the neural processing unit 126 can operate as a neuron or node within the artificial neural network to perform a typical multiply-accumulate function or operation. That is to say, in general, the neural network unit 126 (neuron) is used to: (1) from each A neuron with its associated input receives an input value, which is usually, but not necessarily, from a previous layer in the artificial neural network; (2) multiplying each output value by a corresponding weight value associated with one of its links to produce a product; (3) summing all the products to produce a total; (4) performing a start function on the total to generate the output of the neuron. However, unlike conventional methods that require performing all multiplication operations associated with all of the concatenated inputs and summing their products, the various neurons of the present invention can perform weight multiplication operations associated with one of the concatenated inputs in a given time-frequency period. The product is added (accumulated) to the accumulated value of the product of the join input executed in the time-frequency period associated with the time point before the point in time. Assuming that a total of M links are connected to this neuron, after the M products are summed (approximately M time-frequency cycles are required), the neuron will perform a start function on this accumulated number to produce an output or result. The advantage of this approach is that it reduces the number of multipliers required, and requires only a smaller, simpler, and faster adder circuit (eg, an adder using two inputs) within the neuron, without the need for Use an adder that is capable of summing the products of all the connected inputs or even adding one of the subsets. This approach also facilitates the use of a very large number (N) of neurons (neural processing unit 126) within neural network unit 121, such that neural network unit 121 can produce this large number after approximately M time-frequency periods. (N) Output of neurons. Finally, for a large number of different connection inputs, the neural network unit 121 composed of these neurons can be effectively implemented as an artificial neural network layer. That is to say, if the number of M in different layers increases or decreases, the number of time-frequency cycles required to generate the memory cell output will correspondingly increase and decrease, and resources (such as multipliers and accumulators) will be fully utilized. In contrast, traditional designs are smaller In terms of the M value, some of the multipliers and adders are not utilized. Therefore, in view of the number of connected outputs of the neural network unit, the embodiments described herein combine the advantages of flexibility and efficiency to provide extremely high performance.

The neural processing unit 126 includes a register 205, a dual input multiplex register 208, an arithmetic logic unit (ALU) 204, an accumulator 202, and an enable function unit (AFU) 212. The register 205 receives a weighted text 206 from the weighted random access memory 124 and provides its output 203 at a subsequent time-frequency period. Multiplex register 208 selects one of the two inputs 207, 211 to store in its register and provide its output 209 for a subsequent time-frequency period. Input 207 receives the data text from data random access memory 122. Another input 211 receives the output 209 of the adjacent neural processing unit 126. The neural processing unit 126 shown in the second figure is labeled as the neural processing unit J among the N neural processing units shown in the first figure. That is, the neural processing unit J is an example of one of the N neural processing units 126. In a preferred embodiment, the input 211 of the multiplex register 208 of the J example of the neural processing unit 126 receives the output 209 of the multiplex register 208 of the J-1 example of the neural processing unit 126, and the neural The output 209 of the multiplex register 208 of the processing unit J is provided to the input 211 of the J+1 paradigm multiplex register 208 of the neural processing unit 126. Thus, the multiplex registers 208 of the N neural processing units 126 can operate together, such as the same N-word rotator or cyclic shifter, which will be described in more detail in the subsequent third figure. The multiplex register 208 utilizes a control input 213 to control which of the two inputs is selected by the multiplex register 208 for storage in its register and for subsequent supply to the output 209.

The arithmetic logic unit 204 has three inputs. one of The input is received by the register 205 by the weight text 203. The other input receives the output 209 of the multiplex register 208. Still another input receives the output 217 of the accumulator 202. This arithmetic logic unit 204 performs arithmetic and/or logical operations on its inputs to produce a result for its output. For a preferred embodiment, the arithmetic and/or logical operations performed by arithmetic logic unit 204 are specified by instructions stored in program memory 129. For example, the multiply-accumulate instruction in the fourth figure specifies a multiply-accumulate operation, that is, the result 215 would be the product of the accumulator 202 value 217 and the weight text 203 and the data text of the output 209 of the multiplex register 208. total. However, other operations may be specified, including but not limited to: result 215 is the value passed by multiplex register output 209; result 215 is the value passed by weight text 203; result 215 is a zero value; result 215 is accumulator 202 The sum of the value 217 and the weight 203; the result 215 is the sum of the accumulator 202 value 217 and the multiplex register output 209; the result 215 is the maximum value of the accumulator 202 value 217 and the weight 203; the result 215 is the accumulator The maximum value of 202 value 217 and multiplex register output 209.

Arithmetic logic unit 204 provides its output 215 to accumulator 202 for storage. The arithmetic logic unit 204 includes a multiplier 242 that multiplies the weight text 203 and the data text of the multiplex register 208 output 209 to produce a product 246. In one embodiment, multiplier 242 multiplies two 16-bit operands to produce a 32-bit result. The arithmetic logic unit 204 also includes an adder 244 that adds a product 246 to the output 217 of the accumulator 202 to produce a total, which is the result 215 of the accumulation operation stored in the accumulator 202. In one embodiment, adder 244 is coupled to a 41-bit value 217 of accumulator 202 plus one of multipliers 242. A 32-bit result to produce a 41-bit result. Thus, by utilizing the rotator characteristics of the multiplex register 208 during a plurality of time-frequency periods, the neural processing unit 126 can achieve the product summation operation of the neurons required by the neural network. This arithmetic logic unit 204 may also include other circuit elements to perform other arithmetic/logic operations as previously described. In one embodiment, the second adder subtracts the weight text 203 from the data text output 209 of the multiplex register 208 to generate a difference, and then the adder 244 adds the difference to the output 217 of the accumulator 202. The value is used to produce a result 215 which is the cumulative result in accumulator 202. Thus, during a plurality of time-frequency periods, the neural processing unit 126 can perform the summation of the differences. In a preferred embodiment, although the weight text 203 and the data text 209 are the same size (in terms of bits), they may have different binary decimal point positions, as will be described later. In a preferred embodiment, multiplier 242 and adder 244 are integer multipliers and adders. This arithmetic logic unit 204 has low complexity, small size, and fast compared to arithmetic logic units that use floating point operations. With the advantages of low energy consumption. However, in other embodiments of the invention, arithmetic logic unit 204 may also perform floating point operations.

Although only one multiplier 242 and adder 244 are shown in the arithmetic logic unit 204 of the second figure, in a preferred embodiment, the arithmetic logic unit 204 includes other components to perform the other different operations described above. For example, the arithmetic logic unit 204 can include a comparator (not shown) to compare the accumulator 202 with a data/weight text, and a multiplexer (not shown) to select between two values specified by the comparator. The larger (maximum) is stored to accumulator 202. In another example, the arithmetic logic unit 204 includes selection logic (not shown), utilizing a data/ The weight text is used to skip the multiplier 242, causing the adder 224 to add the data/weight text to the value 217 of the accumulator 202 to generate a total amount to store to the accumulator 202. These additional operations are described in more detail in subsequent sections, such as Figures 18 through 29A, which also contribute to the implementation of convolution operations and common source operations.

The start function unit 212 receives the output 217 of the accumulator 202. The start function unit 212 performs a start function on the output of the accumulator 202 to produce the result 133 of the first map. In general, the start function in the neurons of the intervening layer of the artificial neural network can be used to normalize the total number of multiply accumulates, especially in a non-linear manner. To "normalize" the accumulated total, the current neuron's start function will produce a result value within the range of values that other neurons connected to the current neuron are expected to receive as input. (The result of normalization is sometimes referred to as "startup". In this paper, the start is the output of the current node, and the receiving node multiplies this output by a weight associated with the link between the output node and the receiving node to produce a product. And this product will be multiplied by the other input links associated with the receiving node.) For example, if the received/linked neurons are expected to receive as input values between 0 and 1, the output neurons It may be necessary to non-linearly squeeze and/or adjust (eg, shift up to convert a negative value to a positive value) beyond the cumulative total of the range of 0 and 1 to fall within the expected range. Thus, the operation performed by the start function unit 212 on the accumulator 202 value 217 will bring the result 133 to a known range. The results 133 of the N neural execution units 126 can all be written back to the data random access memory 122 or the weighted random access memory 124. In a preferred embodiment, the boot function unit 212 is operative to execute a plurality of boot functions, and for example, an input from the control register 127 selects one of the start functions to execute on the output of the accumulator 202. . These start functions may include, but are not limited to, step functions, correction functions, sigmoid functions, hyperbolic tangent functions, and soft addition functions (also referred to as smoothing correction functions). The analytical formula of the soft addition function is f(x)=ln(1+e ^x ), which is the total natural logarithm of the sum of 1 and e ^x , where “e” is the Euler's number and x is this Input 217 for the function. In a preferred embodiment, the start function may also include a pass-through function that directly passes the value 217 of the accumulator 202 or a portion thereof, as will be described later. In an embodiment, the circuitry of the startup function unit 212 performs the startup function in a single time-frequency cycle. In an embodiment, the start function unit 212 includes a plurality of forms that receive the accumulated value and output a value. For some start functions, such as a sigmoid function, a double take tangent function, a soft add function, etc., the value is approximated. The value provided by the real startup function.

In a preferred embodiment, the width (in terms of bits) of accumulator 202 is greater than the width of output 133 of start function function 212. For example, in one embodiment, the accumulator has a width of 41 bits to avoid accumulation in the product of up to 512 32-bits (this portion corresponds to the thirty-th figure in subsequent sections). There will be a more detailed description of the loss accuracy, and the result 133 has a width of 16 bits. In one embodiment, in a subsequent time-frequency period, the start function unit 212 passes the other unprocessed portions of the output 217 of the accumulator 202 and writes the portions back to the data random access memory 122 or the weights are stored in random. The memory 124 is taken, and this section will be described in more detail in the subsequent sections corresponding to the eighth figure. Thus, the unprocessed accumulator 202 value can be loaded back to the media register 118 via the MFNN instruction, whereby the instructions executed by the other execution units 112 of the processor 100 can perform complex startups that the boot function unit 212 cannot perform. Functions, such as the common softmax function, are also called normalized exponential functions. In one embodiment, the instruction set architecture of processor 100 includes instructions to perform one of the exponential functions, generally denoted as e ^x or exp(x), which may be used by other execution units 112 of processor 100 to promote soft maximal startup. The execution speed of the function.

In an embodiment, the neural processing unit 126 is a pipeline design. For example, the neural processing unit 126 can include a temporary register of the arithmetic logic unit 204, such as a register between the multiplier and the adder and/or other circuits of the arithmetic logic unit 204. The neural processing unit 126 can also include A register that loads the output of the boot function function 212. Other embodiments of this neural processing unit 126 are described in subsequent sections.

The third diagram is a block diagram showing N multiplex registers 208 of the N neural processing units 126 of the neural network unit 121 of the first figure, obtained for the data random access memory 122 of the first figure. One of the column data words 207 performs the operation of a rotator or a circular shifter such as the same N characters. In the embodiment of the third figure, N is 512. Thus, neural network unit 121 has 512 multiplex registers 208, labeled 0 through 511, corresponding to 512 neural processing units 126, respectively. Each multiplex register 208 receives the corresponding data text 207 on one of the D columns of the data random access memory 122. That is to say, the multiplex register 0 will receive the data word 0 from the data random access memory 122 column, and the multiplex register 1 will randomly access the data from the data. The memory 122 column receives the data 1 and the multiplex register 2 receives the data 2 from the data random access memory 122, and so on, the multiplex register 511 will randomly access the memory 122. Receive data text 511. In addition, the multiplex register 1 receives the output 209 of the multiplex register 0 as another input 211, and the multiplex register 2 receives the output 209 of the multiplex register 1 as another input 211, multiplexing. The register 3 receives the output 209 of the multiplex register 2 as another input 211, and so on, the multiplex register 511 receives the output 209 of the multiplex register 510 as another input 211, and more The scratchpad 0 receives the output 209 of the multiplex register 511 as the other input 211. Each multiplex register 208 receives a control input 213 to control its selection profile 207 or loop input 211. In one of the modes of operation, the control input 213 controls each of the multiplex registers 208 to select the data text 207 for storage in the scratchpad and to provide to the arithmetic logic unit 204 in subsequent steps during a first time-frequency period. And in subsequent time-frequency periods (such as the aforementioned M-1 time-frequency periods), control input 213 controls each multiplex register 208 to select loop input 211 for storage to the scratchpad and provide it to subsequent steps. Arithmetic logic unit 204.

Although in the embodiment depicted in the third figure (and subsequent seventh and nineteenth figures), a plurality of neural processing units 126 may be used to rotate the values of these multiplex registers 208/705 to the right, ie by the nerves The processing unit J moves toward the neural processing unit J+1, although the invention is not limited thereto, and in other embodiments (e.g., corresponding to the embodiments of the twenty-fourth to twenty-sixth embodiments), the plurality of neural processing units 126 It can be used to rotate the value of the multiplex register 208/705 to the left, that is, to move from the neural processing unit J toward the neural processing unit J-1. Furthermore, other embodiments of the invention These neural processing units 126 can selectively rotate the value of the multiplex register 208/705 to the left or right. For example, this selection can be specified by the neural network unit instructions.

The fourth diagram is a table showing a program stored in the program memory 129 of the neural network unit 121 of the first figure and executed by the neural network unit 121. As mentioned earlier, this sample program performs calculations related to one layer of the artificial neural network. The table in Figure 4 shows four columns and three rows. Each column corresponds to an address in the program memory 129 labeled in the first row. The second line specifies the corresponding instruction, and the third line indicates the number of time-frequency cycles associated with this instruction. In the preferred embodiment, the number of time-frequency cycles is indicative of the number of effective time-frequency cycles per instruction time-frequency period value in the embodiment of the pipeline execution, rather than the instruction delay. As shown in the figure, because the neural network unit 121 has the nature of pipeline execution, each instruction has an associated time-frequency period. The instruction at address 2 is an exception, and the instruction actually repeats 511 times. Therefore, 511 time-frequency cycles are required, as will be described later.

All neural processing units 126 will process each instruction in the program in parallel. That is to say, all N neural processing units 126 execute the instructions of the first column in the same time-frequency cycle, and all N neural processing units 126 execute the instructions of the second column in the same time-frequency cycle. analogy. However, the invention is not limited thereto, and in other embodiments of the subsequent sections, some instructions are executed in the form of a sequence of partially parallel portions, for example, as described in the embodiment of the eleventh embodiment, in multiple neural processing In an embodiment where unit 126 shares a start function unit, the start function and the output instructions at addresses 3 and 4 are in this manner. carried out. The example in the fourth graph assumes that one layer has 512 neurons (neural processing unit 126), and each neuron has 512 linked inputs from 512 neurons from the previous layer, for a total of 256K links. Each neuron receives a 16-bit data value from each of the link inputs and multiplies the 16-bit data value by an appropriate 16-bit weight value.

The first column of address 0 (which can also be assigned to another address) specifies an initialization neuroprocessing unit instruction. This initialization instruction clears the value of accumulator 202 to zero. In an embodiment, the initialization command may also be loaded into the column of the data random access memory 122 or the weight random access memory 124 in the accumulator 202, thereby instructing the corresponding text specified by the instruction. This initialization command also loads the configuration values into the control register 127, which is described in more detail in subsequent twenty-ninth and twenty-ninth B-pictures. For example, the width of the data text 207 and the weight text 209 can be loaded for use by the arithmetic logic unit 204 to confirm the size of the operation performed by the circuit, which width also affects the result 215 stored in the accumulator 202. In one embodiment, the neural processing unit 126 includes a circuit that fills the output 215 before the output 215 of the arithmetic logic unit 204 is stored in the accumulator 202, and the initialization command loads a configuration value into the circuit. Affects the aforementioned filling operation. In an embodiment, it may also be specified in an arithmetic logic unit function instruction (such as a multiplication accumulating instruction of address 1) or an output instruction (such as a write start function unit output instruction of address 4) to accumulate the accumulator. 202 clears to zero value.

The second column located in address 1 specifies a multiply accumulate instruction to instruct the 512 neural processing units 126 to load a corresponding data text from a column of data random access memory 122 and to store the random data from the weights. A column of memory 124 is loaded with a corresponding weighting text, and a first multiply-accumulate operation is performed on the data text input 207 and the weight text input 206, i.e., the initial value of the initializer 202 is added. Further, this command will instruct sequencer 128 to generate a value at control input 213 to select data entry 207. In the example of the fourth figure, the designated column of the data random access memory 122 is column 17, and the designated column of the weight random access memory 124 is column 0, so the sequencer is instructed to output the value 17 as a data random. The memory address address 123 is accessed and the value 0 is output as a weighted random access memory address 125. Thus, 512 data words from column 17 of data random access memory 122 are provided as corresponding data inputs 207 for 512 neural processing units 126, and 512 weights from column 0 of weight random access memory 124. The text is provided as a corresponding weight input 206 for 512 neural processing units 126.

The third column located in address 2 specifies a multiply-accumulate rotation instruction having a count of 511 to indicate that the 512 neural processing units 126 perform 511 multiply-accumulate operations. This instruction instructs the 512 neural processing units 126 to input the data word 209 of the arithmetic logic unit 204 into the rotation value 211 from the adjacent neural processing unit 126 in each of the 511 multiply-accumulate operations. That is, this command will instruct sequencer 128 to generate a value at control input 213 to select rotation value 211. In addition, the instruction will instruct the 512 neural processing units 126 to load one of the 511 multiply-accumulate operations into the "next" column of the weighted random access memory 124. That is, the instruction will instruct sequencer 128 to increment the weighted random access memory address 125 from the value of the previous time-frequency period by one, in this example, the instruction The first time-frequency period is column 1, the next time-frequency period is column 2, the next time-frequency period is column 3, and so on, the 511th time-frequency period is column 511. In each of the 511 multiply-accumulate operations, the product of the rotary input 211 and the weighted text input 206 is added to the previous value of the accumulator 202. The 512 neural processing units 126 perform the 511 multiply-accumulate operations in 511 time-frequency cycles, and each of the neural processing units 126 will have a different data for the column 17 from the data random access memory 122. The adjacent neural processing unit 126 performs the arithmetic data of the operation in the previous time-frequency cycle, and performs a multiplication and accumulation operation on the different weight words associated with the data characters, which are conceptually different input inputs of the neurons. This example assumes that each neural processing unit 126 (neuron) has 512 linked inputs, thus involving the processing of 512 data words and 512 weighted words. After the last iteration of the multiply-accumulate rotation instruction of column 2, the sum of the products of the 512 connection inputs is stored in the accumulator 202. In one embodiment, the instruction set of the neural processing unit 126 includes an "execute" instruction to instruct the arithmetic logic unit 204 to perform an arithmetic logic unit operation specified by the initialization neural processing unit instruction, such as the arithmetic of the twenty-ninth A-picture. The logic unit function 2926 specifies, rather than having separate instructions for each of the different types of arithmetic logic operations (e.g., the aforementioned multiply accumulate, accumulator and weight max, etc.).

The fourth column located in address 3 specifies a start function instruction. This start function instruction instructs the start function unit 212 to execute the specified start function for the accumulator 202 value to produce the result 133. The embodiment of the startup function will be described in more detail in subsequent chapters.

The fifth column of address 4 specifies a write start function unit output instruction to instruct the 512 neural processing units 216 to write their boot function unit 212 output as a result 133 back to one of the data random access memory 122 columns. In this example, column 16. That is, this instruction will instruct sequencer 128 to output a value of 16 as data random access memory address 123 and a write command (corresponding to the read command specified by the multiply accumulate instruction of address 1). In a preferred embodiment, the write start function unit output instructions can be executed concurrently with other instructions because of the nature of the pipeline execution, so the write start function unit output instructions can actually be executed in a single time-frequency cycle.

In a preferred embodiment, each neural processing unit 126 acts as a pipeline having various functional components, such as multiplex register 208 (and multiplex register 705 of the seventh diagram), arithmetic. Logic unit 204, accumulator 202, start function unit 212, multiplexer 802 (please refer to FIG. 8), column buffer 1104 and start function unit 1112 (please refer to FIG. 11), etc., some of which are themselves Can be executed in pipeline. In addition to the data text 207 and the weight text 206, this pipeline also receives instructions from the program memory 129. These instructions flow along the pipeline and control multiple functional units. In another embodiment, the program does not include a start function instruction, but the initialization neural processing unit instruction specifies a start function that executes the value 217 of the accumulator 202, indicating that the value of one of the specified start functions is stored in a configuration. The register, the portion of the start function unit 212 for the pipeline, is utilized after the last accumulator 202 value 217 is generated, that is, after the last execution of the multiply-accumulate rotation instruction of address 2 is repeated. In a preferred embodiment, in order to save energy, the pipeline is activated. The motion function unit 212 portion is in an inactive state before the write start function unit output instruction arrives. Upon arrival of the instruction, the startup function unit 212 starts and executes the start function for the accumulator 202 output 217 specified by the initialization instruction.

The fifth diagram shows a timing diagram of the execution of the program of the fourth diagram by the neural network unit 121. Each column of this timing diagram corresponds to the continuous time-frequency period indicated by the first row. The other lines correspond to the different neural processing units 126 of the 512 neural processing units 126, respectively, and indicate their operations. Only the operation of the neural processing unit 0, 1, 511 is shown in the figure to simplify the explanation.

At time-frequency period 0, each of the 512 neural processing units 126 performs an initialization instruction of the fourth map, and in the fifth diagram, a zero value is assigned to the accumulator 202.

In time-frequency period 1, each of the 512 neural processing units 126 performs a multiply-accumulate instruction of address 1 in the fourth figure. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 (i.e., zero) to the text 0 of the column 17 of the data random access memory 122 and the text 0 of the column 0 of the weight random access memory 124. Product; the neural processing unit 1 adds the value of the accumulator 202 (i.e., zero) to the product of the text 1 of the column 17 of the data random access memory 122 and the text 1 of the column 0 of the weight random access memory 124; Similarly, the neural processing unit 511 adds the value of the accumulator 202 (i.e., zero) to the product of the text 511 of the column 17 of the data random access memory 122 and the text 511 of the column 0 of the weight random access memory 124.

In time-frequency period 2, the 512 neural processing units 126 Each of the neural processing units 126 performs the first iteration of the multiply-accumulate rotation instruction of address 2 in the fourth figure. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 to the rotated data text 211 received by the multiplexer 208 output 209 of the neural processing unit 511 (i.e., received by the data random access memory 122). The data text 511) is multiplied by the text 0 of column 1 of the weight random access memory 124; the neural processing unit 1 adds the value of the accumulator 202 to the rotation received by the output 209 of the multiplex register 208 of the neural processing unit 0. The product text 211 (i.e., the data text 0 received by the data random access memory 122) and the text 1 of the column 1 of the weight random access memory 124; and so on, the neural processing unit 511 will increment the value of the accumulator 202. The rotated data text 211 (ie, the data text 510 received by the data random access memory 122) received by the multiplexer 208 output 209 of the neural processing unit 510 is added to the weight random access memory 124. The product of the text 511.

In time-frequency period 3, each of the 512 neural processing units 126 performs a second iteration of the multiply-accumulate rotation instruction of address 2 in the fourth figure. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 to the rotated data text 211 received by the multiplexer 208 output 209 of the neural processing unit 511 (i.e., received by the data random access memory 122). The data text 510) is multiplied by the text 0 of column 2 of the weight random access memory 124; the neural processing unit 1 adds the value of the accumulator 202 to the rotation received by the output 209 of the multiplex register 208 of the neural processing unit 0. The product text 211 (i.e., the data text 511 received by the data random access memory 122) and the text 1 of the column 2 of the weight random access memory 124; and so on, the neural processing unit 511 The value of the accumulator 202 is added to the rotated data text 211 (ie, the data text 509 received by the data random access memory 122) received by the multiplexer 208 output 209 of the neural processing unit 510 and the weight random access memory. The product of the text 511 of column 2 of body 124. As the ellipsis of the fifth figure shows, the next 509 time-frequency periods will continue until the time-frequency period 512.

At time-frequency period 512, each of the 512 neural processing units 126 performs the 511th iteration of the multiply-accumulate rotation instruction of address 2 in the fourth diagram. As shown in the figure, the neural processing unit 0 adds the value of the accumulator 202 to the rotated data text 211 received by the multiplexer 208 output 209 of the neural processing unit 511 (i.e., received by the data random access memory 122). The data text 1) is multiplied by the text 0 of the column 511 of the weight random access memory 124; the neural processing unit 1 adds the value of the accumulator 202 to the rotation received by the output 209 of the multiplex register 208 of the neural processing unit 0. The product text 211 (i.e., the data text 2 received by the data random access memory 122) and the text 1 of the column 511 of the weight random access memory 124; and so on, the neural processing unit 511 will increment the value of the accumulator 202. The rotated data text 211 (ie, the data text 0 received by the data random access memory 122) received by the multiplexer 208 output 209 of the neural processing unit 510 and the column 511 of the weight random access memory 124 are added. The product of the text 511. In one embodiment, multiple time-frequency periods are required to read the data text and weight text from the data random access memory 122 and the weight random access memory 124 to perform the multiplication accumulation instruction of the address 1 in the fourth figure; however, The data random access memory 122, the weight random access memory 124, and the neural processing unit 126 are pipelined. Therefore, after the first multiply-accumulate operation starts (as shown in the time-frequency period 1 of the fifth figure), the subsequent multiply-accumulate operation (as shown in the time-frequency period 2-512 of the fifth figure) begins. Execute in a continuous time-frequency cycle. In a preferred embodiment, the data random access memory 122 and/or the weight random access memory are utilized in response to architectural instructions, such as MTNN or MFNN instructions (described in subsequent fourteenth and fifteenth figures). The access to the body 124, or the micro-instructions translated by the architectural instructions, will temporarily rest on these neural processing units 126.

At time-frequency period 513, the start function unit 212 of each of the 512 neural processing units 126 performs the start function of address 3 in the fourth diagram. Finally, at time-frequency period 514, each of the 512 neural processing units 126 will perform a fourth by writing the result 133 back to the corresponding text in column 16 of the data random access memory 122. The write start function unit output instruction of the address 4 in the figure, that is, the result 133 of the neural processing unit 0 is written to the text 0 of the data random access memory 122, and the result 133 of the neural processing unit 1 is written. The text 1 of the data random access memory 122 is entered, and so on, the result 133 of the neural processing unit 511 is written to the text 511 of the data random access memory 122. Corresponding block diagrams corresponding to the operations of the aforementioned fifth figure are shown in Figure 6A.

Figure 6A is a block diagram showing the execution of the fourth diagram of the neural network unit 121 of the first figure. The neural network unit 121 includes 512 neural processing units 126, a data random access memory 122 that receives the address input 123, and a weighted random access memory 124 that receives the address input 125. In the time-frequency period 0, these 512 nerve processing Unit 126 executes an initialization instruction. This operation is not shown in the figure. As shown in the figure, at time 1 of the time-frequency period, 512 16-bit data words of column 17 are read from data random access memory 122 and provided to the 512 neural processing units 126. During the time-frequency period 1 to 512, 512 16-bit weighted texts of columns 0 through 511 are read from weight random access memory 122 and provided to the 512 neural processing units 126, respectively. During the time-frequency period 1, the 512 neural processing units 126 perform their corresponding multiply-accumulate operations on the loaded data text and weight text. This operation is not shown in the figure. During the time-frequency period 2 to 512, the multiplexer 208 of the 512 neural processing unit 126 operates as a rotator with 512 16-bit characters, and will be previously accessed by the data random access memory. The data characters loaded in column 122 are rotated to the adjacent neural processing unit 126, and the neural processing units 126 perform the corresponding data characters after the rotation and the corresponding weight characters loaded by the weight random access memory 124. Multiply accumulate operation. At time FIFO period 513, the 512 start function units 212 execute a start command. This operation is not shown in the figure. During time-frequency period 514, the 512 neural processing units 126 write their corresponding 512 16-bit results 133 back to column 16 of data random access memory 122.

As shown in the figure, the number of time-frequency cycles required to generate the resulting text (neuron output) and write back to the data random access memory 122 or the weighted random access memory 124 is approximately the data received by the current layer of the neural network. Enter the square root of the number of (links). For example, if the current layer has 512 neurons and each neuron has 512 links from the previous layer, the total number of these links is 256K, and the current layer is generated. As a result, the number of time-frequency cycles required will be slightly greater than 512. Therefore, the neural network unit 121 can provide extremely high performance in neural network calculation.

FIG. 6B is a flow chart showing that the processor 100 of the first figure executes an architectural program to perform a typical multiply-accumulate start function operation of a neuron associated with a hidden layer of an artificial neural network using the neural network unit 121. The operation is as if it were performed by the program of the fourth figure. The example in Figure 6B assumes that there are four hidden layers (variables labeled NUM_LAYERS in initialization step 602), each hidden layer has 512 neurons, and each neuron is connected to all 512 neurons in the previous layer (through the fourth Figure program). However, it is to be understood that the selection of the number of these layers and neurons is illustrative of the present invention. The neural network unit 121 can apply similar calculations to different numbers of hidden layers, with different numbers of nerves in each layer. Embodiments of the element, or embodiments in which the neurons are not all connected. In one embodiment, the weight value for a neuron that does not exist in this layer or a neuron that does not exist will be set to zero. In a preferred embodiment, the architectural program writes the first set of weights to the weighted random access memory 124 and activates the neural network unit 121 when the neural network unit 121 is performing the calculation associated with the first layer. The architecture program writes the second set of weights to the weighted random access memory 124. Thus, once the neural network unit 121 completes the calculation of the first hidden layer, the neural network unit 121 can begin the calculation of the second layer. Thus, the architecture program will travel to and from the two regions of the weighted random access memory 124 to ensure that the neural network unit 121 can be fully utilized. This process begins in step 602.

At step 602, as in the relevant section of Figure 6A The processor 100 executing the architectural program writes the input value to the current neuron hidden layer of the data random access memory 122, that is, the column 17 of the data random access memory 122. These values may also already be in column 17 of data random access memory 122 as the result 133 (e.g., convolution, common source, or input layer) of neural network unit 121 for the previous layer. Second, the architecture program initializes the variable N to a value of 1. The variable N represents the current layer in the hidden layer that is to be processed by the neural network unit 121. In addition, the architecture program initializes the variable NUM_LAYERS to a value of 4 because there are four hidden layers in this example. The flow then proceeds to step 604.

At step 604, processor 100 writes the weight text of layer 1 into weight random access memory 124, such as columns 0 through 511 shown in Figure 6A. The flow then proceeds to step 606.

In step 606, the processor 100 writes a multiply-accumulate start function program (as shown in the fourth figure) to the neural network unit 121 program memory by specifying a function 1432 to write the MTNN instruction 1400 of the program memory 129. 129. The processor 100 then utilizes an MTNN instruction 1400 to initiate a neural network unit program that specifies a function 1432 to begin execution of the program. The flow then proceeds to step 608.

In decision step 608, the architecture program determines if the value of the variable N is less than NUM_LAYERS. If so, the flow proceeds to step 612; otherwise, it proceeds to step 614.

In step 612, processor 100 writes the weight text of layer N+1 into weighted random access memory 124, such as columns 512 through 1023. Therefore, the architecture program can write the weight text of the next layer to the weight random access memory when the neural network unit 121 performs the hidden layer calculation of the current layer. The body 124 is thereby used, whereby after the calculation of the current layer is completed, that is, after the data random access memory 122 is written, the neural network unit 121 can immediately start performing the hidden layer calculation of the next layer. Next, proceed to step 614.

In step 614, the processor 100 confirms that the neural network unit program being executed (in the case of layer 1, starting at step 606, and in the case of layers 2 through 4, starting at step 618) has been completed. . In a preferred embodiment, processor 100 reads neural network unit 121 status register 127 by executing an MFNN instruction 1500 to confirm whether execution has been completed. In another embodiment, the neural network unit 121 generates an interrupt indicating that the multiply-accumulate start function layer program has been completed. The flow then proceeds to decision step 616.

In decision step 616, the architecture program determines if the value of the variable N is less than NUM_LAYERS. If so, the flow will proceed to step 618; otherwise, proceed to step 622.

In step 618, the processor 100 updates the multiply-accumulate start function program to enable the hidden layer calculation of layer N+1. Further, the processor 100 updates the column value of the data random access memory 122 of the multiply-accumulate instruction of the address 1 in the fourth figure to the column in which the calculation result of the previous layer in the data random access memory 122 is written ( For example, update to column 16) and update the output column (for example, update to column 15). The processor 100 then begins updating the neural network unit program. In another embodiment, the program of the fourth figure is the same column of the output instruction of the specified address 4 as the column specified by the multiply-accumulate instruction of the address 1 (that is, the column read by the data random access memory 122). ). In this embodiment, the current column of the input data text is overwritten (because the column data has been read into the multiplex register 208 and passed through The N-text rotator rotates between these neural processing units 126, as long as the list of texts does not need to be used for other purposes, such processing is allowed. In this case, there is no need to update the neural network unit program in step 618, but only need to restart it. The flow then proceeds to step 622.

In step 622, processor 100 reads the result of the neural network unit program of layer N from data random access memory 122. However, if these results are only used in the next layer, the architecture program does not need to read the results from the data random access memory 122, but can retain it in the data random access memory 122 for the next hidden layer calculation. Use. The flow then proceeds to step 624.

In decision step 624, the architecture program determines if the value of the variable N is less than NUM_LAYERS. If so, the flow proceeds to step 626; otherwise, the process is terminated.

In step 626, the architecture program increments the value of N by one. The flow then returns to decision step 608.

As shown in the example of FIG. B, the neuroprocessing unit 126 performs a read and a write to the data random access memory 122 substantially every 512 time-frequency cycles (through the neural network of the fourth figure). The effect of the operation of the road unit program). In addition, the neural processing unit 126 reads the weighted random access memory 124 for each time-frequency period to read a list of weighted words. Therefore, the full bandwidth of the weighted random access memory 124 is consumed because the neural network unit 121 performs the hidden layer operation in a mixed manner. Furthermore, it is assumed that in one embodiment there is a write and read buffer, such as the buffer of the seventeenth diagram. 1704, while the neural processing unit 126 is reading, the processor 100 writes the weight random access memory 124, such that the buffer 1704 performs the weight random access memory 124 once every 16 time-frequency cycles. Write to write weight text. Thus, in an embodiment where the weighted random access memory 124 is a single unit (as described in the corresponding section of Figure 17), the neuroprocessing units 126 are temporarily placed on the weights for every 16 time-frequency periods. The random access memory 124 reads, and the buffer 1704 can write to the weight random access memory 124. However, in embodiments of the dual weight random access memory 124, the neural processing units 126 need not be placed on hold.

The seventh figure is a block diagram showing another embodiment of the neural processing unit 126 of the first figure. The neural processing unit 126 of the seventh diagram is similar to the neural processing unit 126 of the second figure. However, the neural processing unit 126 of the seventh diagram additionally has a dual input multiplex register 705. The multiplex register 705 selects one of the inputs 206 or 711 to be stored in its register and provides its output 203 for subsequent time-frequency periods. Input 206 receives the weight text from weight random access memory 124. Another input 711 is the output 203 of the second multiplex register 705 that receives the adjacent neural processing unit 126. In a preferred embodiment, the input 711 of the neural processing unit J receives the output 203 of the multiplex register 705 of the neural processing unit 126 arranged at J-1, and the output 203 of the neural processing unit J is provided. The input 711 to the multiplex register 705 of the neural processing unit 126 arranged at J+1. Thus, the multiplex registers 705 of the N neural processing units 126 can operate together, such as the same N text rotator, and the operation is similar to that shown in the third figure above, but for the weight text. Non-data Text. The multiplex register 705 uses a control input 213 to control which of the two inputs is selected by the multiplex register 705 for storage in its register and for subsequent supply to the output 203.

Utilizing the multiplex register 208 and/or the multiplex register 705 (and the multiplex registers in other embodiments as shown in the eighteenth and twenty-third figures) actually constitutes a large rotator The data/weights from one of the data random access memory 122 and/or the weighted random access memory 124 are rotated, and the neural network unit 121 does not need to be stored in the data random access memory 122 and/or the weights. A very large multiplexer is used between memory 124 to provide the required data/weight text to the appropriate neural network unit.

Write back the accumulator value in addition to the start function result

For some applications, processor 100 is received (eg, received through MFNN instruction of FIG. 15 to media register 118) unprocessed accumulator 202 value 217 for execution by other execution units 112. The instruction execution calculations do have their usefulness. For example, in an embodiment, the startup function unit 212 is not configured for execution of the soft maximal startup function to reduce the complexity of the startup function unit 212. Therefore, the neural network unit 121 outputs the unprocessed accumulator 202 value 217 or one of the subsets to the data random access memory 122 or the weighted random access memory 124, and the architecture program can be randomly randomized in subsequent steps. The access memory 122 or the weight random access memory 124 reads and calculates the unprocessed value. However, the application of the value 217 for the unprocessed accumulator 202 is not limited to performing soft maximal operations. Other applications are also covered by the present invention.

The eighth figure is a block diagram showing still another embodiment of the neural processing unit 126 of the first figure. The nerve processing unit 126 of the eighth diagram is similar to the neural processing unit 126 of the second figure. However, the neural processing unit 126 of the eighth diagram includes a multiplexer 802 within the startup function unit 212, and the activation function unit 212 has a control input 803. The width (in bits) of the accumulator 202 is greater than the width of the data text. Multiplexer 802 has a plurality of inputs to receive the data text width portion of accumulator 202 output 217. In one embodiment, the accumulator 202 has a width of 41 bits, and the neural processing unit 216 can be used to output a 16-bit result text 133; thus, for example, the multiplexer 802 (or the thirty-th diagram) The multiplexer 3032 and/or multiplexer 3037) has three inputs that receive the bits [15:0], the bits [31:16], and the bits [47:32] of the output 217 of the accumulator 202, respectively. In a preferred embodiment, output bits (e.g., bits [47:41]) that are not provided by accumulator 202 are forced to be zero-valued bits.

The sequencer 128 will generate a value at the control input 803, and the control multiplexer 802 selects one of the words (e.g., 16 bits) of the accumulator 202 to respond to an accumulator instruction, such as in the subsequent ninth diagram. Write accumulator instruction at address 3 through 5. In a preferred embodiment, multiplexer 802 has one or more inputs to receive the output of a start function circuit (e.g., elements 3022, 3024, 3026, 3018, 3014, and 3016 in FIG. 30). The width of the output produced by these start function circuits is equal to one data word. The sequencer 128 will generate a value at the control input 803 to control the multiplexer 802 to select one of these start function circuit outputs instead of selecting one of the words of the accumulator 202 to respond to The start function unit of the address 4 in the fourth figure outputs an instruction.

The ninth diagram is a table showing a program stored in the program memory 129 of the neural network unit 121 of the first figure and executed by the neural network unit 121. The example program in the ninth figure is similar to the program in the fourth figure. In particular, the two instructions are exactly the same at addresses 0 through 2. However, the instructions for addresses 3 and 4 in the fourth figure are replaced by a write accumulator instruction in the ninth figure, which instructs 512 neural processing units 126 to write back their accumulator 202 output 217 as a result 133. The three columns of data random access memory 122, in this example, columns 16 through 18. That is, the write accumulator instruction instructs the sequencer 128 to output a data random access memory address 123 of a value of 16 and a write command during the first time-frequency period, and output a second time-frequency period. The data random access memory address 123 and a write command of value 17 output a data random access memory address 123 and a write command of a value of 18 in the third time-frequency cycle. In a preferred embodiment, the execution time of the write accumulator instruction can overlap with other instructions, so that the write accumulator instruction can actually be executed during the three time-frequency periods, each of which is time-frequency. The cycle is written to one of the columns of data random access memory 122. In one embodiment, the user specifies the value of the start function 2934 and the output command 2956 of the control register 127 (FIG. 29A) to write the desired portion of the accumulator 202 to the data random access. Memory 122 or weight random access memory 124. Additionally, the write accumulator instruction can selectively write back a subset of the accumulators 202 instead of writing back all of the accumulator 202. In an embodiment, the standard type accumulator 202 can be written back. This section will be described in more detail in the subsequent sections corresponding to the twenty-ninth to thirty-first.

The tenth diagram shows a timing diagram of the execution of the program of the ninth diagram by the neural network unit 121. The timing diagram of the tenth figure is similar to the timing diagram of the fifth figure, in which the time-frequency periods 0 to 512 are the same. However, in time-frequency periods 513-515, the start function unit 212 of each of the 512 neural processing units 126 performs one of the write accumulator instructions of addresses 3 through 5 in the ninth figure. . In particular, in the time-frequency period 513, each of the 512 neural processing units 126 will write back the bit random access memory 122 to the data random access memory 122 as the result 133 of the accumulator 202 output 217 bits [15:0]. The corresponding text in column 16; in the time-frequency period 514, each of the 512 neural processing units 126 will write back the data as the result 133 of the 217 bits [31:16] of the accumulator 202 output. Corresponding characters in column 17 of random access memory 122; and in time-frequency period 515, each of 512 neural processing units 126 will output accumulator 202 to bits 217 [40:32] As a result 133, the corresponding text in the column 18 of the data random access memory 122 is written back. In a preferred embodiment, bit [47:41] is forced to a zero value.

Shared boot function unit

The eleventh diagram is a block diagram showing an embodiment of a neural network unit 121 of the first figure. In the embodiment of the eleventh diagram, a neuron is divided into two parts, that is, a start function unit part and an arithmetic logic unit part (this part includes a shift register part), and each start function unit part is composed of The arithmetic logic units are partially shared. In the eleventh diagram, the arithmetic logic unit portion refers to the neural processing unit 126, The shared boot function unit portion refers to the boot function unit 1112. In contrast to the embodiment as in the second figure, each neuron is comprised of its own start function unit 212. Accordingly, in an example of the eleventh embodiment, the neural processing unit 126 (arithmetic logic unit portion) may include the accumulator 202 of the second graph, the arithmetic logic unit 204, the multiplex register 208, and the register. 205, but does not include the start function unit 212. In the embodiment of the eleventh diagram, the neural network unit 121 includes 512 neural processing units 126, but the present invention is not limited thereto. In the example of the eleventh figure, the 512 neural processing units 126 are divided into 64 groups, labeled as groups 0 to 63 in the eleventh figure, and each group has eight neural processing units 126.

The neural network unit 121 includes a column of buffers 1104 and a plurality of shared start function units 1112. The start function units 1112 are coupled between the neural processing unit 126 and the column buffer 1104. The width of the column buffer 1104 (in bits) is the same as one of the data random access memory 122 or the weight random access memory 124, such as 512 words. Each of the neural processing units 126 has a start function unit 1112, that is, each start function unit 1112 corresponds to a group of neural processing units 126; thus, there are 64 in the embodiment of the eleventh embodiment. The start function unit 1112 corresponds to a group of 64 neural processing units 126. The eight neural processing units 126 of the same group share the start function unit 1112 corresponding to this group. The invention is also applicable to embodiments having a different number of activation function units and a different number of neural processing units in each group. For example, the present invention is also applicable to embodiments in which two, four, or sixteen neural processing units 126 share the same start function unit 1112 in each group.

The shared start function unit 1112 helps to reduce the size of the neural network unit 121. Size reduction will sacrifice performance. That is, depending on the sharing rate, an additional time-frequency period may be required to generate the result 133 of the entire array of neural processing units 126, for example, as shown in Figure 12 below, at a sharing ratio of 8:1. In the case of the case, seven additional time-frequency cycles are required. However, in general, compared to the number of time-frequency cycles required to generate the cumulative total (for example, for each neuron with 512 connections, 512 time-frequency periods are required), the aforementioned additional time The number of frequency cycles (for example, 7) is quite small. Therefore, the shared startup function unit has a very small impact on performance (e.g., an increase of approximately one percent of the computation time), which can be a costly reduction for the size of the neural network unit 121 that can be reduced.

In one embodiment, each neural processing unit 126 includes a start function unit 212 for performing a relatively simple start function, these simple start function units 212 having a smaller size to be included in each of the neural processing units 126 Conversely, the shared complex startup function unit 1112 performs a relatively complex startup function that is significantly larger in size than the simple startup function unit 212. In this embodiment, an additional time-frequency period is required only if a complex startup function is specified and needs to be performed by the shared complex startup function unit 1112, in the event that the specified startup function can be executed by the simple startup function unit 212, This additional time-frequency period is not required.

The twelfth and thirteenth drawings are timing charts showing the execution of the program of the fourth diagram by the neural network unit 121 of the eleventh diagram. The timing diagram of the twelfth figure is similar to the timing diagram of the fifth figure, and the time-frequency period of the two is 0 to 512 are the same. However, the operations at time-frequency period 513 are not the same, as the neural processing unit 126 of FIG. 11 shares the start function unit 1112; that is, the neural processing unit 126 of the same group shares the association with the group. The function unit 1112 is started, and the eleventh figure shows this shared architecture.

Each column of the timing diagram of the thirteenth map corresponds to a continuous time-frequency period indicated in the first row. The other lines correspond to different start function units 1112 of the 64 start function units 1112, respectively, and indicate their operations. Only the operations of the neural processing units 0, 1, 63 are shown in the figure to simplify the explanation. The time-frequency period of the thirteenth map corresponds to the time-frequency period of the twelfth figure, but the operation of the neural processing unit 126 sharing the start-up function unit 1112 is displayed in a different manner. As shown in the thirteenth diagram, in the time-frequency period 0 to 512, the 64 start function units 1112 are all in the inactive state, and the neural processing unit 126 performs the initialization of the neural processing unit instruction, the multiply accumulate instruction, and the multiply accumulate rotation instruction. .

As shown in the twelfth and thirteenth diagrams, at time-frequency period 513, the start function unit 0 (the start function unit 1112 associated with group 0) begins the execution of the specified value for the accumulator 202 value 217 of the neural processing unit 0. The function, neural processing unit 0 is the first neural processing unit 216 in group 0, and the output of the start function unit 1112 will be stored in text 0 of column register 1104. Also in time-frequency period 513, each start function unit 1112 begins the execution of the specified start function for the accumulator 202 value 217 of the first neural processing unit 126 in the corresponding neural processing unit 216 group. Therefore, as shown in the thirteenth diagram, at time-frequency period 513, the start function unit 0 begins execution on the accumulator 202 of the neural processing unit 0. The specified start function to generate the result of the text 0 to be stored in the column register 1104; the start function unit 1 begins executing the specified start function on the accumulator 202 of the neural processing unit 8 to generate a column to be stored in the column. The result of the text 8 of the memory 1104; and so on, the start function unit 63 begins executing the specified start function on the accumulator 202 of the neural processing unit 504 to produce the result of the text 504 to be stored in the column register 1104.

In the time-frequency period 514, the start function unit 0 (the start function unit 1112 associated with the group 0) begins executing the specified start function for the accumulator 202 value 217 of the neural processing unit 1, the neural processing unit 1 being in group 0. The second neural processing unit 216, and the output of the start function unit 1112 will be stored in the text 1 of the column register 1104. Also at time-frequency period 514, each start function unit 1112 begins to perform the specified start function for the accumulator 202 value 217 of the second neural processing unit 126 in the corresponding neural processing unit 216 group. Thus, as shown in FIG. 13, at time-frequency period 514, the start function unit 0 begins executing the specified start function on the accumulator 202 of the neural processing unit 1 to generate the text 1 to be stored in the column register 1104. As a result, the start function unit 1 begins executing the specified start function on the accumulator 202 of the neural processing unit 9 to produce the result of the literal 9 to be stored in the column register 1104; and so on, the start function unit 63 begins to The accumulator 202 of the neural processing unit 505 executes the specified start function to produce the result of the text 505 to be stored in the column register 1104. Such processing continues until the time-frequency period 520, and the start function unit 0 (the start function unit 1112 associated with the group 0) begins executing the specified start function for the accumulator 202 value 217 of the neural processing unit 7, the neural processing unit 7 That is the eighth in group 0 (the last one) The neural processing unit 216, and the output of the start function unit 1112 will be stored in the text 7 of the column register 1104. Also at time-frequency period 520, each start function unit 1112 begins the execution of the specified start function for the accumulator 202 value 217 of the eighth neural processing unit 126 in the corresponding neural processing unit 216 group. Thus, as shown in FIG. 13, at time-frequency period 520, the start function unit 0 begins executing the specified start function on the accumulator 202 of the neural processing unit 7 to generate the text 7 to be stored in the column register 1104. As a result, the start function unit 1 begins executing the specified start function on the accumulator 202 of the neural processing unit 15 to produce the result of the literal 15 to be stored in the column register 1104; and so on, the start function unit 63 begins to The accumulator 202 of the neural processing unit 511 executes the specified start function to produce the result of the text 511 to be stored in the column register 1104.

In the time-frequency period 521, once all 512 results of the 512 neural processing units 126 have been generated and written to the column register 1104, the column register 1104 begins to write its contents to the data random access memory. 122 or weight random access memory 124. Thus, the start function unit 1112 of each of the neural processing units 126 group performs a portion of the start function instruction of address 3 in the fourth figure.

The embodiment of sharing the start function unit 1112 in the arithmetic logic unit 204 group as shown in FIG. 11 is particularly helpful in conjunction with the use of the integer arithmetic logic unit 204. This section will be described in subsequent sections as shown in Figures 29 to 33.

MTNN and MFNN architecture instructions

The fourteenth diagram is a block diagram showing the operation of a mobile-to-neural network (MTNN) architecture command 1400 and its portion corresponding to the neural network unit 121 of the first figure. The MTNN instruction 1400 includes an execution code field 1402, a src1 field 1404, a src2 field, a gpr field 1408, and an immediate field 1412. The MTNN instruction is an architectural instruction, that is, the instruction is included in the instruction set architecture of the processor 100. In a preferred embodiment, the instruction set architecture utilizes one of the execution code field bits 1402 to distinguish between the MTNN instruction 1400 and other instructions within the instruction set architecture. The execution code 1402 of the MTNN instruction 1400 may or may not be included in a prefix such as the x86 architecture.

Immediate field 1412 provides a value to specify a function 1432 to control logic 1434 of neural network unit 121. In a preferred embodiment, this function 1432 is an immediate operand as one of the microinstructions 105 of the first figure. These functions 1432, which can be executed by the neural network unit 121, include the write data random access memory 122, the write weight random access memory 124, the write program memory 129, the write control register 127, and start execution. The program in the program memory 129, the execution of the program in the program memory 129, the completion of the notification request (for example, interruption) after executing the program in the program memory 129, and the reset of the neural network unit 121 are not limited thereto. In a preferred embodiment, the neural network unit instruction set includes an instruction, the result of which indicates that the neural network unit program has completed. In addition, the neural network unit instruction set includes an explicitly generated interrupt instruction. In a preferred embodiment, the operation of resetting the neural network unit 121 includes, in addition to the data random access memory 122, the weight random access memory 124, and the program memory in the neural network unit 121. The data of body 129 will remain intact except for the other parts, effectively forcing a return to a reset state (for example, emptying the internal state machine and setting it to an idle state). In addition, internal registers, such as accumulator 202, are not affected by the reset function and must be explicitly cleared, such as using the initialization neural processing unit instruction of address 0 in the fourth figure. In an embodiment, the function 1432 can include a direct execution function, the first source register of which includes a micro-operation (for example, reference to the micro-operation 3418 of the thirty-fourth figure). This direct execution function instructs the neural network unit 121 to directly perform the specified micro-operation. In this way, the architecture program can directly control the neural network unit 121 to perform operations instead of writing the instructions to the program memory 129 and subsequently instructing the neural network unit 121 to execute the instructions in the program memory 129 or through the MTNN instructions. Execution of 1400 (or MFNN instruction 1500 of Figure 15). The fourteenth figure shows an example of the function of this write data random access memory 122.

This gpr field specifies a general purpose register in the general register file 116. In one embodiment, each general purpose register is 64 bits. The universal scratchpad file 116 provides the value of the selected universal register to the neural network unit 121, as shown in the figure, and the neural network unit 121 uses this value as the address 1422. This address 1422 selects one of the columns of memory specified in function 1432. In the case of data random access memory 122 or weighted random access memory 124, this address 1422 additionally selects a data block that is twice the size of the media register in the selected column (eg, 512). Bit). In a preferred embodiment, this location is at a 512 bit boundary. In an embodiment, the multiplexer will select the address 1422 (or in the case of the MFNN instruction 1400 described below). The address of the address 1422) or the address 123/125/131 from the sequencer 128 is provided to the data random access memory 124/weight random access memory 124/program memory 129. In one embodiment, the data random access memory 122 has a double port, enabling the neural processing unit 126 to read/write to the data random access memory 122 using the media register 118, while reading/writing. The data is randomly accessed into the memory 122. In an embodiment, the weighted random access memory 124 also has double turns for similar purposes.

Both the src1 field 1404 and the src2 field 1406 in the figure specify a media register for the media register file 118. In one embodiment, each media register 118 is 256 bits. The media register file 118 provides the connected data (eg, 512 bits) from the selected media register to the data random access memory 122 (or the weighted random access memory 124 or the program memory). 129) The selected column 1428 specified by the write address 1422 and the location specified by the address 1422 in the selected column 1428, as shown in the figure. Through the execution of a series of MTNN instructions 1400 (and the MFNN instructions 1500 described below), the architecture program executing on the processor 100 can fill the data random access memory 122 columns and the weight random access memory 124 columns and A program write program memory 129, such as the one described herein (such as the programs shown in Figures 4 and 9), allows neural network unit 121 to perform operations on data and weights at very fast speeds to complete the manual. Neural network. In one embodiment, the architecture program directly controls the neural network unit 121 rather than writing the program to the program memory 129.

In one embodiment, the MTNN instruction 1400 specifies the number of a starting source register and a source register, ie, Q, rather than Two source registers (as specified by fields 1404 and 1406). This form of MTNN instruction 1400 instructs processor 100 to write media register 118 designated as the originating source register and then Q-1 consecutive media registers 118 to neural network unit 121, also That is, the specified data random access memory 122 or weight random access memory 124 is written. In a preferred embodiment, the instruction translator 104 translates the MTNN instruction 1400 into the number of microinstructions required to write all of the Q designated media registers 118. For example, in one embodiment, when the MTNN instruction 1400 designates the scratchpad MR4 as the starting source register and Q is 8, the instruction translator 104 translates the MTNN instruction 1400 into four microinstructions, where The first microinstruction is written to the scratchpads MR4 and MR5, the second microinstruction is written to the scratchpads MR6 and MR7, the third microinstruction is written to the scratchpads MR8 and MR9, and the fourth micro The instructions are written to the scratchpads MR10 and MR11. In another embodiment, the data path from media register 118 to neural network unit 121 is 1024 bits instead of 512 bits, in which case instruction translator 104 translates MTNN instruction 1400 into two A microinstruction in which the first microinstruction is written to the scratchpads MR4 to MR7, and the second microinstruction is written to the scratchpads MR8 to MR11. The present invention is also applicable to embodiments in which the MFNN instruction 1500 specifies a number of start destination registers and destination registers, such that each MFNN instruction 1500 can be accessed from the data random access memory 122 or the weighted random access memory. One of the columns 124 reads a block of data larger than the single media register 118.

The fifteenth diagram is a block diagram showing a move to neural network (MTNN) architecture instruction 1500 and its corresponding to the first figure The operation of the portion of the neural network unit 121. The MFNN instruction 1500 includes an execution code field 1502, a dst field 1504, a gpr field 1508, and an immediate field 1512. The MFNN instruction is an architectural instruction, that is, the instruction is included in the instruction set architecture of the processor 100. In a preferred embodiment, the instruction set architecture utilizes one of the execution code fields 1502 to distinguish between the MFNN instruction 1500 and other instructions within the instruction set architecture. The execution code 1502 of the MFNN instruction 1500 may or may not be included in a prefix such as the x86 architecture.

Immediate field 1512 provides a value to specify a function 1532 to control logic 1434 of neural network unit 121. In a preferred embodiment, this function 1532 is an immediate operand as one of the microinstructions 105 of the first figure. The functions 1532 that the neural network unit 121 can execute include a read data random access memory 122, a read weight random access memory 124, a read program memory 129, and a read status register 127, but not Limited to this. The example of the fifteenth figure shows a function 1532 of the read data random access memory 122.

This gpr field 1508 specifies a general purpose register within the general register file 116. The universal scratchpad file 116 provides the value of the selected universal register to the neural network unit 121, as shown in the figure, and the neural network unit 121 uses this value as the address 1522 and is similar to the tenth The operation of the address 1422 of the four graphs is performed to select one of the memories specified in the function 1532. In the case of the data random access memory 122 or the weighted random access memory 124, the address 1522 additionally selects a data block whose size is the media register (eg, 256 bits) in the selected column. position. In a preferred embodiment, this location is located A 256-bit boundary.

The dst field 1504 is associated with a media register in a media register file 118. As shown in the figure, the media register file 118 receives data (eg, 256 bits) from the data random access memory 122 (or the weighted random access memory 124 or the program memory 129) to the selected media. The scratchpad, this data is read from the selected column 1528 specified by the data receiving address 1522 and the location specified by the address 1522 in the selected column 1528.

Neural network unit internal random access memory configuration

Figure 16 is a block diagram showing an embodiment of the data random access memory 122 of the first figure. The data random access memory 122 includes a memory array 1606, a read buffer 1602 and a write buffer 1604. The memory array 1606 is loaded with data text. In a preferred embodiment, the data is arranged in an array of N words of D columns as previously described. In one embodiment, the memory array 1606 includes an array of 64 horizontally arranged static random access memory cells, each of which has a width of 128 bits and a height of 64 bits. A 64 KB data random access memory 122 is provided having a width of 8192 bits and having 64 columns, and the data random access memory 122 uses a die area of approximately 0.2 square millimeters. However, the invention is not limited thereto.

In a preferred embodiment, write 埠 1602 is coupled to neural processing unit 126 and media register 118 in a multiplexed manner. Further, the media registers 118 can be coupled through the result bus The read bus is also used to provide data to the rearrangement buffer and/or the result transfer bus to provide to other execution units 112. The neural processing unit 126 shares the read buffer 1602 with the media register 118 to read the data random access memory 122. Moreover, in a preferred embodiment, write 埠 1604 is also multiplexed to neural processing unit 126 and media register 118. The neural processing unit 126 shares the write buffer 1604 with the media register 118 to write the data random access memory 122. In this manner, the media register 118 can write the data random access memory 122 while the neural processing unit 126 reads the data random access memory 122, and the neural processing unit 126 can temporarily store the data in the media. The device 118 is writing to the data random access memory 122 while reading the data random access memory 122. This way of doing things can improve performance. For example, the neural processing unit 126 can read the data random access memory 122 (eg, continuously perform calculations) while the media register 118 can write more data to the data random access memory 122. . In another example, the neural processing unit 126 can write the calculation result to the data random access memory 122, and at the same time, the media register 118 can read the calculation result from the data random access memory 122. In one embodiment, the neural processing unit 126 can write a list of calculation results to the data random access memory 122 while also reading a list of data characters from the data random access memory 122. In one embodiment, memory array 1606 is configured as a memory bank. When the neural processing unit 126 accesses the data random access memory 122, all of the memory blocks are activated to access a complete column of the memory array 1606; however, accessing the data in the media register 118 When the memory 122 is randomly accessed, only the specified memory block will be activated. In one embodiment, each memory block has a width of 128 bits, and the width of the media register 118 is 256 bits. Thus, for example, each time the media register 118 is accessed Need to start two memory blocks. In one embodiment, one of the 埠 1602/1604 is a read/write 埠. In an embodiment, these 埠 1602/1604 are both read/write 埠.

An advantage of having these neural processing units 126 with the capabilities of a rotator as described herein is that the architectural program (via the media register 118) is continuously provided with information to ensure that the neural processing unit 126 is fully utilized. The data random access memory 122 and the memory array required to retrieve the results from the data random access memory 122 while the neural processing unit 126 is performing the calculations, this ability helps to reduce the data random access memory 122 The number of columns of the memory array 1606 can thus be reduced in size.

Internal random access memory buffer

Figure 17 is a block diagram showing an embodiment of a weighted random access memory 124 and a buffer 1704 of the first figure. The weight random access memory 124 includes a memory array 1706 and a stack 1702. The memory array 1706 is loaded with weighting text. For a preferred embodiment, the weighting texts are arranged in an array of N words of W columns as previously described. In one embodiment, the memory array 1706 includes an array of 128 horizontally arranged static random access memory cells, each of which has a width of 64 bits and a height of 2048 bits. Provide a 2MB weight random access memory 124, having a width of 8192 bits and having 2048 columns, and the weight random access memory 124 uses a die area of approximately 2.4 square millimeters. However, the invention is not limited thereto.

In a preferred embodiment, the 埠 1702 is multiplexed to the neural processing unit 126 and the buffer 1704. The neural processing unit 126 and the buffer 1704 are read through the buffer 1702 and written to the weight random access memory 124. The buffer 1704 is coupled to the media register 118 of the first figure. Thus, the media register 118 can read and write the weight random access memory 124 through the buffer 1704. An advantage of this approach is that the media register 118 can also write to or read from the buffer 118 when the neural processing unit 126 is reading or writing to the weighted random access memory 124 (but if The neural processing unit 126 is executing, and preferably, the neural processing unit 126 is placed to avoid accessing the weighted random access memory 124 when the buffer 1704 accesses the weighted random access memory 124. This approach can improve performance, particularly since the read and write of the media scratchpad 118 for the weighted random access memory 124 is significantly less than the read and write by the neural processing unit 126 for the weighted random access memory 124. . For example, in one embodiment, the neural processing unit 126 reads/writes 8192 bits (one column) at a time, however, the width of the media register 118 is only 256 bits, and each MTNN instruction 1400 is only Two media registers 118 are written, ie 512 bits. Therefore, in the case where the architecture program executes sixteen MTNN instructions 1400 to fill the buffer 1704, the collision time between the neural processing unit 126 and the architectural program accessing the weight random access memory 124 may be less than substantially all of the time. Six percent. In another embodiment, instruction translation The processor 104 translates an MTNN instruction 1400 into two microinstructions 105, and each microinstruction writes a single data register 118 to the buffer 1704. Thus, the neural processing unit 126 and the architectural program store the access weights randomly. The frequency of collisions when taking memory 124 is further reduced.

In an embodiment that includes buffer 1704, the use of an architectural program to write weighted random access memory 124 requires a plurality of MTNN instructions 1400. One or more MTNN instructions 1400 specify a function 1432 to write to the data block specified in buffer 1704, and then an MTNN instruction 1400 specifies a function 1432 to instruct neural network unit 121 to write the contents of buffer 1704 to weight random access. One of the memories 124 is selected. The size of a single data block is twice the number of bits in the media register 118, and these data blocks are naturally aligned in the buffer 1704. In one embodiment, each of the specified functions 1432 in the write buffer 1704 specifies that the MTNN instruction 1400 of the data block includes a bitmask having bits corresponding to the respective data blocks of the buffer 1704. The data from the two designated source registers 118 are written into the data block of the buffer 1704, and the corresponding bits in the bit mask are the respective data blocks that are set. This embodiment facilitates the case where there is a duplicate data value in one of the columns of the weighted random access memory 124. For example, to zero the buffer 1704 (and the next column of weighted random access memory 124), the programmer can load zero values into the source register and set all bits of the bit mask. . In addition, the bit mask can also allow the programmer to write only the selected data blocks in buffer 1704, leaving other data blocks to maintain their previous data state.

In an embodiment that includes buffer 1704, reading the weight random access memory 124 using the architectural program requires multiple MFNN instructions. 1500. The initial MFNN instruction 1500 specifies a function 1532 to load a designated column of the weight random access unit 124 into the buffer 1704, and then the one or more MFNN instructions 1500 specify a function 1532 to read the specified data block of one of the buffers 1704 to Destination register. The size of a single data block is the number of bits in the media register 118, and these data blocks are naturally aligned in the buffer 1704. The technical features of the present invention are also applicable to other embodiments. For example, the weight random access memory 124 has a plurality of buffers 1704, which further reduces the number of neural processing units by increasing the number of accesses of the architecture program when the neural processing unit 126 executes. The conflict between the 126 and the architecture program due to the access weight random memory 124 is increased by the buffer 1704 during the time-frequency period in which the neural processing unit 126 does not need to access the weight random access memory 124. Take the possibility.

The sixteenth figure depicts a pair of data random access memory 122, however, the present invention is not limited thereto. The technical features of the present invention are also applicable to other embodiments in which the weighted random access memory 124 is also a dual-turn design. Further, a buffer matching weight random access memory 124 is used in the seventeenth figure, but the present invention is not limited thereto. The technical features of the present invention are also applicable to embodiments in which the data random access memory 122 has a corresponding buffer similar to the buffer 1704.

Dynamically configurable neural processing unit

Figure 18 is a block diagram showing a dynamically configurable neural processing unit 126 of the first figure. The neural processing unit 126 of the eighteenth diagram is similar to the neural processing unit 126 of the second figure. However, the neural processing unit 126 of Fig. 18 is dynamically configurable to operate in one of two different configurations. In the first configuration, the god of the eighteenth The operation via processing unit 126 is similar to neural processing unit 126 of the second figure. That is, in the first configuration, labeled herein as a "wide" configuration or a "single one" configuration, the arithmetic logic unit 204 of the neural processing unit 126 has a single wide text file and a single wide weight text. An operation (eg, 16 bits) is performed to produce a single wide result. In contrast, in the second configuration, which is labeled herein as a "narrow" configuration or a "double" configuration, the neural processing unit 126 will have two narrow data words and two narrow weight texts (eg, eight). Bits) perform operations that produce two narrow results, respectively. In one embodiment, the configuration (wide or narrow) of the neural processing unit 126 is achieved by initializing a neural processing unit instruction (e.g., an instruction located at address 0 in the aforementioned twentieth diagram). Alternatively, this configuration can be accomplished by an MTNN command having a configuration (wide or narrow) that the function 1432 specifies to set the neural processing unit settings. In a preferred embodiment, the program memory 129 instructs or determines that the configuration (wide or narrow) MTNN instruction fills up the configuration register. For example, the output of the configuration register is provided to the logic logic unit 204, the startup function unit 212, and the logic that generates the multiplex register control signal 213. Basically, the elements of the neural processing unit 126 of Fig. 18 and the elements of the same number in the second figure perform similar functions, from which reference can be made to understand the embodiment of Fig. 18. The following description of the embodiment of the eighteenth embodiment includes the differences from the second figure.

The neural processing unit 126 of Fig. 18 includes two registers 205A and 205B, two three-input multiplex registers 208A and 208B, one arithmetic logic unit 204, two accumulators 202A and 202B, and two start-ups. Function units 212A and 212B. Register 205A/205B There are one half of the width of the register 205 of the second figure (such as 8 bits). The registers 205A/205B receive a corresponding narrow weight text 206A/B 206 (eg, 8 bits) from the weight random access memory 124 and provide the output 203A/203B to the arithmetic logic in a subsequent time-frequency period. The operand selection logic 1898 of unit 204. When the neural processing unit 126 is in a wide configuration, the registers 205A/205B operate together to receive a wide weight text 206A/206B (e.g., 16 bits) from the weighted random access memory 124, similar to the second In the embodiment of the embodiment, when the neural processing unit 126 is in a narrow configuration, the registers 205A/205B are actually operated independently, each receiving a narrow weight text from the weighted random access memory 124. 206A/206B (e.g., 8 bits), as such, the neural processing unit 126 is essentially equivalent to two narrow neural processing units operating independently of each other. However, regardless of the configuration of the neural processing unit 126, the same output bits of the weighted random access memory 124 are coupled and provided to the registers 205A/205B. For example, the register 205A of the neural processing unit 0 receives the byte 0, the register 205B of the neural processing unit 0 receives the byte 1, the register 205A of the neural processing unit 1 receives the byte 2, and the nerve The register 205B of the processing unit 1 receives the byte 3, and so on, and the register 205B of the neural processing unit 511 receives the byte 1023.

The multiplex registers 208A/208B have one-half (e.g., 8 bits) of the width of the register 208 of the second figure, respectively. Multiplex register 208A selects one of inputs 207A, 211A, and 1811A to store to its scratchpad and is provided by output 209A for subsequent time-frequency cycles. Multiplexed register 208B selects among inputs 207B, 211B, and 1811B. One store to its temporary storage The processor is provided by the output 209B to the operand selection logic 1898 at a subsequent time-frequency period. Input 207A receives a narrow data text (e.g., 8 bits) from data random access memory 122, and input 207B receives a narrow data text from data random access memory 122. When the neural processing unit 126 is in a wide configuration, the multiplex registers 208A/208B will actually operate together to receive a wide data text 207A/207B from the data random access memory 122 (e.g., 16 bits). ), similar to the multiplex register 208 in the embodiment of the second figure; when the neural processing unit 126 is in a narrow configuration, the multiplex registers 208A/208B will actually operate independently, each receiving random data from One of the narrow data words 207A/207B (e.g., 8 bits) is accessed by the memory 122. Thus, the neural processing unit 126 is essentially equivalent to the operation of the two narrow neural processing units. However, regardless of the configuration of the neural processing unit 126, the same output bits of the data random access memory 122 are coupled and provided to the multiplex registers 208A/208B. For example, the multiplexer 208A of the neural processing unit 0 receives the byte 0, the multiplexer 208B of the neural processing unit 0 receives the byte 1, and the multiplexer 208A of the neural processing unit 1 receives The octet 2, the multiplexer 208B of the neural processing unit 1 receives the byte 3, and so on, and the multiplexer 208B of the neural processing unit 511 receives the byte 1023.

Input 211A receives output 209A of multiplex register 208A of neighboring neural processing unit 126, and input 211B receives output 209B of multiplex register 208B of neighboring neural processing unit 126. The input 1811A receives the output 209B of the multiplexer 208B of the proximity neural processing unit 126, and the input 1811B receives the multiplex of the adjacent neural processing unit 126. The output 209A of the memory 208A. The nerve processing unit 126 shown in Fig. 18 is one of the N nerve processing units 126 shown in the first figure and is labeled as the nerve processing unit J. That is to say, the neural processing unit J is a representative example of one of the N neural processing units. In a preferred embodiment, the multiplexer 208A input 211A of the neural processing unit J receives the output 209A of the multiplexer 208A of the neural processing unit 126 of the example J-1, and the number of the neural processing unit J is The physical register 208A input 1811A will receive the multiplex buffer 208B output 209B of the neural processing unit 126 of the example J-1, and the multiplex register 208A output 209A of the neural processing unit J will be provided to the sample J+1 at the same time. The multiplexer register 208A of the neural processing unit 126 inputs 211A and the multiplexer 208B of the neural processing unit 126 of the example J inputs 211B; the input 211B of the multiplexer 208B of the neural processing unit J receives the sample J The multiplex register 208B of the neural processing unit 126 of the -1 outputs 209B, and the input 1811B of the multiplex register 208B of the neural processing unit J receives the output 209A of the multiplexer 208A of the neural processing unit 126 of the example J. And, the output 209B of the multiplexer 208B of the neural processing unit J is simultaneously provided to the multiplexer 208A input 1811A of the neural processing unit 126 of the example J+1 and the neural processing unit 126 of the example J+1. The multiplex register 208B inputs 211B.

Control input 213 controls each of multiplex registers 208A/208B, selects one of the three inputs to store to its corresponding register, and provides it to the corresponding output 209A/209B in a subsequent step. When the neural processing unit 126 is instructed to load a column from the data random access memory 122 (for example, the multiply-accumulate instruction of address 1 in the twentieth map, as will be described later), regardless of whether the neural processing unit 126 is For a wide or narrow configuration, the control input 213 controls each of the multiplex registers 208A/208B to select a phase from the corresponding narrow text of the selected column of the data random access memory 122. Corresponding narrow data text 207A/207B (such as 8-bit).

When the neural processing unit 126 receives the indication that the previously received data column value needs to be rotated (for example, the multiplication accumulation rotation instruction of the address 2 in the twentieth figure, as will be described later), if the neural processing unit 126 is in a narrow configuration, the control Input 213 controls each of the multiplex registers 208A/208B to select the corresponding input 1811A/1811B. In this case, the multiplex registers 208A/208B will actually operate independently and the neural processing unit 126 will behave as two separate narrow neural processing units. Thus, the multiplexers 208A and 208B of the N neural processing units 126 operate together as the same 2N narrow text rotators, which is described in more detail later in the corresponding FIG.

When the neural processing unit 126 receives an indication that the previously received data column value needs to be rotated, if the neural processing unit 126 is in a wide configuration, the control input 213 controls each of the multiplex registers 208A/208B. The device selection corresponds to input 211A/211B. In this case, the multiplex registers 208A/208B will operate together as if the neural processing unit 126 were a single wide neural processing unit 126. Thus, the multiplex registers 208A and 208B of the N neural processing units 126 operate together as the same N wide-text rotators, similarly to the manner described in the third figure.

Arithmetic logic unit 204 includes operand selection logic 1898, a wide multiplier 242A, a narrow multiplier 242B, a wide dual input multiplexer 1896A, a narrow dual input multiplexer 1896B, a wide adder 244A and a narrow adder 244B. In fact, this arithmetic logic unit 204 can be understood to include operand selection logic, a wide arithmetic logic unit 204A (including the aforementioned wide multiplier 242A, the aforementioned wide multiplexer 1896A and the aforementioned wide adder 244A) and a narrow arithmetic logic unit. 204B (including the aforementioned narrow multiplier 242B, the aforementioned narrow multiplexer 1896B and the aforementioned narrow adder 244B). In a preferred embodiment, wide multiplier 242A can multiply two wide words, similar to multiplier 242 of the second figure, such as a 16-bit by 16-bit multiplier. Narrow multiplier 242B can multiply two narrow words, such as an 8-bit by 8-bit multiplier to produce a 16-bit result. When the neural processing unit 126 is in a narrow configuration, the wide multiplier 242A can be fully utilized as a narrow multiplier to multiply the two narrow words by the assistance of the operand selection logic 1898, so that the neural processing unit 126 Two narrowly operated nerve processing units. In a preferred embodiment, wide adder 244A adds the output of wide multiplexer 1896A to output 217A of wide accumulator 202A to produce a total number 215A for use by wide accumulator 202A, which operates similarly to Adder 244 of the second figure. Narrow adder 244B adds the output of narrow multiplexer 1896B to narrow accumulator 202B output 217B to produce a total number 215B for use by narrow accumulator 202B. In one embodiment, the narrow accumulator 202B has a width of 28 bits to avoid loss of accuracy when performing an accumulation operation of up to 1024 16-bit products. When the neural processing unit 126 is in a wide configuration, the narrow multiplier 244B, the narrow accumulator 202B, and the narrow start function unit 212B are preferably in an inactive state to reduce energy consumption.

The operand selection logic 1898 will select operands from 209A, 209B, 203A, and 203B to provide other elements to the arithmetic logic unit 204, as will be described later. In a preferred embodiment, operand selection logic 1898 also has other functions, such as performing symbolic extensions of signed numeric data and weighted text. For example, if the neural processing unit 126 is in a narrow configuration, the operand selection logic 1898 extends the sign of the narrow data text and the weight text to the width of the wide text and then provides it to the wide multiplier 242A. Similarly, if the arithmetic logic unit 204 accepts an indication that a narrow data/weight text is to be passed (using the wide multiplexer 1896A to skip the wide multiplier 242A), the operand selection logic 1898 extends the sign of the narrow data text and the weight text to The width of the wide text is then provided to the wide adder 244A. For a preferred embodiment, the logic for performing the symbol stretching function is also present within the arithmetic logic operation 204 of the neural processing unit 126 of the second figure.

The wide multiplexer 1896A receives the output of the wide multiplier 242A and one of the operands from the operand selection logic 1898, and selects one of these inputs for the wide adder 244A, and the narrow multiplexer 1896B receives the narrow multiplier 242B. The output is derived from one of the operand selection logic 1898 and one of these inputs is provided to the narrow adder 244B.

The operand selection logic 1898 provides operands in accordance with the configuration of the neural processing unit 126 and the arithmetic and/or logical operations to be performed by the arithmetic logic unit 204, which is based on the function specified by the instructions executed by the neural processing unit 126. Decide. For example, if the instruction instructs the arithmetic logic unit 204 to perform a multiply-accumulate operation The neural processing unit 126 is in a wide configuration, and the operand selection logic 1898 concatenates the outputs 209A and 209B to form one wide text to one input of the wide multiplier 242A, and the outputs 203A and 203B in series to form a wide text. Provided to another input, the narrow multiplier 242B is not activated, such that the neural processing unit 126 operates as a single wide neural processing unit 126 similar to the neural processing unit 126 of the second figure. However, if the instruction instructs the arithmetic logic unit to perform a multiply-accumulate operation and the neural processing unit 126 is in a narrow configuration, the operand selection logic 1898 provides an extended or expanded version of the narrow data word 209A to one of the wide multipliers 242A. Input, while the extended version of the narrow weight text 203A is provided to another input; in addition, the operand selection logic 1898 provides the narrow data text 209B to one of the narrow multipliers 242B and the narrow weight text 203B to the other An input. In order to achieve the operation of extending or expanding the narrow text as described above, if the narrow text has a symbol, the operation element selection logic 1898 will perform symbol extension on the narrow text; if the narrow text has no symbol, the operation element selection logic 1898 The upper zero value is added to the narrow text.

In another example, if the neural processing unit 126 is in a wide configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation operation of a weighted text, the wide multiplier 242A is skipped and the operand selection logic 1898 will output 203A. Serial multiplexer 1896A is provided in series with 203B for supply to wide adder 244A. However, if the neural processing unit 126 is in a narrow configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation operation of a weighted text, the wide multiplier 242A is skipped and the operand selection logic 1898 will have an extended version output. 203A The wide multiplexer 1896A is provided to provide the wide adder 244A; in addition, the narrow multiplier 242B is skipped, and the operand selection logic 1898 provides the extended version of the output 203B to the narrow multiplexer 1896B for narrowing. Adder 244B.

In another example, if the neural processing unit 126 is in a wide configuration and the instruction instructs the arithmetic logic unit 204 to perform an accumulation of data, the wide multiplier 242A is skipped and the operand selection logic 1898 will output 209A. Serial multiplexer 1896A is provided in series with 209B for supply to wide adder 244A. However, if the neural processing unit 126 is in a narrow configuration and the instruction instructs the arithmetic logic unit 204 to perform a data word accumulation operation, the wide multiplier 242A is skipped and the operand selection logic 1898 will have an extended version output. 209A is provided to wide multiplexer 1896A for supply to wide adder 244A; in addition, narrow multiplier 242B is skipped, and operand selection logic 1898 provides extended version 209B to narrow multiplexer 1896B for provision to Narrow adder 244B. The cumulative calculation of weights/data texts contributes to the averaging operation, which can be used in the pooling layer of some artificial neural network applications, such as image processing.

In a preferred embodiment, the neural processing unit 126 further includes a second wide multiplexer (not shown) for skipping the wide adder 244A to facilitate wide data/weight text in a wide configuration. Or a narrow data/weight text loaded into the wide accumulator 202A in a narrow configuration, and a second narrow multiplexer (not shown) to skip the narrow adder 244B to facilitate narrow configuration The next narrow data/weight text is loaded into the narrow accumulator 202B. In a preferred embodiment, the arithmetic logic unit 204 also includes a wide and narrow comparator/multiplexer combination (not shown) that receives the corresponding accumulator value 217A/217B and the corresponding multiplexer 1896A/1896B output. In order to select the maximum value between the accumulator value 217A/217B and a data/weight text 209A/209B/203A/203B, the pooling layer of some artificial neural network applications uses this operation, which is followed by Chapters, for example corresponding to the twenty-seventh and twenty-eighth, will be described in more detail. In addition, the operand selection logic 1898 is used to provide an arithmetic element of zero value (for addition of zeros or to clear the accumulator) and to provide an arithmetic element of value one (for multiplication by one).

Narrow start function unit 212B receives output 217B of narrow accumulator 202B and performs a start function thereon to produce a narrow result 133B. Wide start function unit 212A receives output 217A of wide accumulator 202A and performs a start function thereon to generate a The wide result is 133A. When the neural processing unit 126 is in a narrow configuration, the wide start function unit 212A will understand the output 217A of the accumulator 202A and perform a start function on it to generate a narrow result, such as an 8-bit, which is corresponding in subsequent sections. A more detailed description is given in Figures 29 to 30.

As previously mentioned, a single neural processing unit 126 can actually operate as two narrow neural processing units when in a narrow configuration, so that for smaller text, it is generally comparable to a wide configuration. Up to twice the processing power. For example, suppose the neural network layer has 1024 neurons, and each neuron receives 1024 narrow inputs (and has narrow weighted text) from the previous layer, thus producing one million links. For a neural network with 512 neural processing units 126 In the case of the road unit 121, in a narrow configuration (corresponding to 1024 narrow neural processing units), although the processing is narrow text instead of wide text, the number of links that can be processed can be four times that of the wide configuration (one hundred Ten thousand links are connected to 256K connections, and the required time is roughly half (about 1026 time-frequency periods versus 514 time-frequency periods).

In one embodiment, the dynamic configuration neural processing unit 126 of FIG. 18 includes a three-input multiplexer similar to the multiplex registers 208A and 208B in place of the registers 205A and 205B to form a rotator. The weighted character string received by the weighted random access memory 124 is processed in a manner similar to that described in the embodiment of the seventh embodiment but applied to the dynamic configuration described in the eighteenth figure.

Figure 19 is a block diagram showing the 2N multiplex registers 208A/208B of the N neural processing units 126 of the neural network unit 121 of the first figure according to the embodiment of the eighteenth figure. The data random access memory 122 of the first figure obtains a list of data words 207 to perform operations as the same rotator. In the embodiment of the nineteenth embodiment, N is 512, and the neural processing unit 121 has 1024 multiplex registers 208A/208B, labeled 0 to 511, corresponding to 512 neural processing units 126 and 1024 actually. Narrow nerve processing unit. The two narrow neural processing units within the neural processing unit 126 are labeled A and B, respectively, and in each multiplex register 208, the corresponding narrow neural processing unit is also labeled. Further, the multiplex register 208A of the neural processing unit 126 labeled 0 is labeled 0-A, and the multiplex register 208B of the neural processing unit 126 labeled 0 is labeled 0-B, labeled as The multiplex register 208A of the neural processing unit 126 of 1 is labeled as 1-A, the nerve labeled 1 The multiplex register 208B of the processing unit 126 is labeled 1-B, the multiplex register 208A of the neural processing unit 126 labeled 511 is labeled 511-A, and the number of neural processing units 126 labeled 511 is The work register 208B is labeled 511-B, and its value also corresponds to the narrow nerve processing unit described in the subsequent twenty-first figure.

Each multiplex register 208A receives its corresponding narrow data 207A in one of the D columns of the data random access memory 122, and each multiplex buffer 208B is in the data random access memory. One of the D columns of 122 receives its corresponding narrow data text 207B. That is, the multiplex register 0-A receives the narrow data text 0 of the data random access memory 122 column, and the multiplex register 0-B receives the narrow data text 1 of the data random access memory 122 column. The multiplexed register 1-A receives the narrow data text 2 of the data random access memory 122, the multiplex register 1-B receives the narrow data text of the data random access memory 122, and so on. The multiplexed register 511-A receives the narrow data text 1022 of the data random access memory 122, and the multiplex register 511-B receives the narrow data text 1023 of the data random access memory 122. In addition, the multiplex register 1-A receives the output 209A of the multiplex register 0-A as its input 211A, and the multiplex register 1-B receives the output 209B of the multiplex register 0-B as its input. 211B, and so on, the multiplex register 511-A receives the output 209A of the multiplex register 510-A as its input 211A, and the multiplex register 511-B receives the output of the multiplex register 510-B. 209B as its input 211B, and the multiplex register 0-A receives the output 209A of the multiplex register 511-A as its input 211A, and the multiplex register 0-B receives the multiplex register 511-B. Output 209B is its input 211B. Each multiplex register The 208A/208B will receive the control input 213 to control its selection data text 207A/207B or to rotate the input 211A/211B or to rotate the input 1811A/1811B. Finally, the multiplex register 1-A receives the output 209B of the multiplex register 0-B as its input 1811A, and the multiplex register 1-B receives the output 209A of the multiplex register 1-A as its input. 1811B, and so on, the multiplex register 511-A receives the output 209B of the multiplex register 510-B as its input 1811A, and the multiplex register 511-B receives the output of the multiplex register 511-A. 209A as its input 1811B, and the multiplex register 0-A receives the output 209B of the multiplex register 511-B as its input 1811A, and the multiplex register 0-B receives the multiplex register 0-A Output 209A is its input 1811B. Each multiplex register 208A/208B receives a control input 213 to control its selection data 207A/207B or a post-rotation input 211A/211B or a post-rotation input 1811A/1811B. In an operational mode, during the first time-frequency period, control input 213 controls each multiplex register 208A/208B to select data words 207A/207B for storage to the scratchpad for subsequent supply to arithmetic logic unit 204; Subsequent time-frequency periods (eg, the M-1 time-frequency period described above), control input 213 controls each multiplex register 208A/208B to select the rotated input 1811A/1811B to store to the scratchpad for subsequent supply to the arithmetic logic unit 204, this section will be described in more detail in subsequent chapters.

Figure 20 is a table showing a program stored in the program memory 129 of the neural network unit 121 of the first figure and executed by the neural network unit 121, and the neural network unit 121 has the eighteenth The nerve processing unit 126 shown in the embodiment of the figure. The example program in Figure 20 is similar to the program in Figure 4. The following is to say the difference Bright. The initialization neural processing unit instruction at address 0 specifies that the neural processing unit 126 will enter a narrow configuration. Furthermore, as shown in the figure, the multiply-accumulate rotation instruction at address 2 specifies a count value of 1023 and requires 1023 time-frequency periods. This is because the example in the twentieth diagram assumes that there are actually 1024 narrow (eg, 8-bit) neurons (ie, neural processing units) in one layer, each with 1024 neurons from the previous layer. The link of the yuan is entered, so there are a total of 1024K links. Each neuron receives an 8-bit data value from each of the link inputs and multiplies the 8-bit data value by an appropriate 8-bit weight value.

The twenty-first figure shows a timing chart in which a neural network unit 121 executes the program of the twentieth diagram, and the neural network unit 121 has the neural processing unit 126 as shown in the eighteenth diagram executed in a narrow configuration. The timing chart of the twenty-first figure is similar to the timing chart of the fifth figure. The differences are explained below.

In the timing diagram of the twenty-first figure, these neural processing units 126 will be in a narrow configuration because the initialization neural processing unit instruction at address 0 initializes it to a narrow configuration. Therefore, the 512 neural processing units 126 actually function as 1024 narrow neural processing units (or neurons), which are treated with neuroprocessing units 0-A and neural processing units in the field. 0-B (two narrow neuroprocessing units of neuroprocessing unit 126 labeled 0), neuroprocessing unit 1-A and neuroprocessing unit 1-B (two narrow neural processing units of neuroprocessing unit 126 labeled 1) ), and so on until the neural processing unit 511-A and the neural processing unit 511-B (the two narrow neural processing units of the neural processing unit 126 labeled 511) are indicated. To simplify the description, Only the operations of the narrow neuroprocessing units 0-A, 0-B, and 511-B are shown. Since the count value specified by the multiply-accumulate rotation instruction at address 2 is 1023 and 1023 time-frequency periods are required to operate, the number of timing diagrams in the twenty-first graph includes up to 1026 time-frequency periods.

In a time-frequency period of 0, each of the 1024 neural processing units performs an initialization instruction of the fourth map, i.e., assigns a zero value to the operation of the accumulator 202 as shown in the fifth figure.

In time-frequency period 1, each of the 1024 narrow neural processing units performs a multiply-accumulate instruction at address 1 in the twentieth diagram. As shown in the figure, the narrow neural processing unit 0-A adds the value of the accumulator 202A (i.e., zero) to the column of the data random access unit 122, the narrow text 0, and the weight random access unit 124, the narrow text 0. Product; the narrow neural processing unit 0-B adds the value of the accumulator 202B (i.e., zero) to the product of the narrow text 1 of the data random access unit 122 and the narrow text 1 of the weight random access unit 124; Analogy until the narrow neural processing unit 511-B adds the value of the accumulator 202B (i.e., zero) to the product of the narrow text 1023 of the column 17 of the data random access unit 122 and the narrow text 1023 of the weight random access unit 124.

In time-frequency period 2, each of the 1024 narrow neural processing units performs the first iteration of the multiply-accumulate rotation instruction at address 2 in the twentieth diagram. As shown in the figure, the narrow neural processing unit 0-A adds the value 217A of the accumulator 202A to the rotated narrow data text 1811A received by the multiplexer 208B output 209B of the narrow neural processing unit 511-B (ie, The narrow data text 1023 received by the data random access memory 122 and the narrow text 0 of the column 1 of the weight random access unit 124 The narrow neural processing unit 0-B adds the accumulator 202B value 217B to the rotated narrow data text 1811B received by the narrow neural processing unit 0-A multiplex register 208A output 209A (that is, the data is stored randomly) The product of the narrow data text 0) received by the memory 122 and the narrow text 1 of the column 1 of the weight random access unit 124; and so on, until the narrow neural processing unit 511-B adds the value 217B of the accumulator 202B to the narrow The multiplexed register 208A of the neural processing unit 511-A outputs the rotated narrow data text 1811B (that is, the narrow data text 1022 received by the data random access memory 122) and the weight random access unit 124 received by the 209A. The product of the narrow text 1023.

In time-frequency period 3, each of the 1024 narrow neural processing units performs the second iteration of the multiply-accumulate rotation instruction at address 2 in the twentieth diagram. As shown in the figure, the narrow neural processing unit 0-A adds the value 217A of the accumulator 202A to the rotated narrow data text 1811A received by the multiplexer 208B output 209B of the narrow neural processing unit 511-B (ie, The product of the narrow data text 1022 received by the data random access memory 122 and the narrow text 0 of the column 2 of the weight random access unit 124; the narrow neural processing unit 0-B adds the value 217B of the accumulator 202B by the narrow nerve The multiplexer 208A of the processing unit 0-A outputs the rotated narrow data text 1811B received by the 209A (that is, the narrow data text 1023 received by the data random access memory 122) and the weight random access unit 124. Column 2 is the product of narrow text 1; and so on, until the narrow neural processing unit 511-B adds the accumulator 202B value 217B to the rotation received by the multiplexer 208A output 209A of the narrow neural processing unit 511-A. Narrow data text 1811B (that is, received by data random access memory 122) The narrow data text 1021) is the product of the narrow text 1023 of the column 2 of the weight random access unit 124. As shown in the twenty-first figure, this operation continues for the subsequent 1021 time-frequency periods until the time-frequency period 1024 described below.

In the time-frequency period 1024, each of the 1024 narrow neural processing units performs the 1023th iteration of the multiply-accumulate rotation instruction at address 2 in the twentieth diagram. As shown in the figure, the narrow neural processing unit 0-A adds the value 217A of the accumulator 202A to the rotated narrow data text 1811A received by the multiplexer 208B output 209B of the narrow neural processing unit 511-B (ie, The product of the narrow data text 1) received by the data random access memory 122 and the narrow text 0 of the column 1023 of the weight random access unit 124; the narrow neural processing unit 0-B adds the value 217B of the accumulator 202B to the narrow nerve The multiplexer 208A of the processing unit 0-A outputs the rotated narrow data text 1811B received by the 209A (that is, the narrow data character 2 received by the data random access memory 122) and the weight random access unit 124. Column 1023 is the product of narrow text 1; and so on, until narrow neuroprocessing unit 511-B adds accumulator 202B value 217B to the rotation received by multiplexer 208A output 209A of narrow neural processing unit 511-A. The product of the narrow data text 1811B (that is, the narrow data text 0 received by the data random access memory 122) and the narrow text 1023 of the column 1023 of the weight random access unit 124.

At time-frequency period 1025, the start function unit 212A/212B of each of the 1024 narrow neural processing units performs the start function instruction at address 3 in the twentieth diagram. Finally, in a time-frequency period 1026, each of the 1024 narrow neural processing units writes its narrow result 133A/133B back into column 16 of the data random access memory 122. Corresponding to the narrow text, to execute the write start function unit instruction at address 4 in the twentieth diagram. That is, the narrow result 133A of the neural processing unit 0-A is written to the narrow text 0 of the data random access memory 122, and the narrow result 133B of the neural processing unit 0-B is written to the data random access memory 122. The narrow text 1, and so on, until the narrow result 133B of the neural processing unit 511-B is written to the narrow text 1023 of the data random access memory 122. The twenty-second figure shows the aforementioned operation corresponding to the twenty-first figure in a block diagram.

The twenty-second diagram shows a block diagram of the neural network unit 121 of the first diagram, the neural network unit 121 having the neural processing unit 126 as shown in Fig. 18 to execute the program of the twentieth diagram. The neural network unit 121 includes 512 neural processing units 126, namely 1024 narrow neural processing units, data random access memory 122, and weight random access memory 124. The data random access memory 122 receives its bits. Address input 123, weighted random access memory 124 receives its address input 125. Although not shown in the figure, in the time-frequency period 0, the 1024 narrow neural processing units will execute the initialization instructions of the twentieth map. As shown in the figure, at time-frequency period 1, 1024 octet data words of column 17 are read from data random access memory 122 and provided to the 1024 narrow neural processing units. In the time-frequency period 1 to 1024, 1024 8-bit weight texts of columns 0 to 1023 are read out from the weight random access memory 124 and supplied to the 1024 narrow neural processing units, respectively. Although not shown in the figure, in the time-frequency period 1, the 1024 narrow neural processing units perform the corresponding multiply-accumulate operation on the loaded data text and the weight text. In the time-frequency period 2 to 1024, the number of 1024 narrow neural processing units The scratchpad 208A/208B operates as a 1024 8-bit text rotator that rotates the data text of the column 17 of the previously loaded data random access memory 122 to the adjacent narrow neural processing unit. These narrow neural processing units perform a multiply-accumulate operation on the corresponding rotated data text and the corresponding narrow weight text loaded by the weight random access memory 124. Although not shown in the figure, at time-frequency period 1025, the 1024 narrow start function units 212A/212B execute a start command. At time-frequency period 1026, the 1024 narrow neural processing units write their corresponding 1024 octet results 133A/133B back to column 16 of data random access memory 122.

It can be seen that, compared to the embodiment of the second figure, the embodiment of the eighteenth embodiment gives the programmer the flexibility to choose to use wide data and weight text (such as 16 bits) and narrow data and weight text (such as 8-bit) Perform calculations to meet the accuracy requirements for a particular application. From a single perspective, the embodiment of the eighteenth embodiment provides twice the performance compared to the embodiment of the second figure for narrow data applications, but additional narrow components must be added (eg, multiplexed temporary storage) At the expense of 208B, register 205B, narrow arithmetic logic unit 204B, narrow accumulator 202B, narrow start function unit 212B), these additional narrow elements will increase the neural processing unit 126 by about 50%.

Three-mode neural processing unit

A twenty-third figure is a block diagram showing another embodiment of a dynamically configurable neural processing unit 126 of the first figure. The neural processing unit 126 of the twenty-third figure can be used not only for wide configuration and narrow configuration It can also be used in a third configuration, referred to herein as a "funnel" configuration. The neural processing unit 126 of the twenty-third diagram is similar to the neural processing unit 126 of the eighteenth diagram. However, the wide adder 244A in Fig. 18 is replaced by a three-input wide adder 2344A in the neural processing unit 126 of the twenty-third figure, and the three-input wide adder 2344A receives a third addend 2399. It is an extended version of the output of the narrow multiplexer 1896B. The program executed by the neural network unit having the neural processing unit of the twenty-third figure is similar to the program of the twenty-first figure. However, the initialization neural processing unit instructions located at address 0 will initialize these neural processing units 126 into a funnel configuration rather than a narrow configuration. In addition, the multiply-accumulate rotation instruction at address 2 has a count value of 511 instead of 1023.

When in the funnel configuration, the operation of the neural processing unit 126 is similar to being in a narrow configuration. When performing the multiply accumulate instruction of address 1 in the twentieth diagram, the neural processing unit 126 receives two narrow data words 207A/207B and Two narrow weight texts 206A/206B; wide multiplier 242A multiplies data text 209A by weight text 203A to produce product 246A selected by wide multiplexer 1896A; narrow multiplier 242B compares data text 209B with weight text 203B Multiply by the product 246B that produces the narrow multiplexer 1896B selection. However, the wide adder 2344A adds the product 246A (selected by the wide multiplexer 1896A) and the product 246B/2399 (selected by the wide multiplexer 1896B) to the wide accumulator 202A output 217A, and the narrow adder 244B The narrow accumulator 202B is not activated. In addition, when in the funnel configuration to perform the multiply-accumulate rotation instruction of address 2 in the twentieth diagram, the control input 213 causes the multiplex register 208A/208B to rotate two narrow words (eg, 16 bits), that is, Said that the multiplex register 208A/208B will Select its corresponding input 211A/211B as if it were in a wide configuration. However, wide multiplier 242A multiplies data text 209A by weight text 203A to produce product 246A selected by wide multiplexer 1896A; narrow multiplier 242B multiplies data text 209B by weight text 203B to produce a narrow multiplexer 1896B selects the product 246B; and, the wide adder 2344A adds the product 246A (selected by the wide multiplexer 1896A) and the product 246B/2399 (selected by the wide multiplexer 1896B) to the wide accumulator 202A output 217A, The narrow adder 244B and the narrow accumulator 202B are not activated as described above. Finally, when in the funnel configuration to execute the start function instruction of address 3 as in the twentieth diagram, the wide start function unit 212A performs a start function on the total number of results 215A to produce a narrow result 133A, while the narrow start function unit 212B is Do not start. Thus, only the narrow nerve processing unit labeled A will produce a narrow result 133A, and the narrow result 133B produced by the narrow nerve processing unit labeled B is ineffective. Therefore, writing back the result column (as indicated by the instruction of address 4 in Figure 20) will contain holes because only the narrow result 133A is valid and the narrow result 133B is invalid. Therefore, conceptually, in each time-frequency period, each neuron (the neural processing unit of the twenty-third figure) performs two linked data inputs, ie multiplies two narrow data words by their corresponding weights and The two products are added. In contrast, the second and eighteenth embodiments perform only one link data input per time-frequency period.

In the embodiment of the twenty-third figure, it can be found that the number of result words (neuron output) generated and written back to the data random access memory 122 or the weight random access memory 124 is the received data input (link). Half of the square root of the quantity, and the result is written back to the column with null The hole, that is, every other narrow text result is invalid. More precisely, the result of the narrow neural processing unit labeled B is meaningless. Thus, the embodiment of the twenty-third diagram is particularly efficient for a neural network having two consecutive layers, for example, the first layer has twice as many neurons as the second layer (eg, the first layer has 1024) The neurons are fully connected to the 512 neurons of the second layer). In addition, other execution units 122 (eg, media units, such as x86 advanced vector expansion units) may perform a packing operation on a scatter result column (ie, having holes) to make it compact (ie, not void), if necessary. . Subsequently, when the neural processing unit 121 performs other calculations associated with the data random access memory 122 and/or the other columns of the weight random access memory 124, the processed data column can be used for calculation.

Hybrid neural network unit operation: convolution and common source computing

An advantage of the neural network unit 121 described in the embodiments of the present invention is that the neural network unit 121 can simultaneously operate in a manner similar to a coprocessor executing its own internal program and executing in a processing unit similar to a processor. The published architecture directive (or the microinstruction translated by the architectural directive). The architectural instructions are contained within an architectural program executed by a processor having a neural network unit 121. As such, the neural network unit 121 can operate in a mixed manner while maintaining the high utilization of the neural processing unit 121. For example, the twenty-fourth to twenty-sixth diagrams show the operation of the neural network unit 121 to perform a convolution operation in which the neural network unit is fully utilized, and the twenty-seventh to twenty-eighth diagrams show the nerve The network unit 121 performs the operation of the common source operation. Convolution layer, common source layer, and His digital data calculation applications, such as image processing (such as edge detection, sharpening, blurring, recognition/classification), require these operations. However, the hybrid operation of the neural processing unit 121 is not limited to performing convolution or common source operations. This hybrid feature can also be used to perform other operations, such as the conventional neural network multiply accumulate operation and start function described in the fourth through thirteenth figures. Operation. That is, the processor 100 (more precisely, the reservation station 108) will issue the MTNN command 1400 and the MFNN command 1500 to the neural network unit 121, and in response to this issued command, the neural network unit 121 will write the data to the memory. The body 122/124/129 reads the result from the memory 122/124 written by the neural network unit 121, and at the same time, executes the program of the program memory 129 for executing the processor 100 (via the MTNN1400 instruction). The neural network unit 121 reads and writes to the memory 122/124/129.

The twenty-fourth diagram is a block diagram showing an example of a data structure used by the neural network unit 121 of the first figure to perform a convolution operation. The block diagram includes a convolution kernel 2402, a data array 2404, and the data random access memory 122 and the weight random access memory 124 of the first figure. In a preferred embodiment, data array 2404 (e.g., corresponding to image pixels) is loaded into system memory (not shown) coupled to processor 100 and loaded by processor 100 by executing MTNN instructions 1400. The network unit 121 weights the random access memory 124. The convolution operation convolves a first array with a second array, which is the convolution kernel described herein. As described herein, a convolution kernel is a matrix of coefficients, which may also be referred to as weights, parameters, elements, or values. For a preferred embodiment, the convolution kernel 2042 is static data of the architecture program executed by the processor 100.

The data array 2404 is a two-dimensional array of data values, and the size of each data value (eg, image pixel value) is the size of the text of the data random access memory 122 or the weight random access memory 124 (eg, 16). Bit or 8-bit). In this example, the data value is 16-bit characters, and the neural network unit 121 is configured with 512 wide-configured neural processing units 126. Moreover, in this embodiment, the neural processing unit 126 includes a multiplex register to receive the weight text 206 from the weighted random access memory 124, such as the multiplex register 705 of the seventh figure, whereby the weights are random. The access memory 124 receives a list of data values to perform a collective rotator operation, which is described in more detail in subsequent sections. In this example, data array 2404 is a 2560 row x 1600 column pixel array. As shown in the figure, when the architecture program convolves the data array 2404 with the convolution kernel 2402, the data array 2402 is divided into 20 data blocks, and each data block is a 512x400 data array 2406.

In this example, convolution kernel 2402 is a 3x3 array of coefficients, weights, parameters, or elements. The first column of these coefficients is labeled C0,0; C0,1; and C0,2; the second column of these coefficients is labeled C1,0; C1,1; and C1,2; the third column of these coefficients is labeled C2,0; C2,1; and C2,2. For example, a convolution kernel with the following coefficients can be used to perform edge detection: 0, 1, 0, 1, 4, 1, 0, 1, 0. In another embodiment, a convolution kernel having the following coefficients can be used to perform Gaussian blur operations: 1, 2, 1, 2, 4, 2, 1, 2, 1. In this example, a division is typically performed on the final accumulated value, where the sum of the absolute values of the elements of the divisor-convolution kernel 2042 is 16, in this example. In another example, the divisor may be the number of elements of the convolution kernel 2042. In yet another In the example, the divisor may be a value used to compress the convolution operation to a target value range determined by the element value of the convolution kernel 2042, the target range, and the range of the input value array performing the convolution operation. .

Referring to the twenty-fourth diagram and the twenty-fifth diagram detailing the details therein, the architecture program writes the coefficients of the convolution kernel 2042 into the data random access memory 122. In a preferred embodiment, all of the text on each of the nine consecutive columns of data random access memory 122 (the number of elements in convolution kernel 2402) will utilize different elements of convolution kernel 2402. Columns are written in their primary order. That is to say, as shown in the figure, each character in the same column is written with the first coefficient C0,0; the next column is written with the second coefficient C0,1; the next column is the third coefficient. C0,2 is written; the next column is written with the fourth coefficient C1,0; and so on, until each character of the ninth column is written with the ninth coefficient C2,2. In order to perform a convolution operation on the data matrix 2406 of the data block segmented by the data array 2404, the neural processing unit 126 repeatedly reads the nine columns of the coefficients of the convolution kernel 2042 loaded in the data random access memory 122 in this order. In the subsequent sections, particularly the sections corresponding to the twenty-sixth A diagram, there will be a more detailed description.

Referring to the twenty-fourth diagram and the twenty-fifth diagram detailing the details therein, the architecture program writes the value of the data matrix 2406 into the weight random access memory 124. When the neural network unit program performs the convolution operation, the result array is written back to the weighted random access memory 124. In a preferred embodiment, the architecture program writes a first data matrix 2406 to the weighted random access memory 124 and causes the neural network unit 121 to operate, when the neural network unit 121 is in the first data matrix. 2406 and convolution kernel When the convolution operation is performed by the 2402, the architecture program writes a second data matrix 2406 into the weight random access memory 124. Thus, after completing the convolution operation of the first data matrix 2406, the neural network unit 121 can start executing. The convolution operation of the second data matrix 2406, which is described in more detail later in the corresponding figure. In this manner, the architecture program will travel to and from the two regions of the weighted random access memory 124 to ensure that the neural network unit 121 is fully utilized. Therefore, the example of the twenty-fourth figure shows a first data matrix 2406A and a second data matrix 2406B, and the first data matrix 2406A corresponds to the first data block occupying the columns 0 to 399 of the weight random access memory 124. And the second data matrix 2406B corresponds to the second data block occupying the columns 500 to 899 in the weight random access memory 124. In addition, as shown in the figure, the neural network unit 121 writes the result of the convolution operation back to the columns 900-1299 of the weighted random access memory 124 and the columns 1300-1699, and then the architecture program will use the weight random access memory. Body 124 reads these results. The data value of the data matrix 2406 loaded in the weighted random access memory 124 is labeled as "Dx, y", where "x" is the number of columns of the weighted random access memory 124, and "y" is the weighted random access memory. The text, or the number of lines. For example, the data text 511 located in column 399 is labeled D399, 511 in the twenty-fourth figure, and is received by the multiplex register 705 of the neural processing unit 511.

The twenty-fifth diagram is a flowchart showing that the processor 100 of the first figure executes an architectural program to perform a convolution operation on the convolution kernel 2042 on the data array 2404 of the twenty-fourth map using the neural network unit 121. This process begins in step 2502.

In step 2502, the processor 100 executes the shelf The processor 100 of the program writes the convolution kernel 2402 of the twenty-fourth figure into the data random access memory 122 in the manner described in the twenty-fourth figure. In addition, the architecture program initializes a variable N to a value of 1. The variable N is a data block that the neural network unit 121 is processing in the data array 2404. In addition, the architecture program initializes a variable NUM_CHUNKS to a value of 20. The flow then proceeds to step 2504.

In step 2504, as shown in FIG. 24, the processor 100 writes the data matrix 2406 of the data block 1 into the weight random access memory 124 (eg, the data matrix 2406A of the data block 1). The flow then proceeds to step 2506.

In step 2506, the processor 100 writes a convolution program to the neural network unit 121 program memory 129 using a MTNN instruction 1400 that specifies a function 1432 to write to the program memory 129. The processor 100 then uses a MTNN instruction 1400 that specifies a function 1432 to begin execution of the program to initiate the neural network unit convolution program. An example of a neural network unit convolution program is described in more detail at correspondence to Figure 26A. The flow then proceeds to step 2508.

At decision step 2508, the architecture program determines if the value of the variable N is less than NUM_CHUNKS. If so, the flow will proceed to step 2512; otherwise, proceed to step 2514.

At step 2512, as shown in FIG. 24, processor 100 writes data matrix 2406 of data block N+1 into weighted random access memory 124 (e.g., data matrix 2406B of data block 2). Therefore, when the neural network unit 121 is performing a convolution operation on the current data block, The data matrix 2406 of the next data block can be written into the weight random access memory 124. Thus, after the convolution operation of the current data block is completed, the weight random access memory 124 is written, and the neural network is Unit 121 can immediately begin performing a convolution operation on the next data block.

At step 2514, the processor 100 confirms whether the neural network unit program being executed (for data block 1 but from step 2506, and for block 2-20, execution from step 2518) has completed execution. In a preferred embodiment, processor 100 reads neural network unit 121 status register 127 by executing an MFNN instruction 1500 to confirm whether execution has been completed. In another embodiment, the neural network unit 121 generates an interrupt indicating that the convolution procedure has been completed. The flow then proceeds to decision step 2516.

In decision step 2516, the architecture program determines if the value of the variable N is less than NUM_CHUNKS. If so, the flow proceeds to step 2518; otherwise, the process proceeds to step 2522.

In step 2518, processor 100 updates the convolution program to execute on data block N+1. More precisely, the processor 100 updates the column value of the initialized neural processing unit instruction corresponding to address 0 in the weighted random access memory 124 to the first column of the data matrix 2406 (eg, updated to the data matrix 2406A). Column 0 is either column 500 of the material matrix 2406B and the output column is updated (eg, updated to column 900 or 1300). The processor 100 then begins executing the updated neural network unit convolution program. The flow then proceeds to step 2522.

In step 2522, the processor 100 reads the neural network unit convolution program of the data block N from the weight random access memory 124. Results of the. The flow then proceeds to decision step 2524.

In decision step 2524, the architecture program determines if the value of the variable N is less than NUM_CHUNKS. If so, the flow proceeds to step 2526; otherwise, it terminates.

In step 2526, the architecture program increments the value of N by one. The flow then returns to decision step 2508.

The twenty-sixth A diagram is a list of programs of a neural network unit program that performs a convolution operation on a data matrix 2406 and writes it back using the convolution kernel 2402 of the twenty-fourth graph. Weight random access memory 124. This program loops the instruction loop formed by the instructions of addresses 1 through 9 a certain number of times. The initialization neural processing unit instruction at address 0 specifies the number of times each neural processing unit 126 performs this instruction loop. The example of the twenty-sixth A diagram has a loop count value of 400, corresponding to the twenty-fourth. The number of columns in the data matrix 2406 of the graph, and the loop command at the loop terminal (at address 10) decrements the current loop count value, and if the result is non-zero, it returns to the command loop. The top (ie, the instruction to return to address 1). Initializing the neural processing unit instructions also clears the accumulator 202 to zero. In a preferred embodiment, the loop command at address 10 also clears accumulator 202 to zero. Alternatively, the multiply accumulate instruction at address 1 as described above can also clear accumulator 202 to zero.

For each execution of the in-program instruction loop, the 512 neural processing units 126 simultaneously perform a convolution operation of 512 3x3 convolution kernels and 512 corresponding 3x3 sub-matrices of the data matrix 2406. The convolution operation is performed by the elements of the convolution kernel 2042 and the corresponding sub-matrices. The sum of the nine products calculated by the corresponding elements. In the embodiment of FIG. 26A, the origin (central element) of each of the 512 corresponding 3x3 sub-matrices is the data text Dx+1, y+1 in the twenty-fourth figure, wherein y (row number) is the number of the neural processing unit 126, and x (column number) is the column number read by the multiply-accumulate instruction of the address 1 in the program of the program of the twenty-sixth A picture in the current weight random access memory 124. (This column number is also initialized by the initialization neuroprocessing unit instruction of address 0, and is also incremented when the multiply-accumulate instruction at addresses 3 and 5 is executed, and is also updated by the decrement instruction at address 9.) Thus, in each cycle of the program, the 512 neural processing units 126 compute 512 convolution operations and write the results of the 512 convolution operations back to the instruction sequence of the weighted random access memory 124. Edge handling is omitted herein to simplify the description, but it should be noted that utilizing the collective rotation features of these neural processing units 126 results in a data matrix 2406 (for the image processor, the data matrix of the image) Two rows of multi-line data are wrapped from the vertical edge of one side to the other (eg, from the left edge to the right edge, and vice versa). Now explain the instruction loop.

Address 1 is a multiply-accumulate instruction that specifies column 0 of data random access memory 122 and implicitly utilizes the current weight of random access memory 124, which column is preferably loaded in sequencer 128 (and It is initialized to zero by the instruction at address 0 to perform the operation of the first instruction loop transfer). That is, the instruction at address 1 causes each neural processing unit 126 to read its corresponding text from column 0 of the data random memory 122, and read it from the current weight random access memory 124 column. Corresponding to the text, and performing a multiply-accumulate operation on the two characters. Thus, for example, the neural processing unit 5 multiplies C0,0 by Dx,5 (where "x" is the current weight random access memory 124 column), adds the result to the accumulator 202 value 217, and totals The accumulator 202 is written back.

Address 2 is a multiply-accumulate instruction that specifies that the column of data random access memory 122 is incremented (i.e., increased to one) and then read from the incremented address of data random access memory 122. This instruction will also specify that the value in the multiplex register 705 of each neural processing unit 126 is rotated to the adjacent neural processing unit 126, in this example from the weighted random access memory in response to the instruction of address 1. The body 124 reads the data matrix 2406 values. In the embodiment of the twenty-fourth to twenty-sixth embodiments, the neural processing unit 126 is configured to rotate the value of the multiplex register 705 to the left, that is, from the neural processing unit J to the neural processing unit J- 1, instead of rotating from the neural processing unit J to the neural processing unit J+1 as in the aforementioned third, seventh and nineteenth views. It should be noted that in the embodiment in which the neural processing unit 126 rotates to the right, the architecture program writes the convolution kernel 2042 coefficient values into the data random access memory 122 in different orders (eg, rotating around its center row) to achieve The purpose of similar convolution results. In addition, the architecture program can perform additional convolution kernel preprocessing (such as transposition) when needed. In addition, the instruction specifies a count value of 2. Thus, the instruction at address 2 causes each neural processing unit 126 to read its corresponding text from column 1 of data random access memory 122, and receive the rotated text into multiplex register 705, and The text performs a multiply accumulate operation. Since the count value is 2, this command also causes each neural processing unit 126 to repeat the foregoing operations. In other words, the sequencer 128 will fund The random access memory 122 column address 123 is incremented (i.e., increased to 2), and each neural processing unit 126 reads its corresponding text from the column 2 of the data random access memory 122 and receives the rotated text. Up to the temporary register 705, and a multiply-accumulate operation is performed on the two characters. Thus, for example, assuming that the current weight random access memory 124 is listed as 27, after executing the instruction of address 2, the neural processing unit 5 will multiply the product of C0,1 and D27,6 with C0,2 and D27. The multiplication of 7 is added to its accumulator 202. Thus, after the instruction of address 1 and address 2 is completed, the product of C0, 0 and D27, 5, the product of C0, 1 and D27, 6 and C0, 2 and D27, 7 are accumulated to accumulator 202, and added. All other accumulated values from the previously passed instruction loop.

The operations performed by the instructions of addresses 3 and 4 are similar to the instructions of addresses 1 and 2, using the weighted random access memory 124 column to increment the performance of the indicator, these instructions will be below the weight random access memory 124 The operations are performed and these instructions operate on the next three columns of data random access memory 122, columns 3 through 5. That is to say, taking the neural processing unit 5 as an example, after completing the instructions of the address 1 to 4, the product of C0, 0 and D27, 5, the product of C0, 1 and D27, 6, C0, 2 and D27, 7 The product of product, C1,0 and D28,5, the product of C1,1 and D28,6, and the product of C1,2 and D28,7 are accumulated to accumulator 202, adding all other orders from the previously passed instruction loop. Accumulated value.

The operations performed by the instructions of addresses 5 and 6 are similar to the instructions of addresses 3 and 4, which will be a column below the weighted random access memory 124 and the next three columns of the data random access memory 122, That is, columns 6 to 8 perform calculations. That is to say, taking the neural processing unit 5 as an example, after completing the instructions of addresses 1 to 6, the product of C0, 0 and D27, 5, C0, 1 and The product of D27,6, the product of C0,2 and D27,7, the product of C1,0 and D28,5, the product of C1,1 and D28,6, C1,2 and D28,7,C2,0 and D29, The product of 5, the product of C2, 1 and D29, 6 and the product of C2, 2 and D29, 7 are accumulated to accumulator 202, adding all other accumulated values from the previously passed instruction loop. That is to say, after the instruction of address 1 to 6 is completed, it is assumed that the weight random access memory 124 is listed as 27 when the instruction loop starts, and the neuroprocessing unit 5 is taken as an example, and the convolution kernel 2042 is used for the following 3x3. Submatrix for convolution: D27, 5 D27, 6 D27, 7

D28, 5 D28, 6 D28, 7

D29,5 D29,6 D29,7 In general, after completing the instructions of addresses 1 through 6, the 512 neural processing units 126 have used the convolution kernel 2042 to convolute the following 3x3 sub-matrices: Dr, n Dr, n+1 Dr, n+2

Dr+1, n Dr+1, n+1 Dr+1, n+2

Dr+2, n Dr+2, n+1 Dr+2, n+2 where r is the column address value of the random access memory 124 at the beginning of the command loop, and n is the number of the neural processing unit 126 .

The instruction of address 7 transmits the accumulator 202 value 217 through the start function unit 121. This transfer function passes a text whose size (in bits) is equivalent to the text read by the data random access memory 122 and the weighted random access memory 124 (in this example, 16 bits). . In a preferred embodiment, the user can specify an output format, such as how many bits in the output bit are fractional bits. Yuan, this part will be explained in more detail in the following chapters. In addition, this designation can specify a divide start function instead of specifying a pass start function. This divide start function divides the accumulator 202 value 217 by a divisor, as this article corresponds to the twenty-ninth and thirty-th For example, one of the "dividers" 3014/3016 of the thirtieth figure is utilized. For example, in the case of a convolution kernel 2042 with coefficients, as described above with a Gaussian blur kernel with a factor of one-sixteenth, the instruction of address 7 specifies a division start function (eg, divided by 16), and Do not specify a transfer function. In addition, the architecture program may perform the division by 16 on the coefficients of the convolution kernel 2042 before writing the convolution kernel coefficients into the data random access memory 122, and adjust the binary decimal point of the convolution kernel 2042 value accordingly. The position, for example, uses the data binary point 2922 of the twenty-ninth figure as described below.

The instruction of address 8 writes the output of the start function unit 212 to the column specified in the weight random access memory 124 by the current value of the output column register. This current value is initialized by the instruction of address 0, and this value is incremented by the increment indicator within the instruction after each pass.

As described in the twenty-fourth to twenty-sixth diagrams having an example of a 3x3 convolution kernel 2402, the neural processing unit 126 reads the weighted random access memory 124 approximately every three time-frequency cycles to read the data matrix 2406. One column, and the convolution kernel result matrix is written to the weighted random access memory 124 approximately every twelve time-frequency cycles. Furthermore, it is assumed that in one embodiment, having one of the buffers 1704 of the seventeenth embodiment, the write and read buffers, while the neural processing unit 126 is reading and writing, the processor 100 can have random weights. Access memory 124 for reading and writing, slowing down The buffer 1704 performs a read and write operation on the weight random access memory approximately every sixteen time-frequency cycles to read the data matrix and write the convolution kernel result matrix, respectively. Thus, approximately half of the bandwidth of the weighted random access memory 124 is consumed by the convolution kernel operation performed by the neural network unit 121 in a mixed manner. This example includes a 3x3 convolution kernel 2042. However, the present invention is not limited thereto, and other sizes of convolution kernels, such as 2x2, 4x4, 5x5, 6x6, 7x7, 8x8, etc., may also be applicable to different neural networks. Unit program. In the case of larger convolution kernels, because of the rotated version of the multiply-accumulate instruction (such as the instructions of addresses 2, 4, and 6 in Figure 26A, larger convolution kernels will need to use these instructions) With a larger count value, the time ratio of the neural processing unit 126 reading the weight random access memory 124 is reduced, and therefore, the bandwidth usage ratio of the weight random access memory 124 is also lowered.

In addition, the architectural program can cause the neural network unit program to overwrite the columns that are no longer needed in the input data matrix 2406, rather than writing the convolution operation results back to different columns of the weighted random access memory 124 (eg, column 900). -1299 and 1300-1699). For example, in the case of a 3x3 convolution kernel, the architecture program can write the data matrix 2406 into columns 2-401 of the weighted random access memory 124 instead of writing columns 0-399, and the neural processing unit program The convolution operation result is written from the column 0 of the weight random access memory 124, and the number of columns is incremented every time the instruction loop is passed. In this way, the neural network unit program will only overwrite the columns that are no longer needed. For example, after the first pass through the instruction loop (or more precisely, after loading the instruction of address 1 it loads the column 0 of the weight random access memory 124), the data of column 0 can be overwritten. Write, however, the data of columns 1-3 needs to be left for the second pass through the operation of the instruction loop and cannot be overwritten; similarly, after the second pass through the instruction loop, the data of column 1 can be overwritten. However, the data in columns 2-4 needs to be left for the third pass through the instruction loop and cannot be overwritten; and so on. In this embodiment, the height of each data matrix 2406 (data block) can be increased (e.g., 800 columns) so that fewer data blocks can be used.

In addition, the architecture program can cause the neural network unit program to write the result of the convolution operation back to the data random access memory 122 column above the convolution kernel 2402 (eg, above column 8) instead of writing the convolution operation result back. The weight random access memory 124, when the neural network unit 121 writes the result, the architecture program can read the result from the data random access memory 122 (for example, using the data random access memory 122 in the twenty-sixth figure) Recently written to column 2606 address). This configuration is applicable to embodiments having a weighted random access memory 124 and a dual data random access memory.

According to the operation of the neural network unit 121 in the embodiment of the twenty-fourth to twenty-sixth embodiment, it can be found that each execution of the program of the twenty-sixth A diagram requires about 5000 time-frequency periods, and thus, the second The convolution operation of the entire 2560x1600 data array 2404 in Figure 14 requires approximately 100,000 time-frequency cycles, significantly less than the number of time-frequency cycles required to perform the same task in the conventional manner.

The twenty-sixth B diagram is a block diagram showing one embodiment of certain fields of the control register 127 of the neural network unit 121 of the first figure. The status register 127 includes a field 2602 indicating the last of the weighted random access memory 124 that was written by the neural processing unit 126. Address; a field 2606 indicating the address of the data random access memory 122 that was recently written by the neural processing unit 126; a field 2604 indicating the most recently processed neural processing unit in the weighted random access memory 124 126 reads the address of the column; and a field 2608 indicating the address of the column in the data random access memory 122 that was most recently read by the neural processing unit 126. In this way, the processing program executed by the processor 100 can confirm the processing progress of the neural network unit 121, and read and/or write data to the data random access memory 122 and/or the weight random access memory 124. Time to enter. Using this capability, plus the overwrite of the input data matrix as previously described (or writing the results to the data random access memory 122 as previously described), as described in the following example, the data array 2404 of the twenty-fourth graph It can be viewed as five 512x1600 data blocks instead of 20 512x400 data blocks. The processor 100 writes the first 512x1600 data block from the column 2 of the weight random access memory 124 and causes the neural network unit program to start (the program has a loop count of 1600 values, and the weights are randomized). The output column of the access memory 124 is initialized to 0). When the neural network unit 121 executes the neural network unit program, the processor 100 monitors the output position/address of the weight random access memory 124, thereby (1) (using the MFNN instruction 1500) reading the weight random access memory. Body 124 has a list of valid convolution operations written by neural network unit 121 (starting with column 0); and (2) overwrites the second 512x1600 data matrix 2406 (starting with column 2) The result of the effective convolution operation is read, so that when the neural network unit 121 completes the neural network unit program for the first 512x1600 data block, the processor 100 can immediately update the neural network unit program and start the neural network again if necessary. Road unit program to execute In the second 512x1600 data block. This program will repeat the execution of the remaining three 512x1600 data blocks three times so that the neural network unit 121 can be fully utilized.

In an embodiment, the start function unit 212 has the ability to effectively perform an effective division operation on the accumulator 202 value 217, which in the subsequent sections corresponds in particular to the twenty-ninth, twenty-nine, and twenty-sixth and thirty-th. There will be more detailed instructions. For example, a start function neural network unit instruction that divides the value of accumulator 202 by a divide by 16 can be used for the Gaussian blur matrix described below.

The convolution kernel 2402 used in the example of the twenty-fourth figure is a small static convolution kernel applied to the entire data matrix 2404. However, the present invention is not limited thereto, and the convolution kernel may also be a large matrix. The particular weight corresponds to a different data value of the data array 2404, such as a convolution kernel commonly found in convolutional neural networks. When the neural network unit 121 is used in this manner, the architecture program interchanges the data matrix with the location of the convolution kernel, that is, the data matrix is placed in the data random access memory 122 and the convolution kernel is placed in the weight random. Within memory 124, the number of columns required to execute a neural network unit program is relatively small.

Figure 27 is a block diagram showing an example of a weighted random access memory 124 filled with input data in the first figure. The input data is subjected to a common source operation by the neural network unit 121 of the first figure ( Pooling operation). The common source operation is performed by one common source layer of the artificial neural network, and the sub-region or sub-matrix of the input matrix is obtained and the maximum or average value of the sub-matrix is calculated as a result matrix, ie, the common source moment. Array to reduce the size of the input data matrix (such as an image or a convolved image). In the examples of the twenty-seventh and twenty-eighth diagrams, the common source operation calculates the maximum value of each submatrix. Common source operations are particularly useful for artificial neural networks that perform object classification or detection. In general, the common source operation can actually reduce the input matrix by the number of elements of the detected sub-matrix, and in particular, the direction of each dimension of the input matrix can be reduced by the number of elements in the corresponding dimension direction of the sub-matrix. In the example of the twenty-seventh figure, the input data is a 512 x 1600 matrix of wide text (e.g., 16 bits) stored in columns 0 through 1599 of the weighted random access memory 124. In the twenty-seventh figure, these characters are indicated by their row position. For example, the text in column 0 row 0 is marked as D0, 0; the text in column 0 row 1 is marked as D0, 1; The text of column 0 and line 2 is labeled as D0, 2; and so on, the text in column 0 line 511 is labeled D0, 511. Similarly, the text in column 1 row 0 is labeled D1, 0; the text in column 1 row 1 is labeled D1, 1; the text in column 1 row 2 is labeled D1, 2; and so on, in the column The text of line 1 511 is marked as D1, 511; and so on, the text in column 1599 line 0 is marked as D1599, 0; the text in column 1599 line 1 is marked as D1599, 1 is in column 1599 line 2 The text is labeled D1599, 2; and so on, the text in column 1599, line 511 is labeled D1599, 511.

The twenty-eighth figure is a list of programs of a neural network unit program that performs the common source operation of the input data matrix of the twenty-seventh figure and writes it back to the weight random access memory 124. . In the example of the twenty-eighth figure, the common source operation calculates the maximum of each 4x4 submatrix in the input data matrix. This program will execute the reference multiple times. Let the instruction consisting of 1 to 10 loop. The initialization neural processing unit instruction at address 0 specifies the number of times each neural processing unit 126 executes the instruction loop. In the example of the twenty-eighth diagram, the loop count value is 400, and at the end of the loop (in position) The loop command at address 11) decrements the current loop count value, and if the resulting result is a non-zero value, it returns to the top of the command loop (ie, returns to the address 1 command). The input data matrix in the weighted random access memory 124 is essentially treated by the neural network unit program as 400 mutually exclusive groups of four adjacent columns, namely columns 0-3, columns 4-7, columns. 8-11, and so on, until column 1596-1599. Each group consisting of four adjacent columns includes 128 4x4 sub-matrices, which are 4x4 sub-matrices formed by the elements at the intersection of four columns and four adjacent rows of the group. The adjacent row is line 0-3, line 4-7, line 8-11, and so on until line 508-511. Each of the 512 neural processing units 126, a fourth set of four neural processing units 126 (a total of 128) performs a common source operation on a corresponding 4x4 submatrix, while the other three neural processing units 126 is not used. More precisely, the neural processing units 0, 4, 8, and so on until the neural processing unit 508 performs a common source operation on the corresponding 4x4 submatrix, and the leftmost row number of the 4x4 submatrix corresponds to The neural processing unit number, and the lower column corresponds to the column value of the current weight random access memory 124, this value will be initialized to zero by the initialization instruction of address 0 and will increase by 4 after repeating each instruction loop. Some will be explained in more detail in subsequent chapters. The repetition of the 400 instruction loops corresponds to the number of 4x4 submatrix groups in the input data matrix of the twenty-seventh graph (ie, the input data matrix has 1600 columns divided by 4). Initializing the neural processing unit instructions will also clear The accumulator 202 is reset to zero. In a preferred embodiment, the loop instruction of address 11 also clears accumulator 202 to zero. In addition, the maxwacc instruction of address 1 specifies that the accumulator 202 is cleared to zero.

Each time the instruction loop of the program is executed, the 128 used neural processing units 126 perform 128 common source operations simultaneously on 128 individual 4x4 sub-matrices in the current four-column group of the input data matrix. Further, this common source operation will identify the largest of the 16 elements of the 4x4 submatrix. In the embodiment of the twenty-eighth figure, for each of the 128 used neural processing units 126, the lower left element of the 4x4 submatrix is the element Dx in the twenty-seventh graph. , y, where x is the number of columns of the current weight random access memory 124 at the beginning of the instruction loop, and the column data is read by the maxwacc instruction of the address 1 in the program of the twenty-eighth figure (this column number is also It will be initialized by the initialization neural processing unit instruction of address 0 and incremented each time the maxwacc instruction of addresses 3, 5 and 7 is executed). Thus, for each loop of the program, the 128 used neural processing units 126 will write back the maximum elements of the corresponding 128 4x4 sub-matrices of the current column group back to the weighted random access memory 124. The specified column. The following is a description of this instruction loop.

The maxwacc instruction of address 1 implicitly uses the current weight random access memory 124 column, which is preferably loaded in sequencer 128 (and initialized to zero by the instruction at address 0 to execute the first time) Pass the operation of the instruction loop.) The instruction of address 1 causes each neural processing unit 126 to read its corresponding text from the current column of the weighted random access memory 124, comparing this text to the value 217 of the accumulator 202. The largest of these two values is stored in accumulator 202. Thus, for example, the neural processing unit 8 will confirm the accumulator 202 value 217 and the data word Dx, 8 (where "x" is the current weight random access memory 124 column) and write it back to the accumulator 202.

Address 2 is a maxwacc instruction that specifies that the value in the multiplex register 705 of each neural processing unit 126 is rotated adjacent to the neural processing unit 126, where the instruction for address 1 is just The weighted random access memory 124 reads one of the column input data array values. In the embodiment of the twenty-seventh to twenty-eighth embodiment, the neural processing unit 126 is configured to rotate the multiplexer 705 value to the left, that is, from the neural processing unit J to the neural processing unit J-1, as in the foregoing Corresponds to the chapters of the twenty-fourth to twenty-sixth figures. In addition, this instruction will specify a count value of 3. Thus, the instruction of address 2 causes each neural processing unit 126 to receive the rotated text into the multiplex register 705 and confirm the maximum value of the rotated text and the accumulator 202, and then repeat the operation twice. . That is, each neural processing unit 126 performs three operations of receiving the rotated text to the multiplex register 705 and confirming the maximum value of the rotated text and accumulator 202 values. Thus, for example, assuming that the instruction loop is started, the current weight random access memory 124 is listed as 36. Taking the neural processing unit 8 as an example, after executing the instructions of addresses 1 and 2, the neural processing unit 8 will The maximum value of the accumulator 202 and the four weight random access memory 124 characters D36, 8, D36, 9, D36, 10 and D36, 11 will be stored in its accumulator 202.

The operations performed by the maxwacc instructions of addresses 3 and 4 are similar to the instructions of address 1, using weighted random access memory 124. The addition of the indicator has the effect that the instructions of addresses 3 and 4 are executed in a column below the weighted random access memory 124. That is to say, it is assumed that the current weight random access memory 124 column is 36 at the beginning of the instruction loop, and the neural processing unit 8 is taken as an example. After the instructions of addresses 1 to 4 are completed, the neural processing unit 8 will accumulate in it. The accumulator 202 and the eight weight random access memory 124 characters D36, 8, D36, 9, D36, 10, D36, 11, D37, 8, D37, 9, D37, 10 are stored in the buffer 202. The maximum value of D37,11.

The operations performed by the maxwacc instructions of addresses 5 through 8 are similar to the instructions of addresses 1 through 4, and the instructions of addresses 5 through 8 are executed for the two columns below the weighted random access memory 124. That is to say, it is assumed that the current weight random access memory 124 column is 36 at the beginning of the instruction loop, and the neural processing unit 8 is taken as an example. After the instructions of addresses 1 to 8 are completed, the neural processing unit 8 will accumulate in it. The accumulator 202 and the sixteen weight random access memory 124 characters D36, 8, D36, 9, D36, 10, D36, 11, D37, 8, D37, 9, D37, 10 are stored in the buffer 202. The maximum of D37, 11, D38, 8, D38, 9, D38, 10, D38, 11, D39, 8, D39, 9, D39, 10 and D39, 11. That is to say, assuming that the current weight random access memory 124 column is 36 at the beginning of the instruction loop, the neural processing unit 8 is taken as an example. After completing the instructions of addresses 1 to 8, the neural processing unit 8 will complete the confirmation of the following. Maximum value of 4x4 submatrix: D36, 8 D36, 9 D36, 10 D36, 11

D37, 8 D37, 9 D37, 10 D37, 11

D38, 8 D38, 9 D38, 10 D38, 11

D39, 8 D39, 9 D39, 10 D39, 11 Basically, after completing the instructions of addresses 1 to 8, these 128 gods are used. Each of the neural processing units 126 in the processing unit 126 will complete the confirmation of the maximum values of the following 4x4 sub-matrices: Dr, n Dr, n+1 Dr, n+2 Dr, n+3

Dr+1, n Dr+1, n+1 Dr+1, n+2 Dr+1, n+3

Dr+2,n Dr+2,n+1 Dr+2,n+2 Dr+2,n+3

Dr+3, n Dr+3, n+1 Dr+3, n+2 Dr+3, n+3 where r is the current address weight of the random access memory 124 at the beginning of the command loop, n is The nerve processing unit 126 is numbered.

The instruction of address 9 transmits the accumulator 202 value 217 through the start function unit 212. This transfer function passes a text whose size (in bits) is equivalent to the text read by the weighted random access memory 124 (in this example, 16 bits). In a preferred embodiment, the user can specify an output format, such as how many bits in the output bit are fractional bits, as will be explained in more detail in subsequent sections.

The instruction of address 10 writes the accumulator 202 value 217 into the column of the weighted random access memory 124 specified by the current value of the output column register, which is initialized by the instruction of address 0, and Use the increment indicator in the instruction to increment this value after each pass through the loop. Further, the instruction of address 10 writes a wide text (e.g., 16 bits) of accumulator 202 into weighted random access memory 124. In a preferred embodiment, the instruction will write the 16 bits in accordance with the output binary point 2916, which will be in the following corresponding to the twenty-ninth and twenty-ninth B-pictures. More detailed instructions.

As described above, iterating once the instruction loop write weight is random The column of access memory 124 will contain a hole with an invalid value. That is, the wide text 1 to 3, 5 to 7, 9 to 11, and so on until the wide text 509 to 511 are invalid or unused. In one embodiment, the start function unit 212 includes a multiplexer that enables the result of merging the adjacent words into the column buffer, such as the column buffer 1104 of the eleventh figure, to write back the output weight random access memory. 124 columns. In a preferred embodiment, the start function instruction specifies the number of words in each hole, and the number of words in the hole controls the multiplexer result. In an embodiment, the number of holes may be specified as a value of 2 to 6 to combine the outputs of the 3x3, 4x4, 5x5, 6x6, or 7x7 sub-matrices of the common source. In addition, the architecture program executing on the processor 100 reads the sparse (ie, having holes) result columns generated from the weighted random access memory 124 and utilizes other execution units 112, such as media units that use the schema merge instructions, such as The x86 single instruction multiple data flow extension (SSE) instruction performs the merge function. In a manner similar to that described above and utilizing the hybrid nature of neural network unit 121, the architectural program executing on processor 100 can read state register 127 to monitor the most recently written column of weighted random access memory 124 ( For example, the field 2602 of FIG. 26B is used to read one of the generated sparse result columns, merge them and write them back to the same column of the weighted random access memory 124, so that preparation is completed and can be used as an input data. The matrix is provided for use under the neural network, such as a roll of layers or a traditional neural network layer (ie, a multiply-accumulate layer). In addition, the embodiments described herein perform a common source operation in a 4x4 submatrix, but the present invention is not limited thereto, and the neural network unit program of the twenty-eighth figure can be adjusted to be a sub-matrix of other sizes, such as 3x3. , 5x5, 6x6 or 7x7, perform a common source operation.

As can be seen from the foregoing, the number of result columns written to the weight random access memory 124 is one quarter of the number of columns of the input data matrix. Finally, data random access memory 122 is not used in this example. However, the data random access memory 122, rather than the weighted random access memory 124, can also be used to perform the common source operation.

In the twenty-seventh and twenty-eighth embodiments, the common source operation calculates the maximum value of the sub-region. However, the program of Figure 28 can be adjusted to calculate the average of the sub-regions by replacing the maxwacc instruction with the sumwacc command (adding the weight text to the accumulator 202 value 217) and starting the address 9. The function instruction is modified to divide the accumulated result by the number of elements in each sub-region (preferably by reciprocal multiplication as described below), which is sixteen in this example.

According to the operations of the neural network unit 121 according to the twenty-seventh and twenty-eighth diagrams, each time the program of the twenty-eighth figure is executed, it takes about 6000 time-frequency periods to be used for the twenty-seventh figure. The entire 512x1600 data matrix performs a common source operation, and the number of time-frequency cycles used in this operation is significantly less than the number of time-frequency cycles required to perform similar tasks in the conventional manner.

In addition, the architecture program can cause the neural network unit program to write the result of the common source operation back to the data random access memory 122 column instead of writing the result back to the weight random access memory 124, when the neural network unit 121 will result When the data random access memory 122 is written (for example, using the data of the twenty-sixth B-picture data random access memory 122 recently written to the address of the column 2606), the architecture program reads from the data random access memory 122. result. This configuration is applicable to 單埠 weighted random access memory 124 and dual resources An embodiment of random access memory 122.

Fixed-point arithmetic operation with user-supplied decimal point, full-precision fixed-point accumulation, user-specified reciprocal value, random rounding of accumulator values, and selectable start/output functions

Generally, a hardware unit that performs an arithmetic operation in a digital computing device is an integer or a floating point number according to an object on which an arithmetic operation is performed, and is generally classified into an "integer" unit and a "floating point" unit. A floating point number has a magnitude (or mantissa) and an exponent, usually with a sign. An index is an indicator of the position of a radix point (usually a binary point) relative to a value. In contrast, an integer does not have an exponent, but only has a value, usually with a sign. Floating point units allow programmers to get the numbers they need to work from a very wide range of different values, while hardware is responsible for adjusting the index value of this number when needed, without the programmer having to deal with it. For example, suppose two floating point numbers 0.111 x 10 ^{29 are} multiplied by 0.81 x 10 ³¹ . (Although floating-point units typically work on 2-based floating-point numbers, decimal digits, or 10-based floating-point numbers, are used in this example.) Floating-point units are automatically responsible for multiplying the mantissa, exponential phase Add, then normalize the result to the value .8911 x 10 ⁵⁹ . In another example, assume that the same two floating point numbers are added. The floating point unit is automatically responsible for aligning the decimal point of the mantissa before the addition to produce a total number of .81111 x 10 ³¹ .

However, it is well known that such complicated operations result in an increase in the size of the floating point unit, an increase in energy consumption, an increase in the number of time-frequency cycles required per instruction, and/or an increase in cycle time. For this reason, many Devices such as embedded processors, microcontrollers, and relatively low cost and/or low power microprocessors do not have floating point units. It can be found from the foregoing examples that the complex structure of a floating-point unit includes logic for performing exponential calculations associated with floating-point addition and multiplication/division (ie, performing an addition/subtraction operation on an index of an operand to generate an exponential value of floating-point multiplication/division) The adder, which subtracts the operand index to confirm the subtraction of the binary point offset offset of the floating point addition), and includes an offset for the binary point alignment of the mantissa in the floating point addition, Contains an offset that normalizes floating point results. In addition, the process usually needs to perform the logic of rounding the floating point result, perform conversion between integer format and floating point format, and between different floating point formats (such as amplification precision, double precision, single precision, half precision). Logic, leading zero and preamble detectors, and logic for handling special floating point numbers, such as anomalous, non-numeric, and infinite.

In addition, the verification of the correctness of the floating-point unit will greatly increase the complexity of the design due to the increased numerical space required to be verified, which will extend the product development cycle and time to market. In addition, as mentioned above, floating-point arithmetic operations need to store and use the mantissa field and the exponent field for each floating-point number used for calculation, respectively, to increase the required storage space and/or in a given storage space. Reduce accuracy when storing integers. Many of these shortcomings can be avoided by performing arithmetic operations on integer units.

Programmers usually need to write a program that processes decimals, which is a non-complete number. Such a program may need to be executed on a processor that does not have a floating point unit, or the processor may have a floating point unit, but the integer instructions of the processor may be compared. fast. To take advantage of the performance of integer processors, programmers use well-known fixed-point arithmetic operations on fixed-point numbers. Such programs will include instructions that execute on integer units to process integer or integer data. The software knows that the data is a decimal, and the software contains instructions to perform operations on the integer data to handle the fact that the data is actually a decimal, such as an alignment offset. Basically, the pointing software can manually perform functions that some or all of the floating point units can perform.

In this paper, a "fixed point" number (or value or operand or input or output) is a number whose stored bit is understood to contain a bit to represent a fractional part of the fixed point number, which is referred to herein. It is a "decimal bit". A fixed-point storage location is included in a memory or scratchpad, such as an 8-bit or 16-bit literal in a memory or scratchpad. In addition, the storage bits of the fixed-point number are all used to express a value, and in some cases, one of the bits will be used to express the symbol, but a fixed-bit storage element will be used to express the number. index. In addition, the number of decimal places or the number of decimal places of the fixed point number is specified in a storage space different from the fixed point storage bit, and the number of decimal places or the number of the decimal places is indicated in a shared or common manner. The decimal point position is shared with a set of fixed-point numbers containing the fixed-point number, such as input operands, accumulated values, or a set of output results of the processing unit array.

In the embodiment described herein, the arithmetic logic unit is an integer unit, however, the start function unit is either a floating point arithmetic hardware assist or acceleration. This can make the arithmetic logic unit part smaller and faster to facilitate the use of more arithmetic logic on a given wafer space. Edit unit. This also means that more neurons can be placed on the unit wafer space, which is particularly advantageous for neural network units.

In addition, exponential storage bits are required compared to each floating point number. The fixed point number in the embodiment described herein expresses the number of storage bits belonging to the decimal place in all the digital sets by one index, however, This indicator is located in a single, shared storage space and broadly identifies all numbers of the entire collection, such as the input set of a series of operations, the set of accumulated numbers of a series of operations, the set of outputs, and the number of decimal places. In a preferred embodiment, a user of the neural network unit can specify the number of decimal storage bits for the set of numbers. Therefore, it can be understood that although in many cases (such as general mathematics), the term "integer" refers to a signed complete number, that is, a number that does not have a fractional part, but in the context of this article, " The term "integer" can mean a number with a fractional part. In addition, in the context of this article, the term "integer" is used to distinguish from floating-point numbers. For floating-point numbers, some of the bits in their respective storage spaces are used to express the exponent of floating-point numbers. Similarly, integer arithmetic operations, such as integer multiplication or addition or comparison operations performed by integer units, assume that there are no indices in the operands, therefore, integer components of integer units, such as integer multipliers, integer adders, integer comparators, There is no need to include logic to process the index. For example, there is no need to move the mantissa to align the binary point for addition or comparison operations, and there is no need to add the indices for multiplication.

In addition, the embodiments described herein include a large hardware integer accumulator to accumulate a large series of integer operations (eg, 1000 multiply-accumulate operations) without loss of precision. So Avoiding neural network units processing floating point numbers while maintaining the full precision of the accumulated number without saturating or producing inaccurate results due to overflow. Once the series of integer operations adds a result to the full-precision accumulator, the fixed-point hardware assist performs the necessary scaling and saturation operations to take advantage of the user-specified cumulative value of the fractional number of bits and the output value required. The number of decimal places, this full precision accumulated value is converted into an output value, which will be explained in more detail in subsequent chapters.

When it is desired to compress the accumulated value from full precision form for use in one of the input functions of the start function or for transfer, in a preferred embodiment, the start function unit can selectively perform a random round operation on the accumulated value. This section will be explained in more detail in subsequent chapters. Finally, depending on the different needs of a given layer of one of the neural networks, the neural processing unit can selectively accept indications to use different start functions and/or output many different forms of accumulated values.

The twenty-ninth A diagram is a block diagram showing an embodiment of the control register 127 of the first figure. The control register 127 can include a plurality of control registers 127. As shown in the figure, the control register 127 includes the following fields: configuration 2902, signed data 2912, signed weight 2914, data binary decimal point 2922, weight binary decimal point 2924, arithmetic logic unit function 2926, Rounding control 2932, start function 2934, reciprocal 2942, offset 2944, output random access memory 2952, output binary point 2954, and output command 2956. Controlling the register 127 value can utilize the MTNN instruction 1400 and the NNU program instructions, such as the start command, to perform the write operation.

Configuring the 2902 value specifies that the neural network unit 121 is a genus For narrow, wide, or funnel configurations, as described earlier. Configuration 2902 also sets the size of the input text received by data random access memory 122 and weight random access memory 124. In a narrow configuration and funnel configuration, the size of the input text is narrow (for example, 8-bit or 9-bit), but in a wide configuration, the size of the input text is wide (for example, 12-bit or 16-bit) ). In addition, the configuration 2902 also sets the size of the output result 133 that is the same size as the input text.

When the signed data value 2912 is true, it means that the data characters received by the data random access memory 122 are signed values, and if they are false, the data characters are unsigned values. When the signed weight value 2914 is true, it means that the weighted text received by the weight random access memory 122 is a signed value, and if it is false, it means that the weighted characters are unsigned values.

The data binary decimal point value of 2922 indicates the binary decimal point position of the data text received by the data random access memory 122. In a preferred embodiment, for the position of the binary point, the data binary point 2922 value represents the number of bit positions calculated from the right side of the binary point. In other words, the data binary decimal point 2922 indicates the number of decimal places in the least significant bit of the data text, that is, the number of bits located to the right of the decimal point. Similarly, the weighted binary point 2924 value represents the binary point position of the weighted text received by the weighted random access memory 124. In a preferred embodiment, when the arithmetic logic unit function 2926 is a multiply and accumulate or output accumulation, the neural processing unit 126 determines the number of bits to the right of the binary digit of the value of the accumulator 202 as the data two. Carry decimal point 2922 and The weight of the binary digits is the sum of the decimal points 2924. Thus, for example, if the data binary point 2922 has a value of 5 and the weighted binary point 2924 has a value of 3, the value in the accumulator 202 has 8 bits to the right of the binary decimal point. When the arithmetic logic unit function 2926 is a total/maximum accumulator and data/weight text or a transfer data/weight text, the neural processing unit 126 will load the number of bits to the right of the decimal point of the value of the accumulator 202. It is determined as the data/weight binary decimal point 2922/2924. In another embodiment, a single accumulator binary decimal point 2923 is specified without specifying an individual data binary decimal point 2922 and a weight binary decimal point 2924. This section will be described in more detail later in the corresponding map of Figure 29.

Arithmetic logic unit function 2926 specifies a function that is executed by arithmetic logic unit 204 of neural processing unit 126. As previously described, the arithmetic logic unit function 2926 can include the following operations, but is not limited to: multiplying the data text 209 by the weight text 203 and adding the product to the accumulator 202; adding the accumulator 202 to the weight text 203; The loader 202 is added to the data text 209; the maximum value of the accumulator 202 and the data text 209; the maximum value of the accumulator 202 and the weight text 209; the output accumulator 202; the transfer data 209; the transfer weight text 209; value. In one embodiment, the arithmetic logic unit function 2926 is specified by a neural network unit initialization instruction and is used by the arithmetic logic unit 204 to respond to an execution instruction (not shown). In one embodiment, the arithmetic logic unit function 2926 is specified by individual neural network unit instructions, such as the aforementioned multiply accumulate and maxwacc instructions.

Rounding control 2932 specifies (in the thirtieth figure) rounder The form of the rounding operation used by 3004. In an embodiment, the rounding modes that may be specified include, but are not limited to, no rounding, rounding to the most recent value, and random rounding. For a preferred embodiment, processor 100 includes a random bit source 3003 (see FIG. 30) to generate random bits 3005 that are sampled to perform random rounding to reduce The possibility of rounding deviations. In one embodiment, when the rounding bit 3005 is one and the sticky bit is zero, if the sampled random bit 3005 is true, the neural processing unit 126 rounds up if the random bit is sampled. If the 3005 is false, the neural processing unit 126 will not round up. In one embodiment, the random bit source 3003 is sampled based on the random electronic characteristics of the processor 100 to generate random bits 3005, such random semiconductor characteristics or thermal noise of the semiconductor diode or resistor, although the present invention Not limited to this.

The start function 2934 specifies a function for the accumulator 202 value 217 to generate the output 133 of the neural processing unit 126. As described herein, the start function 2934 includes, but is not limited to, a sigmoid function; a hyperbolic tangent function; a soft addition function; a correction function; divide by a specified power of two; multiply a user-specified reciprocal value to achieve, etc. Dividing the effect; passing the entire accumulator; and passing the accumulator in standard dimensions, as described in more detail in the following sections. In an embodiment, the startup function is specified by a neural network unit startup function instruction. In addition, the startup function may also be specified by the initialization instruction and used in response to an output instruction, such as the start function unit output instruction of address 4 in the fourth figure. In this embodiment, the address 3 is activated in the fourth figure. Function instructions are included in the output instructions.

The reciprocal 2942 value specifies a value that is multiplied by the accumulator 202 value 217 to achieve a value that divides the accumulator 202 value 217. That is, the reciprocal 2942 value specified by the user will be the reciprocal of the divisor that you actually want to execute. This facilitates collocation with convolution or common source operations as described herein. In a preferred embodiment, the user will specify the reciprocal 2942 value as two portions, which will be described in more detail later in the corresponding twenty-ninth C-picture. In one embodiment, the control register 127 includes a field (not shown) that allows the user to specify a division among a plurality of built-in divisor values that are equivalent to commonly used convolutions. The size of the core, such as 9, 25, 36 or 49. In this embodiment, the start function unit 212 stores the reciprocal of these built-in divisors for multiplying the accumulator 202 value 217.

Offset 2944 is the number of bits that specify one of the start function unit 212 shifters to shift the value of the accumulator 202 217 to the right to divide it by the power of two. This facilitates the operation of a convolution kernel with a power of two.

The output random access memory 2952 value specifies one of the data random access memory 122 and the weight random access memory 124 to receive the output result 133.

The output binary point 2954 value represents the position of the decimal point of the output result 133. For a preferred embodiment, for the position of the binary point of the output 133, the output binary point 2954 value represents the number of bit positions calculated from the right side. In other words, the output binary point 2954 represents the number of decimal places in the least significant bit of the output result 133, that is, to the right of the decimal point. The number of bits. The start function unit 212 will be based on the value of the output binary point 2954 (in most cases, it will also be based on the data binary decimal point 2922, the weight binary decimal point 2924, the start function 2934 and/or the configuration 2902 value). Perform rounding, compression, saturation, and size conversion operations.

Output command 2956 will output 133 from many control outputs. In an embodiment, the startup function unit 121 utilizes the concept of a standard size that is twice the width (in bits) specified by the configuration 2902. Thus, for example, if the configuration 2902 sets the size of the input text received by the data random access memory 122 and the weight random access memory 124 to be 8-bit, the standard size will be 16 bits; in another example If the configuration 2902 sets the size of the input text received by the data random access memory 122 and the weight random access memory 124 to be 16 bits, the standard size will be 32 bits. As described herein, the accumulator 202 is relatively large in size (for example, the narrow accumulator 202B is 28 bits and the wide accumulator 202A is 41 bits) to maintain intermediate calculations, such as 1024 and 512 nerves. Network unit multiply accumulates instructions with full precision. Thus, the accumulator 202 value 217 will be greater than (in terms of the bit) standard size, and for most of the value of the start function 2934 (in addition to passing the entire accumulator), the function unit 212 is activated (eg, the following corresponds to the thirty-th figure) The standard size compressor 3008) described in the paragraph compresses the accumulator 202 value 217 to the size of the standard size. The first predetermined value of the output command 2956 will instruct the startup function unit 212 to execute the specified startup function 2934 to generate an internal result and output the internal result as the output result 133, the size of the internal result being equal to the size of the original input text, ie Standard Half of the quasi-size. The second preset value of the output command 2956 will instruct the start function unit 212 to execute the specified start function 2934 to generate an internal result and output the lower half of the inner result as the output result 133, the internal result being equal in size to the original input text. The size of twice the standard size; and the third preset value of the output command 2956 will instruct the start function unit 212 to output the upper half of the internal result of the standard size as the output result 133. The fourth preset value of the output command 2956 will instruct the start function unit 212 to output the unprocessed least significant text of the accumulator 202 as the output result 133; and the fifth preset value of the output command 2956 will indicate that the start function unit 212 will The unprocessed intermediate valid text of the accumulator 202 is output as an output 133; the sixth preset value of the output command 2956 instructs the startup function unit 212 to treat the unprocessed most significant text of the accumulator 202 (its width is configured by The output specified by 2902) is output as output 133, which is described in more detail in the section corresponding to the eighth to tenth figures. As before, the internal result of outputting the entire accumulator 202 size or standard size helps the other execution units 112 of the processor 100 to execute a boot function, such as a soft maximal start function.

The fields described in the twenty-ninth A diagram (and the twenty-ninth and twenty-ninth C diagrams) are located inside the control register 127, however, the invention is not limited thereto, and one or more of the fields It can also be located in other parts of the neural network unit 121. In a preferred embodiment, a plurality of fields may be included within the neural network unit instructions and decoded by the sequencer 128 to generate a microinstruction 3416 (see Figure 34) to control the arithmetic logic unit. 204 and/or start function unit 212. In addition, these fields can also be included in the micro-operations stored in the media register 118. Within 3414 (please refer to the thirty-fourth diagram) to control the arithmetic logic unit 204 and/or the start function unit 212. This embodiment may reduce the use of the initialization neural network unit instructions, while in other embodiments the initialization neural network unit instructions may be removed.

As described above, the neural network unit instructions can specify a memory operand (such as text from the data random access memory 122 and/or the weight random access memory 123) or a post-rotation operand (eg, from a multiplex The memory 208/705) performs an arithmetic logic instruction operation. In one embodiment, the neural network unit instructions may also designate an operand as a scratchpad output of a boot function (such as the output of register 3038 in FIG. 30). Moreover, as previously described, the neural network unit instructions can be specified to increment the current column address of one of the data random access memory 122 or the weight random access memory 124. In an embodiment, the neural network unit instruction may specify an immediate signed integer difference to join the current column for the purpose of incrementing or decrementing a value other than one.

The twenty-ninth B diagram is a block diagram showing another embodiment of the control register 127 of the first figure. The control register 127 of FIG. 29B is similar to the control register 127 of FIG. 29A. However, the control register 127 of FIG. 29B includes an accumulator binary decimal point 2923. . The accumulator binary decimal point 2923 represents the binary position of the accumulator 202. In a preferred embodiment, the accumulator binary point 2923 value indicates the number of bit positions from the right side of the binary point position. In other words, the accumulator binary decimal point 2923 represents the number of decimal places in the least significant bit of accumulator 202, ie, the bit located to the right of the binary decimal point. In this embodiment, the accumulator is binary The decimal point 2923 is explicitly indicated, rather than the implicit confirmation as in the embodiment of Figure 29A.

The twenty-ninth C-picture shows a block diagram of an embodiment in which the inverse of the twenty-ninth A picture is stored in two parts. The first portion 2962 is an offset value indicating the number 2962 of zeros that were previously suppressed by the user in the real reciprocal value of the value 217 of the accumulator 202. The number of leading zeros is the number of zeros consecutively arranged to the right of the binary decimal point. The second portion 2694 is the leading zero suppression reciprocal value, which is the true reciprocal value after all leading zeros have been removed. In one embodiment, the suppressed leading zeros number 2962 is stored in 4 bits, and the leading zero suppression inverted value 2964 is stored in 8-bit unsigned values.

For example, assume that the user wants to multiply the accumulator 202 value 217 by the reciprocal value of the value 49. The reciprocal value of the value 49 is presented in two dimensions and the 13 decimal places are set to be 0.0000010100111 with five leading zeros. Thus, the user will fill the suppressed leading zero number 2962 into the value 5 and the leading zero suppression reversal value 2964 into the value 10100111. After the reciprocal multiplier "divider A" 3014 (see Fig. 30) multiplies the accumulator 202 value 217 by the leading zero suppression reciprocal value 2964, the resulting product is shifted to the right according to the suppressed leading zero number 2962. Such an embodiment facilitates the use of relatively few bits to express the reciprocal 2942 value to achieve high accuracy requirements.

The thirtieth diagram is a block diagram showing one embodiment of the start function unit 212 of the second figure. The start function unit 212 includes control logic 127 of the first diagram, a positive type converter (PFC) and an output binary point aligner (OBPA) 3002 to receive the value of the accumulator 202. 217. A rounder 3004 receives an index of the accumulator 202 value 217 and the number of bits removed by the output binary point aligner 3002, a random bit source 3003 as described above to generate a random bit 3005, a first A multiplexer 3006 receives the output of the positive type converter and the output binary point aligner 3002 and the output of the rounder 3004, a standard size compressor (CCS) and a saturator 3008 to receive the first multiplexer An output of 3006, a bit selector and saturator 3012 to receive the output of the standard size compressor and saturator 3008, a corrector 3018 to receive the output of the standard size compressor and saturator 3008, and a reciprocal multiplier 3014 to receive The output of the standard size compressor and saturator 3008, a rightward shifter 3016 to receive the output of the standard size compressor and saturator 3008, and a dual tangent tanner module 3022 to receive the bit selector and saturator 3012 output, an S-type module 3024 to receive the output of the bit selector and saturator 3012, a soft add module 3026 to receive the output of the bit selector and saturator 3012, a second The device 3032 receives the output of the dual-take tangent module 3022, the S-type module 3024, the soft-add module 3026, the corrector 3018, the reciprocal multiplier 3014 and the right shifter 3016, and the standard size compressor and saturator 3008. The passed standard size output 3028, a symbol restorer 3034 to receive the output of the second multiplexer 3032, a size converter and saturator 3036 to receive the output of the symbol restorer 3034, and a third multiplexer 3037 to receive The output of the size converter and saturator 3036 and the accumulator output 217, and an output register 3038 to receive the output of the multiplexer 3037, the output of which is the result 133 in the first figure.

The positive type converter and the output binary point aligner 3002 receive the accumulator 202 value 217. In a preferred embodiment, accumulator 202 value 217 is a full precision value as previously described. That is, accumulator 202 has a sufficient number of storage bits to load an accumulated number, which is the total number of products produced by integer adder 244 that are summed by a series of integer multipliers 242, and this operation is performed. Any one of the individual products of the multiplier 242 or the respective totals of the adders is not discarded to maintain accuracy. In a preferred embodiment, accumulator 202 has at least a sufficient number of bits to load the maximum number of multiply-accumulates that a neural network unit 121 can be programmed to perform. For example, please refer to the program of the fourth figure. In the wide configuration, the maximum number of multiplications and accumulations that can be generated by the neural network unit 121 can be 512, and the cumulative number of 202 bits is 41. In another example, referring to the program of the twentieth diagram, in a narrow configuration, the maximum number of multiply-accumulates that can be generated by the neural network unit 121 can be 1024, and the cumulative number of 202 bits is 28. Basically, full precision accumulator 202 has at least Q bits, where Q is the sum of M and log ₂ P, where M is the bit width of the integer product of multiplier 242 (for example, for narrow multiplier 242) It is a 16-bit element, which is 32 bits for the wide multiplier 242), and P is the maximum allowable number of products that the accumulator 202 can accumulate. In a preferred embodiment, the maximum number of multipliers is specified in accordance with the program specification of the programmer of neural network unit 121. In one embodiment, assume that a previous multiply-accumulate instruction is used to load the data/weight text 206/207 column from the data/weight random access memory 122/124 (as in the instruction of address 1 in the fourth figure). The maximum value of the count of the sequencer 128 executing the multiply-accumulate neural network unit instruction (such as the instruction of address 2 in the fourth figure) is, for example, 511.

The design of the arithmetic logic unit 204 of the neural processing unit 126 can be simplified by using an accumulator 202 having a sufficient bit width to perform an accumulation operation on one of the maximum number of allowed superimposed values. In particular, such processing can alleviate the need to use logic to perform saturation operations on the total number of integer adder 244 generations, since integer adder 244 causes an overflow accumulator to generate an overflow, while continuing to track the accumulator's binary fraction The point position is used to confirm whether an overflow is generated to confirm whether a saturation operation needs to be performed. For example, for a design with a non-full precision accumulator but with saturation logic to handle the overflow of a non-full accumulator, the following is assumed.

(1) The range of data values is between 0 and 1 and all storage bits are used to store decimal places. The weight text value ranges from -8 to +8 and all but three of the storage bits are used to store the decimal places. The cumulative value of the input as a hyperbolic tangent start function is between -8 and 8, and all but three of the storage bits are used to store the decimal places.

(2) The bit width of the accumulator is not full precision (such as the bit width of only the product).

(3) Assuming that the accumulator is full precision, the final accumulated value will also be between -8 and 8 (eg +4.2); however, the product before "point A" in this sequence will produce positive values more frequently. , and the product after point A will produce negative values more frequently. In this case, it is possible to obtain incorrect results (such as results other than +4.2). This is because at some point in front of point A, when the accumulator needs to reach a value that exceeds its saturation maximum of +8, such as +8.2, it will be damaged. Lost more than 0.2. The accumulator will even keep the remaining multiply-accumulated result at a saturated value, and will lose more positive values. Therefore, the final value of the accumulator may be less than the value calculated using an accumulator with a full precision bit width (ie less than +4.2).

The positive type converter 3004 converts the positive value to 217 when the value 217 of the accumulator 202 is negative, and generates an extra bit to indicate the positive or negative value of the original value. This bit is passed down to the start function unit 212 along with this value. Pipeline. Converting a negative number to a positive type simplifies the operation of the subsequent start function unit 121. For example, after this processing, only the positive value will input the hyperbolic tangent module 3022 and the S-type module 3024, thereby simplifying the design of these modules. In addition, the rounder 3004 and the saturator 3008 can also be simplified.

The output binary point aligner 3002 will shift or scale the positive type value to the right to align with the output binary point 2954 specified in the control register 127. In a preferred embodiment, the output binary point aligner 3002 calculates the number of decimal places of the value 217 of the accumulator 202 (eg, as specified by the accumulator binary decimal point 2923 or the data binary decimal point 2922). The sum of the number of decimal places (as specified by the output binary point 2954) minus the number of decimal places of the output (as specified by the output binary point 2954) is used as the offset. Thus, for example, if the accumulator 202 binary decimal point 2923 is 8 (ie, the above embodiment) and the output binary decimal point 2954 is 3, the output binary decimal point aligner 3002 will have this positive type value right. Five bits are shifted to produce the result provided to multiplexer 3006 and rounder 3004.

The rounder 3004 performs an accumulator 202 value 217 Rounding operation. In a preferred embodiment, the rounder 3004 produces a rounded version of the positive type value generated by the positive type converter and the output binary point aligner 3002, and provides the rounded version. Up to the worker 3006. The rounder 3004 performs a rounding operation in accordance with the rounding control 2932 described above. As described herein, the rounding control described above may include random rounding using random bits 3005. The multiplexer 3006 will select one of its plurality of inputs, that is, from the positive type converter and the output binary point aligner 3002, depending on the rounding control 2932 (which may include random rounding as described herein). The positive type value is either from the rounded version of the rounder 3004 and the selected value is provided to the standard size compressor and saturator 3008. In a preferred embodiment, if the rounding control specifies that rounding is not performed, the multiplexer 3006 selects the output of the positive type converter and the output binary point aligner 3002, otherwise the rounder is selected. The output of 3004. In other embodiments, additional rounding operations may also be performed by the start function unit 212. For example, in one embodiment, when the bit selector 3012 compresses the output of the standard size compressor and saturator 3008 (described below), the bit selector 3012 is based on the missing low order bit. Perform rounding operations. In another example, the product of the reciprocal multiplier 3014 (as described below) is subjected to a rounding operation. In yet another example, the size converter 3036 needs to convert the appropriate output size (as described below), which may involve dropping some of the low order bits used to determine the rounding, and performing a rounding operation.

The standard size compressor 3008 compresses the multiplexer 3006 output value to a standard size. Thus, for example, if the neural processing unit 126 is in a narrow configuration or funnel configuration 2902, a standard size compressor 3008 can compress the 28-bit multiplexer 3006 output value to 16-bit; and if the neural processing unit 126 is in the wide configuration 2902, the standard-size compressor 3008 can compress the 41-bit multiplexer 3006 output value to 32-bit. However, before compressing to the standard size, if the pre-compression value is greater than the maximum value that can be expressed by the standard version, the saturator 3008 fills the pre-compression value to the maximum value that can be expressed by the standard pattern. For example, if any bit in the pre-compression value that is to the left of the most significant pre-compression value bit is a value of 1, the saturator 3008 fills up to the maximum value (if filled to all 1s).

In a preferred embodiment, the hyperbolic tangent module 3022, the S-type module 3024, and the soft-add module 3026 all include a lookup table, such as a programmable logic array (PLA), a read-only memory (ROM). , combination logic gates, and so on. In an embodiment, in order to simplify and reduce the size of the modules 3022/3024/3026, the input values provided to the modules are 3.4, that is, three integer bits and four decimal places, that is, The input value has four bits to the right of the binary decimal point and three bits to the left of the binary decimal point. Since the output value is asymptotically close to its min/max value at the extremes of the input value range (-8, +8) of the 3.4 type, these values are selected. However, the present invention is not limited thereto, and the present invention is also applicable to other embodiments in which a binary decimal point is placed at a different position, such as a 4.3 type or a 2.5 type. The bit selector 3012 selects and selects a bit that satisfies the 3.4 type specification among the bits output by the standard size compressor and the saturator 3008. This involves compression processing, that is, some bits are lost because the standard type has a comparison. More bits. However, in selecting/compressing the standard size compressor and saturator 3008 output values Before, if the value before compression is greater than the maximum value that can be expressed by the 3.4 type, the saturator 3012 will fill the value before compression to the maximum value that can be expressed by the 3.4 type. For example, if any bit in the pre-compression value that is to the left of the most significant 3.4 type bit is a value of 1, the saturator 3012 fills up to the maximum value (eg, fills up to all 1s).

The hyperbolic tangent module 3022, the S-type module 3024, and the soft-add module 3026 perform a corresponding start function (as described above) on the 3.4-type value output by the standard size compressor and the saturator 3008 to produce a result. In a preferred embodiment, the hyperbolic tangent module 3022 and the S-type module 3024 produce a 0.7-bit 7-bit result, ie, zero integer bits and seven decimal places, that is, input. The value has seven bits to the right of the binary decimal point. In a preferred embodiment, the soft add module 3026 produces a 3.4-bit 7-bit result, i.e., the pattern is the same as the input pattern of the module 3026. In a preferred embodiment, the outputs of the hyperbolic tangent module 3022, the S-type module 3024, and the soft-add module 3026 are extended to a standard version (eg, with leading zeros if necessary) and aligned The binary decimal point is specified by the output binary digit 2954.

The corrector 3018 produces a corrected version of the output values of the standard size compressor and saturator 3008. That is, if the output value of the standard size compressor and the saturator 3008 (as described above, the symbol is shifted down the pipeline), the corrector 3018 will output a zero value; otherwise, the corrector 3018 will output its input value. . In a preferred embodiment, the output of the corrector 3018 is of a standard version and has a binary point specified by the output binary point 2954 value.

The reciprocal multiplier 3014 will have a standard size compressor with The output of the saturator 3008 is multiplied by a user-specified reciprocal value assigned to the inverse value 2942 to produce a product of a standard size, which is actually the output of the standard size compressor and saturator 3008, with a reciprocal value of 2942. The reciprocal is the quotient calculated as the divisor. In a preferred embodiment, the output of the reciprocal multiplier 3014 is a standard version and has a binary point specified by the output binary point 2954.

The right shifter 3016 moves the output of the standard size compressor and saturator 3008 by the number of user-specified bits specified by the offset value 2944 to produce a quotient of the standard size. In a preferred embodiment, the output to the right shifter 3016 is a standard version and has a binary point specified by the output binary point 2954.

The multiplexer 3032 selects the appropriate input specified by the value of the start function 2934 and provides its selection to the symbol restorer 3034. If the original accumulator 202 has a negative value of 217, the symbol restorer 3034 outputs the multiplexer 3032. The positive type value is converted to a negative type, for example to a two-complement type.

The size converter 3036 converts the output of the symbol restorer 3034 to the appropriate size in accordance with the value of the output command 2956 as described in FIG. 29A. In a preferred embodiment, the output of symbol restorer 3034 has a binary point specified by the output binary point 2954 value. In a preferred embodiment, the size converter 3036 discards the upper half of the output of the symbol restorer 3034 for the first predetermined value of the output command. In addition, if the output of the symbol restorer 3034 is positive and exceeds the maximum value that can be expressed by the text size specified by the configuration 2902, or the output is negative and smaller than the text size. For small values, the saturator 3036 fills its output to the maximum/minimum expressible value of the text size. For the second and third preset values, the size converter 3036 passes the output of the symbol restorer 3034.

The multiplexer 3037 selects one of the data converter and saturator 3036 output and accumulator 202 output 217 for supply to the output register 3038 in accordance with the output command 2956. Further, for the first and second preset values of the output command 2956, the multiplexer 3037 selects the lower text of the output of the size converter and saturator 3036 (the size is specified by configuration 2902). For the third preset value, the multiplexer 3037 selects the upper text of the output of the size converter and saturator 3036. For the fourth preset value, the multiplexer 3037 selects the lower text of the unprocessed accumulator 202 value 217; for the fifth preset value, the multiplexer 3037 selects the middle of the unprocessed accumulator 202 value 217 The text; and for the sixth preset value, the multiplexer 3037 selects the upper text of the unprocessed accumulator 202 value 217. As previously mentioned, in a preferred embodiment, the start function unit 212 adds a zero value upper orientation element to the upper text of the unprocessed accumulator 202 value 217.

The thirty-first figure shows an example of the operation of the start function unit 212 of the thirty-th figure. As shown in the figure, the configuration 2902 of the neural processing unit 126 is set to a narrow configuration. In addition, the signed material 2912 and the signed weight 2914 are true. In addition, the data binary decimal point value of 2922 indicates that for the data random access memory 122 text, there are 7 bits to the right of the binary decimal point position, and one of the first data characters received by the neural processing unit 126 is an example value. The line is presented as 0.1001110. In addition, the weighted binary decimal point 2924 value indicates the weighted random access memory In the case of 124 characters, there are 3 bits to the right of the decimal point position, and one of the first weight words received by the neural processing unit 126 is 00001.010.

The 16-bit product of the first data and the weight text (this product is added to the initial zero value of the accumulator 202) is presented as 000000.1100001100. Since the data binary decimal point 2912 is 7 and the weighted binary decimal point 2914 is 3, for the implicit accumulator 202 binary decimal point, there will be 10 bits to the right. In the case of a narrow configuration, as shown in this embodiment, the accumulator 202 has 28 bit widths. For example, after all arithmetic logic operations have been completed (eg, all 1024 multiply-accumulate operations in the twentieth diagram), the value 217 of the accumulator 202 would be 000000000000000001.1101010100.

The output binary digit 2954 value indicates that the output binary digit has 7 digits to the right of the decimal point. Thus, after passing the output decimal point aligner 3002 and the standard size compressor 3008, the accumulator 202 value 217 is scaled, rounded, and compressed to the value of the standard version, 000000001.1101011. In this example, the output binary decimal point address represents 7 decimal places, and the accumulator 202 binary decimal point position represents 10 decimal places. Thus, the output binary point aligner 3002 calculates the difference 3 and scales it by shifting the accumulator 202 value 217 to the right by 3 bits. In the thirty-first figure, it is shown that the accumulator 202 value 217 will lose 3 least significant bits (two digits of 100). Moreover, in this example, the rounding control 2932 value indicates that random rounding is used, and in this example it is assumed that the sampling random bit 3005 is true. So, as mentioned above, the least significant bit will be rounded up because of the accumulator 202 rounding bits of value 217 (the three most significant bits of the bit that were removed because of the scaling operation of the accumulator 202 value 217) are one, and the sticky bits (the three because of the accumulator 202 value) Of the bits that are shifted out by the scaling operation of 217, the Boolean or arithmetic result of the two least significant bits is zero.

In this example, the start function 2934 indicates that an sigmoid function is used. Thus, bit selector 3012 selects the bit of the standard pattern value such that the input of S-type module 3024 has three integer bits and four decimal places, as previously described, the value 001.1101 shown. The output value of the S-type module 3024 will be placed in the standard version, which is the value shown as 000000000.1101110.

The output command 2956 of this example specifies the first preset value, i.e., the text size represented by output configuration 2902, in this case narrow text (8 bits). Thus, the size converter 3036 converts the standard S-type output value into an 8-bit quantity having an implicit binary point, ie, 7 bits to the right of the binary point, producing an output The value is 01101110, as shown in the figure.

The thirty-second figure shows a second example of the operation of the start function unit 212 of the thirty-th figure. The example of the thirty-second diagram describes the operation of the start function unit 212 when the start function 2934 indicates that the accumulator 202 value 217 is passed in a standard size. As shown in the figure, this configuration 2902 is set to a narrow configuration of the neural processing unit 216.

In this example, the width of the accumulator 202 is 28 bits, and the accumulator 202 has 10 digits to the right of the decimal point position (this is because in the embodiment the data binary decimal point 2912 and the weight are binary The sum of the decimal point 2914 is a total of 10, or in another embodiment the accumulator binary decimal point 2923 is explicitly designated to have a value of 10). For example, after performing all arithmetic logic operations, the value 217 of the accumulator 202 shown in the thirty-second diagram is 000001100000011011.1101111010.

In this example, the output binary point 2954 value indicates that for the output, there are 4 bits to the right of the binary point. Thus, after passing the output binary point aligner 3002 and the standard size compressor 3008, the accumulator 202 value 217 is saturated and compressed to the standard pattern value 111111111111.1111 shown, which is received by the multiplexer 3032. The value is passed as a standard size 3028.

Two output commands 2956 are shown in this example. The first output command 2956 specifies a second preset value, which is the lower text of the output standard type size. Since the size indicated by configuration 2902 is a narrow text (8-bit), the standard size will be 16-bit, and the size converter 3036 will select the lower 8 bits below the standard size transfer value 3028 to produce as shown in the figure. The 8-bit value is 11111111. The second output command 2956 specifies a third preset value, that is, the upper text of the output standard type size. As such, the size converter 3036 selects the upper 8 bits of the standard size transfer value 3028 to produce the 8-bit value 11111111 as shown.

The thirty-third figure shows a third example of the operation of the start function unit 212 of the thirty-th figure. The example of the thirty-third diagram reveals the operation of the start function unit 212 when the start function 2934 indicates that the entire unprocessed accumulator 202 value 217 is to be passed. As shown in the figure, this configuration 2902 is set to a wide configuration of the neural processing unit 126 (e.g., 16-bit input text).

In this example, the accumulator 202 has a width of 41 bits, and the accumulator 202 has 8 bits to the right of the decimal point position (this is because in the embodiment the data binary decimal point 2912 and the weight binary carry The sum of the decimal points 2914 is a total of 8, or in another embodiment the accumulator binary decimal point 2923 is explicitly designated to have a value of 8). For example, after performing all of the arithmetic logic operations, the value 217 of the accumulator 202 shown in the thirty-third figure is 001000000000000000001100000011011.11011110.

Three output commands 2956 are shown in this example. The first output command specifies a fourth preset value, that is, outputs the text below the value of the unprocessed accumulator 202; the second output command specifies a fifth preset value, that is, the middle of the output of the unprocessed accumulator 202. The text is output; the third output command specifies a sixth preset value, that is, the text above the value of the unprocessed accumulator 202 is output. Because the size indicated by configuration 2902 is a wide text (16-bit), as shown in the thirty-third figure, in response to the first output command 2956, the multiplexer 3037 selects the 16-bit value 0001101111011110; in response to the second output command 2956 The multiplexer 3037 selects the 16-bit value 0000000000011000; and in response to the third output command 2956, the multiplexer 3037 selects the 16-bit value 0000000001000000.

As mentioned above, the neural network unit 121 can execute on integer data instead of floating point data. As such, it is helpful to simplify the individual neural processing units 126, or at least the arithmetic logic unit 204 portions thereof. For example, the arithmetic logic unit 204 does not need to include, for the multiplier 242, an adder that is used in floating point operations to add the exponents of the multipliers. Similarly, this arithmetic logic unit 204 does not need to be used for the adder 234. The shifter that is used in the floating-point operation to align the binary digits of the addendum is included. It will be understood by those of ordinary skill in the art that floating point units are often very complex; therefore, the examples described herein are only simplified for arithmetic logic unit 204, with which the user can specify the relevant two The integer embodiment of the carry decimal point can also be used to simplify other parts. In contrast to floating point embodiments, the use of integer units as arithmetic logic unit 204 can produce a smaller (and faster) neural processing unit 126 that facilitates the integration of a large array of neural processing units 126 into the neural network. Within unit 121. The portion of the start function unit 212 may process the scaling and saturation operations of the value 217 of the accumulator 202 based on the user specified, the number of decimal places required for the accumulated number, and the number of decimal bits required for the output value, preferably based on use. Designated. Any additional complexity and accompanying increase in size, as well as energy and/or time loss within the fixed-point hardware assistance of the start function unit 212, can be shared by sharing the start function unit 212 between the arithmetic logic units 204. This is because, as shown in the embodiment of the eleventh figure, the embodiment using the sharing mode can reduce the number of startup function units 1112.

The embodiments described herein can enjoy many advantages of using integer arithmetic units to reduce hardware complexity (compared to using floating point arithmetic units), while also being used for arithmetic operations of decimals, that is, having a binary decimal point. digital. The advantage of floating-point arithmetic is that it can provide data arithmetic operations to the individual values of the data falling within a very wide range of values (actually limited only by the size of the exponent range, so it will be a very large range). In other words, each floating point number has its potential uniqueness No index value. However, the embodiments described herein understand and utilize features in certain applications where the input data is highly parallel and falls within a relatively narrow range such that all parallel data have the same "index". As such, these embodiments allow the user to assign a binary decimal point position to all input values and/or accumulated values at a time. Similarly, by understanding and utilizing the parallel output having similar range characteristics, these embodiments allow the user to assign the binary point position to all of the output values at a time. Artificial neural networks are an example of such an application, although embodiments of the invention may also be applied to perform calculations for other applications. By assigning a binary decimal point position to multiple inputs instead of giving individual input numbers, embodiments of the present invention can utilize memory space more efficiently than using floating point operations (eg, requiring less memory) Accuracy is improved in the case of using a similar amount of memory, because the bits of the exponent used for floating-point operations can be used to improve numerical precision.

Moreover, embodiments of the present invention understand that accuracy may be lost when performing an accumulation of a large series of integer operations (such as overflow or loss of less important fractional bits), thus providing a solution, primarily using a sufficiently large Accumulator to avoid loss of precision.

Direct execution of neural network unit micro-operation

The thirty-fourth diagram is a block diagram showing a portion of the details of the processor 100 of the first diagram and the neural network unit 121. The neural network unit 121 includes a pipeline stage 3401 of the neural processing unit 126. Each pipeline stage 3401 is distinguished by a stage register and includes combinational logic to achieve the operations of the neural processing unit 126 herein, such as a Boolean logic gate, a multiplexer, and Instruments, multipliers, comparators, etc. Pipeline stage 3401 receives a micro-operation 3418 from multiplexer 3402. Micro-operation 3418 will flow down to pipeline stage 3401 and control its combinational logic. Micro-operation 3418 is a set of bits. For a preferred embodiment, the micro-operation 3418 includes a bit of the data random access memory 122 memory address 123, a bit of the weight random access memory 124 memory address 125, and a program memory 129 memory address. The bit of 131, the multiplex register 208/705 control signal 213/713, and a number of fields that control the register 217 (e.g., the control registers of the twenty-ninth to twenty-ninth C-pictures). In an embodiment, the micro-operation 3418 includes approximately 120 bits. Multiplexer 3402 receives micro-operations from three different sources and selects one of them as a micro-operation 3418 that is provided to pipeline stage 3401.

One micro-operation source of multiplexer 3402 is sequencer 128 of the first figure. The sequencer 128 decodes the neural network unit instructions received by the program memory 129 and accordingly provides a micro-operation 3416 to the first input of the multiplexer 3402.

The second source of micro-computations of multiplexer 3402 is a decoder 10404 that receives microinstructions 105 from reservation station 108 of the first diagram and operands from general register 116 and media register 118. In a preferred embodiment, as previously described, the microinstruction 105 is generated by the instruction translator 104 in response to the translation of the MTNN instruction 1400 and the MFNN instruction 1500. Microinstruction 105 may include an immediate column to specify a particular function (specified by an MTNN instruction 1400 or an MFNN instruction 1500), such as the start and stop execution of a program in program memory 129, executing directly from media register 118. Micro-operation, or memory of one of the neural network units read/written as described above. The decoder 3404 decodes the microinstructions 105 and generates a Micro-operation 3412 provides a second input to the multiplexer. For a preferred embodiment, for certain functions 1432/1532 of the MTNN instruction 1400/MFNN instruction 1500, the decoder 3404 does not need to generate a micro-operation 3412 to pass down to the pipeline 3401, such as write control temporary storage. The program 127 starts executing the program in the program memory 129, suspends the execution of the program in the program memory 129, waits for the execution of the program in the program memory 129, reads from the status register 127, and resets the neural network unit 121. .

The third source of micro-computation for multiplexer 3402 is media register 118 itself. In a preferred embodiment, as previously described in relation to FIG. 14, the MTNN instruction 1400 can specify a function to instruct the neural network unit 121 to directly perform a provision from the media register 118 to the multiplexer 3402. The three-input micro-operation 3414. Direct execution of the micro-operations 3414 provided by the architectural media register 118 facilitates testing of the neural network unit 121, such as built-in self-test (BIST), or debugging.

In a preferred embodiment, decoder 3404 generates a mode indicator 3422 that controls the selection of multiplexer 3402. When the MTNN instruction 1400 specifies a function to begin execution of a program from the program memory 129, the decoder 3404 generates a mode indicator 3422 value causing the multiplexer 3402 to select the micro-operation 3416 from the sequencer 128 until an error occurs or until decoding occurs. The device 3404 encounters an MTNN instruction 1400 that specifies a function to stop execution of the program from the program memory 129. When the MTNN instruction 1400 specifies a function to instruct the neural network unit 121 to directly perform a micro-operation 3414 provided by the media register 118, the decoder 3404 generates a mode indicator 3422 value to cause the multiplexer 3402 to select from the specified media. The micro-operation 3414 of the register 118. Otherwise, the decoder 3404 will generate a mode The value of the formula indicator 3422 causes the multiplexer 3402 to select the micro-operation 3412 from the decoder 3404.

Variable rate neural network unit

In many cases, the neural network unit 121 will enter the standby state after executing the program, waiting for the processor 100 to process some things that need to be processed before executing the next program. For example, assuming that it is in a situation similar to that described in Figures 3-6A, the neural network unit 121 will accumulate a start function program (also known as a pre-nego network layer program). Neural network layer program)) Performs two or more times in succession. Compared to the time it takes for the neural network unit 121 to execute the program, the processor 100 obviously takes a long time to write a weight value of 512 KB into the weight random access memory 124 for use by the next neural network unit program. . In other words, the neural network unit 121 executes the program in a short time and then enters the standby state until the processor 100 writes the next weight value to the weight random access memory 124 for use in the next program execution. For the case, refer to the thirty-sixth A diagram, as will be described later. In this case, the neural network unit 121 can operate at a lower time frequency to extend the execution time of the program, so that the energy consumption required to execute the program is dispersed over a longer time range, and the neural network unit 121, and even the neural network unit 121 Throughout the processor 100, it is maintained at a lower temperature. This case is called a mitigation mode, and can be referred to the thirty-sixth B-picture, as will be described later.

The thirty-fifth diagram is a block diagram showing a processor 100 having a variable rate neural network unit 121. This processor 100 is similar to processor 100 of the first figure, and elements having the same reference numerals are similar in the figures. The processor 100 of the thirty-fifth figure has a time-frequency generating logic 3502 The functional units coupled to the processor 100 are the instruction fetch unit 101, the instruction cache 102, the instruction translator 104, the rename unit 106, the reservation station 108, the neural network unit 121, and other execution units 112, memory. The body subsystem 114, the universal register 116 and the media register 118. The time-frequency generating logic 3502 includes a time-frequency generator, such as a phase-locked loop (PLL), to generate a time-frequency signal having a dominant time frequency or time-frequency. For example, the primary time frequency can be 1 GHz, 1.5 GHz, 2 GHz, and the like. The time frequency means the number of cycles per second, such as the number of oscillations of the time-frequency signal between high and low states. Preferably, the frequency signal has a duty cycle, that is, one half of the cycle is a high state and the other half is a low state; in addition, the frequency signal can also have an unbalanced period, that is, a time frequency. The signal is in a high state for longer than it is at a low state, and vice versa. Preferably, the phase locked loop is used to generate a primary time-frequency signal of a plurality of time frequencies. Preferably, the processor 100 includes a power management module that automatically adjusts the primary time frequency according to various factors, including the dynamic detection operating temperature of the processor 100, the utilization, and the system software (such as the operating system). , Basic Input Output System (BIOS)) A command that indicates the required performance and/or energy efficiency indicators. In an embodiment, the power management module includes microcode of the processor 100.

The time-frequency generation logic 3502 includes a time-frequency spreading network, or a clock tree. The time-frequency tree will spread the main time-frequency signal to the functional unit of the processor 100. As shown in the thirty-fifth figure, the spreading action is to transmit the time-frequency signal 3506-1 to the instruction capturing unit 101, and the time-frequency signal 3506. -2 is transferred to the instruction cache 102, the time frequency signal 3506-10 is transmitted to the instruction translator 104, and the time frequency signal 3506-9 is transmitted. To the rename unit 106, the time-frequency signal 3506-8 is transmitted to the reservation station 108, the time-frequency signal 3506-7 is transmitted to the neural network unit 121, and the time-frequency signal 3506-4 is transmitted to the other execution unit 112. The frequency signal 3506-3 is transmitted to the memory subsystem 114, the time-frequency signal 3506-5 is transmitted to the general purpose register 116, and the time-frequency signal 3506-6 is transmitted to the media register 118. These signals are collectively referred to as Frequency signal 3506. The frequency tree now has a node or line to transmit the primary time-frequency signal 3506 to its corresponding functional unit. In addition, preferably, the time-frequency generating logic 3502 can include a time-frequency buffer, when it is required to provide a clean time-frequency signal and/or need to increase the voltage level of the main time-frequency signal, especially for a distant node. The time-frequency buffer can regenerate the main time-frequency signal. In addition, each functional unit has its own sub-time-frequency tree that regenerates and/or boosts the voltage level of the corresponding primary time-frequency signal 3506 received as needed.

The neural network unit 121 includes time-frequency reduction logic 3504, and the time-frequency reduction logic 3504 receives a mitigation indicator 3512 and a primary time-frequency signal 3506-7 to generate a second time-frequency signal. The second time-frequency signal has a one-time frequency. If the frequency is not the same as the main time frequency, it is in a mode of mitigation. The frequency is reduced by a value from the main time to reduce the heat generation. This value is programmed to the mitigation index 3512. The time-frequency reduction logic 3504 is similar to the time-frequency generation logic 3502, which has a time-frequency spreading network, or a time-frequency tree, to spread the second time-frequency signal to the various functional blocks of the neural network unit 121, and the spreading action is time-consuming. The frequency signal 3508-1 is transmitted to the neural processing unit array 126, and the time-frequency signal 3508-2 is transmitted to the sequencer 128 to transmit the time-frequency signal 3508-3 to the interface logic 3514, collectively referred to as the second time-frequency signal. 3508. Preferably, these nerve treatments Unit 126 includes a plurality of pipeline stages 3401. As shown in FIG. 34, pipeline stage 3401 includes a pipeline staging register for receiving second time-frequency signal 3508-1 from time-frequency reduction logic 3504.

The neural network unit 121 also has interface logic 3514 to receive the primary time-frequency signal 3506-7 and the second time-frequency signal 3508-3. The interface logic 3514 is coupled between the lower portion of the front end of the processor 100 (eg, the reservation station 108, the media register 118 and the universal register 116) and the various functional blocks of the neural network unit 121. These functional blocks are reduced in real time. Logic 3504, data random access memory 122, weighted random access memory 124, program memory 129 and sequencer 128. Interface logic 3514 includes a data random access memory buffer 3522, a weighted random access memory buffer 3524, a thirty-fourth graph decoder 3404, and a mitigation indicator 3512. The mitigation indicator 3512 is loaded with a value that specifies how slow the neural processing unit array 126 will execute the neural network unit program instructions. Preferably, the mitigation indicator 3512 specifies a division value N, and the time-frequency reduction logic 3504 divides the main time-frequency signal 3506-7 by the division value to generate the second time-frequency signal 3508. Thus, the second time-frequency signal is The frequency will be 1/N. Preferably, the value of N can be programmed into any one of a plurality of different preset values. The preset values can cause the time-frequency reduction logic 3504 to generate a plurality of second time-frequency signals 3508 having different time frequencies. The time frequency is less than the primary frequency.

In one embodiment, the time-frequency reduction logic 3504 includes a time-frequency divider circuit for dividing the primary time-frequency signal 3506-7 by the mitigation indicator 3512 value. In an embodiment, the time-frequency reduction logic 3504 includes a time-frequency gate (such as an AND gate), and the time-frequency gate can be gated through a start signal. For time-frequency signal 3506-7, the enable signal will only generate a true value every N cycles of the primary time-frequency signal. For example, a circuit that includes a counter to generate a start signal can count up to N. When the accompanying logic detects that the output of the counter matches N, the logic generates a true pulse at the second time-frequency signal 3508 and resets the counter. Preferably, the mitigation indicator 3512 value can be programmed by an architectural instruction, such as the MTNN instruction 1400 of FIG. Preferably, before the architecture program instructs the neural network unit 121 to start executing the neural network unit program, the architecture program operating on the processor 100 will program the mitigation value to the mitigation indicator 3512, which in the subsequent corresponds to the thirtieth There will be more detailed explanations at the seven maps.

The weighted random access memory buffer 3524 is coupled between the weighted random access memory 124 and the media register 118 as a buffer for data transmission therebetween. Preferably, weighted random access memory buffer 3524 is similar to one or more embodiments of buffer 1704 of FIG. Preferably, the weighted random access memory buffer 3524 receives the data from the media register 118 by using the primary time-frequency signal 3506-7 having the primary time frequency as the time frequency, and the weight random access memory buffer 3524 from The portion of the weighted random access memory 124 that receives the data is the second time-frequency signal 3508-3 having the second time frequency as the time frequency, and the second time frequency can be adjusted from the main time frequency according to the value programmed in the mitigation index 3512. Down or no, that is, whether the neural network unit 121 performs the mitigation or normal mode to perform the down or no. In one embodiment, the weight random access memory 124 is 單埠. As described in the foregoing FIG. 17, the weight random access memory 124 can be buffered by the media register 118 through the weight random access memory buffer. 3524, and accessed by the neural processing unit 126 or the eleventh column buffer 1104, in an arbitrated fashion. In another embodiment, the weight random access memory 124 is a double port. As described in the sixteenth figure above, each port can be transmitted by the media register 118 through the weight random access memory buffer 3524 and by the neural processing unit 126. Or column buffer 1104 is accessed in parallel.

Similar to the weighted random access memory buffer 3524, the data random access memory buffer 3522 is coupled between the data random access memory 122 and the media register 118 as a buffer for data transfer therebetween. Preferably, data random access memory buffer 3522 is similar to one or more embodiments of buffer 1704 of FIG. Preferably, the data random access memory buffer 3522 receives the data from the media register 118 by using the main time-frequency signal 3506-7 having the main time frequency as the time frequency, and the data random access memory buffer 3522 The data random access memory 122 receives the data by using a second time-frequency signal 3508-3 having a second time frequency as a time frequency, and the second time frequency can be adjusted from the main time frequency according to the value programmed in the mitigation index 3512. Down or no, that is, whether the neural network unit 121 performs the mitigation or normal mode to perform the down or no. In one embodiment, the data random access memory 122 is 單埠. As described in FIG. 17 above, the data random access memory 122 can be buffered by the media buffer 118 through the data random access memory buffer 3522. And accessed by the neural processing unit 126 or the column buffer 1104 of the eleventh figure in an arbitration manner. In another embodiment, the data random access memory 122 is a double port. As described in the sixteenth figure above, each port can be accessed by the media register 118 through the data random access memory buffer 3522 and by the neural processing unit 126. or Column buffer 1104 is accessed in parallel.

Preferably, the interface logic 3514 includes a data random access memory buffer 3522 and a weight random access memory buffer, regardless of whether the data random access memory 122 and/or the weight random access memory 124 are 單埠 or 埠. The 3524 synchronizes the primary time-frequency domain with the second time-frequency domain. Preferably, the data random access memory 122, the weight random access memory 124 and the program memory 129 both have a static random access memory (SRAM), which includes individual read enable signals, and writes The signal and memory can be selected to enable the signal.

As before, the neural network unit 121 is one of the execution units of the processor 100. The execution unit is a micro-instruction in the processor that executes the schema instruction or a functional unit that executes the architecture instruction itself, such as executing the micro-instruction 105 or the architecture instruction 103 itself translated by the architecture instruction 103 in the first figure. The execution unit receives operands from the general purpose register of the processor, such as from the general purpose register 116 and the media register 118. The execution unit executes the micro or architectural instructions and produces a result that is written to the general purpose register. The MTNN instructions 1400 and MFNN instructions 1500 described in Figures 14 and 15 are examples of architectural instructions 103. Microinstructions are used to implement architectural instructions. More precisely, the collective execution of one or more microinstructions that are executed by the execution unit for the architectural instructions is the result of the input of the architectural instruction specified by the architectural instruction specified by the architectural instruction to produce the result of the architectural instruction definition.

The thirty-sixth A diagram is a timing diagram showing that the processor 100 has an operational example in which the neural network unit 121 operates in a general mode, which operates at a primary time frequency. In the timing diagram, time The process between the left and the right. The processor 100 executes the architectural program at the primary time frequency. More precisely, the front end of the processor 100 (eg, the instruction fetch unit 101, the instruction cache 102, the instruction translator 104, the rename unit 106, and the reservation station 108) captures, decodes, and issues architectural instructions to the neural network at the primary time frequency. Network unit 121 and other execution units 112.

Initially, the architecture program executes an architectural instruction (e.g., MTNN instruction 1400), and the processor front end 100 issues the architectural instruction to the neural network unit 121 to instruct the neural network unit 121 to begin executing the neural network in the program memory 129. Unit program. Previously, the architecture program executed an architectural instruction to write the value of a specified primary time frequency to the mitigation indicator 3512, even if the neural network unit was in the normal mode. More precisely, the value programmed to the mitigation indicator 3512 causes the time-frequency reduction logic 3504 to generate the second time-frequency signal 3508 at the dominant time frequency of the primary time-frequency signal 3506. Preferably, in this example, the time-frequency buffer of the time-frequency reduction logic 3504 simply boosts the voltage level of the primary time-frequency signal 3506. In addition, the architecture program executes the architectural instructions to write the data random access memory 122, the weighted random access memory 124, and the neural network unit program to the program memory 129. In response to the neural network unit program MTNN command 1400, the neural network unit 121 will begin executing the neural network unit program at the primary time frequency because the mitigation indicator 3512 is programmed with the primary time frequency value. After the neural network unit 121 begins execution, the architecture program continues to execute the architectural instructions at the primary time frequency, including writing and/or reading the data random access memory 122 and the weight random access memory 124 primarily by the MTNN instruction 1400. To complete an instance of the neural network unit program, or invocation Or preparation for the run.

In the example of the thirty-sixth A diagram, the neural network unit 121 is capable of performing the writing/reading of the data random access memory 122 and the weight random access memory 124 compared to the architecture program. Significantly less time (eg, a quarter of the time) completes the execution of the neural network unit program. For example, in the case of operating at the primary time frequency, the neural network unit 121 spends approximately 1000 time-frequency cycles to execute the neural network unit program, however, the architecture program would take approximately 4000 time-frequency cycles. As such, the neural network unit 121 will be in a standby state for the remainder of the time, which in this example is a relatively long period of time, such as approximately 3000 major time frequency periods. As shown in the example of Figure 36A, the foregoing mode is executed again depending on the size and configuration of the neural network, and may continue to be executed many times. Since the neural network unit 121 is a relatively large and transistor-intensive functional unit in the processor 100, the operation of the neural network unit 121 will generate a large amount of thermal energy, especially when operating at a primary time frequency.

The thirty-sixth B diagram is a timing diagram, and the display processor 100 has an operation example in which the neural network unit 121 operates in the mitigation mode, and the mitigation mode operates at a frequency lower than the main time frequency. The timing chart of the thirty-sixth B diagram is similar to the thirty-sixth A diagram. In the thirty-sixth A diagram, the processor 100 executes an architecture program at the main time frequency. This example assumes that the architecture program and the neural network unit program in Figure 36B are identical to the architecture program and neural network unit program in Figure 36A. However, before starting the neural network unit program, the architecture program executes an MTNN instruction 1400 to program a mitigation indicator 3512 with a numerical value. The frequency reduction logic 3504 generates a second time-frequency signal 3508 at a second time frequency that is less than the primary time frequency. That is to say, the architecture program causes the neural network unit 121 to be in the mitigation mode of the thirty-sixth B-picture, instead of the general mode of the thirty-sixth A-picture. Thus, the neural processing unit 126 executes the neural network unit program at the second time frequency, and in the mitigation mode, the second time frequency is less than the primary time frequency. In this example, it is assumed that the mitigation indicator 3512 is programmed with a value that specifies the second time frequency as a quarter of the primary frequency. Thus, the time taken by the neural network unit 121 to execute the neural network unit program in the mitigation mode is four times that of the normal mode, as shown in the thirty-sixth and thirty-sixth B-pictures. Comparing these two figures, it can be seen that the length of time during which the neural network unit 121 is in the standby state is significantly shortened. Thus, the duration of the neural network unit 121 performing the neural network unit program consuming energy in the thirty-sixth B diagram is approximately four times that of the neural network unit 121 executing the program in the normal mode in the thirty-sixth A picture. Therefore, the thermal energy generated by the neural network unit 121 in the unit of the neural network unit 121 in the unit time is approximately one quarter of the thirty-sixth A picture, and has the advantages described herein.

Figure 37 is a flow chart showing the operation of the processor 100 of the thirty-fifth figure. The operation described in this flowchart is similar to the previous operation corresponding to the thirty-fifth, thirty-sixth and thirty-sixth B diagrams. This process begins in step 3702.

In step 3702, processor 100 executes MTNN instruction 1400 to write weights to weight random access memory 124 and writes the data to data random access memory 122. The flow then proceeds to step 3704.

In step 3704, the processor 100 executes the MTNN command 1400 to program the mitigation indicator 3512 with a value that specifies a frequency that is lower than the primary time frequency, even if the neural network unit 121 is in the mitigation mode. The flow then proceeds to step 3706.

In step 3706, the processor 100 executes the MTNN instruction 1400 to instruct the neural network unit 121 to begin executing the neural network unit program, i.e., in a manner similar to that presented in Figure 36B. The flow then proceeds to step 3708.

In step 3708, neural network unit 121 begins executing the neural network unit program. At the same time, the processor 100 executes the MTNN instruction 1400 to write new weights to the weight random access memory 124 (and possibly also write new data to the data random access memory 122), and/or execute the MFNN instruction 1500. The result is read from the data random access memory 122 (the result may also be read from the weighted random access memory 124). The flow then proceeds to step 3712.

In step 3712, the processor 100 executes the MFNN instruction 1500 (e.g., the read status register 127) to detect that the neural network unit 121 has finished executing the program. Assuming that the architecture program selects a good mitigation indicator 3512 value, the time taken by the neural network unit 121 to execute the neural network unit program is the same as the processor 100 executing the partial architecture program to access the weighted random access memory 124 and / Or the time it takes for the data to randomly access the memory 122, as shown in Figure 36B. The flow then proceeds to step 3714.

At step 3714, the processor 100 executes the MTNN instruction 1400 and utilizes a numerically programmed mitigation indicator 3512, which specifies the primary The timing frequency is also in the normal mode even if the neural network unit 121 is in the normal mode. Next, proceed to step 3716.

In step 3716, the processor 100 executes the MTNN instruction 1400 to instruct the neural network unit 121 to begin executing the neural network unit program, i.e., in a manner similar to that presented in the thirty-sixth A diagram. The flow then proceeds to step 3718.

In step 3718, neural network unit 121 begins executing the neural network unit program in a normal mode. This process ends at step 3718.

As described above, execution in the mitigation mode can disperse the execution time while avoiding high temperatures compared to executing the neural network unit program in the normal mode (i.e., at the main time frequency of the processor). Further, when the neural network unit executes the program in the mitigation mode, the neural network unit generates thermal energy at a lower time frequency, and the thermal energy can smoothly pass through the neural network unit (for example, a semiconductor device, a metal layer and a lower layer) The substrate) is discharged from the surrounding package and the cooling mechanism (such as the heat sink, fan). Therefore, devices in the neural network unit (such as transistors, capacitors, wires) are more likely to operate at lower temperatures. Overall, operating in a mitigation mode also helps to reduce device temperature in other parts of the processor die. Lower operating temperatures, especially for junction temperatures of these devices, can reduce leakage current generation. In addition, because the amount of current flowing in per unit time decreases, the inductance noise and IR drop noise are also reduced. In addition, the temperature reduction has a positive effect on the negative bias temperature instability (NBTI) and positive bias instability (PBSI) of the metal oxide half field effect transistor (MOSFET) in the processor, and can improve the reliability. And / or device and the life of the processor part. The temperature is reduced and the Joule heat and electromigration effects within the metal layer of the processor are mitigated.

Communication mechanism between architecture programs and non-architecture programs for sharing resources between neural network units

As described above, in the examples of the twenty-fourth to twenty-eighth and thirty-fifth to thirty-seventh figures, the data random access memory 122 and the weight random access memory 124 are shared. The neural processing unit 126 shares the data random access memory 122 and the weight random access memory 124 with the front end of the processor 100. More precisely, the neural processing unit 126 and the front end of the processor 100, such as the media register 118, both read and write the data random access memory 122 and the weight random access memory 124. In other words, the architecture program executed by the processor 100 and the neural network unit program executing in the neural network unit 121 share the data random access memory 122 and the weight random access memory 124, and in some cases As mentioned earlier, it is necessary to control the flow between the architecture program and the neural network unit program. The resources of program memory 129 are also shared to some extent because the architecture program will write to it and sequencer 128 will read it. The embodiments described herein provide a high performance solution to control the flow of shared resources between the architectural program and the neural network unit.

In the embodiments described herein, neural network unit programs are also referred to as non-architectural programs, neural network unit commands are also referred to as non-architectural instructions, and neural network unit instruction sets (also referred to as neural processing as previously described). The unit instruction set) is also known as the non-architectural instruction set. A non-architectural instruction set is different from a architectural instruction set. Including the instruction translator 104 within the processor 100 In embodiments where the architectural instructions translate microinstructions, the non-architectural instruction set is also different from the microinstruction set.

The thirty-eighth diagram is a block diagram showing the sequencer 128 of the neural network unit 121 in detail. Sequencer 128 provides a memory address to program memory 129 to select the non-architected instructions provided to sequencer 128, as previously described. As shown in the thirty-eighth figure, the memory address is loaded in a program counter 3802 of the sequencer 128. The sequencer 128 will typically increment sequentially in the address sequence of the program memory 129 unless the sequencer 128 encounters a non-architectural instruction, such as a loop or branch instruction, and in this case, the sequencer 128 will The program counter 3802 is updated to the target address of the control instruction, i.e., updated to the address of the non-architected instruction located at the target of the control instruction. Thus, the address 131 loaded in the program counter 3802 specifies the address of the non-architected instruction in the program memory 129 that is currently being fetched for execution by the neural processing unit 126. The value of the program counter 3802 can be obtained by the architecture program through the neural network unit program counter field 3912 of the status register 127, as described in the subsequent thirty-first figure. In this way, the architecture program can determine the location of the data random access memory 122 and/or the weight random access memory 124 to read/write data according to the progress of the non-architecture program.

The sequencer 128 includes a loop counter 3804 which operates in conjunction with a non-architected loop command, such as a loop of address 10 to a command and a twenty-eighth in the twenty-sixth A diagram. In the figure, the loop of address 11 is up to 1 command. In the examples of the twenty-sixth and twenty-eighth diagrams, the loop counter 3804 is loaded with the value specified by the non-architectural initialization instruction of address 0, such as the value 400. Each sequencer 128 When the loop command is encountered and jumps to the target instruction (such as the multiply accumulate instruction at address 1 in the 26th A picture or the maxwacc instruction at address 1 in the twenty-eighth picture), the sequencer 128 will The loop counter 3804 is decremented. Once the loop counter 3804 is reduced to zero, the sequencer 128 turns to the next non-architected instruction. In another embodiment, the loop count value specified in a loop command is loaded in the loop counter for the first time the loop command is encountered, thereby eliminating the need to initialize the loop counter 3804 with the non-architectural initialization command. Therefore, the value of the loop counter 3804 will indicate the number of times the loop group of the non-architected program has yet to be executed. The value of the loop counter 3804 can be obtained by the architecture program through the loop count field 3914 of the status register 127, as shown in the subsequent thirty-ninth figure. In this way, the architecture program can determine the location of the data random access memory 122 and/or the weight random access memory 124 to read/write data according to the progress of the non-architecture program. In one embodiment, the sequencer includes three additional loop counters to match the nest loops in the non-architected program, and the values of the three loop counters are also readable by the status register 127. The loop instruction has a bit to indicate which of the four loop counters is available for use by the current loop instruction.

Sequencer 128 also includes an iteration count counter 3806. The iteration number counter 3806 is paired with non-architectural instructions, such as the multiply-accumulate instruction of address 2 in the fourth, nine, twenty- and twenty-sixth A pictures, and the maxwacc instruction of address 2 in the twenty-eighth figure. This will be referred to as the "execute" instruction. In the foregoing example, each execution instruction specifies an execution count 511, 511, 1023, 2, and 3, respectively. Sequencing when sequencer 128 encounters an execution instruction that specifies a non-zero iteration count The 128 will load the iteration count counter 3806 with this specified value. In addition, sequencer 128 generates an appropriate micro-operation 3418 to control the logic execution within neuroprocessing unit 126 pipeline stage 3401 in the thirty-fourth diagram and decrements the iteration count counter 3806. If the iteration count counter 3806 is greater than zero, the sequencer 128 will again generate an appropriate micro-operation 3418 to control the logic within the neural processing unit 126 and decrement the iteration count counter 3806. Sequencer 128 will continue to operate in this manner until the value of iteration count counter 3806 is zeroed. Therefore, the value of the iteration count counter 3806 is the number of operations to be executed in the non-architectural execution instruction (these operations are multiply and accumulate for the accumulated value and a data/weight text, take the maximum value, add the total operation, etc.). The value of the iteration count counter 3806 can be obtained by the architecture program through the iteration count field 3916 of the state register 127, as described in the subsequent thirty-ninth figure. In this way, the architecture program can determine the location of the data random access memory 122 and/or the weight random access memory 124 to read/write data according to the progress of the non-architecture program.

The thirty-ninth diagram is a block diagram showing the fields of the control and status register 127 of the neural network unit 121. These fields include an address 2602 including a weighted random access memory column that the neural processing unit 126 performs a non-architected program to write recently, and the neural processing unit 126 performs a weighted random access memory column that is recently read by the non-architected program. At address 2604, the neural processing unit 126 executes the address 2606 of the data random access memory column recently written by the non-architected program, and the neural processing unit 126 performs the address of the data random access memory column recently read by the non-architected program. 2608, as shown in the aforementioned twenty-sixth B. In addition, these fields also include a neural network unit program counter 3912 field, a circle count Field 3914 field, with an iteration count counter 3916 field. As described above, the architecture program can read the data in the state register 127 to the media register 118 and/or the general register 116, for example, by reading the MFNN instruction 1500, including the neural network unit program counter 3912, looping The value of the counter 3914 and the number of iteration counters 3916. The value of the program counter field 3912 reflects the value of the program counter 3802 in the thirty-eighth figure. The value of the loop counter field 3914 reflects the value of the loop counter 3804. The value of the iteration count counter field 3916 reflects the value of the iteration count counter 3806. In one embodiment, the sequencer 128 updates the program counter field 3912, the loop counter field 3914 and the iteration count counter column each time the program counter 3802, loop counter 3804, or iteration number counter 3806 needs to be adjusted. The value of bit 3916, so that when the architecture program reads it, the value of these fields will be the current value. In another embodiment, when the neural network unit 121 executes the architectural instructions to read the status register 127, the neural network unit 121 only takes the value of the program counter 3802, the loop counter 3804 and the iteration count counter 3806 and It provides back architecture instructions (eg, to media register 118 or general purpose register 116).

It can be seen that the value of the field of the state register 127 of the thirty-ninth figure can be understood as the information of the progress of the execution of the non-architectural instruction by the neural network unit. Some specific aspects of the execution schedule of the non-architecture program, such as the program counter 3802 value, the loop counter 3804 value, the iteration count counter 3806 value, and the most recent read/write weight random access memory 124 address 125 field. 2602/2604, and recently read/write data random access memory The 122 address 123 field 2606/2608 has been described in the previous section. The architecture program executing on the processor 100 can read the non-architected program progress values of the thirty-ninth map from the state register 127 and use the information to make decisions, such as through architectural instructions such as comparison and branch instructions. For example, the architecture program may determine the data/weight reading/writing queue for the data random access memory 122 and/or the weight random access memory 124 to control the data random access memory 122 or weight. The inflow and outflow of data from random access memory 124, especially for overlapping of large data sets and/or different non-architectural instructions. Examples of these decisions using the architecture program can be found in the previous and subsequent sections of this article.

For example, as described in the foregoing figure 26A, the architecture program sets the non-architecture program to write the result of the convolution operation back to the column above the convolution kernel 2402 in the data random access memory 122 (eg, above column 8). When the neural network unit 121 writes the result using the address of the recently written data random access memory 122 column 2606, the architecture program reads the result from the data random access memory 122.

In another example, as described in the foregoing section 26B, the architecture program uses the information from the field register 127 field of the thirty-eighth figure to confirm that the non-architecture program is the data array 2404 of the twenty-fourth figure. Divided into five 512 x 1600 data blocks to perform the progress of the convolution operation. The architecture program writes the first 512 x 1600 data block of the 2560 x 1600 data array into the weighted random access memory 124 and initiates the non-architected program with a loop count of 1600 and an output of the weighted random access memory 124 initialization. Listed as 0. When the neural network unit 121 executes the non-architecture program, the architecture program reads the status register 127 to confirm the weight random access memory 124. The column 2602 is recently written so that the architecture program can read the result of the effective convolution operation written by the non-architected program, and overwrite the result of the effective convolution operation with the next 512 x 1600 data block after reading, thus, After the neural network unit 121 completes execution of the non-architected program for the first 512 x 1600 data block, the processor 100 can immediately update the non-architected program and restart the non-architected program to execute the next 512 x 1600 data if necessary. Piece.

In another example, assume that the architectural program causes the neural network unit 121 to perform a series of typical neural network multiply-accumulate start functions, wherein the weights are stored in the weighted random access memory 124 and the results are written back to the data randomly. The memory 122 is accessed. In this case, the architecture program reads the weight of one of the random access memories 124 and then does not read it. Thus, after the current weight has been read/used by the non-architect program, the architecture program can be used to begin to overwrite the weights on the weighted random access memory 124 to provide an example of the non-architectural program ( For example, the next neural network layer). In this case, the architecture program reads the state register 127 to obtain the address of the most recent read column 2604 of the weighted random access memory to determine its write of the new weight in the weighted random access memory 124. position.

In another example, assume that the architectural program knows that the non-architected program includes an execution instruction with a large number of iteration counts, such as the non-architectural multiply-accumulate instruction of address 2 in the twentieth diagram. In this case, the architecture program needs to know the iteration count of 3916 to know how many time-frequency cycles are needed to complete the non-architectural instruction to determine which of the two or more actions the architecture program will take next. . Lift For example, if it takes a long time to complete execution, the architecture program will give up control to another architecture program, such as the operating system. Similarly, assume that the architecture program knows that the non-architecture program includes a loop group with a fairly large loop count, such as the non-architected program of Figure 28. In this case, the architecture program will need to know the loop count 3914 to know how many time-frequency cycles are needed to complete the non-architectural instruction to determine which of the two or more actions to take next.

In another example, it is assumed that the architectural program causes the neural network unit 121 to perform a common source operation similar to that described in the twenty-seventh and twenty-eighthth embodiments, wherein the data to be co-sourced is stored in the weighted random access memory 124. The result is written back to the weighted random access memory 124. However, unlike the examples of the twenty-seventh and twenty-eighth diagrams, it is assumed that the results of this example are written back to the top 400 columns of the weighted random access memory 124, such as columns 1600 through 1999. In this case, after the non-architecture program finishes reading the data of the weighted random access memory 124 of the four common sources, the non-architected program will not read it again. Therefore, once the current four columns of data have been read/used by the non-architected program, the architecture program can be used to start the new data (such as the weight of the next paradigm of the non-architected program, for example, for example, the typical execution of the data. The multiplication accumulates the non-architectural program of the function operation to overwrite the data of the weighted random access memory 124. In this case, the architecture program reads the state register 127 to obtain the address of the most recent read column 2604 of the weighted random access memory to determine the location of the new weight rewrite write weight random access memory 124. .

Time recurrent neural network acceleration

Traditional feedforward neural networks do not have a storage network previously Input memory. The feedforward neural network is typically used to perform multiple tasks in which the input to the network over time is independent of each other, and multiple outputs are also the same task. In contrast, time-recurrent neural networks often help to perform tasks that are important in the order in which tasks are input to the neural network over time. (The order here is often referred to as the time step.) Thus, the time recurrent neural network includes a conceptual memory or internal state that loads the information generated by the network in response to calculations performed by previous inputs in the sequence. The output of the time recurrent neural network is associated with the input of this internal state and the next time step. The following tasks, such as speech recognition, language modeling, text generation, language translation, image description generation, and some forms of handwriting recognition, are good examples of time recursive neural networks that can perform well.

Examples of three conventional time recurrent neural networks are the Elman time recurrent neural network, the Jordan time recurrent neural network and the long and short term memory (LSTM) neural network. The Elman time recurrent neural network contains content nodes to remember the hidden layer state of the time recurrent neural network in the current time step, which will be used as input to the hidden layer in the next time step. The Jordan time recurrent neural network is similar to the Elman time recurrent neural network except that the content nodes remember the output layer state of the time recurrent neural network rather than the hidden layer state. Long- and short-term memory neural networks include long-term and short-term memory layers composed of long- and short-term memory cells. Each long-term and short-term memory cell has one of the current time steps of the current state with a current output, and one of the new or subsequent time steps of the new state with a new output. The long-term and short-term memory cells include an input gate and an output gate, and a forgotten gate, which can cause the neuron to lose its memory. state. These three time recurrent neural networks are described in more detail in subsequent chapters.

As described herein, for a time recurrent neural network, such as the Elman or Jordan time recurrent neural network, the neural network unit uses a time step each time to perform a set of input layer node values and perform the necessary calculations. It propagates through a time recurrent neural network to produce output layer node values as well as hidden layer and content layer node values. Thus, the input layer node value is associated with the time step of computing the hidden, outputting the value with the content layer node; while the hidden, output and content layer node values are associated with the time step of generating these node values. The input layer node values are samples of systems simulated by the time recurrent neural network, such as images, speech samples, and snapshots of commercial market data. For long- and short-term memory neural networks, each execution of the neural network unit uses a time step to obtain a set of memory cell input values and perform the necessary calculations to generate memory cell output values (as well as memory cell states and input gates, The forgetting gate and the output gate value) can also be understood as the memory cell input value through the long-term and short-term memory layer memory. Therefore, the memory cell input value is associated with the time step of calculating the memory cell state and the input gate, the forgetting gate and the output gate value; and the memory cell state and the input gate, the forgetting gate and the output gate value are associated with the time at which these node values are generated. step.

The content layer node value, also known as the state node, is the state value of the neural network based on the input layer node values associated with the previous time step, and not only the input layer node values associated with the current time step. The calculation performed by the neural network unit for the time step (eg for hidden layer nodes of the Elman or Jordan time recurrent neural network) The value calculation is a function of the content layer node value generated by the previous time step. Therefore, the network status value (content node value) at the beginning of the time step affects the output layer node value generated during this time step. In addition, the network status value at the end of the time step is affected by the input node value for this time step and the network status value at the beginning of the time step. Similarly, for long- and short-term memory cells, the memory cell state value is associated with the memory cell input value of the previous time step, rather than just the memory cell input value of the current time step. Since the calculation of the neural network unit for the time step execution (eg, the calculation of the next memory cell state) is a function of the memory cell state value generated by the previous time step, the network state value (memory cell state value) at the beginning of the time step will The memory cell output value generated in this time step is affected, and the network state value at the end of this time step is affected by the memory cell input value and the previous network state value of this time step.

The fortieth figure is a block diagram showing an example of an Elman time recurrent neural network. The Elman time recurrent neural network of the fortieth figure includes input layer nodes, or neurons, denoted as D0, D1 to Dn, collectively referred to as multiple input layer nodes D and individually referred to as input layer nodes D; hidden layer nodes/ Neurons, labeled Z0, Z1 to Zn, collectively referred to as multiple hidden layer nodes Z and individually referred to as hidden layer nodes Z; output layer nodes/neurons, labeled Y0, Y1 to Yn, collectively referred to as multiple outputs Layer nodes Y are collectively referred to as output layer nodes Y; and content layer nodes/neurons, designated C0, C1 through Cn, collectively referred to as multiple content layer nodes C and individually referred to as content layer nodes C. In the example of the Elman time recurrent neural network in the fortieth figure, each hidden layer node Z has an input link. To the output of each input layer node D, and having an input coupled to the output of each content layer node C; each output layer node Y has an input coupled to the output of each hidden layer node Z; and each content layer node C has an input Link to the output of a corresponding hidden layer node Z.

In many ways, the operation of the Elman time recurrent neural network is similar to the traditional feed-forward artificial neural network. That is to say, for a given node, each input link of this node will have an associated weight; the value received by the node in an input link will be multiplied by the associated weight to generate a product; this node will Adding the product associated with all input joins produces a total (this offset may also contain an offset); in general, a total is also executed for this total to generate the output value of the node. Sometimes referred to as the startup value of this node. For traditional feedforward networks, data always flows in the direction from the input layer to the output layer. That is, the input layer provides a value to the hidden layer (usually there are multiple hidden layers), while the hidden layer produces its output value to the output layer, and the output layer produces an output that can be fetched.

However, unlike the traditional feedforward network, the Elman time recurrent neural network also includes some feedback links, that is, the connection from the hidden layer node Z to the content layer node C in the fortieth figure. The operation of the Elman time recurrent neural network is as follows. When the input layer node D provides an input value to the hidden layer node Z in a new time step, the content node C provides a value to the hidden layer Z, which is the hidden layer node Z. The output value in response to the previous input, which is the current time step. In this sense, the content node C of the Elman time recursive neural network is a memory based on the input values of the previous time steps. The forty-first and forty-two maps will An operational embodiment of a neural network unit 121 that performs calculations associated with the Elman time recurrent neural network of the fortieth map will be described.

To illustrate the present invention, an Elman time recurrent neural network is a time recurrent neural network comprising at least one input node layer, one hidden node layer, one output node layer and one content node layer. For a given time step, the content node layer stores the results of the hidden node layer generated in the previous time step and fed back to the content node layer. The result of this feedback to the content layer can be the result of the execution of the startup function or the result of the hidden node layer performing the accumulation operation without executing the startup function.

The forty-first diagram is a block diagram showing the data random access memory 122 and the weight of the neural network unit 121 when the neural network unit 121 performs the calculation associated with the Elman time recurrent neural network of the fortieth map. An example of data configuration within random access memory 124. In the example of the 41st graph, it is assumed that the Elman time recurrent neural network of the fortieth graph has 512 input nodes D, 512 hidden nodes Z, 512 content nodes C, and 512 output nodes Y. In addition, it is also assumed that the Elman time recurrent neural network is fully connected, that is, all 512 input nodes D are connected to each hidden node Z as an input, and all 512 content nodes C are connected to each hidden node Z as an input, and all 512 The hidden nodes Z are each connected to each output node Y as an input. Moreover, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, such as a widened configuration. Finally, this example assumes that the weights associated with the links from content node C to hidden node Z are all a value of 1, so that these weight values are not required to be stored.

As shown in the figure, weight random access memory 124 The lower 512 columns (columns 0 to 511) are loaded with weight values associated with the connections between the input node D and the hidden node Z. More precisely, as shown in the figure, column 0 is loaded with the weight associated with the input connection from the input node D0 to the hidden node Z, that is, the text 0 is loaded with the link between the input node D0 and the hidden node Z0. The weight of the text 1 will load the weight associated with the link between the input node D0 and the hidden node Z1, the text 2 will load the weight associated with the link between the input node D0 and the hidden node Z2, and so on, the text 511 will load the association. The weight of the connection between the input node D0 and the hidden node Z511; the column 1 is loaded with the weight associated with the input connection from the input node D1 to the hidden node Z, that is, the text 0 is loaded and associated with the input node D1 and the hidden node Z0. The weight of the link between the two, the text 1 will load the weight associated with the connection between the input node D1 and the hidden node Z1, the text 2 will load the weight associated with the connection between the input node D1 and the hidden node Z2, and so on, the text 511 The weight associated with the connection between the input node D1 and the hidden node Z511 is loaded; until column 511, the column 511 is loaded with the weight associated with the input connection from the input node D511 to the hidden node Z, That is, the text 0 will load the weight associated with the connection between the input node D511 and the hidden node Z0, and the text 1 will load the weight associated with the connection between the input node D511 and the hidden node Z1, and the text 2 will be loaded with the input node D511 and The weight of the link between the nodes Z2 is hidden, and so on, the text 511 loads the weight associated with the link between the input node D511 and the hidden node Z511. This configuration and use is similar to the embodiment described above corresponding to Figures 4-6A.

As shown in the figure, the subsequent 512 columns of weight random memory 124 (columns 512 through 1023) are loaded in a similar manner to The weight of the connection between node Z and output node Y is hidden.

The data random access memory 122 is loaded with Elman time recursive neural network node values for use in a series of time steps. Further, the data random access memory 122 provides a node value for a given time step in three columns as a group load. As shown in the figure, taking a data random access memory 122 having 64 columns as an example, the data random access memory 122 can load node values for use in 20 different time steps. In the example of the 41st graph, columns 0 through 2 are loaded with node values for time step 0, columns 3 through 5 are loaded with node values for time step 1, and so on, columns 57 through 59 are loaded for time steps. 19 node value used. The first column in each group is loaded with the value of input node D for this time step. The second column in each group loads the value of the hidden node Z for this time step. The third column in each group loads the value of the output node Y for this time step. As shown in the figure, each row of data random access memory 122 loads the node value of its corresponding neuron or neural processing unit 126. That is to say, row 0 is loaded with node values associated with nodes D0, Z0 and Y0, and its calculation is performed by neural processing unit 0; row 1 is loaded with node values associated with nodes D1, Z1 and Y1, and the calculation is Executed by the neural processing unit 1; and so on, the row 511 is loaded with node values associated with nodes D511, Z511 and Y511, the calculation of which is performed by the neural processing unit 511, which in the subsequent corresponds to the forty-second map There will be more detailed instructions.

As indicated in the 41st figure, for a given time step, the value of the hidden node Z in the second column of each group of three columns of memory will be the value of the content node C of the next time step. That is, the node that the neural processing unit 126 calculates and writes in a time step The value of Z will be the value of node C used by this neural processing unit 126 to calculate the value of node Z in the next time step (along with the value of input node D for this next time step). The initial value of the content node C (the value of the node C used to calculate the value of the node Z in the column 1 at time step 0) is assumed to be zero. This will be explained in more detail in the subsequent sections of the non-architected program corresponding to the 42nd figure.

Preferably, the value of the input node D (column 0, 3 in the example of the 41st graph, and so on to the value of column 57) is written by the architecture program executed by the processor 100 through the MTNN instruction 1400 / The data random access memory 122 is populated and is read/used by a non-architected program executed by the neural network unit 121, such as the non-architected program of the forty-second graph. Conversely, the value of the hidden/output node Z/Y (columns 1 and 2, 4 and 5 in the example of the 41st graph, and so on to the values of columns 58 and 59) are performed by the neural network. The non-architectural program of unit 121 writes/fills data random access memory 122 and is read/used by MFNN instruction 1500 by an architectural program executing on processor 100. The example in the 41st figure assumes that the architecture program performs the following steps: (1) For 20 different time steps, the value of the input node D is filled in the data random access memory 122 (column 0, 3, This type is pushed to column 57); (2) to start the non-architecture program of the 42nd chart; (3) to detect whether the non-architecture program is executed; (4) to read the output node Y from the data random access memory 122. The values (columns 2, 5, and so on to column 59); and (5) repeat steps (1) through (4) several times until the task is completed, such as the calculation required to identify the phone user's words.

In another implementation, the architecture program will execute The following steps: (1) For a single time step, fill in the data random access memory 122 (such as column 0) with the value of the input node D; (2) start the non-architecture program (the 42nd non-architectural program) A modified version, no loops are required, and only a single set of three columns of data storage memory 122 are accessed; (3) detecting whether the non-architected program is executed; (4) random access memory from the data 122 reads the value of the output node Y (e.g., column 2); and (5) repeats steps (1) through (4) several times until the task is completed. Which of the two methods is preferred depends on the sampling method of the input value of the time recurrent neural network. For example, if this task allows sampling of inputs (eg, about 20 time steps) and performing calculations in multiple time steps, the first approach is ideal because it may result in more computational resource efficiency and / Or better performance, however, if this task only allows sampling in a single time step, the second method is required.

The third embodiment is similar to the second method described above. However, unlike the second method, a single group of three columns of data random access memory 122 is used. In this manner, the non-architectural program uses multiple sets of three columns of memory, that is, Each time step uses a different set of three columns of memory, which is similar to the first. In this third embodiment, preferably, the architecture program includes a step before step (2), in which the architecture program updates the non-architecture program before starting, for example, the instruction of the address 1 is The data random access memory 122 column is updated to point to the next set of three columns of memory.

The forty-second figure is a table showing a program stored in the program memory 129 of the neural network unit 121. The program is executed by the neural network unit 121, and the data is used according to the configuration of the 41st figure. Weights to achieve the Elman time recurrent neural network. Some of the instructions in the non-architecture of Figure 42 (and the forty-fifth, forty-eight, fifty-one, fifty-four and fifty-seven) are as described above (eg multiply accumulate (MULT-ACCUM), loop (LOOP), INITIALIZE instruction, the following paragraphs assume that these instructions are consistent with the foregoing description, unless otherwise stated.

The sample program in Figure 42 contains 13 non-architected instructions located at addresses 0 through 12. The instruction of address 0 (INITIALIZE NPU, LOOPCNT = 20) clears the accumulator 202 and initializes the loop counter 3804 to a value of 20 to perform 20 loop groups (instructions of addresses 4 through 11). Preferably, the initialization command also causes the neural network unit 121 to be in a wide configuration. Thus, the neural network unit 121 is configured as 512 neural processing units 126. As described in the subsequent sections, during the execution of the instructions of addresses 1 through 3 and addresses 7 through 11, the 512 neural processing units 126 operate as 512 corresponding hidden layer nodes Z, and at address 4 During the execution of the instructions up to 6, the 512 neural processing units 126 operate as 512 corresponding output layer nodes Y.

The instructions of addresses 1 through 3 are not part of the program's loop group and will only be executed once. These instructions calculate the initial value of the hidden layer node Z and write it to column 1 of the data random access memory 122 for the first execution of the instructions of addresses 4 through 6 to calculate the first time step (time step) 0) Output layer node Y. In addition, the values of the hidden layer nodes Z calculated by the instructions of the addresses 1 to 3 and written into the column 1 of the data random access memory 122 become the value of the content layer node C for the addresses of the addresses 7 and 8. Use once to calculate the value of the hidden layer node Z for the second The time step (time step 1) is used.

During the execution of the instructions of addresses 1 and 2, each of the 512 neural processing units 126 performs 512 multiplication operations, placing 512 input nodes in column 0 of the data random access memory 122. The D value is multiplied by the weight of the row corresponding to the neural processing unit 126 in columns 0 to 511 of the weighted random access memory 124 to generate 512 multiply accumulators for the accumulator 202 of the corresponding neural processing unit 126. During the execution of the instruction of address 3, the values of the 512 accumulators 202 of the 512 neural processing units are passed and written to column 1 of the data random access memory 122. That is to say, the output instruction of address 3 writes the value of the accumulator 202 of each of the 512 neural processing units into the data random access memory 122, which is the initial hidden layer. The Z value, then, this instruction clears the accumulator 202.

The operations performed by the instructions of addresses 1 through 2 of the non-architected program of the 42nd graph are similar to those performed by the instructions of addresses 1 through 2 of the non-architectural instructions of the fourth figure. Further, the instruction of address 1 (MULT_ACCUM DR ROW 0) indicates that each of the 512 neural processing units 126 reads the corresponding text of column 0 of the data random access memory 122 into it. The scratchpad 208 reads the corresponding text of the column 0 of the weight random access memory 124 into its multiplex register 705, multiplies the data text by the weight text to generate a product, and adds the product to the accumulator 202. The instruction of address 2 (MULT-ACCUM ROTATE, WR ROW +1, COUNT=511) indicates that each of the 512 neural processing units 126 transfers the text from the adjacent neural processing unit 126 to its multiplex 208 (utilizing 512 by neural network unit 121 The multiplexer 208 collectively operates to form a 512-character rotator, which is the register of the address 1 indicating that the data random access memory 122 is read into the register, and the weight is The corresponding text in a column below the random access memory 124 is read into its multiplex register 705, the data word is multiplied by the weight text to produce a product, and the product is added to the accumulator 202, and the foregoing operation is performed 511 times.

In addition, the single non-architectural output instruction (OUTPUT PASSTHRU, DR OUT ROW 1, CLR ACC) of address 3 in Figure 42 will write the operation of the start function instruction and the write output of addresses 3 and 4 in the fourth figure. The instructions are merged (although the program of the 42nd graph transfers the value of the accumulator 202, and the program of the fourth graph executes a start function for the accumulator 202 value). That is, in the program of the 42nd graph, the start function of the value of the accumulator 202, if any, is specified in the output instruction (also specified in the output instructions of addresses 6 and 11), and Not shown in the program in Figure 4 is specified in a different non-architectural start function instruction. Another embodiment of the non-architectural program of the fourth figure (and the twentieth, twenty-sixth and twenty-eighth) is also about to start the operation of the function instruction and the write output instruction (such as the address of the fourth figure 3 It is also within the scope of the present invention to incorporate 4) into a single non-architectural output instruction as shown in FIG. The example in Figure 42 assumes that the node of the hidden layer (Z) does not perform a start function on the accumulator value. However, the embodiment in which the hidden layer (Z) performs the start function on the accumulator value is also within the scope of the present invention. These embodiments can perform operations using the instructions of addresses 3 and 11, such as S-type, hyperbolic tangent, correction function, etc. .

The instruction will only be executed once compared to the address 1 to 3. The instructions of addresses 4 through 11 are located within the program loop and are executed a number of times, as specified by the loop count (for example, 20). The first nine executions of the instructions of addresses 7 through 11 calculate the value of the hidden layer node Z and write it to the data random access memory 122 for the second to twenty executions of the instructions of addresses 4 through 6. To calculate the output layer node Y of the remaining time step (time steps 1 to 19). (The last/twentieth execution of the instructions of addresses 7 through 11 calculates the value of the hidden layer node Z and writes it to column 61 of the data random access memory 122, however, these values are not used.)

In the first execution of the instructions of addresses 4 and 5 (MULT-ACCUM DR ROW +1, WR ROW 512 and MULT-ACCUM ROTATE, WR ROW +1, COUNT=511) (corresponding to time step 0), this Each of the 512 neural processing units 126 performs 512 multiplication operations to randomly store the values of the 512 hidden nodes Z of the column 1 of the memory 122 (these values are from addresses 1 to 3). The single execution of the instruction is generated and written. The weights of the rows corresponding to the neural processing unit 126 in the columns 512 to 1023 of the weight random access memory 124 are generated to generate 512 multiplication accumulations for the corresponding neural processing unit 126. Accumulator 202. In the first execution of the instruction of address 6 (OUTPUT ACTIVATION FUNCTION, DR OUT ROW +1, CLR ACC), a start function (eg S-type, hyperbolic tangent, correction function) is executed for the 512 accumulated values. To calculate the value of the output layer node Y, the execution result is written to column 2 of the data random access memory 122.

In the second execution of the instructions of addresses 4 and 5 (corresponding to time step 1), each of the 512 neural processing units 126 performs 512 multiplication operations to randomize the data. The value of the 512 hidden nodes Z of column 4 of the body 122 (these values are generated and written by the first execution of the instructions of the addresses 7 to 11) is multiplied by the column 512 of the weighted random access memory 124 to The weight of the row corresponding to the neural processing unit 126 in 1023 is generated to generate 512 multiply accumulates for the accumulator 202 of the corresponding neural processing unit 126, and in the second execution of the instruction of the address 6, the 512 The accumulated value performs a start function to calculate the value of the output layer node Y, which is written to column 5 of the data random access memory 122; in the third execution of the instructions of addresses 4 and 5 (corresponding to the time step) 2), each of the 512 neural processing units 126 performs 512 multiplication operations to randomly access the values of the 512 hidden nodes Z of the column 7 of the data 122 (these values are determined by the address) The second execution of the instructions from 7 to 11 is generated and written. The weights of the rows corresponding to the neural processing unit 126 in the columns 512 to 1023 of the weighted random access memory 124 are multiplied to generate 512 multiply accumulates for the phase. Corresponding to the accumulator 202 of the neural processing unit 126, and in place In the third execution of the instruction of 6, a start function is executed for the 512 accumulated values to calculate the value of the output layer node Y, and the result is written into the column 8 of the data random access memory 122; and so on. In the twentieth execution of the instructions of addresses 4 and 5 (corresponding to time step 19), each of the 512 neural processing units 126 performs 512 multiplication operations to randomize the data to the memory. The value of 512 hidden nodes Z of column 58 of body 122 (these values are generated and written by the nineteenth execution of the instructions of addresses 7 through 11) multiplied by column 512 of weighted random access memory 124 to The weight corresponding to the row of the neural processing unit 126 in 1023 is generated to generate 512 multiply accumulates to the accumulator 202 of the corresponding neural processing unit 126, and the reference at address 6 For the twentieth execution, a start function is executed for the 512 accumulated values to calculate the value of the output layer node Y, and the execution result is written to the column 59 of the data random access memory 122.

In the first execution of the instructions of addresses 7 and 8, each of the 512 neural processing units 126 accumulates the values of the 512 content nodes C of the data array 1 of the memory 122. Accumulator 202, these values are generated by a single execution of the instructions of addresses 1 through 3. Further, the instruction of address 7 (ADD_D_ACC DR ROW +0) will indicate that each of the 512 neural processing units 126 will store the current column of the data random access memory 122 (in the first execution process) The corresponding text in column 0) is read into its multiplex register 208 and the text is added to accumulator 202. The instruction of address 8 (ADD_D_ACC ROTATE, COUNT=511) indicates that each of the 512 neural processing units 126 transfers the text from the adjacent neural processing unit 126 to its multiplex register 208 (using The 512 multiplex registers 208 of the neural network unit 121 collectively operate to form a 512-word rotator, and the multiplex registers are the instructions of the address 7 to read the data random access memory 122. The register is added to the accumulator 202 and the aforementioned operation is performed 511 times.

In the second execution of the instructions of addresses 7 and 8, each of the 512 neural processing units 126 will accumulate the values of the 512 content nodes C of the data 4 of the memory 122. To its accumulator 202, these values are generated and written by the first execution of the instructions of addresses 9 through 11; in the third execution of the instructions of addresses 7 and 8, the 512 neural processing units 126 Individual nerve processing The unit 126 will accumulate the values of the 512 content nodes C of the data row 7 of the memory 122 to its accumulator 202, which are generated and written by the second execution of the instructions of addresses 9 through 11. And so on, in the twentieth execution of the instructions of addresses 7 and 8, each of the 512 neural processing units 126 will randomly fetch the data into 512 of the columns 58 of the memory 122. The value of the content node C is accumulated to its accumulator 202, which is generated and written by the nineteenth execution of the instructions of addresses 9 through 11.

As described above, the example of the forty-second diagram assumes that the weight associated with the link of the content node C to the hidden layer node Z has a value of one. However, in another embodiment, the links within the Elman time recurrent neural network have non-zero weight values that are placed in the weighted random access memory 124 prior to execution of the program in the forty-second graph. (For example, columns 1024 to 1535), the program instructions for address 7 are MULT-ACCUM DR ROW+0, WR ROW 1024, and the program instructions for address 8 are MULT-ACCUM ROTATE, WR ROW +1, COUNT=511. Preferably, the instruction of address 8 does not access the weighted random access memory 124, but the instruction of the rotated address 7 reads the value of the multiplexed register 705 from the weighted random access memory 124. The bandwidth is not accessed during the time-frequency period of the 511 execution address 8 instructions to reserve more bandwidth for use by the architectural program access weight random access memory 124.

In the first execution of the instructions of addresses 9 and 10 (MULT-ACCUM DR ROW +2, WR ROW 0 and MULT-ACCUM ROTATE, WR ROW +1, COUNT=511) (corresponding to the time step) Step 1), each of the 512 neural processing units 126 performs 512 multiplication operations, multiplying the values of the 512 input nodes D of the column 3 of the data random access memory 122 by weight random access. The columns 0 to 511 of the memory 124 correspond to the weights of the rows of the neural processing unit 126 to generate 512 products, and the accumulating operations performed on the 512 content node C values by the instructions of the addresses 7 and 8 are added to the corresponding The accumulator 202 of the neural processing unit 126 calculates the value of the hidden layer node Z. In the first execution of the instruction of the address 11 (OUTPUT PASSTHRU, DR OUT ROW +2, CLR ACC), the 512 neural processing units 126 The 512 accumulators 202 values are passed and written to column 4 of the data random access memory 122, and the accumulator 202 is cleared; in the second execution of the instructions of addresses 9 and 10 (corresponding to the time step) 2), each of the 512 neural processing units 126 performs 512 multiplication operations, multiplying the values of the 512 input nodes D of the column 6 of the data random access memory 122 by the weight random access memory. Corresponding to this nerve in the column 124 to 511 The weight of the row of the unit 126 is generated to generate 512 products, and the accumulating operations performed on the 512 content node C values by the instructions of the addresses 7 and 8 are accumulated in the accumulator 202 of the corresponding neural processing unit 126 to calculate the hidden The value of the layer node Z, in the second execution of the instruction of the address 11, the 512 accumulators 202 of the 512 neural processing units 126 are passed and written to the column 7 of the data random access memory 122, and The accumulator 202 will be cleared; and so on, in the nineteenth execution of the instructions of addresses 9 and 10 (corresponding to time step 19), each of the 512 neural processing units 126 will Performing 512 multiplication operations, 512 input sections of column 57 of data random access memory 122 The value of point D is multiplied by the weight of the row corresponding to the neural processing unit 126 in columns 0 to 511 of the weighted random access memory 124 to generate 512 products, together with the instructions of addresses 7 and 8 for 512 content nodes C. The accumulation operation performed by the value is accumulated in the accumulator 202 of the corresponding neural processing unit 126 to calculate the value of the hidden layer node Z, and in the nineteenth execution of the instruction of the address 11, the 512 neural processing unit 126 The 512 accumulator 202 values are passed and written to column 58 of data random access memory 122, and accumulator 202 is cleared. As previously mentioned, the value of the hidden layer node Z generated and written in the twentieth execution of the instructions of addresses 9 and 10 is not used.

The instruction of address 12 (LOOP 4) decrements the loop counter 3804 and returns to the instruction of address 4 if the new loop counter 3804 has a value greater than zero.

A forty-third figure is a block diagram showing an example of a Jordan time recurrent neural network. The 43rd chart of the Jordan time recurrent neural network is similar to the 40th Elman time recurrent neural network with input layer nodes/neuron D, hidden layer nodes/neurons Z, output layer nodes/neurons Y , with content layer nodes / neurons C. However, in the Jordan time recurrent neural network of the 43rd graph, the content layer node C uses the output feedback from its corresponding output layer node Y as its input link instead of the Elman time recursion as in the fortieth chart. The neural network is the output from the hidden layer node Z as its input link.

To illustrate the present invention, the Jordan Time Recurrent Neural Network is a time recurrent neural network comprising at least one input node layer, one hidden node layer, one output node layer and one content node layer. road. At the beginning of a given time step, the content node layer stores the results of the output node layer generated in the previous time step and fed back to the content node layer. The result of this feedback to the content layer can be the result of the startup function or the result of the output node performing the accumulation operation without executing the startup function.

The forty-fourth figure is a block diagram showing the data random access memory 122 of the neural network unit 121 when the neural network unit 121 performs the calculation of the Jordan time recurrent neural network associated with the forty-third figure. An example of data configuration within weight random access memory 124. In the example of the forty-fourth figure, it is assumed that the Jordan time recurrent neural network of the forty-third figure has 512 input nodes D, 512 hidden nodes Z, 512 content nodes C, and 512 output nodes Y. In addition, it is also assumed that the Jordan time recurrent neural network is fully connected, that is, all 512 input nodes D are connected to each hidden node Z as an input, and all 512 content nodes C are connected to each hidden node Z as an input, and all 512 The hidden nodes Z are each connected to each output node Y as an input. The example of the Jordan time recurrent neural network of the forty-fourth figure applies a start function to the value of the accumulator 202 to generate the value of the output layer node Y. However, this example assumes that the accumulation function will be applied before the start function. The value of the 202 is passed to the content layer node C instead of the true output layer node Y value. In addition, the neural network unit 121 is provided with 512 neural processing units 126, or neurons, for example, in a wide configuration. Finally, this example assumes that the weights associated with the links from content node C to hidden node Z all have a value of one; thus, it is not necessary to store these as a weight value.

As in the example of the 41st figure, as shown in the figure, the lower 512 columns (columns 0 to 511) of the weighted random access memory 124 are loaded. The weight value associated with the connection between the input node D and the hidden node Z is loaded, and the subsequent 512 columns (columns 512 to 1023) of the weighted random access memory 124 are loaded with links between the hidden node Z and the output node Y. The weight value.

The data random access memory 122 is loaded with Jordan time recursive neural network node values for use in a series of time steps similar to the example of the 41st graph; however, the example of the forty-fourth graph is a set of four The column's memory load provides the node value for a given time step. As shown in the figure, in an embodiment having 64 columns of data random access memory 122, data random access memory 122 can carry node values required for 15 different time steps. In the example of the forty-fourth figure, columns 0 to 3 are loaded with node values for time step 0, columns 4 to 7 are loaded with node values for time step 1, and so on, columns 60 to 63 are loaded. The node value used for time step 15. The first column of the four columns of memory is loaded with the value of input node D for this time step. The second column of the four columns of memory loads the value of the hidden node Z for this time step. The third column of the four columns of memory is loaded with the value of the content node C of this time step. The fourth column of the four columns of memory is the value of the output node Y that is loaded for this time step. As shown in the figure, each row of data random access memory 122 loads the node value of its corresponding neuron or neural processing unit 126. That is to say, row 0 is loaded with node values associated with nodes D0, Z0, C0 and Y0, and its calculation is performed by neural processing unit 0; row 1 is loaded with node values associated with nodes D1, Z1, C1 and Y1, The calculation is performed by the neural processing unit 1; and so on, the row 511 is loaded with the node values associated with the nodes D511, Z511, C511 and Y511, the calculation is performed by The neural processing unit 511 performs. This section will be described in more detail later in the corresponding figure 44.

The value of the content node C for a given time step in Figure 44 is generated during this time step and is entered as the next time step. That is, the value of the node C calculated and written by the neural processing unit 126 during this time step becomes the value of the node C used by the neural processing unit 126 to calculate the value of the node Z in the next time step ( Together with the value of input node D for this next time step). The initial value of the content node C (i.e., the value of the node C used to calculate the value of the node 1 node Z at time step 0) is assumed to be zero. This section will be described in more detail in the subsequent sections of the non-architected program corresponding to the 45th.

As described in the foregoing 41st, preferably, the value of the input node D (columns 0, 4 in the example of the forty-fourth graph, and so on, to the value of the column 60) is executed by the processor 100. The architecture program writes/fills the data random access memory 122 through the MTNN command 1400 and is read/used by a non-architected program executed by the neural network unit 121, such as the non-architected program of the forty-fifth figure. Conversely, the value of the hidden node Z/content node C/output node Y (in the example of the forty-fourth figure is column 1/2/3, 5/6/7, and so on, to column 61/62/63) The value is written/filled into the data random access memory 122 by the non-architectural program executed by the neural network unit 121, and is read/used by the architecture program executed by the processor 100 through the MFNN instruction 1500. The example in the forty-fourth figure assumes that the architecture program performs the following steps: (1) For 15 different time steps, the value of the input node D is filled in the data random access. Memory 122 (columns 0, 4, and so on to column 60); (2) start the non-architectural program of the 45th figure; (3) detect whether the non-architected program is executed; (4) store the data randomly The memory 122 reads the value of the output node Y (columns 3, 7, and so on to column 63); and (5) repeats steps (1) through (4) several times until the task is completed, such as for a mobile phone user. The calculations required for the discourse to identify.

In another implementation, the architecture program performs the following steps: (1) for a single time step, filling the data random access memory 122 (such as column 0) with the value of the input node D; (2) starting the non-architectural The program (the revised version of the non-architected program in the forty-fifth figure does not require loops, and only accesses a single group of four columns of data storage memory 122); (3) detects whether the non-architected program is executed. (4) reading the value of the output node Y from the data random access memory 122 (e.g., column 3); and (5) repeating steps (1) through (4) several times until the task is completed. Which of the two methods is preferred depends on the sampling method of the input value of the time recurrent neural network. For example, if this task allows sampling of inputs (for example, about 15 time steps) and performing calculations in multiple time steps, the first method is ideal because it can bring more computing resource efficiency and / or better performance, however, if this task only allows sampling in a single time step, you need to use the second method.

The third embodiment is similar to the second method described above. However, unlike the second method, a single group of four data random access memory 122 columns is used. The non-architectural program in this manner uses multiple sets of four columns of memory, that is, Different sets of four columns of memory are used at various time steps, which is similar to the first way. In this third embodiment, preferably, the architecture The program includes a step before the step (2). In this step, the architecture program updates the non-architecture program before starting the non-architecture program, for example, updating the data random access memory 122 column in the instruction of the address 1 to point. The next set of four columns of memory.

The forty-fifth figure is a table showing a program stored in the program memory 129 of the neural network unit 121, which is executed by the neural network unit 121, and uses data and weights according to the configuration of the forty-fourth figure. To achieve the Jordan time recurrent neural network. The non-architectural program of the 45th figure is similar to the non-architectural program of the 42nd figure. The difference between the two can be referred to the description of the relevant section of this article.

The sample program of Figure 45 includes 14 non-architected instructions located at addresses 0 through 13. The instruction of address 0 is an initialization instruction to clear accumulator 202 and initialize loop counter 3804 to a value of 15 to perform 15 loop groups (instructions of addresses 4 through 12). Preferably, this initialization command causes the neural network unit 121 to be in a wide configuration and configured as 512 neural processing units 126. As described herein, during the execution of instructions for addresses 1 through 3 and addresses 8 through 12, the 512 neural processing units 126 correspond to and operate as 512 hidden layer nodes Z, at addresses 4, 5 During execution of the instruction with 7, these 512 neural processing units 126 correspond and operate as 512 output layer nodes Y.

The instructions of addresses 1 through 5 and address 7 are identical to the instructions of addresses 1 through 6 in the forty-second figure and have the same function. The instructions of address 1 to 3 calculate the initial value of the hidden layer node Z and write it to the column 1 of the data random access memory 122 for the first execution of the instructions of the addresses 4, 5 and 7 to calculate The output layer node Y of the first time step (time step 0).

During the first execution of the output instruction of address 6, the 512 accumulators 202 are incremented by the instructions of addresses 4 and 5 (these values are then used by the output instruction of address 7 to calculate And the value written to the output layer node Y) is transferred and written to the data random access memory 122 column 2, these values are the content layer node C values generated in the first time step (time step 0) and Used in the second time step (time step 1); during the second execution of the output instruction of address 6, the 512 accumulators 202 are incremented by the instructions of addresses 4 and 5 (next, These values will be used by the output instruction of address 7 to calculate and write the value of the output layer node Y) and will be passed to and written to column 6 of the data random access memory 122, which is the second time step (time) The content layer node C value generated in step 1) is used in the third time step (time step 2); and so on, during the fifteenth execution of the output instruction of address 6, the 512 The instructions of addresses 4 and 5 accumulate the resulting accumulator 202 values (then these The value will be used by the output instruction of address 7 to calculate and write the value of the output layer node Y) and will be passed to and written to column 58 of data random access memory 122, which is the fifteenth time step (time) The content layer node C value generated in step 14) (and read by the instruction of address 8 but not used).

The instructions of addresses 8 through 12 are substantially identical to the instructions of addresses 7 through 11 in the forty-second figure and have the same function, with only one difference point. The difference is that the instruction of address 8 (ADD_D_ACC DR ROW +1) in the forty-fifth figure increases the number of columns of data random access memory 122 by one, and the instruction of address 7 in the forty-second figure. (ADD_D_ACC DR ROW +0) increments the number of columns of data random access memory 122 by zero. This difference is due to the difference in the configuration of the data in the data random access memory 122. In particular, the configuration of the four columns in the forty-fourth figure includes a separate column for the value of the content layer node C (eg, column 2). , 6, 10, etc.), and the configuration of the three columns in the 41st graph does not have this independent column, but the value of the content layer node C and the value of the hidden layer node Z share the same column (such as column 1,4,7, etc.). Fifteen executions of the instructions of addresses 8 through 12 calculate the value of the hidden layer node Z and write it to the data random access memory 122 (write columns 5, 9, 13 and so on until column 57). The second to sixteenth execution of the instructions of addresses 4, 5 and 7 is used to calculate the output layer node Y of the second to fifteenth time steps (time steps 1 to 14). (The last/fifth execution of the instructions of addresses 8 through 12 calculates the value of the hidden layer node Z and writes it to column 61 of the data random access memory 122, but these values are not used.)

The loop command of address 13 causes the loop counter 3804 to decrement and return to the instruction of address 4 if the new loop counter 3804 has a value greater than zero.

In another embodiment, the design of the Jordan time recurrent neural network utilizes the content node C to load the start function value of the output node Y, which is the accumulated value after the execution of the function. In this embodiment, since the value of the output node Y is the same as the value of the content node C, the non-architectural instruction of the address 6 is not included in the non-architectural program. Therefore, the number of columns used in the data random access memory 122 can be reduced. More precisely, the columns of the values of the loaded content nodes C (e.g., columns 2, 6, 59) in the forty-fourth graph are not present in this embodiment. In addition, this embodiment Each time step requires only three columns of data random access memory 122, and will be paired with 20 time steps instead of 15, and the address of the instruction of the non-architected program in the forty-fifth figure will also be appropriate. Adjustment.

Long-term and short-term memory

The use of long and short term memory cells for time recurrent neural networks is a concept well known in the art. For example, Long Short-Term Memory, Sepp Hochreiter and Jürgen Schmidhuber, Neural Computation, November 15, 1997, Vol. 9, No. 8, Pages 1735-1780; Learning to Forget: Continual Prediction with LSTM, Felix A. Gers Jürgen Schmidhuber, and Fred Cummins, Neural Computation, October 2000, Vol. 12, No. 10, Pages 2451-2471; these documents are available from the MIT Press Journals. Long- and short-term memory cells can be constructed in many different types. The long-short-term memory cell 4600 of the 46th diagram described below is based on the website http://deeplearning.net/tutorial/lstm.html titled LSTM Networks for Sentiment Analysis. The long and short-term memory cells described in the tutorial are models. A copy of this tutorial was downloaded on October 19, 2015 (hereafter referred to as the “long and short-term memory tutorial”) and is provided in the US application data disclosure case in this case. This long and short term memory cell 4600 can be used to generally describe the ability of the neural network unit 121 embodiments described herein to be effective in performing calculations associated with long and short term memory. It should be noted that the embodiments of the neural network unit 121, including the embodiment described in the forty-ninth figure, can effectively perform other long-term and short-term relationships other than the long-term and short-term memory cells described in the forty-sixth figure. Calculation of memory cells.

Preferably, the neural network unit 121 can be used to perform calculations for a time recurrent neural network with long and short term memory cell layers connected to other levels. For example, in this long-term and short-term memory tutorial, the network includes a mean common source layer to receive the long-short-term memory cell output (H) of the long- and short-term memory layer, and a logistic regression layer to receive the output of the mean common source layer. .

The forty-sixth diagram is a block diagram showing one embodiment of long and short term memory cells 4600.

As shown in the figure, the long-and short-term memory cell 4600 includes a memory cell input (X), a memory cell output (H), an input gate (I), an output gate (O), and a forget gate (F). A memory state (C) and a candidate memory state (C'). The input gate (I) can gate the signal input from the memory cell input (X) to the memory cell state (C), while the output gate (O) can gate the signal state of the memory cell state (C) to the memory cell output (H). . This memory cell state (C) is fed back as a candidate memory cell state (C') for a time step. The forgetting gate (F) can gate this candidate memory cell state (C'), and the candidate memory cell state will feed back and become the memory cell state (C) of the next time step.

The embodiment of the forty-sixth embodiment uses the following equations to calculate the various different values described above:

(1) I = SIGMOID (Wi * X + Ui * H + Bi)

(2) F = SIGMOID (Wf * X + Uf * H + Bf)

(3) C'=TANH(Wc * X+Uc * H+Bc)

(4) C=I * C'+F * C

(5)O=SIGMOID(Wo * X+Uo * H+Bo)

(6) H=O * TANH(C)

Wi and Ui are weight values associated with the input gate (I), and Bi is the offset value associated with the input gate (I). Wf and Uf are weight values associated with the forgotten gate (F), and Bf is the offset value associated with the forgotten gate (F). Wo and Uo are weight values associated with the output gate (O), and Bo is the offset value associated with the output gate (O). As described above, equations (1), (2), and (5) calculate the input gate (I), the forget gate (F), and the output gate (O), respectively. Equation (3) calculates the candidate memory cell state (C'), and Equation (4) calculates the candidate memory cell state (C') with the current memory cell state (C) as input, and the current memory cell state (C) Memory cell state (C) of the current time step. Equation (6) calculates the memory cell output (H). However, the invention is not limited thereto. Embodiments for calculating input gates, forgetting gates, output gates, candidate memory cell states, memory cell states, and memory cell output long-term memory cells using other methods are also encompassed by the present invention.

To illustrate the invention, the long and short term memory cells include a memory cell input, a memory cell output, a memory cell state, a candidate memory cell state, an input gate, an output gate, and a forget gate. For each time step, the input gate, output gate, forget gate, and candidate memory cell state are functions of the memory cell input of the current time step and the memory cell output and associated weight of the previous time step. The memory cell state of this time step is a function of the memory cell state of the previous time step, the candidate memory cell state, the input gate and the output gate. In this sense, the state of the memory cell feeds back the state of the memory cell used to calculate the next time step. This time step The memory cell output is a function of the state of the memory cell and the output gate calculated at this time step. The long- and short-term memory neural network is a neural network with a long-term and short-term memory cell layer.

The forty-seventh figure is a block diagram showing the data of the neural network unit 121 when the neural network unit 121 performs the calculation of the long-short-term memory cell 4600 layer associated with the long-short-term memory neural network of the forty-sixth figure. An example of data configuration in random access memory 122 and weighted random access memory 124. In the example of the forty-seventh diagram, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, such as a widened configuration, however, there are only 128 neural processing units 126 (eg, neural processing units 0 to 127) The resulting value will be used because the long and short term memory layer in this example has only 128 long and short term memory cells 4600.

As shown in the figure, the weight random access memory 124 loads the weight values, offset values and intermediate values of the corresponding neural processing units 0 to 127 of the neural network unit 121. The rows 0 to 127 of the weight random access memory 124 load the weight values, offset values and intermediate values of the corresponding neural processing units 0 to 127 of the neural network unit 121. Each of the columns 0 to 14 is loaded with 128 values corresponding to the above equations (1) to (6) to be supplied to the neural processing units 0 to 127, which are: Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, C', TANH(C), C, Wo, Uo, Bo. Preferably, the weight value and the offset value -Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, Wo, Uo, Bo (located in columns 0 to 8 and columns 12 to 14) are executed The architecture program of the processor 100 writes/fills the weight random access memory 124 through the MTNN instruction 1400, and is read/used by the non-architecture program executed by the neural network unit 121, such as the 48th figure. Architecture program. Preferably, living The inter-values -C', TANH(C), C (in columns 9 to 11) are written/filled into the weighted random access memory 124 by the non-architectural program executing in the neural network unit 121 and read/ Use, as described later.

As shown in the figure, data random access memory 122 loads input (X), output (H), input gate (I), forget gate (F) and output gate (O) values for a series of time steps. Further, the five columns of this memory are loaded with the values of X, H, I, F and O for a given time step. Taking a random access memory 122 having 64 columns as an example, as shown in the figure, the data random access memory 122 can carry the memory cell values for use in 12 different time steps. In the example of the 47th figure, columns 0 to 4 are loaded with the memory cell values used for time step 0, columns 5 to 9 are loaded with the memory cell values used for time step 1, and so on, columns 55 to 59 The memory cell value used for time step 11 is loaded. The first of the five columns of memory is loaded with the X value of this time step. The second of the five columns of memory is loaded with the H value for this time step. The third of the five columns of memory is loaded with the I value for this time step. The fourth of the five columns of memory is loaded with the F value for this time step. The fifth of the five columns of memory is loaded with the O value for this time step. As shown in the figure, each line within the data random access memory 122 is loaded with values for use by the corresponding neuron or neural processing unit 126. That is to say, the row 0 is loaded with the value associated with the long and short term memory cell 0, and its calculation is performed by the neural processing unit 0; the row 1 is loaded with the value associated with the long and short term memory cell 1, and its calculation is performed by the nerve The processing unit 1 performs; and so on, the row 127 is loaded with the value associated with the long- and short-term memory cells 127, and the calculation is performed by the neural processing unit 127. As described in the next forty-eighth figure.

Preferably, the X value (in columns 0, 5, 9, and so on to column 55) is written/filled into the data random access memory 122 by the architecture program executed by the processor 100 through the MTNN instruction 1400, and Read/use by a non-architected program executing on neural network unit 121, such as the non-architected program shown in FIG. Preferably, the I value, the F value and the O value (located in columns 2/3/4, 7/8/9, 12/13/14, and so on to column 57/58/59) are performed by neural processing. The non-architectural program of the unit 121 writes/fills the data random access memory 122 as will be described later. Preferably, the H values (in columns 1, 6, 10, and so on to column 56) are written/filled into the random access memory 122 by the non-architectural program executed by the neural processing unit 121 and read. / used, and read by the MFNN instruction 1500 by an architectural program executing on the processor 100.

The example in the 47th figure assumes that the architecture program performs the following steps: (1) For 12 different time steps, the value of the input X is filled in the data random access memory 122 (column 0, 5, according to this) Analogy to column 55); (2) start the non-architectural program of the 48th figure; (3) detect whether the non-architected program is executed; (4) read the value of the output H from the data random access memory 122 ( Columns 1, 6, and so on to column 59); and (5) repeat steps (1) through (4) several times until the task is completed, such as the calculation required to identify the phone user's words.

In another implementation, the architecture program performs the following steps: (1) for a single time step, filling in the data random access memory 122 (such as column 0) with the value of input X; (2) starting the non-architect program (The forty-eighth figure is a modified version of one of the non-architecture programs, no loops are required, and Only accessing the data random storage memory 122 of a single group of five columns); (3) detecting whether the non-architectural program is executed; (4) reading the value of the output H from the data random access memory 122 (such as column 1) And (5) repeat steps (1) through (4) several times until the task is completed. Which of the two methods is superior depends on the sampling method of the input X value of the long-term and short-term memory layers. For example, if this task allows sampling of inputs (eg, about 12 time steps) and performing calculations in multiple time steps, the first approach is ideal because it may result in more computational resource efficiency and / Or better performance, however, if this task only allows sampling in a single time step, the second method is required.

The third embodiment is similar to the second method described above. However, unlike the second method, a single group of five columns of data random access memory 122 is used. In this manner, the non-architectural program uses multiple sets of five columns of memory, that is, Each time step uses a different five-column set of memory, which is similar to the first. In this third embodiment, preferably, the architecture program includes a step before the step (2), in which the architecture program updates the non-architecture program before starting, for example, the instruction of the address 0 is The data random access memory 122 column is updated to point to the next set of five columns of memory.

The forty-eighth figure is a table showing a program stored in the program memory 129 of the neural network unit 121. The program is executed by the neural network unit 121 and uses data and weights according to the configuration of the 47th figure. To achieve the calculation associated with the long- and short-term memory cell layer. The sample program of the forty-eighth figure includes 24 non-architectural instructions located at addresses 0 through 23, respectively. Instruction of address 0 (INITIALIZE NPU, CLR ACC, LOOPCNT=12, DR IN ROW=-1, DR OUT ROW=2) will clear accumulator 202 and initialize loop counter 3804 to a value of 12 to perform 12 loop groups (instructions of addresses 1 through 22). This initialization command initializes the column to be read of the data random access memory 122 to a value of -1, and the value is incremented to zero after the first execution of the instruction of the address 1. This initialization instruction initializes the columns to be written to the data random access memory 122 (e.g., the registers 2660 of the twenty-sixth and thirty-ninth figures) into column 2. Preferably, the initialization command causes the neural network unit 121 to be in a wide configuration. Thus, the neural network unit 121 is configured with 512 neural processing units 126. As described in subsequent sections, 128 of the 512 neural processing units 126 correspond to and operate as 128 long-term memory cells 4600 during the execution of instructions at addresses 0 through 23.

In the first execution of the instructions of addresses 1 through 4, each of the 128 neural processing units 126 (i.e., neural processing units 0 through 127) will be directed to the first of the corresponding long and short term memory cells 4600. The time step (time step 0) calculates the input gate (I) value and writes the I value to the corresponding text of column 2 of the data random access memory 122; in the second execution of the instructions of addresses 1 to 4, Each of the 128 neural processing units 126 calculates an I value for the second time step (time step 1) of the corresponding long and short term memory cell 4600 and writes the I value to the data random access memory 122. Corresponding text of column 7; and so on, in the twelfth execution of the instructions of addresses 1 to 4, each of the 128 neural processing units 126 will target the corresponding long-term and short-term memory cells 4600 The twelfth time step (time step 11) The I value is calculated and the I value is written to the corresponding text of column 57 of the data random access memory 122, as shown in Fig. 47.

Further, the multiply accumulate instruction of address 1 reads a column below the current column of the data random access memory 122 (column 0 in the first execution, column 5 in the second execution, and so on, In the twelfth execution, column 55), the column contains the memory cell input (X) value associated with the current time step, and the instruction reads the column 0 of the Wi value contained in the weight random access memory 124. And the aforementioned read values are multiplied to produce an accumulator 202 that adds the first multiply accumulation to the instruction that was just cleared by the initialization instruction of address 0 or the address of address 22. Subsequently, the multiply accumulate instruction of address 2 reads the next data random access memory 122 column (column 1 in the first execution, column 6 in the second execution, and so on, in the twelfth execution) That is, column 56), this column contains the memory cell output (H) value associated with the current time step, and this instruction will read the column 1 containing the Ui value in the weight random access memory 124, and the aforementioned value Multiply by to generate a second multiply accumulation and add to accumulator 202. The H value associated with the current time step is read by the data random access memory 122 by the instruction of address 2 (and the instructions of addresses 6, 10 and 18), generated at the previous time step, and by address 22 The output instruction is written to the data random access memory 122; however, in the first execution, the instruction of the address 2 writes the column 1 of the data random access memory as an H value with an initial value. Preferably, the architecture program writes the initial H value to column 1 of the data random access memory 122 (eg, using the MTNN instruction 1400) prior to initiating the non-architecture of the twenty-eighth figure; however, the invention is not limited Therefore, the non-architecture program includes an initialization command to write the initial H value into the column 1 of the data random access memory 122. It also falls within the scope of the invention. In an embodiment, this initial H value is zero. Next, the instruction of the address 3 to add the weight text to the accumulator (ADD_W_ACC WR ROW 2) reads the column 2 containing the Bi value in the weight random access memory 124 and adds it to the accumulator 202. Finally, the output instruction of address 4 (OUTPUT SIGMOID, DR OUT ROW +0, CLR ACC) performs an S-type start function on the value of the accumulator 202 and writes the execution result to the current output column of the data random access memory 122. (Column 2 in the first execution, column 7 in the second execution, and column 57 in the twelfth execution) and the accumulator 202 is cleared.

In the first execution of the instructions of addresses 5 through 8, each of the 128 neural processing units 126 will calculate its first time step (time step 0) for the corresponding long and short term memory cells 4600. Forgotten gate (F) values and write F values to the corresponding text of column 3 of data random access memory 122; in the second execution of instructions of addresses 5 through 8, the 128 neural processing units 126 Each of the neural processing units 126 calculates its forgetting gate (F) value for the second time step (time step 1) of the corresponding long-term and short-term memory cell 4600 and writes the F value to the column 8 of the data random access memory 122. Corresponding text; and so on, in the twelfth execution of the instructions of addresses 5 to 8, each of the 128 neural processing units 126 will be directed to the tenth of the corresponding long-term and short-term memory cells 4600. The second time step (time step 11) calculates its forgetting gate (F) value and writes the F value to the corresponding text of column 58 of data random access memory 122, as shown in FIG. The instructions of addresses 5 through 8 calculate the F value in a manner similar to the above instructions of addresses 1 through 4, however, the instructions of addresses 5 through 7 will be from the weighted random access memory 124, respectively. Column 3, column 4 and column 5 read the Wf, Uf and Bf values to perform multiplication and/or addition operations.

In the twelve executions of the instructions of addresses 9 through 12, each of the 128 neural processing units 126 calculates its candidate memory cell state for the corresponding time step of the corresponding long-term and short-term memory cells 4600 ( The C') value and the C' value are written to the corresponding text of column 9 of the weighted random access memory 124. The instructions of addresses 9 through 12 calculate the C' value in a manner similar to the above-mentioned instructions of addresses 1 through 4, however, the instructions of addresses 9 through 11 will be respectively from the weighted random access memory 124, column 6, column 7 and Column 8 reads the Wc, Uc and Bc values to perform multiplication and/or addition operations. In addition, the output instruction of address 12 performs a hyperbolic tangent start function instead of (as the output instruction execution of address 4) an S-type start function.

Further, the multiply accumulate instruction of address 9 reads the current column of the data random access memory 122 (column 0 in the first execution, column 5 in the second execution, and so on, The twelfth execution is column 55). This current column contains the memory cell input (X) value associated with the current time step. This instruction also reads the column 6 of the weighted random access memory 124 that contains the Wc value. And multiplying the aforementioned values to produce a first multiply accumulation plus an accumulator 202 that has just been cleared by the instruction of address 8. Next, the multiply-accumulate instruction of address 10 reads the next column of data random access memory 122 (column 1 in the first execution, column 6 in the second execution, and so on, in the The eleventh execution is column 56), which includes the memory cell output (H) value associated with the current time step, and the instruction reads the column 7 containing the Uc value in the weighted random access memory 124, and Multiplying the aforementioned values to produce a second multiply accumulate and add to the accumulator 202. Next, the instruction of the address 11 to add the weight text to the accumulator reads the column 8 containing the Bc value in the weight random access memory 124 and adds it to the accumulator 202. Finally, the output instruction of address 12 (OUTPUT TANH, WR OUT ROW 9, CLR ACC) performs a hyperbolic tangent start function on the accumulator 202 value and writes the execution result to column 9 of the weighted random access memory 124. And the accumulator 202 is cleared.

In the twelve executions of the instructions of addresses 13 through 16, each of the 128 neural processing units 126 calculates a new memory cell state for the corresponding time step of the corresponding long-term and short-term memory cells 4600 ( C) the value and writing the new C value to the corresponding text of column 11 of the weighted random access memory 122, each neural processing unit 126 also calculates tanh(C) and writes it into the weighted random access memory. Corresponding text of column 10 of 124. Further, the multiply-accumulate instruction of address 13 reads a column below the current column of the data random access memory 122 (column 2 in the first execution and column 7 in the second execution, By analogy, in the twelfth execution, column 57), this column contains the input gate (I) value associated with the current time step, and the instruction reads the weighted random access memory 124 containing the candidate memory cell state (C ') The value of column 9 (just written by the instruction of address 12), and the aforementioned value is multiplied to produce a first multiply accumulated plus accumulator 202 that has just been cleared by the instruction of address 12. Next, the multiply accumulate instruction of address 14 reads a column below the data random access memory 122 (column 3 in the first execution, column 8 in the second execution, and so on, in the The eleventh execution is column 58), this column contains the forgetting gate (F) value associated with the current time step, and the instruction is read in the weighted random access memory 124 included in the previous time step. The current memory cell state (C) value (written by the most recent execution of the instruction of address 15) is column 11, and the aforementioned values are multiplied to produce a second product added to accumulator 202. Next, the output instruction of the address 15 (OUTPUT PASSTHRU, WR OUT ROW 11) passes the value of the accumulator 202 and writes it to the column 11 of the weighted random access memory 124. It should be understood that the C value read by the address of the address 14 by the column 11 of the data random access memory 122 is the C value generated and written by the instruction of the address 13 to 15 in the most recent execution. The output instruction of address 15 does not clear accumulator 202, so its value can be used by the instruction of address 16. Finally, the output command of address 16 (OUTPUT TANH, WR OUT ROW 10, CLR ACC) performs a hyperbolic tangent start function on the value of accumulator 202 and writes its execution result to column 10 of weight random access memory 124. The instructions for address 21 are used to calculate the memory cell output (H) value. The instruction of address 16 clears accumulator 202.

In the first execution of the instructions of addresses 17 through 20, each of the 128 neural processing units 126 will calculate its first time step (time step 0) for the corresponding long- and short-term memory cells 4600. The gate (O) value is output and the O value is written to the corresponding text of column 4 of the data random access memory 122; in the second execution of the instructions of addresses 17 through 20, the 128 neural processing units 126 Each of the neural processing units 126 calculates its output gate (O) value for the second time step (time step 1) of the corresponding long-term memory cell 4600 and writes the O value to the column 9 of the data random access memory 122. Corresponding text; and so on, in the twelfth execution of the instructions of addresses 17 to 20, each of the 128 neural processing units 126 will target the phase Corresponding to the twelfth time step (time step 11) of the long and short term memory cell 4600, the output gate (O) value is calculated and the O value is written into the corresponding text of the column 58 of the data random access memory 122, such as the fortieth The seven figures are shown. The instructions of addresses 17 through 20 calculate the value of O similarly to the instructions of addresses 1 through 4 described above, however, the instructions of addresses 17 through 19 will be from column 12, column 13 and column of weight random access memory 124, respectively. 14 Read the Wo, Uo and Bo values to perform multiplication and/or addition operations.

In the first execution of the instructions of addresses 21 through 22, each of the 128 neural processing units 126 will calculate its first time step (time step 0) for the corresponding long-term and short-term memory cells 4600. The memory cell outputs (H) values and writes the H values to the corresponding text of column 6 of the data random access memory 122; in the second execution of the instructions of addresses 21 through 22, the 128 neural processing units 126 Each of the neural processing units 126 calculates its memory cell output (H) value for the second time step (time step 1) of the corresponding long-term and short-term memory cell 4600 and writes the H value into the data random access memory 122. 11 corresponding text; and so on, in the twelfth execution of the instructions of addresses 21 to 22, each of the 128 neural processing units 126 will target the corresponding long-term and short-term memory cells 4600 The twelfth time step (time step 11) calculates its memory cell output (H) value and writes the H value to the corresponding text of column 60 of the data random access memory 122, as shown in FIG.

Further, the multiply accumulate instruction of the address 21 reads the third column behind the current column of the data random access memory 122 (column 4 in the first execution and column 9 in the second execution, Such kind of push, in The twelfth execution is column 59), which contains the output gate (O) value associated with the current time step, and the instruction reads the column 10 of the value of tanh(C) in the weighted random access memory 124 ( It is written by the instruction of address 16) and the aforementioned values are multiplied to produce a multiply accumulator plus accumulator 202 that has just been cleared by the instruction of address 20. Subsequently, the output instruction of address 22 transfers the value of accumulator 202 and writes it to the next second output column 11 of data random access memory 122 (on the first execution, column 6, in the second time) Execution is column 11, and so on, at the twelfth execution, column 61), and the accumulator 202 is cleared. It should be understood that the H value of the data random access memory 122 column is written by the instruction of the address 22 (column 6 in the first execution, column 11 in the second execution, and so on, in The twelfth execution is the value of H consumed/read in the subsequent execution of the instructions of the instructions of addresses 2, 6, 10 and 18. However, the H value written in column 61 during the twelfth execution is not consumed/read by the execution of the instructions of addresses 2, 6, 10 and 18; for a preferred embodiment, this value will It is consumed/read by the architecture program.

The instruction of bit 23 (LOOP 1) decrements the loop counter 3804 and returns to the instruction of address 1 if the value of the new loop counter 3804 is greater than zero.

A forty-ninth diagram is a block diagram showing an embodiment of a neural network unit 121 having an output buffer masking and feedback capability within the neural processing unit group of this embodiment. The forty-ninth diagram shows a single neural processing unit group 4901 consisting of four neural processing units 126. Although the forty-ninth diagram shows only a single neural processing unit group 4901, it is to be understood that each neural processing unit in the neural network unit 121 126 will be included in a neural processing unit group 4901, so there will be a total of N/J neural processing unit groups 4901, where N is the number of neural processing units 126 (for example, 512 for a wide configuration) In the case of a narrow configuration, it is 1024) and J is the number of neural processing units 126 in a single group 4901 (for example, four in the case of the forty-ninth embodiment). In the forty-ninth diagram, the four neural processing units 126 in the neural processing unit group 4901 are referred to as a neural processing unit 0, a neural processing unit 1, a neural processing unit 2, and a neural processing unit 3.

Each of the neural processing units in the embodiment of the forty-ninth embodiment is similar to the neural processing unit 126 of the seventh embodiment described above, and elements having the same reference numerals are similar in the drawings. However, multiplex register 208 is tuned to include four additional inputs 4905 that are tuned to include four additional inputs 4907 that are adjusted to be input from the original Selections 211 and 207 and additional inputs 4905 are provided to output 209, and selection input 713 is adjusted to provide selection from the original inputs 711 and 206 and additional inputs 4907 to output 203.

As shown in the figure, the column buffer 1104 of the eleventh diagram is the output buffer 1104 in the forty-ninth figure. Further, the characters 0, 1, 2, and 3 of the output buffer 1104 shown in the figure receive the corresponding outputs associated with the four start function units 212 of the neural processing units 0, 1, 2, and 3. The output buffer 1104 of this section contains N words corresponding to a neural processing unit group 4901, which is referred to as an output buffer text group. In the embodiment of the forty-ninth embodiment, N is four. The four characters of the output buffer 1104 are fed back to the multiplex register 208 and 705, and received by multiplex register 208 as four additional inputs 4905 and received by multiplex register 705 as four additional inputs 4907. The feedback buffer text group is fed back to the feedback action of its corresponding neural processing unit group 4901, so that the arithmetic instructions of the non-architectural program can be from the text associated with the output buffer 1104 of the neural processing unit group 4901 (ie, the output buffer text group) Select one or two words as the input in the group). For an example, please refer to the non-architectural program in the following 51st figure, as shown in the figure 4, 8, 11, 12 and 15. That is, the output buffer 1104 text specified in the non-architected instruction will confirm the value generated by the selection input 213/713. This capability actually allows the output buffer 1104 to act as a scratch pad memory that allows non-architected programs to reduce the write data random access memory 122 and/or the weighted random access memory 124 and subsequent The number of reads, such as the value of the intervening generation and use during the process. Preferably, the output buffer 1104, or column buffer 1104, includes a one-dimensional array of registers for storing 1024 narrow words or 512 wide words. Preferably, the reading of the output buffer 1104 can be performed in a single time-frequency period, while the writing to the output buffer 1104 can be performed in a single time-frequency period. Different from the data random access memory 122 and the weighted random access memory 124, the data can be accessed by the architecture program and the non-architecture program. The output buffer 1104 cannot be accessed by the architecture program, but can only be stored by the non-architecture program. take.

Output buffer 1104 will be adjusted to receive a mask input 4903. Preferably, the mask input 4903 includes four words corresponding to the four characters of the output buffer 1104, the four characters The four neural processing units 126 are associated with the neural processing unit group 4901. Preferably, if the mask input 4903 of the text corresponding to the output buffer 1104 is true, the text of the output buffer 1104 maintains its current value; otherwise, the text of the output buffer 1104 is The output of the start function unit 212 is updated. That is, if the mask input 4903 of the text corresponding to the output buffer 1104 is false, the output of the start function unit 212 is written to the text of the output buffer 1104. Thus, the output command of the non-architected program can selectively write the output of the start function unit 212 to some characters of the output buffer 1104 and maintain the current value of the other characters of the output buffer 1104. For an example, please refer to the example. The instructions of the non-architecture program in the subsequent fifty-first figure are as shown in the instructions of addresses 6, 10, 13 and 14. That is, the text of the output buffer 1104 specified in the non-architected program is generated from the value of the mask input 4903.

To simplify the description, the input 1811 of the multiplex register 208/705 is not shown in the forty-ninth figure (as shown in Figures 18, 19 and 23). However, embodiments that simultaneously support feedback/masking that can dynamically configure the neural processing unit 126 and the output buffer 1104 are also within the scope of the present invention. Preferably, in these embodiments, the output buffer text group is dynamically configurable accordingly.

It should be understood that although the number of neural processing units 126 in the neural processing unit group 4901 of this embodiment is four, the present invention is not limited thereto, and the number of intra-group neural processing units 126 is large or small. The examples are all within the scope of the invention. Moreover, in the case of an embodiment having a shared start function unit 1112, as in the fifty-second figure It is shown that the number of neural processing units 126 within a group of neural processing units 4901 can have a synergistic effect on the number of neural processing units 126 within a group of activation function units 212. The occlusion and feedback capabilities of the output buffer 1104 within the neural processing unit group are particularly useful for improving the computational efficiency associated with the long and short term memory cells 4600, as described in subsequent fifty and fifty-first figures.

Figure 50 is a block diagram showing the neural network of the forty-ninth figure when the neural network unit 121 performs the calculation associated with a hierarchy of 128 long- and short-term memory cells 4600 in the forty-sixth figure. An example of data configuration in the data random access memory 122 of the unit 121, the weighted random access memory 124 and the output buffer 1104. In the example of Fig. 50, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, for example, in a wide configuration. As in the forty-seventh and forty-eighth examples, there are only 128 long- and short-term memory cells 4600 in the long- and short-term memory layers of the fifty and fifty-first examples. However, in the example of Fig. 50, all of the values generated by 512 neural processing units 126 (e.g., neural processing units 0 through 127) will be used. When performing the non-architectural program of Figure 51, each neural processing unit group 4901 will collectively operate as a long-term and short-term memory cell 4600.

As shown in the figure, the data random memory 122 loads the memory cell input (X) and output (H) values for use in a series of time steps. Further, for a given time step, a pair of two columns of memory are loaded with X values and H values, respectively. Taking a random access memory 122 having 64 columns as an example, as shown in the figure, the memory cell value loaded by the data random access memory 122 can be used in 31 different time steps. In the example in Figure 50, columns 2 and 3 are loaded for the time step 0. Values, columns 4 and 5 are loaded with the values used for time step 1, and so on, columns 62 and 63 are loaded with the values used for time step 30. The first column of the two columns of memory is loaded with the X value of this time step, and the second column is the H value for the time step of loading. As shown in the figure, each of the four rows in the data random access memory 122 corresponds to a value stored in the memory system of the neural processing unit group 4901 for use by its corresponding long-term and short-term memory cells 4600. That is, rows 0 to 3 are loaded with values associated with long- and short-term memory cells 0, which are calculated by neuroprocessing unit 0-3, ie, neuroprocessing unit group 0 is executed; rows 4 to 7 are loaded associated with long The value of the short-term memory cell 1 is calculated by the neural processing unit 4-7, that is, the neural processing unit group 1 is executed; and so on, the lines 508 to 511 are loaded with the values associated with the long- and short-term memory cells 127, and the calculation is performed. It is performed by the neural processing unit 508-511, that is, the neural processing unit group 127, as shown in the subsequent fifty-first figure. As shown in the figure, column 1 is not used, column 0 is loaded with the initial memory cell output (H) value. In a preferred embodiment, the value can be filled in by the architecture program, however, the invention is not limited Thus, it is also within the scope of the present invention to fill in the initial memory cell output (H) values of column 0 using non-architectural program instructions.

Preferably, the X value (in columns 2, 4, 6 and so on to column 62) is written/filled into the data random access memory 122 by the architecture program executed by the processor 100 through the MTNN instruction 1400, and The non-architecture program executed by the neural network unit 121 performs reading/using, such as the non-architected program shown in FIG. Preferably, the H value (in columns 3, 5, 7 and so on to column 63) is written/filled into the data random access memory 122 by the non-architectural program executed by the neural network unit 121 and read. / Use, as described later. Preferably, the H value is read by the MFNN instruction 1500 by an architectural program executing on the processor 100. It should be noted that the non-architecture scheme of the fifty-first diagram assumes that each group of four rows of memory corresponding to the neural processing unit group 4901 (such as row 0-3, row 4-7, row 5-8, according to this) Analogy to lines 508-511), the four X values in a given column are filled with the same value (eg, filled in by the architecture program). Similarly, the non-architectural program of the fifty-first graph calculates and writes the same value for the four H values of a given column in each of the four rows of memory corresponding to the neural processing unit group 4901.

As shown in the figure, the weighted random access memory 124 is loaded with the weights, offsets, and memory cell state (C) values required by the neural processing unit of the neural network unit 121. In each group of four rows of memory corresponding to the neural processing unit group 121 (eg, rows 0-3, rows 4-7, rows 5-8, and so on to rows 508-511): (1) the row number divided by The remainder of 4 is equal to 3, and the values of Wc, Uc, Bc, and C are loaded in columns 0, 1, 2, and 6, respectively; (2) the line number divided by 4 is equal to 2, and will be in Columns 3, 4, and 5 respectively load the values of Wo, Uo, and Bo; (3) the row number divided by the remainder of 4 is equal to 1, and the values of Wf, Uf, and Bf are loaded in columns 3, 4, and 5, respectively; And (4) the row number divided by the remainder of 4 is equal to 0, and the values of Wi, Ui and Bi are loaded in columns 3, 4 and 5, respectively. Preferably, these weights and offset values - Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, Wo, Uo, Bo (in columns 0 to 5) - are executed by the processor 100. The architecture program writes/fills the weighted random access memory 124 through the MTNN instruction 1400 and is read/used by a non-architected program executed by the neural network unit 121, such as the non-architected program of FIG. Preferably, the intervening C value The non-architecture program executed by the neural network unit 121 writes/fills the weight random access memory 124 and reads/uses it, as will be described later.

The example in Figure 50 assumes that the architecture program performs the following steps: (1) For 31 different time steps, the value of the input X is filled in the data random access memory 122 (column 2, 4, and so on. Column 62); (2) start the non-architecture program of the 51st figure; (3) detect whether the non-architecture program is executed; (4) read the value of the output H from the data random access memory 122 (column 3) , 5, and so on to column 63); and (5) repeat steps (1) through (4) several times until the task is completed, such as the calculation required to identify the phone user's words.

In another implementation, the architecture program performs the following steps: (1) for a single time step, filling in the data random access memory 122 (such as column 2) by inputting the value of X; (2) starting the non-architectural program (The 51st version of the non-architected program is a modified version that does not require loops and only accesses a single pair of two columns of data storage memory 122); (3) detects whether the non-architected program has been executed; (4) Reading the value of the output H from the data random access memory 122 (e.g., column 3); and (5) repeating steps (1) through (4) several times until the task is completed. Which of the two methods is superior depends on the sampling method of the input X value of the long-term and short-term memory layers. For example, if this task allows sampling of inputs (eg, approximately 31 time steps) and performing calculations in multiple time steps, the first approach is desirable because it may result in more computational resource efficiency and / Or better performance, however, if this task only allows sampling in a single time step, the second method is required.

The third embodiment is similar to the second method described above, However, unlike the second method, which uses a single pair of two columns of data random access memory 122, the non-architectural program in this manner uses multiple pairs of memory columns, that is, different pairs of memory columns are used in each time step, and this portion is similar. In the first way. Preferably, the architecture program of the third embodiment includes a step before the step (2), in which the architecture program updates the non-architecture program before starting the non-architecture program, for example, in the instruction of the address 1 The data random access memory 122 column is updated to point to the next pair of two columns of memory.

As shown in the figure, for the neural processing units 0 to 511 of the neural network unit 121, after the execution of the instructions of the different addresses in the non-architectural program of the fifty-first figure, the output buffer 1104 is loaded with the memory cell output (H). ), candidate memory cell state (C'), input gate (I), forget gate (F), output gate (O), memory cell state (C) and tanh (C), each output buffer text group In the group (for example, the output buffer 1104 corresponds to the group of four words of the neural processing unit group 4901, such as words 0-3, 4-7, 5-8 and so on to 508-511), the text number is divided by The text with a remainder of 4 is represented as OUTBUF[3], the text with the remainder of the text number divided by 4 is represented as OUTBUF[2], and the text with the remainder of the text number divided by 4 is represented as OUTBUF [ 1], and the text with the remainder of the text number divided by 4 is represented as OUTBUF[0].

As shown in the figure, after the instruction of address 2 in the non-architecture program of the fifty-first figure is executed, for each neural processing unit group 4901, all four characters of the output buffer 1104 are written correspondingly. The initial memory cell output (H) value of the long-term and short-term memory cells 4600. After the instruction of address 6 is executed, for each neural processing unit group 4901, the OUTBUF[3] text of the output buffer 1104 is written into the corresponding long-term and short-term records. The candidate memory cell state (C') value of cell 4600 is recalled, while the other three words of output buffer 1104 maintain their previous values. After the instruction of address 10 is executed, for each neural processing unit group 4901, the OUTBUF[0] text of the output buffer 1104 is written to the input gate (1) value of the corresponding long-term memory cell 4600, OUTBUF[ 1] The text will be written to the value of the forgotten gate (F) of the corresponding long-term memory cell 4600, and the OUTBUF[2] text will be written to the output gate (O) value of the corresponding long-term memory cell 4600, and the OUTBUF[3] text It is to maintain its previous value. After the instruction of address 13 is executed, for each neural processing unit group 4901, the OUTBUF[3] text of the output buffer 1104 is written to the new memory cell state (C) value of the corresponding long-term memory cell 4600. (For the output buffer 1104, the C value containing the slot 3 is written to the column 6 of the weight random access memory 124, as described in the subsequent fifty-first figure), and the output buffer 1104 The other three words are to maintain their previous values. After the instruction of address 14 is executed, for each neural processing unit group 4901, the OUTBUF[3] text of the output buffer 1104 is written to the tanh(C) value of the corresponding long-term memory cell 4600, and the output buffer is output buffer. The other three words of the device 1104 are to maintain their previous values. After the instruction of address 16 is executed, for each neural processing unit group 4901, all four words of output buffer 1104 are written to the new memory cell output (H) value of the corresponding long-term memory cell 4600. The execution flow of the above address 6 to 16 (that is, the execution of the address 2 is excluded, because the address 2 is not part of the program loop) is repeated 30 times, and the program returns to the address 3 as the address 17. Loop.

The fifty-first figure is a table showing the storage in the neural network. A program of the program memory 129 of the circuit unit 121. The program is executed by the neural network unit 121 of the forty-ninth figure and uses data and weights according to the configuration of the fifty-fifth figure to achieve association with the long-term and short-term memory layers. Calculation. The sample program of Figure 51 contains 18 non-architected instructions located at addresses 0 through 17. The instruction of address 0 is an initialization instruction to clear accumulator 202 and initialize loop counter 3804 to a value of 31 to perform 31 loop groups (instructions of addresses 1 through 17). The initialization command initializes the column to be written of the data random access memory 122 (for example, the register 2606 of the twenty-sixth/39th figure) to a value of 1, and the first instruction of the address 16 After the second execution, this value will increase to 3. Preferably, the initialization command causes the neural network unit 121 to be in a wide configuration. Thus, the neural network unit 121 is configured with 512 neural processing units 126. As described in the subsequent sections, during the execution of the instructions of addresses 0 through 17, the 128 neural processing unit groups 490 of the 512 neural processing units 126 operate as 128 corresponding long-term and short-term memory cells 4600.

The instructions for addresses 1 and 2 are not part of the program loop group and will only be executed once. These instructions will generate an initial memory cell output (H) value (eg, 0) and write it to all of the text in output buffer 1104. The instruction of address 1 reads the initial H value from column 0 of data random access memory 122 and places it in accumulator 202 cleared by instruction of address 0. The address 2 instruction (OUTPUT PASSTHRU, NOP, CLR ACC) passes the accumulator 202 value to the output buffer 1104 as shown in FIG. The "NOP" flag in the output instruction of address 2 (and the other output instructions in Figure 51) indicates that the output value will only be written to the output buffer 1104 and will not be written to the memory, ie it will not Data random access The memory 122 or the weight random access memory 124. The instruction of address 2 will clear accumulator 202.

The instructions of addresses 3 through 17 are located in the loop group, and the number of executions is the value of the loop count (such as 31).

Each execution of the instructions of addresses 3 through 6 calculates the tanh(C') value of the current time step and writes it to the text OUTBUF[3], which will be used by the instruction of address 11. More precisely, the multiply-accumulate instruction of address 3 reads the memory associated with this time step from the current read column of data random access memory 122 (e.g., columns 2, 4, 6 and so on to column 62). The cell inputs (X) value, reads the Wc value from column 0 of the weighted random access memory 124, and multiplies the aforementioned values to produce a product that is added to the accumulator 202 that is cleared by the instruction of address 2.

The multiply accumulate instruction of address 4 (MULT-ACCUM OUTBUF[0], WR ROW 1) reads the H value from the word OUTBUF[0] (ie all four neural processing units 126 of the neural processing unit group 4901), from Column 1 of weight random access memory 124 reads the Uc value and multiplies the aforementioned values to produce a second product to add accumulator 202.

Adding the weight text to the accumulator instruction (ADD_W_ACC WR ROW 2) of address 5 reads the Bc value from column 2 of the weight random memory 124 and adds it to accumulator 202.

The output instruction of address 6 (OUTPUT TANH, NOP, MASK[0:2], CLR ACC) performs a hyperbolic tangent start function on the accumulator 202 value, and only writes the execution result to the text OUTBUF[3] (also That is, only the neural processing unit 126 having the remainder of the number 4 except the number 4 in the neural processing unit group 4901 writes the result), and the accumulator 202 Will be cleared. That is, the output instruction of address 6 will mask the text OUTBUF[0], OUTBUF[1] and OUTBUF[2] (as indicated by the instruction term MASK[0:2]) and maintain its current value, such as the fiftyth. The figure shows. In addition, the output instruction of address 6 is not written to the memory (as indicated by the instruction term NOP).

Each execution of the instruction of address 7 to 10 calculates the input gate (I) value of the current time step, forgets the gate (F) value and the output gate (O) value and writes them to the text OUTBUF[0], OUTBUF respectively. [1], and OUTBUF[2], these values will be used by the instructions of addresses 11, 12 and 15. More precisely, the multiply-accumulate instruction of address 7 reads the memory associated with this time step from the current read column of data random access memory 122 (e.g., columns 2, 4, 6 and so on to column 62). The cell inputs (X) values, the Wi, Wf and Wo values are read from column 3 of the weighted random access memory 124, and the aforementioned values are multiplied to produce a product that is added to the accumulator 202 cleared by the instruction of address 6. More precisely, in the neural processing unit group 4901, the neural processing unit 126 with the remainder of the number divided by 4 calculates the product of X and Wi, and the neural processing unit 126 with the remainder of the number divided by 1 calculates X and The product of Wf, and the remainder of the number divided by 4, the neural processing unit 126 calculates the product of X and Wo.

The multiply accumulate instruction of address 8 reads the H value from the word OUTBUF[0] (i.e., all four neural processing units 126 of the neural processing unit group 4901), and reads Ui from column 4 of the weighted random access memory 124. The Uf and Uo values are multiplied by the aforementioned values to produce a second product that is added to the accumulator 202. More precisely, in the neural processing unit group 4901, the neural processing unit 126 is numbered with a remainder of 4 divided by zero. The product of H and Ui is calculated. The neural processing unit 126, which has a remainder of the number divided by 1, calculates the product of H and Uf, and the neural processing unit 126, which has a remainder of 4 except the number 2, calculates the product of H and Uo.

Adding the weight text to the accumulator instruction (ADD_W_ACC WR ROW 2) of address 9 reads the Bi, Bf and Bo values from column 5 of the weighted random memory 124 and adds it to accumulator 202. More precisely, in the neural processing unit group 4901, the neural processing unit 126 with the remainder of the number divided by 4 performs the addition calculation of the Bi value, and the neural processing unit 126 with the remainder of the number divided by 1 performs the Bf. The addition of the values is calculated, and the neural processing unit 126, which has a remainder of 4 except the number 4, performs the addition calculation of the Bo value.

The output instruction of address 10 (OUTPUT SIGMOID, NOP, MASK[3], CLR ACC) performs an S-type start function on the value of the accumulator 202 and writes the calculated I, F and O values to the text OUTBUF[0, respectively. ], OUTBUF[1] and OUTBUF[2], this instruction will clear the accumulator 202 without writing to the memory. That is, the output instruction of address 10 masks the text OUTBUF[3] (as indicated by the instruction term MASK[3]) and maintains the current value of the text (ie, C'), as shown in Figure 50.

Each execution of the instructions of addresses 11 through 13 computes a new memory cell state (C) value generated by the current time step and writes it to column 6 of the weighted random access memory 124 for use in the next time step (also That is, the instruction for the address 12 is used when the next loop is executed.) More precisely, this value is written in the column 6 corresponding to the four lines of the neural processing unit group 4901. . In addition, bit Each execution of the instruction at address 14 writes the tanh(C) value to OUTBUF[3] for instruction of address 15.

More precisely, the multiply accumulate instruction of address 11 (MULT-ACCUM OUTBUF[0], OUTBUF[3]) reads the input gate (I) value from the word OUTBUF[0] and reads from the text OUTBUF[3]. The candidate memory cell state (C') value is multiplied by the aforementioned value to produce a first product added to the accumulator 202 cleared by the instruction of address 10. More precisely, each of the four neural processing units 126 of the neural processing unit group 4901 calculates the first product of the I value and the C' value.

The multiply accumulate instruction of address 12 (MULT-ACCUM OUTBUF[1], WR ROW 6) instructs the neural processing unit 126 to read the forgetting gate (F) value from the text OUTBUF[1] from the weighted random access memory 124. Column 6 reads its corresponding text and multiplies it to produce a second product that is summed with the address of address 11 resulting from the first product in accumulator 202. More precisely, for the neural processing unit 126 with the remainder of the labeling 4 in the neural processing unit group 4901, the text read from column 6 is the current memory cell state (C) value calculated in the previous time step. The sum of the first product and the second product is the new memory cell state (C). However, for the other three neural processing units 126 of the neural processing unit group 4901, the text read from column 6 is an unneeded value because the accumulated values produced by these values will not be used, That is, the instructions of addresses 13 and 14 are not placed in the output buffer 1104 and are cleared by the instruction of the address 14. That is to say, only the new record generated by the neural processing unit 126 having the remainder of the label 4 except the number 4 in the neural processing unit group 4901 The recall state (C) value will be used, that is, the instructions of addresses 13 and 14. For the second to thirty-one execution of the instruction of address 12, the C value read from column 6 of weight random access memory 124 is the instruction written by address 13 in the previous execution of the loop group. The value. However, for the first execution of the instruction at address 12, the value of C in column 6 is written by the architecture program before the non-architecture program of the fifty-first diagram is started or the version is modified by one of the non-architecture programs. Enter the initial value.

The output instruction of address 13 (OUTPUT PASSTHRU, WR ROW 6, MASK[0:2]) will only pass the value of accumulator 202, ie the calculated C value, to the text OUTBUF[3] (that is, only neural processing) The neural processing unit 126 having the remainder of the label 4 in the unit group 4901 divides the calculated C value into the output buffer 1104), and the column 6 of the weighted random access memory 124 is updated. The output buffer 1104 is written as shown in Fig. 50. That is to say, the output instruction of address 13 will obscure the characters OUTBUF[0], OUTBUF[1] and OUTBUF[2] while maintaining their current values (ie, I, F and O values). As described above, only the column 6 corresponds to the C value in the four-line text of the neural processing unit group 4901 in which the remainder of the label except 4 is 3, that is, the instruction of the address 12 is used; therefore, the non-architecture The program does not care about the values of row 0-2, row 4-6, and so on to row 508-510 in column 6 of the random access memory 124, as shown in Fig. 50 (i.e., I, F and O value).

The output instruction of address 14 (OUTPUT TANH, NOP, MASK[0:2], CLR ACC) performs a hyperbolic tangent start function on the accumulator 202 value and writes the calculated tanh(C) value to the text. OUTBUF[3], this instruction will clear accumulator 202 without writing to memory. The output instruction of address 14, like the output instruction of address 13, will mask the words OUTBUF[0], OUTBUF[1] and OUTBUF[2] and maintain their original values, as shown in Fig. 50.

Each execution of the instructions of addresses 15 through 16 calculates the memory cell output (H) value generated by the current time step and writes it to the second column after the current output column of the data random access memory 122, the value of which will be Read by the architecture program and used for the next time step (ie, used by the instructions of addresses 3 and 7 in the next loop execution). More precisely, the multiply accumulate instruction at address 15 reads the output gate (O) value from the word OUTBUF[2], reads the tanh(C) value from the text OUTBUF[3], and multiplies it to produce a The product is added to accumulator 202 which is cleared by the instruction of address 14. More precisely, each of the four neural processing units 126 of the neural processing unit group 4901 calculates the product of the value O and tanh (C).

The output instruction of address 16 will pass the value of accumulator 202 and write the calculated H value to column 3 in the first execution, the calculated H value to column 5 in the second execution, and so on. In the thirty-first execution, the calculated H value is written to column 63. As shown in Fig. 50, these values are used by the instructions of addresses 4 and 8. In addition, as shown in Fig. 50, these calculated H values are placed in the output buffer 1104 for subsequent use by the instructions of addresses 4 and 8. The output instruction of address 16 will clear accumulator 202. In one embodiment, the long- and short-term memory cell 4600 is designed such that the output command of address 16 (and/or the output command of address 22 in the forty-eighth figure) has a start function, such as an S-type or a hyperbolic The tangent function, rather than the value of the accumulator 202.

The loop command of address 17 causes the loop counter 3804 to decrement and return to the instruction of address 3 if the new loop counter 3804 has a value greater than zero.

It can be seen that, because of the feedback and shielding capability of the output buffer 1104 in the embodiment of the neural network unit 121 of the forty-ninth figure, the number of instructions in the loop group of the non-architectural program of the fifty-first graph is compared. The non-architectural instructions in Figure 48 are roughly reduced by 34%. In addition, because of the feedback and shielding capability of the output buffer 1104 in the embodiment of the neural network unit 121 of the forty-ninth figure, the memory configuration in the random access memory 122 of the non-architectural program of the fifty-first embodiment is shown in FIG. The number of time steps for collocation is roughly three times that of the forty-eighth figure. The foregoing improvements facilitate some architectural applications that utilize neural network unit 121 to perform long- and short-term memory cell layer calculations, particularly for applications where the number of long- and short-term memory cells 4600 in the long- and short-term memory cell layer is less than or equal to 128.

The embodiments of the forty-seventh to fifty-first embodiments assume that the weights and offset values in the various time steps remain unchanged. However, the present invention is not limited thereto, and embodiments in which other weights and offset values are changed over time are also within the scope of the present invention, wherein the weight random access memory 124 is not as shown in the forty-seventh to fifty-th Fill in a single set of weights and offset values, but fill in different sets of weights and offset values at each time step and the weights of the non-architectural programs of the 48th to 51st maps. Adjustment.

Basically, in the aforementioned embodiments of the forty-seventh to fifty-firstth embodiments, weights, offsets, and intermediate values (e.g., C, C' values) are stored. The random access memory 124 is weighted, and the input and output values (e.g., X, H values) are stored in the data random access memory 122. This feature facilitates the embodiment in which the random access memory 122 is double-sided and the random access memory 124 is the same, since there are more instances from the non-architecture and architecture programs to the data random access memory 122. More traffic. However, because the weighted random access memory 124 is large, in another embodiment of the present invention, the memory of the non-architectural and architectural program write values is exchanged (ie, the interchangeable data random access memory 122 and the random weights are randomized). Access memory 124). That is, the W, U, B, C', tanh (C) and C values are stored in the data random access memory 122 and the X, H, I, F and O values are stored in the weighted random access memory. Body 124 (adjusted embodiment of the forty-seventh figure); and W, U, B, and C values are stored in the data random access memory 122 and the X and H values are stored in the weighted random access memory 124 (the adjusted example of the fifty-fifth figure). Because the weight random access memory 124 is large, these embodiments can handle more time steps in one batch. For applications that use the neural network unit 121 to perform computational architectural programs, this feature facilitates certain applications that can benefit from more time steps and can be designed for memory (such as weighted random access memory). 124) Provide sufficient bandwidth.

The fifty-second diagram is a block diagram showing an embodiment of a neural network unit 121 having an output buffer masking and feedback capability within the group of neural processing units of this embodiment, and sharing a start function unit 1112. The neural network unit 121 of the fifty-second diagram is similar to the neural network unit 121 of the forty-seventh diagram, and elements having the same reference numerals are similar in the drawings. However, the four start function unit 212 of the forty-ninth figure is in the present In the embodiment, it is replaced by a single shared start function unit 1112. The single start function unit receives four outputs 217 from four accumulators 202 and generates four outputs to the text OUTBUF[0], OUTBUF[1 ], OUTBUF[2] and OUTBUF[3]. The neural network unit 212 of the fifty-second diagram operates in a manner similar to the embodiment described in the forty-ninth to fifty-first figures above, and the manner in which it operates to share the start function unit 1112 is similar to that of the foregoing eleventh The embodiment described in the thirteenth figure.

The fifty-third figure is a block diagram showing the neural network of the forty-ninth figure when the neural network unit 121 performs the calculation associated with a hierarchy of 128 long- and short-term memory cells 4600 in the forty-sixth figure. Another embodiment of the data configuration in the data random access memory 122 of the unit 121, the weighted random access memory 124 and the output buffer 1104. The example of the fifty-third figure is similar to the example of the fifty-fifth figure. However, in the fifty-third figure, the Wi, Wf, and Wo values are in column 0 (instead of column 3 in Figure 50); Ui, Uf and Uo values are in column 1 (rather than fifth) Ten graphs are in column 4); Bi, Bf and Bo values are in column 2 (instead of column 5 in Fig. 50); C values are in column 3 (instead of column 50 in column 50) ). In addition, the content of the output buffer 1104 of the fifty-third figure is similar to the fifty-fifth figure, but because of the difference between the non-architectural programs of the fifty-fourth and fifty-first figures, the content of the third column (ie, The I, F, O and C' values are present in the output buffer 1104 after the instruction of address 7 is executed (instead of the instruction of address 10 as in Fig. 50); the content of the fourth column (i.e., I, The F, O and C values are present in the output buffer 1104 after the instruction of address 10 is executed (instead of the instruction of address 13 as in the fifty-fifth figure); the fifth column The contents (ie, I, F, O, and tanh(C) values) appear in the output buffer 1104 after the instruction of address 11 is executed (instead of the instruction of address 14 as in FIG. 50); and sixth The contents of the column (i.e., the H value) appear in the output buffer 1104 after the execution of the instruction of the address 13 (instead of the instruction of the address 16 as in the fifty-fifth figure), as will be described later.

The fifty-fourth figure is a table showing a program stored in the program memory 129 of the neural network unit 121, which is executed by the neural network unit 121 of the forty-ninth figure and according to the fifty-third figure. Configure usage data and weights to achieve calculations associated with long- and short-term memory cell layers. The example program of the fifty-fourth figure is similar to the program of the fifty-first figure. More precisely, in the fifty-fourth and fifty-first figures, the instructions of the addresses 0 to 5 are the same; the instructions of the addresses 7 and 8 in the fifty-fourth figure are the same as the addresses in the fifty-first figure. The instructions of 10 and 11; and the instructions of addresses 10 through 14 in the fifty-fourth figure are identical to the instructions of addresses 13 to 17 in the fifty-first figure.

However, the instruction of address 6 in the fifty-fourth figure does not clear the accumulator 202 (in contrast, the instruction of address 6 in the fifty-first figure clears the accumulator 202). In addition, the instructions of addresses 7 through 9 in the fifty-first figure do not appear in the non-architectural program of the fifty-fourth figure. Finally, with respect to the instruction of address 9 in the fifty-fourth figure and the instruction of address 12 in the fifty-first figure, except that the instruction of address 9 in the fifty-fourth figure reads the weight random access memory The order of 124 is the same as that of the address 12 of the fifty-first figure except that the index of the weighted random access memory is 6 and the other parts are the same.

Because of the difference between the non-architecture program of the fifty-fourth figure and the non-architecture program of the fifty-first figure, the number of columns of the weighted random access memory 124 used in the configuration of the fifty-third figure is reduced by three, and the program is returned. Inside the circle The number of instructions will also be reduced by three. The size of the loop group in the non-architectural program of the fifty-fourth figure is substantially only half of the size of the loop group in the non-architectural program of the forty-eighth figure, and is substantially only in the non-architectural program of the fifty-first figure. 80% of the circle group size.

Fifty-fifth is a block diagram showing a portion of a neural processing unit 126 in accordance with another embodiment of the present invention. More precisely, for a single one of the plurality of neural processing units 126 of the forty-ninth diagram, the multiplex register 208 is shown with its associated inputs 207, 211 and 4905, and more The scratchpad 705 is associated with its inputs 206, 711 and 4907. In addition to the input of the forty-ninth figure, the multiplexer 208 and the multiplexer 705 of the neural processing unit 126 individually receive an in-group number (index_within_group) input 5599. The intra-group numbering input 5599 indicates the numbering of the particular neural processing unit 126 within its neural processing unit group 4901. Thus, for example, taking an embodiment in which each neural processing unit group 4901 has four neural processing units 126, within each neural processing unit group 4901, one of the neural processing units 126 is numbered in its group. A value of zero is received in 5599, wherein one of the neural processing units 126 receives a value of one in its group number entry 5599, wherein one of the neural processing units 126 receives the value two in its group number input 5599, and one of the neural processing units 126 receives the value three in the numbering entry 5599 in its group. In other words, the intra-group number input 5599 value received by the neural processing unit 126 is the remainder of the number of the neural processing unit 126 in the neural network unit 121 divided by J, where J is within the neural processing unit group 4901. The number of neural processing units 126. Thus, for example, the neural processing unit 73 is in its group The inner number input 5599 receives the value one, the neural processing unit 353 receives the value three in its group number input 5599, and the neural processing unit 6 receives the value two in its group number 5599.

In addition, when control input 213 specifies a predetermined value, denoted herein as "SELF," multiplex register 208 selects output buffer 1104 output 4905 corresponding to the number of group input 5599 in the group. Therefore, when a non-architectural instruction specifies the receipt of data from the output buffer 1104 by the value of SELF (indicated as OUTBUF[SELF] in the instructions of addresses 57 and 7 in Figure 57), each neural processing unit 126 The multiplex register 208 receives its corresponding text from the output buffer 1104. Thus, for example, when the neural network unit 121 performs the non-architectural instructions of addresses 2 and 7 in the fifty-fifth diagram, the multiplexer 208 of the neural processing unit 73 selects the second of the four inputs 4905. The number (number 1) is input to receive the text 73 from the output buffer 1104, and the multiplex register 208 of the neural processing unit 353 selects the fourth (number 3) input among the four inputs 4905 to receive the output buffer. The text 353 of 1104, and the multiplex register 208 of the neural processing unit 6 selects the third (number 2) input among the four inputs 4905 to receive the text 6 from the output buffer 1104. Although not used in the non-architecture of Figure 57, the non-architected instructions may also use the SELF value (OUTBUF[SELF]) to specify that the data from the output buffer 1104 is received and the control input 713 to specify a preset value. The multiplex register 705 of each neural processing unit 126 receives its corresponding text from the output buffer 1104.

Figure 56 shows a block diagram showing when the neural network unit performs the Jordan time recurrent neural network associated with the 43rd map. An example of data configuration in the data random access memory 122 and the weight random access memory 124 of the neural network unit 121 when calculating and utilizing the embodiment of the fifty-fifth figure. The weighting in the weight random access memory 124 in the figure is the same as the example in the forty-fourth figure. The configuration of the values in the data random access memory 122 is similar to the example of the forty-fourth figure, except that in this example, each time step has a corresponding one pair of two columns of memory to load the input layer node D. The value is compared to the output layer node Y value, rather than the set of four columns of memory as in the example of the forty-fourth figure. That is, in this example, the hidden layer Z value and the content layer C value are not written to the data random access memory 122. Rather, the output buffer 1104 is used as a hidden layer Z value and a content layer C value as a category of draft memory, as described in the non-architectural program of FIG. The feedback feature of the OUTBUF[SELF] output buffer 1104 can make the operation of the non-architected program faster (this is the two write and read operations to be performed on the data random access memory 122 for The output buffer 1104 performs two writes and two read operations instead of) and reduces the space of the data random access memory 122 used in each time step, so that the data random access memory 122 of the present embodiment The loaded data can be used for approximately twice the time steps of the fourty-fourth and forty-fifth embodiments, as shown in the figure, i.e., 32 time steps.

The fifty-seventh diagram is a table showing a program stored in the program memory 129 of the neural network unit 121. The program is executed by the neural network unit 121 and uses data and weights according to the configuration of the fifty-sixth figure. To achieve the Jordan time recurrent neural network. The non-architectural program in Figure 57 is similar to the non-architected program in Figure 45. As described below.

The example program of Figure 57 has 12 non-architected instructions located at addresses 0 through 11. The initialization instruction of address 0 clears the accumulator 202 and initializes the value of the loop counter 3804 to 32, causing the loop group (instructions of addresses 2 through 11) to execute 32 times. The output instruction of address 1 places the zero value of accumulator 202 (cleared by the instruction of address 0) into output buffer 1104. It can thus be observed that during the execution of the instructions of addresses 2 to 6, the 512 neural processing units 126 correspond to and operate as 512 hidden layer nodes Z, and the execution of the instructions at addresses 7 to 10. In the process, it corresponds to and operates as 512 output layer nodes Y. That is, 32 executions of the instructions of addresses 2 through 6 calculate the hidden layer node Z values for the 32 corresponding time steps and place them in the output buffer 1104 for the corresponding address of the instructions 7 through 9. 32 executions are used to calculate the output layer node Y of the 32 corresponding time steps and write it to the data random access memory 122, and provide the corresponding 32 executions of the instruction of the address 10 to use this The content layer node C of the 32 corresponding time steps is placed in the output buffer 1104. (The content layer node C placed in the 32nd time step in the output buffer 1104 is not used.)

In the first execution of the instructions of addresses 2 and 3 (ADD_D_ACC OUTBUF[SELF] and ADD_D_ACC ROTATE, COUNT=511), each of the 512 neural processing units 126 will have 512 of the output buffer 1104. The content node C values are accumulated to their accumulators 202, which are generated and written by instruction execution of addresses 0 through 1. In the second execution of the instructions of addresses 2 and 3, each of the 512 neural processing units 126 126 will accumulate 512 content node C values of output buffer 1104 to its accumulator 202, which are generated and written by instruction execution of address bits 7 through 8 and 10. More precisely, the instruction of address 2 will instruct the multiplexer 208 of each neural processing unit 126 to select its corresponding output buffer 1104 text, as previously described, and add it to accumulator 202; address 3 The instructions instruct the neural processing unit 126 to rotate the content node C value within a 512-character rotator that is collectively operated by the multiplexed registers 208 of the 512 neural processing units. It is constructed such that each neural processing unit 126 can accumulate the 512 content node C values to its accumulator 202. The instruction of address 3 does not clear the accumulator 202, so the instruction of the address 4 and 5 can add the input layer node D value (multiplied by its corresponding weight) by the instruction of the address 2 and 3. Content layer node C value.

In each execution of the instructions of addresses 4 and 5 (MULT-ACCUM DR ROW +2, WR ROW 0 and MULT-ACCUM ROTATE, WR ROW +1, COUNT=511), the 512 neural processing units 126 Each neural processing unit 126 performs 512 multiplication operations to associate the data random access memory 122 with the current time step (eg, column 0 for time step 0 and column 0 for time step 1) Column 2, and so on, for the 512 input node D values for column 62) for time step 31, multiplied by columns 0 through 511 of weight random access memory 124 corresponding to the neural processing unit 126 The weights are weighted to produce 512 products, and the accumulated results of the 512 content node C values, together with the instructions of the addresses 2 and 3, are added to the accumulator 202 of the corresponding neural processing unit 126 to calculate the concealment. Node Z layer value.

In each execution of the instruction of address 6 (OUTPUT PASSTHRU, NOP, CLR ACC), the 512 accumulators 202 of the 512 neural processing units 126 are passed and written to the corresponding characters of the output buffer 1104, and The accumulator 202 will be cleared.

During the execution of the instructions of addresses 7 and 8 (MULT-ACCUM OUTBUF[SELF], WR ROW 512 and MULT-ACCUM ROTATE, WR ROW +1, COUNT=511), each of the 512 neural processing units 126 The neural processing unit 126 performs 512 multiplication operations, and the 512 hidden node Z values in the output buffer 1104 (generated and written by the corresponding execution of the instructions of the addresses 2 to 6) are multiplied by the weights. The weights corresponding to the rows of the neural processing unit 126 in the columns 512 through 1023 of the memory 124 are taken to generate 512 multiply accumulates to the accumulator 202 of the corresponding neural processing unit 126.

In each execution of the instruction of the address 9 (OUTPUT ACTIVATION FUNCTION, DR OUT ROW + 2), a start function (such as a hyperbolic tangent function, a sigmoid function, a correction function) is performed on the 512 accumulated values to calculate Output node Y value, this output node Y value will be written into the data random access memory 122 corresponding to the current time step column (for example: for time step 0 is column 1, for time step 1 For column 3, and so on, for time step 31 it is column 63). The instruction of address 9 does not clear accumulator 202.

In each execution of the instruction of address 10 (OUTPUT PASSTHRU, NOP, CLR ACC), the 512 values accumulated by the instructions of addresses 7 and 8 are placed in the output buffer 1104 for the instructions of addresses 2 and 3. The next execution is used and the accumulator 202 is cleared.

The loop command of address 11 decrements the value of loop counter 3804, and if the value of the new loop counter 3804 is still greater than zero, it indicates an instruction to return to address 2.

As described in the section corresponding to the forty-fourth figure, in the example of performing the Jordan time recurrent neural network using the non-architectural program of the fifty-seventh figure, a start function is applied to the accumulator 202 value to generate an output. The layer node Y value, however, this example assumes that the accumulator 202 value is passed to the content layer node C before the start function is applied, rather than passing the true output layer node Y value. However, for a Jordan time recurrent neural network that applies a start function to the accumulator 202 value to produce a content layer node C, the instruction of address 10 will be removed from the non-architectural program of the fifty-seventh figure. In the embodiments described herein, the Elman or Jordan time recurrent neural network has a single hidden node layer (such as the fortieth and forty-two maps), however, it is to be understood that these processors 100 and neural networks Embodiments of unit 121 can efficiently perform computations associated with temporal recurrent neural networks having multiple hidden layers, in a manner similar to that described herein.

As described above in the section corresponding to the second figure, each neural processing unit 126 operates as a neuron within an artificial neural network, and all of the neural processing units 126 in the neural network unit 121 are processed in large scale parallel. The way to effectively calculate the neuron output value at one level of this network. The parallel processing of this neural network unit, especially the rotator collectively constructed using the neural processing unit multiplex register, is not intuitively thought of as a conventional way of calculating the output of the neuron layer. Further, the traditional approach usually involves being associated with a single neuron or The calculation of a very small subset of neurons (for example, using parallel arithmetic units to perform multiplication and addition calculations), then continuing to perform calculations associated with a neuron under the same level, and so on in a sequential manner until Completion of calculations for all neurons in this hierarchy. In contrast, the present invention, in each time-frequency period, all of the neural processing units 126 (neurons) of the neural network unit 121 perform parallel execution of a small set associated with the generation of all neuron outputs (eg, a single A multiplication and accumulation calculation). At the end of approximately M time-frequency periods, an M is the number of nodes connected in the current hierarchy. The neural network unit 121 calculates the output of all neurons. In many artificial neural network configurations, because of the large number of neural processing units 126, the neural network unit 121 can calculate its neuron output values for all neurons of the entire hierarchy at the end of the M time-frequency periods. As described herein, this calculation is efficient for all types of artificial neural network computations, including but not limited to feedforward and time recurrent neural networks such as Elman, Jordan, and long- and short-term memory networks. . Finally, although in the embodiments herein, the neural network unit 121 is configured as 512 neural processing units 126 (eg, taking a wide text configuration) to perform the calculation of the time recurrent neural network, the invention is not limited thereto, The neural network unit 121 is configured as 1024 neural processing units 126 (e.g., employing a narrow text configuration) to perform the calculation of the time recurrent neural network unit, and other numbers of neural processing units 126 having 512 and 1024 as previously described. The neural network unit 121 is also within the scope of the present invention.

However, the above is only the preferred embodiment of the present invention, and the scope of the present invention cannot be limited thereto. The simple equivalent changes and modifications made by the scope of the invention and the description of the invention are still within the scope of the invention. For example, the software can perform the functions, manufacture, shaping, simulation, description, and/or testing of the devices and methods described herein. This can be achieved by a general programming language (such as C, C++), a hardware description language (HDL) containing Verilog HDL, VHDL, etc., or other existing programs. The software can be placed on any known computer usable medium such as magnetic tape, semiconductor, magnetic disk, optical disk (such as CD-ROM, DVD-ROM, etc.), network cable, wireless or other communication medium. Embodiments of the apparatus and methods described herein can be included in a semiconductor intellectual property core, such as a micro processing core (such as an embodiment in a hardware description language) and converted to hardware by fabrication of integrated circuits. In addition, the devices and methods described herein may also comprise a combination of hardware and software. Therefore, any embodiments described herein are not intended to limit the scope of the invention. Furthermore, the present invention is applicable to a microprocessor device of a general-purpose computer. Finally, it is common for those skilled in the art to make use of the concepts and embodiments disclosed herein to form and adapt various structures to achieve the same objectives without departing from the scope of the invention.

100‧‧‧ processor

101‧‧‧Command capture unit

102‧‧‧ instruction cache

103‧‧‧Architecture Instructions

104‧‧‧Instruction Translator

105‧‧‧Microinstructions

106‧‧‧Renaming unit

108‧‧‧Reservation station

112‧‧‧Other execution units

114‧‧‧ memory subsystem

116‧‧‧Universal register

118‧‧‧Media register

121‧‧‧Neural Network Unit

122‧‧‧ Data Random Access Memory

124‧‧‧ weighted random access memory

126‧‧‧Neural Processing Unit

127‧‧‧Control and Status Register

128‧‧‧Sequencer

129‧‧‧Program memory

123,125,131‧‧‧ memory address

133‧‧‧ Results

Claims (21)

An apparatus comprising: an array of N processing units, each processing unit comprising: an accumulator having an output; an arithmetic unit having first, second and third inputs and performing thereon Computing to generate a result stored to the accumulator, the first input receiving the output of the accumulator; a weight input received by the second input to the arithmetic unit; and a multiplex register having a first a second data input, an output and a control input, the output being received by the third input to the arithmetic unit, the control input controlling a selection of the first and second data inputs; wherein the multiplex temporary storage The output of the device is received by the second data input of the multiplexed register of an adjacent processing unit, and when the control input selects the second data input, the multiplexed temporary of the N processing units The memory system operates together as a rotator of the same N characters; a first memory, loads N weight words of W columns, and supplies the N weight words of one of the W columns to the processing single Corresponding to the weight input of the N processing units of the array; and a second memory, loading N data words of the D columns, and providing the N data words of one of the D columns to The relative of the multiplexed registers of the N processing units of the processing unit array The first data should be entered. The apparatus of claim 1, wherein the arithmetic unit performs the operation of multiplying the second and third inputs to generate a product, and accumulating the product with the first input to produce the result. The apparatus of claim 1, wherein the arithmetic unit performs the operation to generate a difference between the second and third inputs, and accumulating the difference with the first input to produce the result. The device of claim 1, wherein the device is included in a processor, the processor having an instruction set, and one of the instruction instructions in the instruction set commands the processor to use the plurality of data characters by the processor One or more architectural registers are written to the second memory by a location specified by the architectural instruction. The device of claim 4, wherein the second memory comprises: a first buffer providing the N data characters of the column of the D columns to the N processing units; and a second UI And providing the plurality of data characters to the position in the second memory while the first data file is provided to the N processing units. The device of claim 1, wherein the device is packaged Included in a processor, the processor has an instruction set, and one of the instruction instructions in the instruction set commands the processor to write the plurality of weighted words to the first one of the one or more architectural registers of the processor. The memory is specified by one of the locations of the architecture instruction. The device of claim 6, further comprising: a buffer storing the N weight words, wherein the W is written to the first memory after being filled in by the plurality of examples of the architectural instructions One of the columns. The device of claim 1, wherein the device is included in a processor, the processor having an instruction set, and one of the instruction instructions in the instruction set commands the processor to use the plurality of data characters by the second The memory is read by one of the locations specified by the architectural instruction to one or more architectural registers of the processor. The device of claim 1, wherein W is at least N. The device of claim 1, wherein W is at least 512. A processor comprising: an instruction set having architectural instructions to command the processor to operate; an array of N processing units, each processing unit comprising: an accumulator having an output; An arithmetic unit having first, second, and third inputs, and Executing an operation to generate a result stored to the accumulator, the first input receiving the output of the accumulator; a weight input received by the second input to the arithmetic unit; and a multiplex register having a And a second data input, an output and a control input, the output being received by the third input to the arithmetic unit, the control input controlling a selection of the first and second data inputs; wherein the multiplexing The output of the register is received by the second data input of the multiplex register of an adjacent processing unit, and when the control input selects the second data input, the plurality of N processing units The work registers are jointly operated as a rotator of the same N characters; a first memory is loaded with N weight words of W columns, and the N weight words of one of the W columns are provided to Corresponding to the weight input of the N processing units of the processing unit array; and a second memory, loading N data words of D columns, and listing the N data of one of the D columns Text provided to the processing unit array The registers of the multiplexing of the N processing units of the first data corresponding to the input. The processor of claim 11, wherein the arithmetic unit performs the operation of multiplying the second and third inputs to generate a product, and accumulating the product with the first input to produce the result. The processor of claim 11, wherein the arithmetic unit performs the operation to generate a difference between the second and third inputs, and the difference is added to the first input to generate the result. The processor of claim 11, further comprising an architectural instruction instructing the processor to write the plurality of data words into the second memory by one or more architecture registers of the processor, the architecture instruction One location. The processor of claim 14, wherein the second memory comprises: a first file, providing the N data characters of the column of the D columns to the N processing units; and a second And receiving the plurality of data characters to write the position in the second memory while the first data file provides the N data characters to the N processing units. The processor of claim 1, further comprising an architectural instruction instructing the processor to write the plurality of weighted words into the first memory by the one or more architecture registers of the processor, the architecture instruction One location. The processor of claim 16 further comprising: a buffer storing the N weight texts, thereby After a plurality of examples are executed and filled in, they are written to one of the W columns of the first memory. The processor of claim 11, further comprising an architectural instruction instructing the processor to read the plurality of data words from the location in the second memory specified by the architecture instruction to one or more of the processor Architecture register. The device of claim 1, wherein W is at least N. The device of claim 1, wherein W is at least 512. A computer program product encoded in at least one non-transitory computer usable medium for use by a computer device, comprising: a computer usable code embodied in the medium for describing a device, the computer can use the code The first code includes: an array consisting of N processing units, each processing unit includes: an accumulator having an output; an arithmetic unit having first, second, and third inputs, and Performing an operation thereon to generate a result stored to the accumulator, the first input receiving the output of the accumulator; a weight input received by the second input to the arithmetic unit; and a multiplex register having First and second data input, one output And a control input, the output is received by the third input to the arithmetic unit, the control input controls a selection of the first and second data inputs; wherein the output of the multiplex register is Receiving, by the second data input of the multiplexed register of the adjacent processing unit, when the control input selects the second data input, the multiplexed registers of the N processing units work together as the same a rotator of N characters; a second code specifying a first memory to load N weight words of W columns, and providing the N weight words of one of the W columns to the processing Corresponding to the weight input of the N processing units of the cell array; and the third code, specifying a second memory to load N data words of the D columns, and one of the D columns The N data words are provided to the corresponding first data input of the multiplexed register of the N processing units of the processing unit array.