TWI793224B - Integrated circuit chip apparatus and related product - Google Patents

Abstract

The present disclosure provides an integrated circuit chip device and related products. The integrated circuit chip device includes: a main processing circuit and a plurality of basic processing circuits; at least one of the main processing circuit or a plurality of basic processing circuits includes: A type operation circuit, the data type operation circuit is used to perform conversion between floating-point type data and fixed-point type data. The technical solution provided by the present disclosure has the advantages of small calculation amount and low power consumption.

Description

Integrated circuit chip devices and related products

The present disclosure relates to the field of neural networks, in particular to an integrated circuit chip device and related products.

Artificial Neural Network (ANN) is a research hotspot in the field of artificial intelligence since the 1980s. It abstracts the human brain neuron network from the perspective of information processing, establishes a simple model, and forms different networks according to different connection methods. In engineering and academia, it is often referred to directly as a neural network or a neural network. A neural network is an operational model consisting of a large number of nodes (or neurons) connected to each other. The calculation of the existing neural network is based on a CPU (Central Processing Unit, central processing unit) or a GPU (Graphics Processing Unit, graphics processing unit) to realize the calculation of the neural network, which requires a large amount of calculation and high power consumption.

Embodiments of the present disclosure provide an integrated circuit chip device and related products, which can increase the processing speed of a computing device and improve efficiency.

In a first aspect, an integrated circuit chip device is provided, and the integrated circuit chip device includes: a main processing circuit and a plurality of basic processing circuits; at least one of the main processing circuit or a plurality of basic processing circuits includes: a data type An operation circuit, the data type operation circuit is used to perform conversion between floating-point type data and fixed-point type data;

The main processing circuit is used to execute each continuous operation in the neural network operation and transmit data with the basic processing circuit;

The plurality of basic processing circuits are used to execute calculations in the neural network in parallel according to the data transmitted by the main processing circuit, and transmit the calculation results to the main processing circuit.

In a second aspect, a neural network computing device is provided, and the neural network computing device includes one or more integrated circuit chip devices provided in the first aspect.

A third aspect provides a combined processing device, which includes: the neural network computing device provided in the second aspect, a general interconnection interface, and a general processing device;

The neural network computing device is connected to the general processing device through the general interconnection interface.

In a fourth aspect, a chip is provided, and the chip integrates the device of the first aspect, the device of the second aspect, or the device of the third aspect.

In a fifth aspect, an electronic device is provided, and the electronic device includes the chip of the fourth aspect.

In a sixth aspect, there is provided a method for calculating a neural network, the method is applied in an integrated circuit chip device, and the integrated circuit chip device includes: the integrated circuit chip device described in the first aspect, and the integrated circuit chip device is used for Perform neural network operations.

It can be seen that, through the embodiment of the present disclosure, the data conversion operation circuit is provided to perform operation after conversion of the type of the data block, which saves transmission resources and computing resources, so it has the advantages of low power consumption and small amount of calculation.

In order to enable those skilled in the art to better understand the present disclosure, the following will clearly and completely describe the technical solutions in the present disclosure embodiments in conjunction with the drawings in the present disclosure embodiments. Obviously, the described embodiments are only It is a part of the embodiments of the present disclosure, but not all of them. Based on the embodiments in the present disclosure, all other embodiments obtained by persons with ordinary knowledge in the technical field without creative efforts fall within the protection scope of the present disclosure.

In the device provided in the first aspect, the integrated circuit chip device further includes: a branch processing circuit, the branch processing circuit is arranged between the main processing circuit and at least one basic processing circuit; the branch processing circuit is configured to Data is forwarded between the main processing circuit and the at least one base processing circuit.

In the device provided in the first aspect, the main processing circuit is used to obtain the data block to be calculated and the operation instruction, and convert the data block to be calculated into a fixed-point type data block through the data type operation circuit, according to the The arithmetic instruction divides the fixed-point data block to be calculated into a distribution data block and a broadcast data block; splitting the distribution data block to obtain multiple basic data blocks, and distributing the multiple basic data blocks to the at least one base processing circuit, broadcasting the broadcast data block to the at least one base processing circuit;

The basic processing circuit is configured to perform an inner product operation on the basic data block and the broadcast data block in a fixed-point type to obtain an operation result, and send the operation result to the main processing circuit;

The main processing circuit is configured to process the operation result to obtain the data block to be calculated and an instruction result of the operation instruction.

In the device provided in the first aspect, the main processing circuit is specifically configured to broadcast the broadcast data block to the multiple basic processing circuits in one pass.

In the device provided in the first aspect, the basic processing circuit is specifically configured to perform inner product processing on the basic data block and the broadcast data block in a fixed-point data type to obtain an inner product processing result, and process the inner product The results are accumulated to obtain an operation result, and the operation result is sent to the main processing circuit.

In the device provided in the first aspect, the main processing circuit is configured to accumulate the operation results to obtain an accumulation result when the operation result is the result of inner product processing, and arrange the accumulation results to obtain the The data block to be calculated and the instruction result of the operation instruction.

In the device provided in the first aspect, the main processing circuit is specifically configured to divide the broadcast data block into multiple partial broadcast data blocks, and broadcast the multiple partial broadcast data blocks to the multiple Basic processing circuit.

In the device provided in the first aspect, the basic processing circuit is specifically configured to perform an inner product processing on the partial broadcast data block and the basic data block in a fixed-point data type to obtain an inner product processing result, and convert the The inner product processing results are accumulated to obtain a partial operation result, and the partial operation result is sent to the main processing circuit.

In the device provided in the first aspect, the basic processing circuit is specifically configured to multiplex the part of the broadcast data block n times and perform an inner product operation between the part of the broadcast data block and the n basic data blocks to obtain n partial processing results, The n partial processing results are respectively accumulated to obtain n partial operation results, and the n partial operation results are sent to the main processing circuit, and the n is an integer greater than or equal to 2.

In the device provided in the first aspect, the main processing circuit includes: a main register or a main on-chip cache circuit;

The basic processing circuit includes: a basic register or a basic on-chip cache circuit.

In the device provided in the first aspect, the main processing circuit includes: a vector arithmetic unit circuit, an arithmetic logic unit circuit, an accumulator circuit, a matrix transposition circuit, a direct memory access circuit, a data type operation circuit or a data rearrangement circuit one or any combination of them.

In the device provided in the first aspect, the main processing circuit is configured to obtain a data block to be calculated and an operation instruction, and divide the data block to be calculated into a distribution data block and a broadcast data block according to the operation instruction; performing splitting processing on the distribution data block to obtain multiple basic data blocks, distributing the multiple basic data blocks to the at least one basic processing circuit, and broadcasting the broadcast data block to the at least one basic processing circuit;

The basic processing circuit is used to convert the basic data block and the broadcast data block into a fixed-point type data block, perform an inner product operation according to the fixed-point type data block to obtain an operation result, and convert the operation result into a float Send the point data to the main processing circuit;

The main processing circuit is configured to process the operation result to obtain the data block to be calculated and an instruction result of the operation instruction.

In the device provided in the first aspect, the branch processing circuit includes a plurality of branch processing circuits, the main processing circuit is respectively connected to the plurality of branch processing circuits, and each branch processing circuit is connected to at least one basic processing circuit.

In the device provided in the first aspect, the data is one or any combination of vectors, matrices, three-dimensional data blocks, four-dimensional data blocks, and n-dimensional data blocks.

In the device provided in the first aspect, if the operation instruction is a multiplication instruction, the main processing circuit determines that the multiplier data block is a broadcast data block, and the multiplicand data block is a distribution data block;

If the operation instruction is a convolution instruction, the main processing circuit determines that the input data block is a broadcast data block, and the convolution kernel is a distribution data block.

In the method provided in the sixth aspect, the operation of the neural network includes: one or more of convolution operation, matrix multiplication matrix operation, matrix multiplication vector operation, paranoid operation, full connection operation, GEMM operation, GEMV operation, and activation operation random combination.

Referring to FIG. 1a, FIG. 1a is a schematic structural diagram of an integrated circuit chip device. As shown in FIG. 1a, the chip device includes: a main processing circuit, a basic processing circuit and a branch processing circuit (optional). in,

The main processing circuit may include a register and/or an on-chip buffer circuit, and the main processing circuit may also include: a control circuit, a vector operator circuit, an ALU (arithmetic and logic unit, arithmetic logic unit) circuit, an accumulator circuit, a DMA (Direct Memory Access, direct memory access) circuits and other circuits, of course, in practical applications, the above-mentioned main processing circuit can also be added, conversion circuits (such as matrix transposition circuits), data rearrangement circuits or activation circuits and other circuits;

Optionally, the main processing circuit may include: a data type conversion operation circuit, and the data type conversion operation circuit may be used to convert the received or sent data from floating-point type data to fixed-point type data. Of course, in practical applications, the Fixed-point type data is converted to floating-point type data. The present invention does not limit the specific form of the above-mentioned data type conversion operation circuit.

The main processing circuit also includes a data sending circuit, a data receiving circuit or an interface. The data sending circuit can integrate a data distribution circuit and a data broadcast circuit. Of course, in practical applications, the data distribution circuit and the data broadcast circuit can also be set separately; The above-mentioned data sending circuit and data receiving circuit can also be integrated together to form a data sending and receiving circuit. For broadcast data, that is the data that needs to be sent to each underlying processing circuit. For distribution data, that is, data that needs to be selectively sent to some basic processing circuits, the specific selection method can be specifically determined by the main processing circuit according to the load and calculation method. For the broadcast sending mode, the broadcast data is sent to each basic processing circuit in a broadcast form. (In practical applications, the broadcast data is sent to each basic processing circuit through one broadcast, and the broadcast data can also be sent to each basic processing circuit through multiple broadcasts. The specific implementation of this application does not limit the above-mentioned The number of broadcasts), for the distribution transmission method, the distribution data is selectively sent to some basic processing circuits.

When distributing data, the control circuit of the main processing circuit transmits data to some or all of the basic processing circuits (the data can be the same or different. Specifically, if the data is sent in a distributed manner, each basic processing circuit that receives data receives The received data can be different, of course, some basic processing circuits can also receive the same data;

Specifically, when broadcasting data, the control circuit of the main processing circuit transmits data to some or all of the basic processing circuits, and each basic processing circuit receiving data can receive the same data.

Optionally, the vector operator circuit of the above-mentioned main processing circuit can perform vector operations, including but not limited to: addition, subtraction, multiplication, and division of two vectors, addition, subtraction, multiplication, and division operations between vectors and constants, or performing operations on each element in the vector Arbitrary operations. Wherein, the continuous operation may specifically be, vector and constant addition, subtraction, multiplication, division operation, activation operation, accumulation operation and so on.

Each basic processing circuit may include a basic register and/or a basic on-chip cache circuit; each basic processing circuit may also include: one or any combination of an inner product operator circuit, a vector operator circuit, an accumulator circuit, and the like. The above-mentioned inner product operator circuit, vector operator circuit, and accumulator circuit can all be integrated circuits, and the above-mentioned inner product operator circuit, vector operator circuit, and accumulator circuit can also be independently arranged circuits.

The chip device may also optionally include one or more branch processing circuits. For example, when there are branch processing circuits, the main processing circuit is connected to the branch processing circuits, the branch processing circuits are connected to the basic processing circuit, and the internal processing circuit of the basic processing circuit The product operator circuit is used to perform the inner product operation between data blocks, the control circuit of the main processing circuit controls the data receiving circuit or the data sending circuit to send and receive external data, and the control circuit controls the data sending circuit to distribute the external data to the branch processing A circuit, the branch processing circuit is used to send and receive data from the main processing circuit or the basic processing circuit. The structure shown in Figure 1a is suitable for the calculation of complex data, because for the main processing circuit, the number of connected units is limited, so it is necessary to add branch processing circuits between the main processing circuit and the basic processing circuit to achieve more The basic processing circuit is connected to realize the calculation of complex data blocks. The connection structure of the branch processing circuit and the basic processing circuit can be arbitrary, and is not limited to the H-shaped structure in FIG. 1a. Optionally, the main processing circuit to the basic processing circuit is a broadcast or distribution structure, and the basic processing circuit to the main processing circuit is a gather (gather) structure. Broadcasting, distribution and collection are defined as follows, for distribution or broadcasting structure, the number of basic processing circuits at this time is greater than the main processing circuit, that is, 1 main processing circuit corresponds to multiple basic processing circuits, that is, from the main processing circuit to multiple basic processing circuits The processing circuit is a broadcast or distribution structure, conversely, from multiple basic processing circuits to the main processing circuit can be a collection structure.

The basic processing circuit receives the data distributed or broadcast by the main processing circuit and saves it in the on-chip cache of the basic processing circuit, and can perform calculations to generate results, and can send data to the main processing circuit.

The data involved in the basic processing circuit can be data of any data type, data represented by floating-point numbers of any bit width or data represented by fixed-point numbers of any bit width; all involved arithmetic circuits and storage The circuits can be arithmetic circuits and storage circuits of any type of data that can be processed, can be arithmetic circuits and storage circuits of floating-point numbers of any bit width, and can be arithmetic circuits and storage circuits of fixed-point numbers of any bit width.

Optionally, each basic processing circuit can include a data type conversion operation circuit, and a data type conversion operation circuit can also be configured in part of the basic processing circuit; the data type conversion operation circuit can be used to convert the received or sent data from the floating point type to Data can be converted into fixed-point type data, and fixed-point type data can also be converted into floating-point type data. The present invention does not limit the specific form of the above-mentioned data type conversion operation circuit.

Optionally, the vector operator circuit of the basic processing circuit can perform vector operations on the two vectors after the data type conversion. Of course, in practical applications, the inner product operator circuit of the basic processing circuit can perform the vector operation on the two vectors after the data type conversion The inner product operation is performed on two vectors, and the accumulator circuit can also accumulate the result of the inner product operation.

In an optional solution, the two vectors can be stored in on-chip caches and/or registers, and the basic processing circuit can extract the two vectors to perform operations according to actual calculation needs. The operation includes but not limited to: inner product operation, multiplication operation, addition operation or other operations.

In an optional solution, the result of the inner product operation can be accumulated to the on-chip cache and/or register; the advantage of the optional solution is that it reduces the amount of data transmission between the basic processing circuit and the main processing circuit, and improves the calculation efficiency. efficiency, reducing data transfer power consumption.

In an optional solution, the result of the inner product operation is not accumulated, but directly transmitted as the result; the advantage of this technical solution is that it reduces the amount of calculation inside the basic processing circuit and improves the calculation efficiency of the basic processing circuit.

In an optional solution, each basic processing circuit can perform multiple sets of inner product operations of two vectors, and can also separately accumulate the results of multiple sets of inner product operations;

In an optional solution, multiple sets of two vector data may be stored in on-chip caches and/or registers;

In an optional solution, the results of multiple sets of inner product operations can be respectively accumulated in on-chip caches and/or registers;

In an optional solution, the results of the inner product operations of each group may not be accumulated, but directly transmitted as the result;

In an optional solution, each basic processing circuit can perform the inner product operation of the same vector and multiple vectors ("one-to-many" inner product, that is, the inner product of multiple groups of two vectors in each group One vector is shared), and the inner product results corresponding to each vector are accumulated separately. This technical solution can realize multiple calculations of different input data with the same set of weights, increase data multiplexing, reduce the amount of data transmission of internal data in the basic processing circuit, improve calculation efficiency, and reduce power consumption.

Specifically, in the data used to calculate the inner product, the data sources of the shared vector of each group and the other vector of each group (that is, the vector that is different between each group) can be different:

In an optional solution, when calculating the inner product, the vectors shared by each group are broadcast or distributed from the main processing circuit or the branch processing circuit;

In an optional solution, when calculating the inner product, the vectors shared by each group come from the on-chip cache;

In an optional solution, when calculating the inner product, the vector shared by each group comes from a register;

In an optional solution, when calculating the inner product, another non-shared vector of each group is broadcast or distributed from the main processing circuit or the branch processing circuit;

In an alternative, when calculating the inner product, another non-shared vector for each group comes from the on-chip cache;

In an alternative, another non-shared vector for each group comes from a register when computing the inner product;

In an optional solution, when multiple groups of inner product operations are performed, each group of shared vectors retains any number of copies in the on-chip cache and/or register of the basic processing circuit;

In an optional solution, the shared vector can be reserved for each group of inner products;

In an optional solution, only one copy of the shared vector can be reserved;

Specifically, the results of multiple sets of inner product operations can be respectively accumulated in on-chip caches and/or registers;

Specifically, the results of the inner product operations of each group may not be accumulated, but directly transmitted as the result;

Refer to the structure shown in Figure 1a, which includes a main processing circuit (capable of performing vector operations), and multiple basic processing circuits (capable of performing inner product operations). The advantage of this combination is that the device can not only use the basic processing circuit to perform matrix and vector multiplication operations, but also use the main processing circuit to perform other arbitrary vector operations, so that the device can complete more quickly under the limited hardware circuit configuration. More calculations reduce the number of data transmissions with the outside of the device, improve computing efficiency, and reduce power consumption. In addition, the chip can be equipped with a data type conversion operation circuit in the basic processing circuit and/or the main processing circuit, so that the floating point type data can be converted into fixed point type data when performing neural network calculations, and the fixed point type data can also be converted into fixed point type data. Floating-point type data, and this chip can dynamically allocate the calculation amount (ie load) of each circuit (mainly the main processing circuit and the basic processing circuit), and the circuit will convert the data type, which can reduce the complexity of data calculation programs, reduce power consumption, and dynamically allocate data type conversions to achieve computing efficiency without affecting the chip. The distribution method includes but is not limited to: load balancing, load minimum value distribution, and the like.

A structural diagram of the fixed-point data is shown in Figure 1d. As shown in Figure 1d, it is an expression method for fixed-point data. For a computing system, the number of storage bits for a floating-point data is 32 bits, while for fixed-point data , especially using floating-point data as shown in Figure 1d for data representation, the number of storage bits for one fixed-point data can be less than 16Bit, so for this conversion, it can greatly reduce the number of calculations between calculators. In addition, for calculators, the data storage space with fewer bits is also smaller, that is, the storage overhead will be smaller, and the amount of calculation will be reduced, that is, the calculation overhead will be reduced, so it can reduce the calculation overhead and Storage overhead, but some overhead is also required for data type conversion, hereinafter referred to as conversion overhead. For data with a large amount of calculation and data storage, the conversion overhead is relatively higher than the subsequent calculation overhead, storage overhead, and transmission overhead. It is almost negligible, so for data with a large amount of calculation and a large amount of data storage, this disclosure adopts the technical solution of converting the data type into a fixed-point type of data. On the contrary, for data with a small amount of calculation and a small amount of data storage, At this time, since the calculation overhead, storage overhead, and transmission overhead are relatively small, if fixed-point data is used at this time, the precision of fixed-point data will be slightly lower than that of floating-point data. Accuracy, so the data of the fixed-point type is converted into floating-point data here, that is, the purpose of improving the accuracy of the calculation by adding a small overhead.

Referring to the device shown in FIG. 1b, the device shown in FIG. 1b is a computing device without branch processing circuits. The device shown in FIG. 1b includes: a main processing circuit and N basic processing circuits, wherein the main processing circuit ( The specific structure is shown in Figure 1c) and N basic processing circuits can be directly or indirectly connected, such as indirect connection, an optional solution can include N/4 branch processing circuits as shown in Figure 1a, each The four branch processing circuits are respectively connected to four basic processing circuits. For the main processing circuit and the circuits respectively included in the N basic processing circuits, please refer to the above description shown in FIG. The processing circuit can also be set in the branch processing circuit. In addition, the number of each branch processing circuit connected to the basic processing circuit is not limited to 4, and the manufacturer can configure it according to actual needs. The above-mentioned main processing circuit and/or the N basic processing circuits may all include a data type conversion operation circuit. Specifically, the main processing circuit may include a data type operation circuit, or the N basic processing circuits or a part of them may include a data type conversion circuit. The type conversion circuit may also include the main processing circuit and N basic processing circuits or a part thereof. The above-mentioned main processing circuit can dynamically allocate the operation entity of the data type conversion step according to the neural network calculation instruction. Specifically, the main processing circuit can determine whether to perform the data type conversion step on the received data according to its own load. Specifically, the The value of the load is set in multiple intervals, and each interval corresponds to the execution subject of the assigned data type conversion step. For example, taking three intervals as an example, the load value of interval 1 is relatively low, and the data type conversion step can be performed independently by the main processing circuit. The load value of interval 2 is between interval 1 and interval 3, and the data type conversion step can be performed jointly by the main processing circuit or N basic processing circuits. The load value of interval 3 is relatively high, and the data type conversion step can be performed by N basic processing circuits . In this regard, it can be implemented in an explicit manner. For example, the main processing circuit can configure a special instruction or instruction. When the basic processing circuit receives the special instruction or instruction, it will determine to perform the data type conversion step. If the basic processing circuit does not receive When there are special instructions or instructions, it is determined not to perform the data type conversion step. As another example, it may be performed in an implicit manner, for example, when the basic processing circuit receives data of a floating-point type and determines that an inner product operation needs to be performed, it converts the data type into fixed-point data.

The following provides a method for realizing calculation using the device shown in Figure 1a. The calculation method can specifically be the calculation method of the neural network, such as the forward operation of the neural network, the training of the neural network, and in practical applications, the forward operation Operations such as matrix multiplication, convolution operation, activation operation, transformation operation, etc. can be performed according to different input data, and the above operations can be realized by the device shown in FIG. 1a.

The data conversion operation circuit of the main processing circuit first converts the type of data and then the control circuit transmits it to the basic processing circuit for operation. For example, the data conversion operation circuit of the main processing circuit can convert floating-point numbers into fixed-point numbers with lower bit width Then transmit to the basic processing circuit, the advantage is that the bit width of the transmitted data can be reduced, the total number of bits transmitted can be reduced, and the basic processing circuit can perform position-wide fixed-point operations with higher efficiency and lower power consumption.

If the data received by the basic processing circuit is floating-point data, then the basic processing circuit can convert the data type after receiving the data and then perform calculations by the data conversion operation circuit. For example, the basic processing circuit receives the transmission from the main processing circuit The floating-point number, the data conversion operation circuit is then converted into a fixed-point number, and then the inner product operator circuit, vector operator circuit or accumulator circuit of the basic processing circuit performs operations to improve operation efficiency and reduce power consumption.

After the basic processing circuit calculates the result, it can perform data type conversion first and then transmit it to the main processing circuit. For example, the floating-point number calculation result calculated by the basic processing circuit can be converted into a low-bit-width fixed-point number and then transmitted to the main processing circuit. , which has the advantage of reducing the data bit width in the transmission process, higher efficiency, and saving power consumption.

The main processing circuit transmits the data to be calculated to all or part of the basic processing circuit; taking matrix multiplication by vector calculation as an example, the control circuit of the main processing circuit can split the matrix data into each column as a basic data, such as m*n The matrix can be split into n vectors with m rows, and the control circuit of the main processing circuit distributes the split n vectors with m rows to multiple basic processing circuits. For vectors, the control circuit of the main processing circuit may broadcast the vector as a whole to each basic processing circuit. If the value of m is relatively large, then the control circuit can first split the m*n matrix into x*n vectors, taking x=2 as an example, it can be split into 2n vectors, each vector contains m/ 2 lines, that is, each vector of n m-line vectors is divided into 2 vectors. Taking the first line as an example, if the first vector of n m-line vectors is 1000 lines, then it can be divided into 2 vectors as , the first 500 lines form the first vector, and the last 500 lines form the second vector, and the control circuit broadcasts the two vectors to multiple basic processing circuits through two broadcasts.

The data transmission method may be broadcast or distribution, or any other possible transmission method;

After the basic processing circuit receives the data, it executes the calculation and obtains the calculation result;

The basic processing circuit transmits the operation result back to the main processing circuit;

The operation result may be an intermediate operation result or a final operation result.

Use the device shown in Figure 1a to complete the operation of multiplying the vector by the matrix;

(Matrix multiplication by vector can be that each row in the matrix performs an inner product operation with the vector, and arranges these results into a vector in the order of the corresponding rows.)

The following describes the operation of calculating the multiplication of a matrix S with M rows and L columns and a vector P with a length of L, as shown in Figure 2a below, (each row in the matrix S has the same length as the vector P, and the data in them is by position One-to-one correspondence) The neural network computing device has K basic processing circuits:

Referring to Figure 2, Figure 2 provides an implementation method of matrix multiplication by vector, which may specifically include:

Step S201, the data conversion operation circuit of the main processing circuit converts each row of data in the matrix S into fixed-point data, the control circuit of the main processing circuit distributes it to one of the K basic processing circuits, and the basic processing circuit receives The received distribution data is stored in the on-chip cache and/or register of the basic processing circuit;

In an optional solution, if the number of rows of the matrix S M<=K, the control circuit of the main processing circuit distributes one row of the S matrix to the K basic processing circuits respectively;

In an optional solution, if the number of rows M>K of the matrix S, the control circuit of the main processing circuit distributes the data of one or more rows in the matrix S to each basic processing circuit.

The set of rows in S distributed to the i-th basic processing circuit is Ai, and there are a total of Mi rows. Figure 2c shows the calculation to be performed on the i-th basic processing circuit.

In an optional solution, in each basic processing circuit, for example, in the i-th basic processing circuit, the received distribution data such as the matrix Ai may be stored in the register and/or on-chip cache of the i-th basic processing circuit ; The advantage is to reduce the amount of data transmission of the subsequent distribution data, improve the calculation efficiency, and reduce the power consumption.

Step S202, the data type operation circuit of the main processing circuit converts the vector P into fixed-point data, and the control circuit of the main processing circuit transmits each part of the fixed-point vector P to K basic processing circuits in a broadcast manner;

In an optional solution, the control circuit of the main processing circuit can only broadcast each part of the vector P to the registers or on-chip caches of each basic processing circuit once, and the i-th basic processing circuit can process the data of the vector P obtained this time Full multiplexing is performed to complete the inner product operation corresponding to each row in the matrix Ai. The advantage is that it reduces the data transmission amount of repeated transmission of the vector P from the main processing circuit to the basic processing circuit, improves execution efficiency, and reduces transmission power consumption.

In an optional solution, the control circuit of the main processing circuit can broadcast each part of the vector P to the registers or on-chip caches of each basic processing circuit for multiple times, and the i-th basic processing circuit can obtain the data of the vector P each time Without multiplexing, the inner product operation corresponding to each row in the matrix Ai is completed in batches; the advantage is that the data transmission amount of the single-transmission vector P inside the basic processing circuit is reduced, and the basic processing circuit cache and/or Or the capacity of registers, improve execution efficiency, reduce transmission power consumption, and reduce costs.

In an optional solution, the control circuit of the main processing circuit can broadcast each part of the vector P to the registers or on-chip caches of each basic processing circuit for multiple times, and the i-th basic processing circuit can obtain the data of the vector P each time Perform partial multiplexing to complete the inner product operation corresponding to each row in the matrix Ai; the advantage is that it reduces the amount of data transmission from the main processing circuit to the basic processing circuit, and also reduces the amount of data transmission inside the basic processing circuit, improving execution efficiency , to reduce transmission power consumption.

Step S203, the inner product operator circuit of the K basic processing circuits calculates the inner product of the data of the matrix S and the vector P, for example, the ith basic processing circuit calculates the inner product of the data of the matrix Ai and the data of the vector P;

In step S204, the accumulator circuits of the K basic processing circuits accumulate the results of the inner product operation to obtain an accumulation result, and transmit the accumulation result back to the main processing circuit in the form of a fixed-point type.

In an optional solution, the partial sum (partial sum is a part of the accumulation result obtained by performing the inner product operation of the basic processing circuit each time, for example, the accumulation result is: F1*G1+ F2*G2+ F3*G3+ F4*G4+ F5* G5, then the partial sum can be: the value of F1*G1+ F2*G2+ F3*G3) is transmitted back to the main processing circuit for accumulation; the advantage is that it reduces the amount of computation inside the basic processing circuit and improves the computing efficiency of the basic processing circuit.

In an optional solution, the partial sum obtained by the inner product operation performed by the basic processing circuit each time can also be stored in the register and/or on-chip cache of the basic processing circuit, and transferred back to the main processing circuit after the accumulation is completed; the advantage is that The amount of data transmission between the basic processing circuit and the main processing circuit is reduced, the operation efficiency is improved, and the power consumption of data transmission is reduced.

In an optional solution, the sum obtained from the inner product operation performed by the basic processing circuit each time may also be stored in the register and/or on-chip cache of the basic processing circuit for accumulation in some cases, and transferred to the main The processing circuit accumulates, and after the accumulation is completed, it is transmitted back to the main processing circuit; the advantage is that the amount of data transmission between the basic processing circuit and the main processing circuit is reduced, the calculation efficiency is improved, the power consumption of data transmission is reduced, and the basic processing circuit is reduced. The amount of internal calculations can be increased to improve the calculation efficiency of the basic processing circuit.

Referring to Fig. 2b, use the device as shown in Fig. 1a to complete the operation of matrix multiplication matrix;

The following describes the operation of calculating the multiplication of a matrix S with a size of M rows and L columns and a matrix W with a size of L rows and N columns, (each row in the matrix S has the same length as each column of the matrix W, as shown in Figure 2d) The neural network computing device has K basic processing circuits:

Step S201b, the control circuit of the main processing circuit distributes each row of data in the matrix S to one of the K basic processing circuits, and the basic processing circuit stores the received data in the on-chip cache and/or register;

In an optional solution, if the number of rows of S M<=K, the control circuit of the main processing circuit distributes one row of the S matrix to the M basic processing circuits respectively;

In an optional solution, if the number of rows of S is M>K, the control circuit of the main processing circuit distributes the data of one or more rows in the matrix S to each basic processing circuit.

There are Mi rows in S distributed to the i-th basic processing circuit, and the set of Mi rows is called Ai, as shown in Figure 2e, which shows the computation to be performed on the i-th basic processing circuit.

In an optional solution, in each basic processing circuit, for example, in the i-th basic processing circuit:

The matrix Ai distributed by the main processing circuit is received, and the matrix Ai is stored in the i-th basic processing circuit register and/or on-chip cache; the advantage is that the subsequent data transmission amount is reduced, the calculation efficiency is improved, and the power consumption is reduced.

Step S202b, the control circuit of the main processing circuit transmits each part in the matrix W to each basic processing circuit in a broadcast manner;

In an alternative solution, each part of the matrix W can be broadcast only once to the registers or on-chip caches of each basic processing circuit, and the i-th basic processing circuit fully multiplexes the data of the matrix W obtained this time, Complete the inner product operation corresponding to each row in the matrix Ai; the multiplexing in this embodiment can be the repeated use of the basic processing circuit in the calculation, for example, the multiplexing of the data of the matrix W can be the data of the matrix W in multiple times used.

In an optional solution, the control circuit of the main processing circuit can broadcast each part of the matrix W to the registers or on-chip caches of each basic processing circuit for multiple times, and the i-th basic processing circuit performs a calculation on the data of the matrix W obtained each time No multiplexing is performed, and the inner product operation corresponding to each row in the matrix Ai is completed in stages;

In an optional solution, the control circuit of the main processing circuit can broadcast each part of the matrix W to the registers or on-chip caches of each basic processing circuit for multiple times, and the i-th basic processing circuit performs a calculation on the data of the matrix W obtained each time Perform partial multiplexing to complete the inner product operation corresponding to each row in the matrix Ai;

In an optional solution, each basic processing circuit, such as the i-th basic processing circuit, calculates the inner product of the data of the matrix Ai and the data of the matrix W;

In step S203b, the accumulator circuit of each basic processing circuit accumulates the result of the inner product operation and transmits it back to the main processing circuit.

In an optional solution, the basic processing circuit may transfer the partial sum obtained by performing the inner product operation each time back to the main processing circuit for accumulation;

In an optional solution, the partial sum obtained by the inner product operation performed by the basic processing circuit each time may also be stored in the register and/or on-chip cache of the basic processing circuit, and transferred back to the main processing circuit after the accumulation is completed;

In an optional solution, the sum obtained from the inner product operation performed by the basic processing circuit each time may also be stored in the register and/or on-chip cache of the basic processing circuit for accumulation in some cases, and transferred to the main The processing circuit performs accumulation, and after the accumulation is completed, it is transmitted back to the main processing circuit.

Referring to Figure 3a, use the device shown in Figure 1a to complete the full connection operation:

If the input data of the fully connected layer is a vector (that is, the input of the neural network is a single sample), then the weight matrix of the fully connected layer is used as the matrix S, and the input vector is used as the vector P, according to the use method of the device. Carry out the operation of matrix multiplication vector as shown in Figure 2;

If the input data of the fully connected layer is a matrix (that is, the input of the neural network is the case where multiple samples are used as a batch), the weight matrix of the fully connected layer is used as the matrix S, and the input vector is used as the matrix W, or the fully connected layer The weight matrix is used as matrix W, and the input vector is used as matrix S, according to the execution operation of the matrix multiplication matrix shown in Figure 2c according to the use of the device;

Referring to Figure 3b, the convolution operation is performed using the setup shown in Figure 1a:

For a convolutional layer, record the number of convolution kernels as M;

Step S301, the control circuit of the main processing circuit distributes the weight of each convolution kernel in the convolution layer weights to one of the K basic processing circuits, and stores them in the on-chip cache and/or register of the basic processing circuit ;

In an optional solution, if the number of convolution kernels M<=K, the control circuit of the main processing circuit distributes the weight of a convolution kernel to the M basic processing circuits respectively;

In an optional solution, if the number of convolution kernels M>K, the control circuit of the main processing circuit distributes weights of one or more convolution kernels to each basic processing circuit.

A total of Mi convolution kernels are distributed to the i-th basic processing circuit, and the set of these convolution kernel weights is called Ai.

In an optional solution, in each basic processing circuit, for example, in the i-th basic processing circuit:

Save the received convolution kernel weight Ai distributed by the main processing circuit in its register and/or on-chip cache;

Step S302, the control circuit of the main processing circuit transmits each part of the input data T to each basic processing circuit in a broadcast manner;

In an optional solution, the control circuit of the main processing circuit can broadcast each part of the input data T to the registers or on-chip caches of each basic processing circuit only once, and the i-th basic processing circuit performs a The data is fully multiplexed, and the inner product operation corresponding to each convolution kernel in Ai is completed;

In an optional solution, the control circuit of the main processing circuit can broadcast each part of the input data T to the registers or on-chip buffers of each basic processing circuit for multiple times, and the i-th basic processing circuit performs each input data T The data of is not multiplexed, and the inner product operation corresponding to each convolution kernel in Ai is completed in stages;

In an optional solution, the control circuit of the main processing circuit can broadcast each part of the input data T to the registers or on-chip buffers of each basic processing circuit for multiple times, and the i-th basic processing circuit performs each input data T The data of Ai is partially multiplexed, and the inner product operation corresponding to each convolution kernel in Ai is completed;

Step S303, each basic processing circuit calculates the data inner product of the convolution kernel and the input data T, for example, the i-th basic processing circuit calculates the inner product of each convolution kernel of Ai and the data of the input data T;

Step S304, the accumulator circuit of each basic processing circuit accumulates the result of the inner product operation and transmits it back to the main processing circuit:

In an optional solution, the basic processing circuit can transfer the partial sum obtained by performing the inner product operation each time back to the main processing circuit for accumulation;

In an optional solution, the basic processing circuit may also store the partial sum obtained by the inner product operation performed each time in the register and/or on-chip cache of the basic processing circuit, and transfer it back to the main processing circuit after the accumulation is completed;

In an optional solution, the basic processing circuit can also store the sum of the inner product operation performed each time in the register and/or on-chip cache of the basic processing circuit for accumulation, and in some cases transfer it to the main The processing circuit performs accumulation, and after the accumulation is completed, it is transmitted back to the main processing circuit;

The method of updating weights using the device shown in Figure 1a:

The vector arithmetic unit circuit of the main processing circuit is used to implement the weight update function in the training process of the neural network. Specifically, the weight update refers to a method of using the gradient of the weight to update the weight.

In an optional solution, the vector operator circuit of the main processing circuit is used to perform addition and subtraction operations on the two vectors of the weight and the weight gradient to obtain an operation result, which is the updated weight.

In an optional solution, the vector operator circuit of the main processing circuit is used to multiply or divide the weight and the weight gradient by a number to obtain the intermediate weight and the intermediate weight gradient value, and the vector operator circuit is used for the intermediate weight and The intermediate weight gradient value is added and subtracted to obtain the operation result, and the operation result is the update weight.

In an optional solution, a set of momentum can be calculated using the gradient of the weight, and then the momentum and the weight can be added and subtracted to obtain the updated weight;

The method of realizing the reverse operation of the fully connected layer using the device shown in Figure 1a

The reverse operation of the fully connected layer can be divided into two parts, as shown in Figure 4a below, the solid arrow indicates the forward calculation process of the fully connected layer, as shown in Figure 4b, indicates the reverse calculation process of the fully connected layer.

The reverse operation of the fully connected layer shown in Figure 4a and Figure 4b can be completed by using the matrix multiplication matrix method shown in Figure 2b as shown in Figure 1a;

Use the device shown in Figure 1a to realize the reverse operation of the convolutional layer;

The reverse operation of the convolutional layer can be divided into two parts. In Figure 5a below, the solid arrow indicates the forward calculation process of the convolutional layer. As shown in Figure 5b, it indicates the reverse calculation process of the convolutional layer.

For the reverse operation of the convolutional layer shown in Figure 5a and Figure 5b, the device shown in Figure 1a can be used to complete the reverse operation of the convolutional layer using the method shown in Figure 3b.

A method of implementing BLAS (Basic Linear Algebra Subprograms) functions using the device shown in Figure 1a

GEMM calculation refers to: the matrix-matrix multiplication operation in the BLAS library. The usual representation of this operation is: C = alpha*op( S )*op( P ) + beta*C, where S and P are the two input matrices, C is the output matrix, alpha and beta are scalars, op Represents some kind of operation on the matrix S or P. In addition, there will be some auxiliary integers as parameters to illustrate the width and height of the matrix S and P;

The steps to realize GEMM calculation using the device as shown in Fig. 1a include:

The data type conversion operation circuit of the main processing circuit can perform data type conversion on the matrix S and the matrix W;

The conversion circuit of the main processing circuit performs respective op operations on the input matrix S and the matrix W;

In an optional solution, op can be a matrix transposition operation; the matrix transposition operation can be realized by using the matrix transposition circuit of the main processing circuit;

In an optional solution, after performing the OP operation of matrix S and matrix W, the data type conversion operation can also be performed by the data conversion operation circuit of the main processing circuit, that is, the data conversion operation circuit converts op( S ) and op( The data type of P) is converted from floating-point type data to fixed-point type data, and then the matrix multiplication operation as shown in Figure 2b is performed.

In an optional solution, the op of a matrix can be empty, and the op operation is not performed;

Completing the matrix multiplication calculation between op(S) and op(P) with the use of the device shown in Figure 1a using the calculation method of matrix multiplication matrix as described in Figure 2b;

Use the arithmetic logic unit of the main processing circuit to multiply each value in the result of op(S)*op(P) by alpha;

In an optional solution, the operation of multiplying alpha is not performed when alpha is 1;

Use the arithmetic logic unit of the main processing circuit to realize the operation of beta*C;

In an optional solution, when beta is 1, the operation of multiplying by beta is not performed;

Use the vector operator circuit of the main processing circuit to realize the step of adding corresponding positions between the matrix alpha*op( s)*op( P ) and beta*C to obtain the result of GEMM calculation.

In an optional solution, when beta is 0, this step is not performed;

GEMV calculation refers to the operation of matrix-vector multiplication in the BLAS library. The usual representation of this operation is: C = alpha*op( S )*P+ beta*C, where S is the input matrix, P is the input vector, C is the output vector, alpha and beta are scalars, and op represents the pair matrix some operation of S;

The steps to realize GEMV calculation using the device shown in Figure 1a are:

The data type conversion operation circuit of the main processing circuit can perform data type conversion on the input matrix S and the matrix W;

The conversion circuit of the main processing circuit performs a corresponding op operation on the input matrix S;

In an optional solution, op can be a matrix transposition operation; the matrix transposition operation is realized by using the conversion circuit of the main processing circuit;

In an optional solution, the op of a certain matrix can be empty, and the transpose operation is not performed;

Complete the matrix-vector multiplication calculation between the matrix op(S) and the vector P by using the calculation method of matrix multiplication vector as described in Figure 2a with the device shown in Figure 1a;

Use the arithmetic logic unit of the main processing circuit to multiply each value in the result of op(S)*P by alpha;

In an optional solution, the operation of multiplying alpha is not performed when alpha is 1;

Use the arithmetic logic unit of the main processing circuit to realize the operation of beta*C;

In an optional solution, when beta is 1, the operation of multiplying by beta is not performed;

Use the vector operator circuit of the main processing circuit to realize the step of adding corresponding positions between the matrix alpha*op( S )*P and beta*C to obtain the result of GEMV.

In an optional solution, when beta is 0, the step of adding is not performed;

Method for implementing an activation function using a device as shown in Figure 1a

The activation circuit of the main processing circuit is used to input a vector, and the activation vector of the vector is calculated;

In an optional solution, the activation circuit of the main processing circuit passes each value in the input vector through an activation function (the input of the activation function is a value, and the output is also a value), and a value is calculated and output to the corresponding position of the output vector ;

In an alternative, the activation function can be: y=max(m, x), where x is the input value, y is the output value, and m is a constant;

In an alternative, the activation function can be: y=tanh(x), where x is the input value and y is the output value;

In an alternative, the activation function can be: y=sigmoid(x), where x is the input value and y is the output value;

In an alternative, the activation function can be a piecewise linear function;

In an optional solution, the activation function can be any function that inputs a number and outputs a number.

In an alternative, sources of input vectors include (but are not limited to):

an external data source for the device;

In an optional solution, the input data comes from the operation result of matrix multiplication by the device;

In an optional solution, the input data comes from the operation result of matrix multiplication performed by the device;

The calculation result of the main processing circuit of the device;

In an optional solution, the input data comes from a calculation result after biasing is implemented by the main processing circuit of the device.

It should be noted that the above activation operation can be realized by the arithmetic logic circuit and the accumulator circuit in the main processing circuit, or an activation circuit can be added separately in the main processing circuit to realize the activation operation.

Use the device shown in Figure 1a to realize the biasing operation:

The function of adding two vectors or two matrices can be realized by using the vector operator circuit of the main processing circuit;

The function of adding a vector to each row or column of a matrix can be realized by using the vector operator circuit of the main processing circuit.

In an optional solution, the matrix may be obtained from the result of a matrix multiplication matrix operation performed by the device;

In an optional solution, the matrix may be obtained from the result of matrix multiplication and vector operation performed by the device;

In an optional solution, the matrix may come from external data received by the main processing circuit of the device.

In an optional solution, the vector may come from data received externally by the main processing circuit of the device.

Including but not limited to the above data sources.

Use the device shown in Figure 1a to realize data type conversion:

Using the data type conversion operation circuit of the main processing circuit to realize the conversion of the data type;

In an optional solution, the data type conversion of a set of data is realized by using the data type conversion operation circuit of the main processing circuit;

In an optional solution, the form of data type conversion includes but is not limited to: converting floating-point numbers to fixed-point numbers and fixed-point numbers to floating-point numbers;

The present invention also provides a chip, the chip includes a computing device, and the computing device includes:

It includes a main processing circuit. The data involved in the main processing circuit can be data of any data type. In an optional solution, the data represented by floating-point numbers of any bit width can also be fixed-point numbers of any bit width. The data represented by the number; all the operation circuits and storage circuits involved can be operation circuits and storage circuits of any data type, and in an optional solution, the operation circuits and storage circuits that can be floating-point numbers of any bit width can also be It can be an arithmetic circuit and a storage circuit of a fixed-point number of any bit width.

In an optional solution, the main processing circuit includes a data type conversion operation circuit;

In an optional solution, the main processing circuit includes a vector operation unit for performing data type conversion;

Specifically, it includes a data input interface for receiving input data;

In an optional solution, the received data source may be: the exterior of the neural network computing circuit device or part or all of the basic processing circuits of the neural network computing circuit device;

In an optional solution, there may be multiple data input interfaces; specifically, it may include a data output interface for outputting data;

In an optional solution, the destination of the output data may be: the exterior of the neural network computing device or part or all of the basic processing circuits of the neural network computing circuit device;

In an optional solution, there may be multiple data output interfaces;

In an optional solution, the main processing circuit includes on-chip caches and/or registers;

In an optional solution, the main processing circuit includes an operation unit, which can perform data operations;

In an optional solution, the main processing circuit includes an arithmetic operation unit;

In an optional solution, the main processing circuit includes a vector operation unit, which can perform operations on a set of data at the same time; specifically, the arithmetic operation and/or vector operation can be any type of operation, including but not limited to : Addition, subtraction, multiplication, and division of two numbers, addition, subtraction, multiplication, and division of a number and a constant, exponential operations, power operations, logarithmic operations, and various non-linear operations on a number, comparison operations on two numbers, logical operations, etc. Addition, subtraction, multiplication, and division of two vectors, addition, subtraction, multiplication, and division of each element in a vector with a constant, perform exponential operations, power operations, logarithmic operations, and various nonlinear operations on each element in a vector, etc., for each element in a vector Two corresponding elements perform comparison operations, logical operations, etc.

In an optional solution, the main processing circuit includes a data rearranging unit, configured to transmit data to the basic processing circuit in a certain order, or rearrange the data in situ in a certain order;

In an optional solution, the order of data arrangement includes: transforming the dimension order of a multi-dimensional data block; the order of data arrangement may also include: dividing a data block into blocks to send to different basic processing circuit.

The calculation device also includes a plurality of basic processing circuits: each basic processing circuit is used to calculate the inner product of two vectors, and the calculation method is that the two sets of numbers received by the basic processing circuit correspond to the elements in the two sets of numbers Multiply, and accumulate the results of the multiplication; the result of the inner product is transmitted, and the transmission here may be transmitted to other basic processing circuits according to the position of the basic processing circuit, or directly to the main processing circuit.

The data involved in the basic processing circuit can be data of any data type, and in an optional solution, it can be data represented by floating-point numbers of any bit width or data represented by fixed-point numbers of any bit width; All the operation circuits and storage circuits mentioned above can be operation circuits and storage circuits of any data type. In an optional solution, the operation circuits and storage circuits that can be floating-point numbers of any bit width can also be of any bit width. Fixed-point arithmetic circuit and storage circuit.

In an optional solution, the basic processing circuit includes a data type conversion operation circuit;

In an optional solution, the basic processing circuit includes a vector operation unit for performing data type conversion;

Specifically, including a storage unit composed of an on-chip cache and/or register;

Specifically, including one or more data input interfaces for receiving data;

In an optional solution, two data input interfaces are included, and one or more data can be respectively obtained from the two data input interfaces each time;

In an optional solution, the basic processing circuit may store the input data received from the data input interface in registers and/or on-chip caches;

The source of data received by the above data input interface may be: other basic processing circuits and/or main processing circuits.

The main processing circuit of the neural network computing circuit device;

Other basic processing circuits of the neural network computing circuit device (the neural network computing circuit device has multiple basic processing circuits);

Specifically, including one or more data output interfaces for transmitting output data;

In an alternative, one or more data can be transmitted from the data output interface;

Specifically, the data transmitted through the data output interface may be: the data received from the data input interface, the data stored in the on-chip cache and/or register, the result of the multiplier operation, the result of the accumulator operation or the operation of the inner product operator One or any combination of results.

In an optional solution, it includes three data output interfaces, two of which correspond to two data input interfaces, each layer outputs the data received from the data input interface of the previous layer, and the third data output interface is responsible for output calculations result;

Specifically, the destination of the data transmitted by the data output interface may be: the source of the above data and the destination of the data here determine the connection relationship of the basic processing circuit in the device.

The main processing circuit of the neural network computing circuit device;

Other basic processing circuits of the neural network computing circuit device, the neural network computing circuit device has a plurality of basic processing circuits;

Specifically, it includes an arithmetic operation circuit: the arithmetic operation circuit may specifically be: one or more circuits of one or more multipliers, one or more circuits of accumulators, one or more circuits performing inner product operations of two sets of numbers, or any combination.

In an optional solution, the multiplication operation of two numbers can be performed, and the result can be stored in the on-chip cache and/or register, or directly accumulated in the register and/or on-chip cache;

In an optional solution, the inner product operation of two sets of data can be performed, and the result can be stored in the on-chip cache and/or register, or directly accumulated in the register and/or on-chip cache;

In an optional solution, data accumulation operations may be performed to accumulate data into on-chip caches and or registers;

Specifically, the accumulated data of the accumulator circuit may be: data received from the data input interface, data stored in the on-chip cache and/or register, multiplier operation result, accumulator operation result, inner product operator operation One or any combination of results.

It should be noted that the "data input interface" and "data output interface" used in the above description of the basic processing circuit refer to the data input and output interface of each basic processing circuit, not the data input and output of the entire device. interface.

This disclosure also discloses a neural network computing device, which includes one or more chips as shown in Figure 1a or Figure 1b, used to obtain data to be calculated and control information from other processing devices, and execute a specified neural network Operation, the execution result is transmitted to the peripheral device through the I/O interface. Peripherals such as cameras, monitors, mice, keyboards, network cards, wifi interfaces, servers. When more than one chip as shown in Figure 1a or Figure 1b is included, the chips shown in Figure 1a or Figure 1b can be linked and transmit data through a specific structure, for example, interconnected and transmitted through a PCIE bus data to support the operation of larger neural networks. At this time, the same control system can be shared, or there can be independent control systems; the memory can be shared, or each accelerator can have its own memory. In addition, its interconnection method can be any interconnection topology.

The neural network computing device has high compatibility and can be connected with various types of servers through the PCIE interface.

This disclosure also discloses a combined processing device, which includes the above-mentioned neural network computing device, a general interconnection interface, and other processing devices (ie, a general processing device). The neural network computing device interacts with other processing devices to jointly complete operations specified by the user. For example, the figure below in 4c is a schematic diagram of the combined processing device.

Other processing devices include one or more types of general-purpose/special-purpose processors such as central processing unit CPU, graphics processing unit GPU, and neural network processor. The number of processors included in other processing devices is not limited. Other processing devices serve as the interface between the neural network computing device and external data and control, including data transfer, and complete the basic control of starting and stopping the neural network computing device; other processing devices can also cooperate with the neural network computing device to complete computing tasks.

The universal interconnection interface is used for transmitting data and control instructions between the neural network computing device and other processing devices. The neural network computing device obtains the required input data from other processing devices and writes it into the storage device on the neural network computing device; it can obtain control instructions from other processing devices and writes it into the control cache on the neural network computing device chip; The data in the storage module of the neural network computing device can be read and transmitted to other processing devices.

As shown in Figure 4d, optionally, the structure also includes a storage device, which is used to save the data required by the computing unit/computing device or other computing units, especially for the data required to be calculated in the neural network computing device or data that cannot be fully stored in the internal storage of other processing devices.

The combined processing device can be used as a SOC system on a mobile phone, robot, drone, video surveillance equipment and other equipment, effectively reducing the core area of the control part, increasing the processing speed, and reducing the overall power consumption. In this case, the common interconnection interface of the combined processing device is connected with certain components of the equipment. Some components such as camera, monitor, mouse, keyboard, network card, wifi interface.

The embodiment of the present disclosure provides a neural network processor board, which can be used in many general-purpose or special-purpose computing system environments or configurations. Examples: personal computers, server computers, handheld or portable devices, tablet-type devices, smart homes, home appliances, multiprocessor systems, microprocessor-based systems, robots, programmable consumer electronics, networked personal computers , PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

Please refer to FIG. 5c, which is a schematic structural diagram of a neural network processor board provided by an embodiment of the present disclosure. As shown in FIG. 5 c , the neural network processor board 10 includes a neural network chip packaging structure 11 , a first electrical and non-electrical connection device 12 and a first substrate 13 .

The present disclosure does not limit the specific structure of the neural network chip packaging structure 11. Optionally, as shown in FIG. Two substrates 113 .

The specific form of the neural network chip 111 involved in this disclosure is not limited. The above-mentioned neural network chip 111 includes but is not limited to a neural network chip that integrates a neural network processor. The above-mentioned chip can be made of silicon material, germanium material, quantum material or molecular materials etc. According to the actual situation (for example: harsher environment) and different application requirements, the above neural network chip can be packaged so that most of the neural network chip is wrapped, and the pins on the neural network chip are passed through the gold wire The other conductors are connected to the outside of the package structure for circuit connection with the outer layer.

The present disclosure does not limit the specific structure of the neural network chip 111 , as an option, please refer to the device shown in FIG. 1a or FIG. 1b .

The present disclosure does not limit the types of the first substrate 13 and the second substrate 113 , they may be printed circuit boards (printed circuit board, PCB) or (printed wiring board, PWB), and may also be other circuit boards. The manufacturing material of the PCB is not limited either.

The second substrate 113 involved in this disclosure is used to carry the aforementioned neural network chip 111, and the neural network chip packaging structure 11 obtained by connecting the aforementioned neural network chip 111 and the second substrate 113 through the second electrical and non-electrical connection device 112 , used to protect the neural network chip 111 and facilitate further packaging of the neural network chip packaging structure 11 and the first substrate 13 .

There is no limitation on the packaging method and the structure corresponding to the packaging method of the above-mentioned second electrical and non-electrical connecting device 112, and a suitable packaging method can be selected and simply improved according to actual conditions and different application requirements, for example: flip chip Flip Chip Ball Grid Array Package (FCBGAP), Low-profile Quad Flat Package (LQFP), Quad Flat Package with Heat Sink (HQFP), None Package methods such as Quad Flat Non-lead Package (QFN) or Fine-pitch Ball Grid Package (FBGA).

Flip Chip (Flip Chip) is suitable for situations where the area after packaging is high or the inductance of the wire and the transmission time of the signal are sensitive. In addition, wire bonding (Wire Bonding) packaging can be used to reduce costs and improve the flexibility of the packaging structure.

Ball Grid Array can provide more pins, and the average wire length of the pins is short, which can transmit signals at high speed. Among them, the package can be packaged with Pin Grid Array (PGA) , Zero Insertion Force (ZIF), Single Edge Contact Connection (SECC), Land Grid Array (LGA), etc. instead.

Optionally, the neural network chip 111 and the second substrate 113 are packaged in a Flip Chip Ball Grid Array (Flip Chip Ball Grid Array) packaging manner. For a schematic diagram of the packaging structure of the neural network chip, refer to FIG. 6 . As shown in FIG. 6 , the neural network chip packaging structure includes: a neural network chip 21 , pads 22 , solder balls 23 , a second substrate 24 , connection points 25 on the second substrate 24 , and pins 26 .

Wherein, the pad 22 is connected to the neural network chip 21, and the solder ball 23 is formed by soldering between the pad 22 and the connection point 25 on the second substrate 24, and the neural network chip 21 and the second substrate 24 are connected, which realizes The packaging of the neural network chip 21.

The pin 26 is used to connect with the external circuit of the package structure (for example, the first substrate 13 on the neural network processor board 10), which can realize the transmission of external data and internal data, and facilitate the neural network chip 21 or the neural network chip 21 The corresponding neural network processor processes the data. The present disclosure does not limit the type and quantity of the pins, and different pin forms can be selected according to different packaging technologies, and the arrangement must follow certain rules.

Optionally, the above neural network chip packaging structure further includes insulating fillers placed in the gaps between the pads 22 , solder balls 23 and connection points 25 to prevent interference between solder balls.

Wherein, the material of the insulating filler may be silicon nitride, silicon oxide or silicon oxynitride; interference includes electromagnetic interference, inductive interference, and the like.

Optionally, the above neural network chip packaging structure further includes a heat dissipation device for dissipating the heat generated by the neural network chip 21 during operation. Wherein, the heat dissipation device may be a metal sheet with good thermal conductivity, a heat sink or a heat sink, for example, a fan.

For example, as shown in FIG. 6a, the neural network chip packaging structure 11 includes: a neural network chip 21, pads 22, solder balls 23, a second substrate 24, connection points 25 on the second substrate 24, pins 26, Insulating filler 27, heat dissipation paste 28 and heat dissipation fins 29 of the metal casing. Wherein, the thermal paste 28 and the metal shell heat sink 29 are used to dissipate the heat of the neural network chip 21 during operation.

Optionally, the neural network chip packaging structure 11 further includes a reinforcement structure connected to the pad 22 and embedded in the solder ball 23 to enhance the connection strength between the solder ball 23 and the pad 22 .

Wherein, the reinforcement structure may be a metal wire structure or a column structure, which is not limited here.

This disclosure does not limit the specific form of the first electrical and non-electrical device 12. You can refer to the description of the second electrical and non-electrical device 112, that is, the neural network chip packaging structure 11 is packaged by welding, or it can be connected The way of connecting the second substrate 113 and the first substrate 13 by wire connection or plugging is convenient for subsequent replacement of the first substrate 13 or the packaging structure 11 of the neural network chip.

Optionally, the first substrate 13 includes an interface of a memory unit for expanding the storage capacity, such as: synchronous dynamic random access memory (Synchronous Dynamic Random Access Memory, SDRAM), double rate synchronous dynamic random access memory (Double Date Rate SDRAM, DDR), etc., improve the processing capacity of the neural network processor by expanding the memory.

The first substrate 13 may also include a Peripheral Component Interconnect-Express (PCI-E or PCIe) interface, a Small Form-factor Pluggable (SFP) interface, an Ethernet interface, Controller Area Network bus (Controller Area Network, CAN) interface, etc., are used for data transmission between the package structure and external circuits, which can improve the computing speed and the convenience of operation.

The neural network processor is packaged as a neural network chip 111, the neural network chip 111 is packaged as a neural network chip packaging structure 11, and the neural network chip packaging structure 11 is packaged as a neural network processor board 10, through the interface on the board ( socket or ferrule) to perform data interaction with an external circuit (for example: computer motherboard), that is, directly use the neural network processor board 10 to realize the function of the neural network processor and protect the neural network chip 111 . Moreover, other modules can also be added to the neural network processor board 10, which improves the application range and computing efficiency of the neural network processor.

In one embodiment, the present disclosure discloses an electronic device, which includes the neural network processor board 10 or the neural network chip packaging structure 11 described above.

Electronic devices include data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, mobile phones, driving recorders, navigators, sensors, cameras, servers, cameras, video cameras, projectors, watches, earphones, mobile storage , wearable devices, vehicles, home appliances, and/or medical devices.

Said vehicles include airplanes, ships and/or vehicles; said household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, range hoods; said medical equipment includes nuclear magnetic resonance machine, B-ultrasound machine and/or electrocardiograph.

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the disclosure in detail. It should be understood that the above descriptions are only specific embodiments of the disclosure and are not intended to limit the disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

S201, S202, S203, S204, S201b, S202b, S203b, S301, S302, S303, S304‧‧‧Step A, Ai, B, S‧‧‧Matrix P‧‧‧Vector 10‧‧‧Neural Network Processor Board Card 11‧‧‧neural network chip packaging structure 12‧‧‧first electrical and non-electrical connection device 13‧‧‧first substrate 111‧‧‧neural network chip 112‧‧second electrical and non-electrical connection device 113‧ ‧‧Second Substrate 1111‧‧‧Storage Unit 1112‧‧‧Direct Memory Access Unit 1113‧‧‧Instruction Cache Unit 1114‧‧‧Weight Value Cache Unit 1115‧‧‧Input Neuron Cache Unit 1116‧‧‧Output Neuron Meta cache unit 1117‧‧‧control unit 1118‧‧‧operation unit 21‧‧‧neural network chip 22‧‧‧pad 23‧‧‧solder ball 24‧‧‧second substrate 25‧‧‧on the second substrate 24 Connection point 26‧‧‧pin 27‧‧‧insulating filler 28‧‧‧thermal paste 29‧‧‧metal shell heat sink

Fig. 1a is a schematic structural diagram of an integrated circuit chip device.

Fig. 1b is a schematic structural diagram of another integrated circuit chip device.

Fig. 1c is a schematic structural diagram of a basic processing circuit.

Fig. 1d is a schematic structural diagram of a fixed-point data type.

Fig. 2 is a schematic flow chart of matrix multiplication by vector.

Figure 2a is a schematic diagram of matrix multiplication by vector.

Fig. 2b is a schematic flow chart of matrix-by-matrix multiplication.

Fig. 2c is a schematic diagram of matrix Ai multiplied by a vector.

Figure 2d is a schematic diagram of matrix A multiplied by matrix B.

Fig. 2e is a schematic diagram of matrix Ai multiplied by matrix B.

Figure 3a is a schematic diagram of neural network training.

Fig. 3b is a schematic diagram of convolution operation.

Fig. 4a is a schematic diagram of the forward operation of the neural network.

Fig. 4b is a schematic diagram of the reverse operation of the neural network.

FIG. 4c is a schematic structural diagram of a combination processing device disclosed in the present disclosure.

FIG. 4d is a schematic diagram of another structure of a combined processing device disclosed in this disclosure.

Fig. 5a is a schematic diagram of another forward operation of the neural network.

Fig. 5b is a schematic diagram of another reverse operation of the neural network.

Fig. 5c is a schematic structural diagram of a neural network processor board provided by an embodiment of the present disclosure;

Fig. 5d is a schematic structural diagram of a neural network chip packaging structure provided by an embodiment of the present disclosure;

Fig. 5e is a schematic structural diagram of a neural network chip provided by an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of a neural network chip packaging structure provided by an embodiment of the present disclosure;

Fig. 6a is a schematic diagram of another neural network chip packaging structure provided by the embodiment of the present disclosure.

Claims (21)

An integrated circuit chip device, wherein the integrated circuit chip device includes: a main processing circuit and a plurality of basic processing circuits, the main processing circuit or at least one basic processing circuit in the plurality of basic processing circuits includes: a data type operation circuit , the data type operation circuit is used to perform mutual conversion between floating-point type data and fixed-point type data according to the calculation amount and data storage amount; the main processing circuit is used to perform each continuous operation in the neural network operation and with The basic processing circuit transmits data; the plurality of basic processing circuits are used to execute calculations in the neural network in parallel according to the data transmitted by the main processing circuit, and transmit the calculation results to the main processing circuit. According to the integrated circuit chip device of claim 1, the integrated circuit chip device further includes: a branch processing circuit, the branch processing circuit is arranged between the main processing circuit and at least one basic processing circuit; the branch processing circuit Circuitry for forwarding data between the main processing circuit and at least one base processing circuit. According to the integrated circuit chip device of item 1 of the patent scope of the application, the main processing circuit is used to obtain a data block to be calculated and an operation instruction, and the data type operation circuit converts the data block to be calculated into a certain Point-type data blocks, divide the fixed-point-type data blocks to be calculated into a distribution data block and a broadcast data block according to the operation instruction; split the distribution data block to obtain multiple basic data blocks, and divide the multiple distribute basic data blocks to the at least one basic processing circuit, and broadcast the broadcast data block to the at least one basic processing circuit; The basic processing circuit is used to perform an inner product operation on the basic data block and the broadcast data block in a fixed-point type to obtain an operation result, and send the operation result to the main processing circuit; the main processing circuit is used for the operation The result processing obtains the data block to be calculated and the instruction result of the operation instruction. According to the integrated circuit chip device according to item 2 or 3 of the scope of the patent application, the main processing circuit is specifically used to broadcast the broadcast data block to the plurality of basic processing circuits once. According to the integrated circuit chip device of item 4 of the scope of the patent application, the basic processing circuit is specifically used to perform inner product processing on the basic data block and the broadcast data block in a fixed-point data type to obtain an inner product processing result, and the The inner product processing results are accumulated to obtain an operation result, and the operation result is sent to the main processing circuit. According to the integrated circuit chip device of item 4 of the scope of patent application, wherein, the main processing circuit is used to accumulate the calculation results to obtain an accumulation result when the calculation result is the result of inner product processing, and the accumulation result Arranging to obtain the data block to be calculated and the instruction result of the operation instruction. According to the integrated circuit chip device according to item 2 or 3 of the scope of patent application, the main processing circuit is specifically used to divide the broadcast data block into multiple partial broadcast data blocks, and broadcast the multiple partial broadcast data blocks to the plurality of underlying processing circuits. The integrated circuit chip device according to item 7 of the scope of the patent application, wherein the basic processing circuit is specifically used to perform an inner product processing on the part of the broadcast data block and the basic data block in a fixed-point data type to obtain the inner product processing result , accumulating the inner product processing results to obtain a partial calculation result, and sending the partial calculation result to the main processing circuit. The integrated circuit chip device according to item 8 of the scope of the patent application, wherein the basic processing circuit is specifically used to multiplex the part of the broadcast data block n times and perform the inner product operation between the part of the broadcast data block and n basic data blocks to obtain n As for the partial processing results, n partial processing results are respectively accumulated to obtain n partial operation results, and the n partial operation results are sent to the main processing circuit, where n is an integer greater than or equal to 2. According to the integrated circuit chip device of claim 1, the main processing circuit includes: a main register or a main on-chip cache circuit; the basic processing circuit includes: a basic register or a basic on-chip cache circuit. According to the integrated circuit chip device of item 10 of the scope of the patent application, the main processing circuit includes: a vector arithmetic unit circuit, an arithmetic logic unit circuit, an accumulator circuit, a matrix transposition circuit, a direct memory access circuit, and a data type operation circuit Or one or any combination of data rearrangement circuits. According to the integrated circuit chip device according to item 1 of the scope of the patent application, the main processing circuit is used to obtain the data block to be calculated and the operation instruction, and divide the data block to be calculated into the distribution data according to the operation instruction block and the broadcast data block; the distributed data block is split and processed to obtain multiple basic data blocks, the multiple basic data blocks are distributed to the at least one basic processing circuit, and the broadcast data block is broadcast to the at least one basic Processing circuit; the basic processing circuit is used to convert the basic data block and the broadcast data block into the fixed-point type data block, perform an inner product operation according to the fixed-point type data block to obtain an operation result, and convert the operation result The floating-point data is sent to the main processing circuit; the main processing circuit is used to process the operation result to obtain the data block to be calculated and the instruction result of the operation instruction. According to the integrated circuit chip device of claim 2, the branch processing circuit includes a plurality of branch processing circuits, the main processing circuit is respectively connected to the plurality of branch processing circuits, and each branch processing circuit is connected to at least one basic processing circuit . According to the integrated circuit chip device of item 1 of the patent scope of the application, the data is one or any combination of vectors, matrices, three-dimensional data blocks, four-dimensional data blocks and n-dimensional data blocks. According to the integrated circuit chip device according to item 3 of the scope of the patent application, if the operation instruction is a multiplication instruction, the main processing circuit determines that the multiplier data block is the broadcast data block, and the multiplicand data block is the distribution data block; if The operation instruction is a convolution instruction, the main processing circuit determines that the input data block is the broadcast data block, and the convolution kernel is the distribution data block. A neural network computing device, wherein the neural network computing device includes one or more integrated circuit chip devices according to any one of items 1-15 of the scope of the patent application. A combined processing device, wherein the combined processing device includes: a neural network computing device, a general interconnection interface, and a general processing device according to the 16th patent scope of the application; the neural network computing device communicates with the general processing device through the general interconnection interface device connection. A chip, wherein the chip integrates an integrated circuit chip device as described in any one of the patent scopes 1-15, or a neural network computing device as described in the patent scope 16, or as described in the patent scope of the application The combined processing device of item 17. A smart device, wherein, the smart device includes the chip as claimed in item 18 of the scope of the patent application. An operation method of a neural network, wherein the method is applied in an integrated circuit chip device, and the integrated circuit chip device includes: an integrated circuit chip device according to any one of items 1-15 of the scope of patent application, and the integrated circuit chip device is used for for performing neural network operations. According to the method of item 18 of the scope of patent application, the operations of the neural network include: convolution operation, matrix multiplication matrix operation, matrix multiplication vector operation, paranoid operation, full connection operation, GEMM operation, GEMV operation, activation operation One or any combination.

Images