TW201928796A - Integrated circuit chip device and related product having the advantages of small calculation amount and low power consumption - Google Patents

Abstract

The present disclosure provides an integrated circuit chip device and a related product, the integrated circuit chip device comprising: a main processing circuit and a plurality of basic processing circuits. At least one of the main processing circuit or the plurality of basic processing circuits includes a data type operation circuit; the data type operation circuit is configured to perform conversion between floating point type data and fixed point type data; the plurality of basic processing circuits are arranged in an array; each of the basic processing circuits is connected to other adjacent basic processing circuits, and the main processing circuit connects n basic processing circuits in the first row, n basic processing circuits in the mth row, and m basic processing circuits in the first column. The technical solution provided by the present disclosure has the advantages of small calculation amount and low power consumption.

Description

Integrated circuit chip device and related products

The present disclosure relates to the field of neural networks, and in particular, to an integrated circuit chip device and related products.

Artificial neural network (Artificial Neural Network, ANN) is a research hotspot that has emerged in the field of artificial intelligence since the 1980s. It abstracts the human brain neuron network from the perspective of information processing, establishes some simple model, and forms different networks according to different connection methods. In engineering and academia, it is often referred to as neural network or neural network. A neural network is a computing model that consists of a large number of nodes (or neurons) connected to each other. The existing neural network operations are based on a CPU (Central Processing Unit, Central Processing Unit) or a GPU (Graphics Processing Unit, Graphics Processor) to implement the operations of the neural network. Such operations have a large amount of calculation and high power consumption.

The embodiments of the present disclosure provide an integrated circuit chip device and related products, which can increase the processing speed and efficiency of a computing device.

According to a first aspect, there is provided an integrated circuit chip device including: a main processing circuit and a plurality of basic processing circuits;

The multiple basic processing circuits are distributed in an array; each basic processing circuit is connected to an adjacent other adjacent basic processing circuit, and the main processing circuit is connected to k basic processing circuits in the multiple basic processing circuits. The k basic processing circuits are: n basic processing circuits in the first row, n basic processing circuits in the m row, and m basic processing circuits in the first column;

The plurality of basic processing circuits include: a data type conversion circuit for performing conversion between floating point type data and fixed point type data;

The main processing circuit is configured to perform each continuous operation in a neural network operation and transmit data to and from the k basic processing circuits;

The k basic processing circuits are used for data forwarding between the main processing circuit and a plurality of basic processing circuits;

The plurality of basic processing circuits are configured to determine whether to enable the data type conversion circuit to perform data type conversion of the transmission data according to the transmission data and the type of operation, perform the operations in the neural network in a parallel manner according to the transmission data, and The operation result is transmitted to the main processing circuit through the k basic processing circuits.

In a second aspect, a neural network computing device is provided. The neural network computing device includes one or more integrated circuit chip devices provided in the first aspect.

According to a third aspect, a combined processing device is provided. The combined processing device includes a neural network computing device, a universal interconnection interface, and a universal processing device provided in the second aspect;

The neural network computing device is connected to the universal processing device through the universal interconnection interface.

According to a fourth aspect, a chip is provided, which integrates the device of the first aspect, the device of the second aspect, or the device of the third aspect.

According to a fifth aspect, an electronic device is provided, and the electronic device includes a chip according to the fourth aspect.

According to a sixth aspect, a method for calculating a neural network is provided. The method is applied in an integrated circuit chip device. The integrated circuit chip device includes the integrated circuit chip device according to the first aspect, and the integrated circuit chip device is used for: Perform operations on neural networks.

It can be seen that, according to the embodiment of the present disclosure, a data conversion operation circuit is provided to perform a conversion operation on the type of the data block, which saves transmission resources and calculation resources, so it has the advantages of low power consumption and small calculation amount.

In order to enable those skilled in the art to better understand the disclosure scheme, the technical solutions in the embodiments of the present disclosure will be described clearly and completely in combination with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are merely These embodiments are part of, but not all of the embodiments of this disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by persons with ordinary knowledge in the technical field without making creative labor fall into the scope of protection of the present disclosure.

In the apparatus provided by the first aspect, the main processing circuit is configured to obtain a data block to be calculated and an operation instruction, and divide the data block to be calculated into a transmission data block and a broadcast data block according to the operation instruction; Performing split processing on the distribution data block to obtain multiple basic data blocks, distributing the multiple basic data blocks to the K basic processing circuits, and broadcasting the broadcast data block to the k basic processing Circuit

The multiple basic processing circuits are configured to convert the basic data block and the broadcast data block into a fixed-point data type basic data block and a broadcast data block according to the received basic data block, broadcast data block, and operation instructions. The data block and the broadcast data block perform an inner product operation with a fixed-point data type to obtain a fixed-point data type operation result, and convert the fixed-point data type operation result to a floating-point data type operation result to the k basic processing circuits and transmit the result to the Mentioned main processing circuit;

The main processing circuit is configured to process the operation result to obtain the data block to be calculated and an instruction result of an operation instruction.

Optionally, the main processing circuit is specifically configured to broadcast the broadcast data block to the k basic processing circuits through one broadcast.

In the device provided by the first aspect, the main processing circuit is configured to, when the operation result is a result of an inner product processing, accumulate the operation result to obtain an accumulation result, and arrange the accumulation result to obtain the result. The data block to be calculated and the instruction result of the operation instruction.

In the apparatus provided by the first aspect, the main processing circuit is specifically configured to divide the broadcast data block into multiple partial broadcast data blocks, and broadcast the multiple partial broadcast data blocks to the k multiple times. Basic processing circuit.

In the apparatus provided by the first aspect, the plurality of basic processing circuits are specifically configured to convert the partial broadcast data block and the basic data block into a fixed-point data type, and obtain the result after performing an inner product process once with the fixed-point data type. The inner product processing result of the fixed-point data type, accumulating the inner product processing result of the fixed-point data type to obtain a partial operation result of the fixed-point data type, and converting the partial operation result of the fixed-point type into a floating-point type operation result through the k The basic processing circuits are sent to the main processing circuit.

In the apparatus provided by the first aspect, the plurality of basic processing circuits are specifically configured to multiplex the partial broadcast data block n times to obtain a fixed-point data type by performing an inner product operation between the partial broadcast data block and the n basic data blocks. Fixed-point data type n partial processing results, the fixed-point data type n partial processing results are accumulated respectively to obtain fixed-point data type n partial operation results, and the data type operation circuit is started to convert the fixed-point data type n partial operation results to conversion Floating-point data type n partial operation results are sent to the main processing circuit through the k basic processing circuits, where n is an integer greater than or equal to 2.

In the apparatus provided by the first aspect, the main processing circuit includes: a main register or a main on-chip cache circuit;

The plurality of basic processing circuits include a basic register or a basic on-chip cache circuit.

In the device provided by the first aspect, the main processing circuit includes: a vector operator circuit, an arithmetic logic unit circuit, an accumulator circuit, a matrix transposition circuit, a direct memory access circuit, a data type operation circuit, or a data rearrangement circuit One or any combination.

In the device provided by the first aspect, the data is one or any combination of a vector, a matrix, a three-dimensional data block, a four-dimensional data block, and an n-dimensional data block.

In the apparatus provided by the first aspect, if the operation instruction is a multiplication instruction, the main processing circuit determines that the multiplier data block is a broadcast data block and the multiplicand data block is a distribution data block;

If the operation instruction is a convolution instruction, the main processing circuit determines that the input data block is a broadcast data block and the convolution kernel is a distribution data block.

In the method provided by the sixth aspect, the operation of the neural network includes one of convolution operation, matrix multiplication matrix operation, matrix multiplication vector operation, paranoid operation, fully connected operation, GEMM operation, GEMV operation, and activation operation or random combination.

Referring to FIG. 1a, FIG. 1a is an integrated circuit chip device provided by the present disclosure. The integrated circuit chip device includes a main processing circuit and a plurality of basic processing circuits. The plurality of basic processing circuits are arranged in an array (m * n array). ), Where the range of m and n is an integer greater than or equal to 1 and at least one of m and n is greater than or equal to 2. For a plurality of basic processing circuits distributed in an m * n array, each basic processing circuit is connected to an adjacent basic processing circuit, and the main processing circuit is connected to k basic processing circuits of the multiple basic processing circuits, and the k basic processing circuits are connected to each other. The processing circuits may be: n basic processing circuits in the first row, n basic processing circuits in the m row, and m basic processing circuits in the first column. As shown in the integrated circuit chip device shown in FIG. 1a, the main processing circuit and / or the plurality of basic processing circuits may include a data type conversion operation circuit, and some of the specific multiple basic processing circuits may include a data type conversion circuit. For example, in an optional technical solution, k basic processing circuits may be configured with data type conversion circuits, so that n basic processing circuits may be respectively responsible for performing data type conversion steps on the data of the m basic processing circuits in this column. This setting can improve the operation efficiency and reduce the power consumption, because for the n basic processing circuits in the first row, since it first receives the data sent by the main processing circuit, the received data is converted into fixed-point type. Data can reduce the amount of calculation of subsequent basic processing circuits and the amount of data transmission with subsequent basic processing circuits. Similarly, the configuration of data type conversion circuits for the m basic processing circuits in the first column also has a small amount of calculation and low power consumption. advantage. In addition, according to this structure, the main processing circuit can adopt a dynamic data transmission strategy. For example, the main processing circuit broadcasts data to the m basic processing circuits in the first column, and the main processing circuit sends and distributes the data to the n basic processing circuits in the first row. Data, this advantage is that different data is transmitted to the basic processing circuit through different data input ports, so that the basic processing circuit does not distinguish what kind of data the received data is, and only needs to determine from which receiving port the data is received. You can know what kind of data it belongs to.

The main processing circuit is configured to perform each continuous operation in a neural network operation and transmit data to the basic processing circuit connected thereto; the above-mentioned continuous operation is not limited to: an accumulation operation, an ALU operation, an activation operation, and the like .

The plurality of basic processing circuits are configured to perform operations in the neural network in a parallel manner according to the transmitted data, and transmit the operation results to the main processing circuit through the basic processing circuit connected to the main processing circuit. The above-mentioned parallel manner for performing operations in the neural network includes, but is not limited to, inner product operations, matrix or vector multiplication operations, and the like.

The main processing circuit may include: a data sending circuit, a data receiving circuit, or an interface. The data sending circuit may integrate a data distribution circuit and a data broadcasting circuit. Of course, in practical applications, the data distribution circuit and the data broadcasting circuit may be separately provided. For broadcast data, that is, data that needs to be sent to each basic processing circuit. For distribution data, that is, data that needs to be selectively sent to some basic processing circuits, specifically, such as convolution operation, convolution input data of convolution operation needs to be sent to all basic processing circuits, all of which are broadcast data, volume The convolution kernel needs to be selectively sent to some basic data blocks, so the convolution kernel is to distribute data. The specific selection of the distribution data to be sent to that basic processing circuit can be determined by the main processing circuit according to the load and other distribution methods. For the broadcast transmission method, broadcast data is transmitted to each basic processing circuit in a broadcast form. (In practical applications, broadcast data is sent to each basic processing circuit by one broadcast, and broadcast data can also be sent to each basic processing circuit by multiple broadcasts. The specific implementation of this disclosure does not limit the above. The number of broadcasts). For the distribution sending method, the distribution data is selectively sent to some basic processing circuits.

The main processing circuit (as shown in FIG. 1d) may include a register and / or an on-chip cache circuit. The main processing circuit may further include: a control circuit, a vector operator circuit, an arithmetic and logic unit (ALU) circuit, and an accumulation circuit. Circuit, DMA (Direct Memory Access) circuit, etc. Of course, in practical applications, the above main processing circuit can also be added, conversion circuit (such as matrix transposition circuit), data rearrangement circuit or activation circuit, etc. And other circuits.

Each basic processing circuit may include a basic register and / or a basic on-chip cache circuit; each basic processing circuit may further include one or any combination of an inner product operator circuit, a vector operator circuit, an accumulator circuit, and the like. The inner product operator circuit, the vector operator circuit, and the accumulator circuit may all be integrated circuits, and the inner product operator circuit, the vector operator circuit, and the accumulator circuit may also be separately provided circuits.

Optionally, the accumulator circuit for the nth basic processing circuit in the mth row can perform the accumulation operation of the inner product operation, because for the mth basic processing circuit, it can receive the product of all the basic processing circuits in this column. As a result, the accumulation operation of the inner product operation is performed by the n basic processing circuits in the m-th row, so that the calculation resources can be effectively allocated, which has the advantage of saving power consumption. This technical solution is especially applicable when the number of m is large.

For data type conversion, the main processing circuit can allocate and execute the circuit. Specifically, the display or implicit mode can be used to allocate and execute the circuit. For the display mode, the main processing circuit can be configured with a special instruction or instruction. When the circuit receives the special instruction or instruction, it determines to perform data type conversion. For example, when the basic processing circuit does not receive the special instruction or instruction, it determines to not perform data type conversion. As another example, it may be performed in an implied manner. For example, when the basic processing circuit receives data of a floating point type and determines that an inner product operation needs to be performed, the data type is converted into fixed-point data. For the display configuration mode, a special instruction or instruction can be configured with a decreasing sequence. The value of the decreasing sequence decreases by 1 each time the decreasing sequence passes through the basic processing circuit, and the basic processing circuit reads the value of the decreasing sequence. Data type conversion. If the value is equal to or less than zero, no data type conversion is performed. This setting is configured according to the basic processing circuit allocated by the array. For example, for the m basic processing circuits in the ith column, the main processing circuit needs the first five basic processing circuits to perform data type conversion, and the main processing circuit issues a A special instruction that contains a decreasing sequence. The initial value of the decreasing sequence can be 5, and the value of the decreasing sequence is reduced by 1 after each basic processing circuit. When the fifth basic processing circuit is reached, the The value is 1. When the sixth basic processing circuit is reached, the decrementing sequence is 0. At this time, the sixth basic processing circuit will not perform the data type conversion. In this way, the main processing circuit can dynamically configure the data type conversion. And the number of executions.

An embodiment of the present disclosure provides an integrated circuit chip device, including a main processing circuit (also referred to as a main unit) and a plurality of basic processing circuits (also referred to as a basic unit); the structure of the embodiment is shown in FIG. 1b; The dashed box is the internal structure of the neural network computing device; the gray-filled arrows indicate the data transmission path between the main processing circuit and the basic processing circuit array, and the hollow arrows indicate the basic processing circuits (phases) in the basic processing circuit array. (Neighboring basic processing circuits). The length, width, and length of the basic processing circuit array may be different, that is, the values of m and n may be different, and of course, they may be the same. The present disclosure does not limit the specific values of the foregoing values.

The circuit structure of the basic processing circuit is shown in Figure 1c; the dashed box in the figure indicates the boundary of the basic processing circuit, and the thick arrows crossing the dashed box indicate the data input and output channels (pointing to the dashed box is the input channel, and the dotted box is the output channel ); The rectangular box in the dashed box indicates the storage unit circuit (register and / or on-chip cache), including input data 1, input data 2, multiplication or inner product results, and accumulated data; diamond box indicates the arithmetic circuit, including multiplication or internal Product operator, adder.

In this embodiment, the neural network computing device includes a main processing circuit and 16 basic processing circuits (the 16 basic processing circuits are for illustration only, and other values may be used in practical applications);

In this embodiment, the basic processing circuit has two data input interfaces and two data output interfaces; in the subsequent description of this example, the horizontal input interface (the horizontal arrow pointing to this unit in Figure 1b) is called input 0, The vertical input interface (the vertical arrow pointing to this unit in Figure 1b) is called input 1; each horizontal data output interface (the horizontal arrow pointed out from this unit in Figure 1b) is called output 0, and the vertical The data output interface (the vertical arrow pointed out from this unit in Figure 1b) is called output 1.

The data input interface and data output interface of each basic processing circuit can be connected to different units, including the main processing circuit and other basic processing circuits;

In this example, the input 0 of the four basic processing circuits 0, 4, 8, 12 (numbered as shown in Figure 1b) is connected to the data output interface of the main processing circuit;

In this example, the input 1 of the four basic processing circuits 0,1,2,3 is connected to the data output interface of the main processing circuit;

In this example, the outputs 1 of the four basic processing circuits 12, 13, 14, 15 are connected to the data input interface of the main processing circuit;

In this example, the connection between the output interface of the basic processing circuit and the input interface of other basic processing circuits is shown in Figure 1b, which is no longer listed one by one;

Specifically, the output interface S1 of the S unit is connected to the input interface P1 of the P unit, which means that the P unit can receive data from its P1 interface and the data sent by the S unit to its S1 interface.

This embodiment includes a main processing circuit, which is connected to an external device (that is, an input interface also has an output interface), a part of the data output interface of the main processing circuit is connected to a part of the data input interface of the basic processing circuit; the main processing circuit A part of the data input interface is connected to a part of the data output interface of the basic processing circuit.

Method for using integrated circuit chip device

The data involved in the method of use provided by this disclosure may be data of any data type, for example, data represented by floating point numbers of arbitrary bit widths or data represented by fixed point numbers of arbitrary bit widths.

A structure diagram of the fixed-point type data is shown in FIG. 1e. As shown in FIG. 1e, it is a method of expressing fixed-point type data. For a computing system, the storage digits of a floating-point data are 32 bits, and for fixed-point data. In particular, the floating-point type data shown in Figure 1e is used to represent the data. The storage digits of one fixed-point data can be less than 16Bit, so for this conversion, the calculator can be greatly reduced. In addition, for the calculator, the storage space for data with fewer bits is also smaller, that is, the storage overhead will be smaller, and the amount of calculation will be reduced, that is, the calculation overhead will be reduced, so it can reduce the calculation overhead and Storage overhead, but also requires some overhead for data type conversion. The following is referred to as conversion overhead. For data with a large amount of calculation and large amount of data storage, the conversion overhead is relative to subsequent calculation overhead, storage overhead, and transmission overhead. It is almost negligible, so for the data with a large amount of calculation and a large amount of data storage, this disclosure uses data type conversion The technical solution of fixed-point data. Conversely, for data with a small amount of calculation and small data storage, the calculation overhead, storage overhead, and transmission overhead are relatively small at this time. If fixed-point data is used at this time, due to the accuracy of fixed-point data It will be slightly lower than floating-point data. Under the premise of a small amount of calculation, the accuracy of the calculation needs to be guaranteed, so the fixed-point data is converted to floating-point data here, that is, the accuracy of the calculation is increased by adding a small overhead. the goal of.

The operations that need to be performed in the basic processing circuit can be performed using the following methods:

The main processing circuit first converts the type of data and then transmits it to the basic processing circuit for operation (for example, the main processing circuit can convert floating-point numbers to fixed-point numbers with lower bit widths and then transmit them to the basic processing circuit. The advantage is that it can reduce The bit width of the transmitted data reduces the total number of bits transmitted, and the basic processing circuit is more efficient in performing wide-status fixed-point operations and consumes less power.

The basic processing circuit can perform data type conversion and then calculate after receiving the data (for example, the basic processing circuit receives the floating-point number transmitted by the main processing circuit, and then converts it to a fixed-point number for operation, improving the operation efficiency and reducing power consumption. ).

After the basic processing circuit calculates the result, it can be converted to a data type and then transmitted to the main processing circuit. , Which has the advantage of reducing the data bit width of the transmission process, higher efficiency, and saving power consumption).

How to use the basic processing circuit (see Figure 2a);

The main processing circuit receives input data to be calculated from outside the device;

Optionally, the main processing circuit uses various arithmetic circuits, vector arithmetic circuits, inner product arithmetic circuits, and accumulator circuits of the unit to perform arithmetic processing on the data;

The main processing circuit sends data to the basic processing circuit array (referring to the collection of all basic processing circuits as the basic processing circuit array) through the data output interface (as shown in Figure 2b);

The method for sending data here may be sending data directly to a part of the basic processing circuit, that is, multiple broadcast mode;

The way of sending data here can send different data to different basic processing circuits, that is, the distribution mode;

The basic processing circuit array calculates the data;

The basic processing circuit performs operations after receiving the input data;

Optionally, after receiving the data, the basic processing circuit transmits the data from the data output interface of the unit; (transmit to other basic processing circuits that do not receive data directly from the main processing circuit.)

Optionally, the basic processing circuit transmits the operation result from the data output interface; (intermediate calculation result or final calculation result)

The main processing circuit receives the output data returned from the basic processing circuit array;

Optionally, the main processing circuit continues to process the data received from the basic processing circuit array (such as an accumulation or activation operation);

The main processing circuit finishes processing and transmits the processing result from the data output interface to the outside of the device.

Using the circuit device to complete a matrix multiplication vector operation;

(The matrix multiplication vector can be the inner product of each row in the matrix and the vector, and the results are placed into a vector in the order of the corresponding rows.)

The following describes the calculation of a multiplication of a matrix S of size M rows and L columns and a vector P of length L, as shown in FIG. 2c.

This method uses all or part of the basic processing circuits of the neural network computing device, assuming that K basic processing circuits are used;

The main processing circuit sends data in part or all of the rows of the matrix S to each of the k basic processing circuits;

In an optional solution, the control circuit of the main processing circuit sends data of a row in the matrix S one number or a part of the number to a basic processing circuit at a time; (for example, for sending a number each time, it can be A certain basic processing circuit sends the first number of the third row of data for the first time, the second number of the third row of data for the second time, the third number of the third row for the third time ..., or Each time a part of the number is sent, the first two numbers of the third line (ie, the first and second numbers) are sent for the first time, the third and third numbers of the third line are sent for the second time, and the third and third lines are sent. 5 and 6th numbers ...;)

In an optional solution, the control circuit of the main processing circuit sends data of some rows in the matrix S to each basic processing circuit at a time; (for example, for a basic processing circuit, the first Send the 3rd, 4th, and 5th lines of the first number each time, send the 2nd number of the 3rd, 4th, and 5th lines each time, and send the 3rd, 4th, and 5th lines each time The third number of ........., or the first time to send the first two numbers of each line 3,4,5, the second time to send the third and fourth numbers of each line 3,4,5, the third Send the 3rd, 4th, and 5th lines 5th and 6th numbers of each line ...)

The control circuit of the main processing circuit sequentially sends the data in the vector P to the 0th basic processing circuit;

After the 0th basic processing circuit receives the data of the vector P, it sends the data to the next basic processing circuit connected to it, namely the basic processing circuit 1.

Specifically, some basic processing circuits cannot directly obtain all the data required for calculation from the main processing circuit. For example, the basic processing circuit 1 in FIG. 2d has only one data input interface connected to the main processing circuit, so it can only be obtained directly from the main processing circuit. The main processing circuit obtains the data of the matrix S, and the data of the vector P needs to be output to the basic processing circuit 1 by the basic processing circuit 0. Similarly, the basic processing circuit 1 also continues to output the data of the vector P after receiving the data. To the base processing circuit 2.

Each basic processing circuit performs operations on the received data, including, but not limited to, inner product operations, multiplication operations, addition operations, and the like;

In an optional solution, the basic processing circuit calculates a multiplication of one or more sets of two data at a time, and then accumulates the results in a register and / or an on-chip buffer;

In an optional solution, the basic processing circuit calculates an inner product of one or more groups of two vectors at a time, and then accumulates the results in a register and / or an on-chip buffer;

After the basic processing circuit calculates the result, the result is transmitted from the data output interface (that is, to other basic processing circuits connected to it);

In an optional solution, the calculation result may be a final result or an intermediate result of the inner product operation;

After receiving the calculation results from other basic processing circuits, the basic processing circuit transmits the data to other basic processing circuits or main processing circuits connected to the data;

The main processing circuit receives the result of the inner product operation of each basic processing circuit, and processes the result to obtain the final result (the processing may be an accumulation operation or an activation operation, etc.).

An embodiment of a method for implementing a matrix multiplication vector by using the above calculation device:

In an optional solution, a plurality of basic processing circuits used in the method are arranged in a manner as shown in FIG. 2d or FIG. 2e;

As shown in Figure 2c, the data conversion operation circuit of the main processing circuit converts the matrix S and the matrix P into fixed-point type data; the control circuit of the main processing unit divides the M rows of data of the matrix S into K groups, which are respectively formed by the i-th base The processing circuit is responsible for the operation of the i-th group (the set of rows in the data is denoted as Ai);

The method for grouping M rows of data here is any grouping method that will not be repeatedly allocated;

In an optional solution, the following allocation method is adopted: the jth row is allocated to the j% K (% is the remainder operation) basic processing circuits;

In an alternative, for a case where the grouping cannot be evenly divided, a part of the rows may be evenly distributed first, and the remaining rows may be distributed in an arbitrary manner.

The control circuit of the main processing circuit sends the data in some or all of the rows of the matrix S to the corresponding basic processing circuit in turn each time;

In an optional solution, the control circuit of the main processing circuit sends one or more data in a row of data of the i-th group of data Mi to the i-th basic processing circuit each time;

In an optional solution, the control circuit of the main processing circuit sends one or more data of each of some or all of the i-th group of data Mi to the i-th basic processing circuit each time;

The control circuit of the main processing circuit sends the data in the vector P to the first basic processing circuit in sequence;

In an optional solution, the control circuit of the main processing circuit may send one or more data in the vector P at a time;

The i-th basic processing circuit receives the data of the vector P and sends it to the i + 1-th basic processing circuit connected to it;

After each basic processing circuit receives one or more data from a row or several rows in the matrix S and one or more data from the vector P, it performs operations (including but not limited to multiplication or addition);

In an optional solution, the basic processing circuit calculates a multiplication of one or more sets of two data at a time, and then accumulates the results in a register and / or an on-chip buffer;

In an optional solution, the basic processing circuit calculates an inner product of one or more groups of two vectors at a time, and then accumulates the results in a register and / or an on-chip buffer;

In an optional solution, the data received by the basic processing circuit may also be intermediate results, which are stored in registers and / or on-chip buffers;

The basic processing circuit transmits the local calculation result to the next basic processing circuit or main processing circuit connected to it;

In an optional solution, corresponding to the structure of FIG. 2d, only the output interface of the last basic processing circuit of each column is connected to the main processing circuit. In this case, only the last basic processing circuit can directly connect the local The calculation results are transmitted to the main processing circuit. The calculation results of other basic processing circuits must be passed to their next basic processing circuit, and the next basic processing circuit is passed to the next basic processing circuit until all of them are transmitted to the last basic processing circuit. The last basic processing circuit performs an accumulation calculation on the local calculation result and the results of other basic processing circuits received in this column to obtain an intermediate result, and sends the intermediate result to the main processing circuit. Of course, the last basic processing circuit can also send The results of the other basic processing circuits in this column and the local processing results are sent directly to the main processing circuit.

In an optional solution, corresponding to the structure of FIG. 2e, each basic processing circuit has an output interface connected to the main processing circuit. In this case, each basic processing circuit directly transmits the local calculation result to Main processing circuit

After the basic processing circuit receives the calculation results transmitted from other basic processing circuits, it transmits it to the next basic processing circuit or main processing circuit connected to it.

The main processing circuit receives the results of the M inner product operations as the operation results of the matrix multiplication vector.

Using the circuit device to complete a matrix multiplication matrix operation;

The following describes the calculation of a multiplication of a matrix S whose size is M rows and L columns and a matrix P whose size is L rows and N columns. (Each row in matrix S has the same length as each column of matrix P, as shown in Figure 2f.)

The method is described by using the embodiment of the device as shown in FIG. 1b;

The data conversion operation circuit of the main processing circuit converts the matrix S and the matrix P into fixed-point type data;

The control circuit of the main processing circuit sends the data in some or all of the rows of the matrix S to those basic processing circuits that are directly connected to the main processing circuit through the horizontal data input interface (for example, the gray-filled vertical data at the top in Figure 1b) path);

In an optional solution, the control circuit of the main processing circuit sends data of a row in the matrix S one number or a part of the number to a certain basic processing circuit at a time; (for example, for a certain basic processing circuit, the first transmission The first number in the third line, the second number in the third line of data is sent for the second time, the third number in the third line is sent in the third time ..., or the first two numbers in the third line are sent for the first time , Send the 3rd and 4th numbers of the 3rd line for the second time, send the 5th and 6th numbers of the 3rd line for the third time ...;)

In an optional solution, the control circuit of the main processing circuit sends data of some rows in the matrix S to each basic processing circuit at a time; (for example, for a basic processing circuit, the first Send the 3rd, 4th, and 5th lines the first number of each line, send the 2nd number of the 3rd, 4th, and 5th lines each time, and the 3rd time to send the 3rd, 4th, and 5th lines of each line The third number ..., or the first two numbers of each of the 3, 4, and 5 lines, the second number of 3, 4, and 5 numbers of the third and fourth numbers, the third time Send the 3rd, 4th, and 5th lines of each 5th and 6th numbers ...;)

The control circuit of the main processing circuit sends data in some or all of the columns in the matrix P to those basic processing circuits that are directly connected to the main processing circuit through the vertical data input interface (for example, the gray to the left of the basic processing circuit array in Figure 1b) Filled horizontal data path);

In an optional solution, the control circuit of the main processing circuit sends data of a column in the matrix P one number or a part of the number to a basic processing circuit at a time; (for example, for a basic processing circuit, the first sending The first number in the third column, the second number in the third column of data is sent for the second time, the third number in the third column is sent for the third time ..., or the first two numbers in the third column are sent for the first time. , Send the 3rd and 3rd numbers in the 3rd column for the second time, and send the 5th and 6th numbers in the 3rd column for the third time ...;)

In an optional solution, the control circuit of the main processing circuit sends data of some columns in the matrix P to the basic processing circuit one at a time; (for example, for a basic processing circuit, the first Send the 3rd, 4th, and 5th columns the first number of each column, send the 2nd number of the 3rd, 4th, and 5th columns each time, and the 3rd time to send the 3rd, 4th, and 5th columns of each column The third number ..., or the first two numbers in the 3rd, 4th, and 5th columns, the third number, the 3rd, 4th, and 5th columns in the third time Send the 3rd, 4th, and 5th columns of each 5th and 6th numbers ...;)

After the basic processing circuit receives the data of the matrix S, it transmits the data to its next basic processing circuit through its horizontal data output interface (for example, the white-filled horizontal data path in the middle of the basic processing circuit array in Figure 1b). ); After receiving the data of matrix P, the basic processing circuit transmits the data to the next basic processing circuit connected to it through its vertical data output interface (for example, the white filled pad in the middle of the basic processing circuit array in Figure 1b) Vertical data path);

Each basic processing circuit performs operations on the received data;

In an optional solution, the basic processing circuit calculates a multiplication of one or more sets of two data at a time, and then accumulates the results in a register and / or an on-chip buffer;

In an optional solution, the basic processing circuit calculates an inner product of one or more groups of two vectors at a time, and then accumulates the results in a register and / or an on-chip buffer;

After the basic processing circuit calculates the result, the result can be transmitted from the data output interface;

In an optional solution, the calculation result may be a final result or an intermediate result of the inner product operation;

Specifically, if the basic processing circuit has an output interface directly connected to the main processing circuit, the result is transmitted from the interface; if not, the result is output to the direction of the basic processing circuit that can directly output to the main processing circuit (for example, FIG. In 1b, the bottom line of the basic processing circuit directly outputs its output result to the main processing circuit, and the other basic processing circuits transmit the calculation results downward from the vertical output interface).

After receiving the calculation results from other basic processing circuits, the basic processing circuit transmits the data to other basic processing circuits or main processing circuits connected to the data;

Output the result in a direction that can be directly output to the main processing circuit (for example, in Figure 1b, the bottom line of the basic processing circuit directly outputs its output result to the main processing circuit, and the other basic processing circuits transfer the calculation from the vertical output interface downward. result);

The main processing circuit receives the result of the inner product operation of each basic processing circuit and can obtain the output result.

An embodiment of the "matrix by matrix" method:

The method uses a basic processing circuit array arranged as shown in Figure 1b, assuming h rows and w columns;

The data conversion operation circuit of the main processing circuit converts the matrix S and the matrix P into fixed-point type data;

The control circuit of the main processing circuit divides the h rows of data of the matrix S into h groups, and the i-th basic processing circuit is responsible for the operation of the i-th group (the set of rows in the data is denoted as Hi);

Here, the method for grouping the data of h rows is any grouping method that will not be repeatedly allocated;

In an optional solution, the following allocation method is adopted: the control circuit of the main processing circuit allocates the jth row to the j% h basic processing circuits;

In an alternative, for a case where the grouping cannot be evenly divided, a part of the rows may be evenly distributed first, and the remaining rows may be distributed in an arbitrary manner.

The control circuit of the main processing circuit divides the W column data of the matrix P into w groups, and the i-th basic processing circuit is responsible for the operation of the i-th group (the set of rows in the data is denoted as Wi);

The method for grouping W column data here is any grouping method that will not be repeatedly assigned;

In an optional solution, the following allocation method is adopted: the control circuit of the main processing circuit divides the jth row to the j% wth basic processing circuits;

In an optional solution, for a case where the grouping cannot be evenly divided, a part of the columns may be evenly distributed first, and the remaining columns may be allocated in an arbitrary manner.

The control circuit of the main processing circuit sends data in part or all of the rows of the matrix S to the first basic processing circuit of each row in the basic processing circuit array;

In an alternative, the control circuit of the main processing circuit sends one or more of a row of data in the i-th group of data Hi to the first basic processing circuit in the i-th row of the basic processing circuit array each time. data;

In an optional solution, the control circuit of the main processing circuit sends each of some or all of the i-th group of data Hi to the first basic processing circuit in the i-th row of the basic processing circuit array. One or more data;

The control circuit of the main processing circuit sends data in some or all of the columns of the matrix P to the first basic processing circuit of each column in the basic processing circuit array;

In an optional solution, the control circuit of the main processing circuit sends one or more of a column of data in the i-th group of data Wi that it is responsible to the first basic processing circuit in the i-th column of the basic processing circuit array. data;

In an optional solution, the control circuit of the main processing circuit sends each of some or all of the i-th group of data Ni to the first basic processing circuit in the i-th column of the basic processing circuit array. One or more data;

After the basic processing circuit receives the data of the matrix S, it transmits the data to its next basic processing circuit through its horizontal data output interface (for example, the white-filled horizontal data path in the middle of the basic processing circuit array in Figure 1b). ); After receiving the data of matrix P, the basic processing circuit transmits the data to the next basic processing circuit connected to it through its vertical data output interface (for example, the white filled pad in the middle of the basic processing circuit array in Figure 1b) Vertical data path);

Each basic processing circuit performs operations on the received data;

In an optional solution, the basic processing circuit calculates a multiplication of one or more sets of two data at a time, and then accumulates the results in a register and / or an on-chip buffer;

In an optional solution, the basic processing circuit calculates an inner product of one or more groups of two vectors at a time, and then accumulates the results in a register and / or an on-chip buffer;

After the basic processing circuit calculates the result, the result can be transmitted from the data output interface;

In an optional solution, the calculation result may be a final result or an intermediate result of the inner product operation;

Specifically, if the basic processing circuit has an output interface directly connected to the main processing circuit, the result is transmitted from the interface; if not, the result is output to the basic processing circuit that can directly output to the main processing circuit (for example, the most The bottom line of the basic processing circuit directly outputs its output result to the main processing circuit, and the other basic processing circuits transmit the calculation results downward from the vertical output interface).

After receiving the calculation results from other basic processing circuits, the basic processing circuit transmits the data to other basic processing circuits or main processing circuits connected to the data;

Output the result in a direction that can be directly output to the main processing circuit (for example, the bottom row of the basic processing circuit outputs its output result directly to the main processing circuit, and the other basic processing circuits transmit the calculation result downward from the vertical output interface);

The main processing circuit receives the result of the inner product operation of each basic processing circuit and can obtain the output result.

The words "horizontal" and "vertical" used in the above description are just to describe the example shown in Figure 1b. In actual use, only the "horizontal" and "vertical" interfaces of each unit need to be distinguished to represent two different interfaces. can.

Use the circuit device to complete a fully connected operation:

If the input data of the fully connected layer is a vector (ie, the input of the neural network is a single sample), then the weight matrix of the fully connected layer is used as the matrix S, and the input vector is used as the vector P. Perform operations in a vector method;

If the input data of the fully connected layer is a matrix (that is, when the input of the neural network is multiple samples), then the weight matrix of the fully connected layer is used as the matrix S, and the input vector is used as the matrix P, or the weight of the fully connected layer is used. The value matrix is used as the matrix P, the input vector is used as the matrix S, and the operation is performed according to the matrix of the device multiplied by the matrix;

Use the circuit device to complete a convolution operation:

The convolution operation is described below. A square in the figure below represents a piece of data, and the input data is represented in Figure 3a (N samples, each sample has C channels, and the feature map of each channel has a height of H and a width of W). The weights, ie, the convolution kernels, are shown in Figure 3b (there are M convolution kernels, each convolution kernel has C channels, and the height and width are KH and KW, respectively). For N samples of the input data, the rules of the convolution operation are the same. The process of performing the convolution operation on one sample is explained below. On one sample, each of the M convolution kernels must be the same. Operation, each convolution kernel operates to obtain a planar feature map, and M convolution kernels finally calculate to obtain M planar feature maps (for a sample, the output of the convolution is M feature maps), for a convolution kernel , To perform an inner product operation at each plane position of a sample, and then slide in the H and W directions. For example, Figure 3c shows a convolution kernel that performs an inner product operation at the lower right corner of a sample of input data. Correspondence map; Figure 3d shows the position of the convolution slide one grid to the left and Figure 3e shows the position of the convolution slide one grid up.

The method is described by using the embodiment of the device as shown in FIG. 1b;

The data conversion operation circuit of the main processing circuit can convert part or all of the data in the weighted convolution kernel into fixed-point type data, and the control circuit of the main processing circuit sends the data in part or all of the weighted convolution kernel to Those basic processing circuits that are directly connected to the main processing circuit through the horizontal data input interface (for example, the gray-filled vertical data path at the top in Figure 1b);

In an optional solution, the control circuit of the main processing circuit sends data of a certain convolution kernel in the weight value to a certain basic processing circuit at a time; (for example, for a certain basic processing circuit, the first Send the first number of the 3rd line once, send the 2nd number of the 3rd line data for the 2nd time, send the 3rd number of the 3rd line for the 3rd time ... or before the 3rd line of the 1st time Two numbers, the second time sends the 3rd and 4th numbers of the 3rd line, the third time sends the 5th and 6th numbers of the 3rd line ...;)

In another alternative, the control circuit of the main processing circuit sends the data of some convolution kernels in the weight value to the basic processing circuit one by one each time; (for example, for a certain A basic processing circuit that sends the first number of each line 3,4,5 for the first time, the second number of each line 3,4,5 for the second time, and the third number of 3, The 3rd number of each line 4,5 ... or the first two numbers of 3,4,5 lines are sent for the first time, the 3rd and 4th of 5th lines are sent for the second time 4 numbers, 3rd, 4th, 5th lines send 5th and 6th numbers per line ...;)

The control circuit of the main processing circuit divides the input data according to the position of the convolution. The control circuit of the main processing circuit sends the data in some or all of the convolution positions in the input data to the main processing circuit directly through the vertical data input interface. Connected basic processing circuits (for example, the gray-filled horizontal data path to the left of the basic processing circuit array in Figure 1b);

In an optional solution, the control circuit of the main processing circuit sends data of a convolution position in the input data to a basic processing circuit at a time or a portion of the data each time; (for example, for a basic processing circuit, the first Send the first number in the third column once, send the second number in the third column data for the second time, send the third number in the third column for the third time ..., or before the first column 3 Two numbers, the third and third numbers are sent in the third column, the third and third and fifth numbers are sent in the third column ...;)

In another optional solution, the control circuit of the main processing circuit sends data of some convolution positions in the input data to a basic processing circuit each time or a part of the data; (for example, for a certain A basic processing circuit that sends the first number of each of the 3, 4, and 5 columns for the first time, sends the second number of each of the 3, 4, 5 columns for the second time, and sends the third number of 3, The 3rd number of each column in 4,5 columns ..., or the first two numbers of 3,4,5 columns are sent for the first time, and the 3rd and 4th of 5th columns are sent for the second time. 4 numbers, send the 3rd, 4th, 5th columns for the third time, 5th and 6th numbers for each column ...;)

After the basic processing circuit receives the weighted data, it transmits the data to its next basic processing circuit through its horizontal data output interface (for example, the white-filled horizontal data path in the middle of the basic processing circuit array in Figure 1b). ); After receiving the data of the input data, the basic processing circuit transmits the data to the next basic processing circuit connected to it through its vertical data output interface (for example, the white filled pad in the middle of the basic processing circuit array in Figure 1b) Vertical data path);

Each basic processing circuit performs operations on the received data;

In an optional solution, the basic processing circuit calculates a multiplication of one or more sets of two data at a time, and then accumulates the results in a register and / or an on-chip buffer;

In an optional solution, the basic processing circuit calculates an inner product of one or more groups of two vectors at a time, and then accumulates the results in a register and / or an on-chip buffer;

After the basic processing circuit calculates the result, the result can be transmitted from the data output interface;

In an optional solution, the calculation result may be a final result or an intermediate result of the inner product operation;

Specifically, if the basic processing circuit has an output interface directly connected to the main processing circuit, the result is transmitted from the interface; if not, the result is output to the direction of the basic processing circuit that can directly output to the main processing circuit (for example, FIG. In 1b, the bottom line of the basic processing circuit directly outputs its output result to the main processing circuit, and the other basic processing circuits transmit the calculation results downward from the vertical output interface).

After receiving the calculation results from other basic processing circuits, the basic processing circuit transmits the data to other basic processing circuits or main processing circuits connected to the data;

Output the result in a direction that can be directly output to the main processing circuit (for example, the bottom row of the basic processing circuit outputs its output result directly to the main processing circuit, and the other basic processing circuits transmit the calculation result downward from the vertical output interface);

The main processing circuit receives the result of the inner product operation of each basic processing circuit and can obtain the output result.

Method for completing biasing operation using said circuit device;

The vector operator circuit of the main processing circuit can realize the function of adding two vectors or two matrices;

The vector operator circuit of the main processing circuit can realize the function of adding a vector to each row or each column of a matrix.

In an optional solution, the matrix may be obtained from a result of performing a matrix multiplication matrix operation by the device;

In an optional solution, the vector may be obtained from a result of performing a matrix multiplication vector operation by the device;

In an alternative, the matrix may come from data received externally by the main processing circuit of the device.

In an alternative, the vector may come from data received externally by the main processing circuit of the device.

Including but not limited to these data sources.

Method for completing activation function operation using said circuit device:

Use the activation circuit of the main processing circuit to input a vector and calculate the activation vector of this vector;

In an optional solution, the activation circuit of the main processing circuit passes each value in the input vector through an activation function (the input of the activation function is a numerical value and the output is also a numerical value), and calculates a corresponding value that is output to the output vector. position;

In an alternative, the activation function may be: y = max (m, x), where x is the input value, y is the output value, and m is a constant;

In an alternative, the activation function may be: y = tanh (x), where x is an input value and y is an output value;

In an alternative, the activation function may be: y = sigmoid (x), where x is an input value and y is an output value;

In an alternative, the activation function may be a piecewise linear function;

In an alternative solution, the activation function can be a function that arbitrarily inputs a number and outputs a number.

In an alternative, the source of the input vector is (including but not limited to):

An external data source of the device;

In an optional solution, the input data comes from a matrix multiplication vector operation result of the device;

In an optional solution, the input data comes from a matrix multiplication matrix operation result of the device;

Calculation result of the main processing circuit of the device;

In an optional solution, the input data comes from a calculation result after the main processing circuit of the device implements the offset.

Method for realizing BLAS (Basic Linear Algebra Subprograms) using the device;

GEMM calculation refers to the matrix-matrix multiplication operation in the BLAS library. The usual representation of this operation is: C = alpha * op (S) * op (P) + beta * C, where A and B are the two matrices of the input, C is the output matrix, alpha and beta are the scalars, and op Represents some kind of operation on the matrix S or P, in addition, there will be some auxiliary integers as parameters to explain the width and height of the matrix A and B;

The steps for implementing GEMM calculation using the device are:

The main processing circuit can perform data type conversion on the input matrix S and matrix P before performing the OP operation;

The conversion circuit of the main processing circuit performs respective op operations on the input matrix S and the matrix P;

In an alternative, op can be a matrix transposition operation; the vector operation function or data rearrangement function of the main processing circuit is used (the main processing circuit has a data rearrangement circuit mentioned earlier) to implement the matrix transposition operation, Of course, in practical applications, the above-mentioned OP can also be directly implemented by a conversion circuit. For example, in a matrix transposition operation, the OP operation is directly implemented by a matrix transposition circuit;

In an optional solution, the op of a matrix can be empty, and the OP operation is not performed.

Use the matrix multiplication matrix calculation method to complete the matrix multiplication calculation between op (S) and op (P);

Use the arithmetic logic circuit of the main processing circuit to multiply each value in the result of op (S) * op (P) by alpha;

In an optional solution, if the alpha is 1, the multiplication by the alpha operation is not performed.

Use the arithmetic logic circuit of the main processing circuit to implement the operation of beta * C;

In an optional solution, if beta is 1, multiplication by beta is not performed.

The arithmetic logic circuit of the main processing circuit is used to implement the step of adding corresponding positions between the matrix alpha * op (S) * op (P) and beta * C;

In an optional solution, if beta is 0, no addition operation is performed;

GEMV calculation refers to the matrix-vector multiplication operation in the BLAS library. The usual representation of this operation is: C = alpha * op (S) * P + beta * C, where S is the input matrix, P is the input vector, C is the output vector, alpha and beta are scalars, and op is the matrix Some operation of S;

The steps for implementing GEMV calculation using the device are:

The main processing circuit can perform data type conversion on the input matrix S and matrix P before performing the OP operation;

The conversion circuit of the main processing circuit performs corresponding op operations on the input matrix S;

In an optional solution, op may be a matrix transposition operation; a matrix transposition circuit of the main processing circuit is used to implement the matrix transposition operation;

In an optional solution, the op of a certain matrix may be empty, and the op operation is not performed.

The matrix-vector multiplication calculation between the matrix op (S) and the vector P is completed by a matrix multiplication vector calculation method;

Use the arithmetic logic circuit of the main processing circuit to multiply each value in the result of op (S) * P by alpha;

In an optional solution, if the alpha is 1, the multiplication by the alpha operation is not performed.

Use the arithmetic logic circuit of the main processing circuit to achieve the operation of beta * C;

In an optional solution, if beta is 1, multiplication by beta is not performed.

Use the arithmetic logic circuit of the main processing circuit to implement the step of adding the corresponding positions between the matrices alpha * op (S) * P and beta * C;

In an optional solution, if beta is 0, no addition operation is performed;

Implementing data type conversion

The data type conversion operation circuit of the main processing circuit is used to convert the data type;

In an optional solution, the form of data type conversion includes, but is not limited to, floating-point number to fixed-point number, fixed-point number to floating-point number, and the like;

Method for updating weights:

The vector operator circuit of the main processing circuit is used to implement the weight update function during the training of the neural network. Specifically, the weight update refers to a method of using the gradient of the weight to update the weight.

In an optional solution, a vector operator circuit of the main processing circuit is used to add and subtract two vectors of weight and weight gradient to obtain an operation result, and the operation result is to update the weight.

In an optional solution, the vector operator circuit of the main processing circuit is used to multiply or divide the weight and the weight gradient by a number to obtain the intermediate weight and the intermediate weight gradient value. The vector operator circuit compares the intermediate weight and The intermediate weight gradient value is added and subtracted to obtain an operation result, and the operation result is an updated weight value.

In an optional solution, a set of momentums may be calculated first using a gradient of weights, and then the momentums and weights may be added and subtracted to obtain updated weights.

Method for realizing reverse operation of fully connected layer

The reverse operation of the fully-connected layer can be divided into two parts. As shown in FIG. 4a, the solid arrow indicates the forward calculation process of the fully-connected layer, and the dotted line indicates the reverse calculation process of the fully-connected layer.

It can be seen from FIG. 4a that the reverse operation of the fully connected layer can be completed by using the device to complete the matrix multiplication operation using the device;

Realize the reverse operation of the convolution layer;

The reverse operation of the convolutional layer can be divided into two parts. As shown in FIG. 4a, the solid arrow indicates the forward calculation process of the convolution layer, and as shown in FIG. 4b, the reverse calculation process of the convolution layer.

The reverse operation of the convolution layer shown in Figs. 4a and 4b can be performed by using the device shown in Fig. 1a and the device shown in Fig. 1b. When performing a forward operation or a reverse operation is actually multiple operations of the neural network, the multiple operations include, but are not limited to, one of matrix multiplication by matrix, matrix multiplication by vector, convolution operation, activation operation, and so on. In any combination, the manner of the above operations can be described in this disclosure, and is not repeated here.

This disclosure also discloses a neural network computing device, which includes one or more chips as shown in FIG. 1a or FIG. 1b, for obtaining data to be calculated and control information from other processing devices, and executing a specified neural network. The calculation and execution results are passed to the peripheral device through the I / O interface. Peripheral equipment such as camera, monitor, mouse, keyboard, network card, wifi interface, server. When more than one chip shown in Figure 1a or Figure 1b is included, the chips shown in Figure 1a or Figure 1b can be linked and transmitted through a specific structure, such as interconnection and transmission through the PCIE bus. Data to support larger-scale neural network operations. At this time, you can share the same control system, or you can have separate control systems; you can share memory, or each accelerator can have its own memory. In addition, its interconnection method can be any interconnection topology.

The neural network computing device has high compatibility and can be connected to various types of servers through a PCIE interface.

The present disclosure also discloses a combined processing device, which includes the aforementioned neural network computing device, a universal interconnection interface, and other processing devices (ie, general processing devices). The neural network computing device interacts with other processing devices to complete a user-specified operation. Figure 4c is a schematic diagram of a combined processing device.

Other processing devices include one or more types of processors such as a central processing unit CPU, a graphics processor GPU, and a neural network processor. The number of processors included in other processing devices is not limited. Other processing devices serve as the interface between the neural network computing device and external data and control, including data transfer, to complete the basic control of the neural network computing device, such as start and stop; other processing devices can also cooperate with the neural network computing device to complete computing tasks.

A universal interconnection interface for transmitting data and control instructions between the neural network computing device and other processing devices. The neural network computing device obtains required input data from other processing devices and writes it to a storage device on the neural network computing device chip; it can obtain control instructions from other processing devices and write it to the control buffer on the neural network computing device chip; also The data in the storage module of the neural network computing device can be read and transmitted to other processing devices.

As shown in FIG. 4d, optionally, the structure further includes a storage device for storing data required by the operation unit / operation device or other operation units, and particularly suitable for the data required for operation in the neural network operation device Or all data that cannot be saved in the internal storage of other processing devices.

The combined processing device can be used as an SOC system-on-chip for devices such as mobile phones, robots, drones, and video surveillance equipment, effectively reducing the core area of the control section, increasing processing speed, and reducing overall power consumption. In this case, the universal interconnection interface of the combined processing device is connected to some parts of the equipment. Some components such as camera, monitor, mouse, keyboard, network card, wifi interface.

The embodiments of the present disclosure provide a neural network processor board, which can be used in many general or special-purpose computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, smart homes, home appliances, multiprocessor systems, microprocessor-based systems, robots, programmable consumer electronics devices, network personal computers , PC), small computers, mainframe computers, distributed computing environments including any of the above systems or equipment, and more.

Please refer to FIG. 5a, which is a schematic structural diagram of a neural network processor board provided by an embodiment of the present disclosure. As shown in FIG. 5 a, the neural network processor board 10 includes a neural network chip package structure 11, a first electrical and non-electrical connection device 12, and a first substrate 13.

The present disclosure does not limit the specific structure of the neural network chip packaging structure 11. Optionally, as shown in FIG. 5b, the aforementioned neural network chip packaging structure 11 includes: a neural network chip 111, a second electrical and non-electrical connection device 112, a first Two substrates 113.

The specific form of the neural network chip 111 involved in this disclosure is not limited. The aforementioned neural network chip 111 includes, but is not limited to, a neural network chip that integrates a neural network processor. The above chip may be made of silicon material, germanium material, quantum material, or molecule. Materials. According to the actual situation (for example: harsh environment) and different application requirements, the above neural network chip can be packaged so that most of the neural network chip is wrapped, and the pins on the neural network chip are passed through gold wires. The isoconductor is connected to the outer side of the package structure for circuit connection with the outer layer.

The present disclosure does not limit the specific structure of the neural network chip 111. For optional, please refer to the device shown in FIG. 1a or FIG. 1b.

The disclosure does not limit the types of the first substrate 13 and the second substrate 113, and may be a printed circuit board (PCB) or a printed wiring board (PWB), or may be other circuit boards. There are no restrictions on the materials used to make the PCB.

The second substrate 113 according to the present disclosure is used to carry the neural network chip 111, and a neural network chip package structure 11 obtained by connecting the neural network chip 111 and the second substrate 113 through a second electrical and non-electrical connection device 112. It is used to protect the neural network chip 111 and facilitate the further packaging of the neural network chip packaging structure 11 and the first substrate 13.

There is no limitation on the above-mentioned specific packaging method of the second electrical and non-electrical connection device 112 and the corresponding structure of the packaging method. A suitable packaging method can be selected and simply improved according to the actual situation and different application needs, such as flip chip Flip Chip Ball Grid Array Package (FCBGAP), Low-profile Quad Flat Package (LQFP), Quad Flat Package with Heat sink (HQFP), None Packaging methods such as Quad Flat Non-lead Package (QFN) or Fine-pitch Ball Grid Package (FBGA).

Flip chip (Flip Chip) is suitable for the case where the area after packaging is high or the inductance of the wire and the signal transmission time are sensitive. In addition, wire bonding can be used to reduce the cost and improve the flexibility of the packaging structure.

Ball Grid Array, which can provide more pins, and the average lead length of the pins is short, which has the function of transmitting signals at high speed. Among them, the package can be packaged with a pin grid array (Pin Grid Array, PGA) , Zero Insertion Force (ZIF), Single Edge Contact Connection (SECC), and Land Grid Array (LGA).

Optionally, the Flip Chip Ball Grid Array (Flip Chip Ball Grid Array) packaging method is used to package the neural network chip 111 and the second substrate 113. For a schematic diagram of a specific neural network chip packaging structure, refer to FIG. 6. As shown in FIG. 6, the aforementioned neural network chip package structure includes: a neural network chip 21, a pad 22, a solder ball 23, a second substrate 24, a connection point 25 on the second substrate 24, and a pin 26.

Among them, the pad 22 is connected to the neural network chip 21, and a solder ball 23 is formed by welding between the pad 22 and the connection point 25 on the second substrate 24, and the neural network chip 21 and the second substrate 24 are connected. Packaging of the neural network chip 21.

Pin 26 is used to connect with the external circuit of the package structure (for example, the first substrate 13 on the neural network processor board 10), and can realize the transmission of external data and internal data, which is convenient for the neural network chip 21 or the neural network chip 21 The corresponding neural network processor processes the data. The type and quantity of pins are not limited in this disclosure. Different pin forms can be selected according to different packaging technologies and arranged in accordance with certain rules.

Optionally, the aforementioned neural network chip package structure further includes an insulating filler placed in a gap between the pad 22, the solder ball 23, and the connection point 25 to prevent interference between the solder ball and the solder ball.

The material of the insulating filler may be silicon nitride, silicon oxide, or silicon oxynitride; interference includes electromagnetic interference, inductive interference, and the like.

Optionally, the aforementioned neural network chip package structure further includes a heat dissipation device for dissipating heat during operation of the neural network chip 21. The heat dissipation device may be a metal sheet, a heat sink, or a heat sink with good thermal conductivity, such as a fan.

For example, as shown in FIG. 6a, the neural network chip package structure 11 includes: a neural network chip 21, a pad 22, a solder ball 23, a second substrate 24, a connection point 25, a pin 26, The insulating filler 27, the heat dissipation paste 28, and the metal case heat sink 29. Among them, the heat dissipation paste 28 and the metal shell heat sink 29 are used to dissipate heat during the operation of the neural network chip 21.

Optionally, the aforementioned neural network chip package structure 11 further includes a reinforcing structure connected to the pad 22 and embedded in the solder ball 23 to enhance the connection strength between the solder ball 23 and the pad 22.

The reinforcing structure may be a metal wire structure or a columnar structure, which is not limited herein.

The present disclosure also does not limit the specific form of the first electrical and non-electrical device 12, and may refer to the description of the second electrical and non-electrical device 112, that is, the neural network chip packaging structure 11 is packaged by soldering, and connection may also be adopted. The method of connecting the second substrate 113 and the first substrate 13 in a line connection or plugging manner is convenient for subsequent replacement of the first substrate 13 or the neural network chip package structure 11.

Optionally, the first substrate 13 includes an interface of a memory unit for expanding the storage capacity, such as: synchronous dynamic random access memory (SDRAM), double-rate synchronous dynamic random access memory (Double Date Rate SDRAM, DDR), etc., to improve the processing capacity of the neural network processor by expanding the memory.

The first substrate 13 may further include a Peripheral Component Interconnect-Express (PCI-E or PCIe) interface, a Small Form-factor Pluggable (SFP) interface, an Ethernet interface, Controller Area Network (CAN) interfaces, etc., are used for data transmission between the package structure and external circuits, which can improve the operation speed and the convenience of operation.

The neural network processor is packaged as a neural network chip 111, the neural network chip 111 is packaged as a neural network chip package structure 11, the neural network chip package structure 11 is packaged as a neural network processor board 10, and an interface on the board ( Slots or inserts) perform data interaction with external circuits (for example, computer motherboards), that is, the function of the neural network processor is realized directly by using the neural network processor board 10 and the neural network chip 111 is protected. In addition, other modules can be added to the neural network processor board 10, which improves the application range and operation efficiency of the neural network processor.

In one embodiment, the present disclosure discloses an electronic device including the neural network processor board 10 or the neural network chip package structure 11 described above.

Electronic devices include data processing devices, robots, computers, printers, scanners, tablets, smart terminals, mobile phones, driving recorders, navigators, sensors, cameras, servers, cameras, camcorders, projectors, watches, headphones, mobile storage , Wearables, vehicles, home appliances, and / or medical devices.

The vehicles include airplanes, ships, and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, cooker hoods, and the medical equipment includes nuclear magnetic resonance Instrument, B-mode and / or electrocardiograph.

The specific embodiments described above further describe the purpose, technical solution and beneficial effects of the present disclosure. It should be understood that the above are only specific embodiments of the present disclosure and are not used to limit the present disclosure. Any modification, equivalent replacement, or improvement made within the spirit and principle of this disclosure shall be included in the protection scope of this disclosure.

A, B, S‧‧‧ Matrix

P‧‧‧ vector

10‧‧‧ Neural Network Processor Board

11‧‧‧ neural network chip packaging structure

12‧‧‧First electrical and non-electrical connection device

13‧‧‧First substrate

111‧‧‧ neural network chip

112‧‧‧Second electrical and non-electrical connection device

113‧‧‧second substrate

1111‧‧‧Storage Unit

1112‧‧‧Direct Memory Access Unit

1113‧‧‧Instruction cache unit

1114‧‧‧weight cache unit

1115‧‧‧Input neuron buffer unit

1116‧‧‧Output neuron buffer unit

1117‧‧‧Control Unit

1118‧‧‧ Computing Unit

21‧‧‧Neural Network Chip

22‧‧‧ pad

23‧‧‧Solder Ball

24‧‧‧second substrate

25‧‧‧ Connection points on the second substrate 24

26‧‧‧pin

27‧‧‧Insulation filler

28‧‧‧ Thermal Paste

29‧‧‧ metal case heat sink

FIG. 1a is a schematic structural diagram of an integrated circuit chip device.

FIG. 1b is a schematic structural diagram of another integrated circuit chip device.

Figure 1c is a schematic structural diagram of a basic processing circuit.

FIG. 1d is a schematic structural diagram of a main processing circuit.

FIG. 1e is a schematic structural diagram of a fixed-point data type.

Figure 2a is a schematic diagram of a method of using a basic processing circuit.

Figure 2b is a schematic diagram of data transmission by a main processing circuit.

Figure 2c is a schematic diagram of a matrix multiplied by a vector.

FIG. 2d is a schematic structural diagram of an integrated circuit chip device.

FIG. 2e is a schematic structural diagram of another integrated circuit chip device.

Figure 2f is a schematic diagram of a matrix multiplied by a matrix.

Figure 3a is a schematic diagram of convolution input data.

Figure 3b is a schematic diagram of a convolution kernel.

FIG. 3c is a schematic diagram of a calculation window of a three-dimensional data block of input data.

FIG. 3d is a schematic diagram of another operation window of a three-dimensional data block of input data.

Figure 3e is a schematic diagram of another operation window of a three-dimensional data block of the input data.

Figure 4a is a schematic diagram of a forward operation of a neural network.

Figure 4b is a schematic diagram of the reverse operation of the neural network.

FIG. 4c also discloses a schematic structural diagram of a combined processing device in the present disclosure.

FIG. 4d also discloses another structural schematic diagram of a combined processing device in the present disclosure.

5a is a schematic structural diagram of a neural network processor board provided in an embodiment of the present disclosure;

5b is a schematic structural diagram of a neural network chip package structure provided by an embodiment of the present disclosure;

5c is a schematic structural diagram of a neural network chip according to an embodiment of the present disclosure;

6 is a schematic diagram of a neural network chip package structure provided by the embodiment of the present disclosure;

FIG. 6a is a schematic diagram of another neural network chip package structure provided by the embodiment of the present disclosure.

Claims (17)

An integrated circuit chip device, wherein the integrated circuit chip device includes: a main processing circuit and a plurality of basic processing circuits; the plurality of basic processing circuits are distributed in an array; each basic processing circuit and adjacent other basic processing circuits Connection, the main processing circuit is connected to k basic processing circuits in the plurality of basic processing circuits, and the k basic processing circuits are: n basic processing circuits in the first row, n basic processing circuits in the m row, and 1 column of m basic processing circuits; the plurality of basic processing circuits include a data type conversion circuit for performing conversion between floating point type data and fixed point type data; the main processing circuit for performing neural network operations Each of the continuous operations and data transmission with the k basic processing circuits; the k basic processing circuits are used for data forwarding between the main processing circuit and the multiple basic processing circuits; the multiple basic processing circuits , Used to determine whether to start the data type conversion circuit to execute the data type of the transmitted data according to the type of the transmitted data and the operation Conversion, carried out in parallel in the neural network computation based on the transmission data, and the calculation results of the main processing circuit and the base-k transferred to the processing circuit. The integrated circuit chip device according to the first patent application scope, wherein the main processing circuit is configured to obtain a data block to be calculated and an operation instruction, and divide the data block to be calculated into a distribution data block according to the operation instruction. And a broadcast data block; split the distribution data block to obtain a plurality of basic data blocks, distribute the plurality of basic data blocks to the k basic processing circuits, and broadcast the broadcast data block to the k Basic processing circuit; the multiple basic processing circuits are configured to convert the basic data block and the broadcast data block into fixed-point data type basic data blocks according to the received basic data block, the broadcast data block, and the operation instruction; Broadcast data block, performing an inner product operation on the basic data block and the broadcast data block with a fixed-point data type to obtain a fixed-point data type operation result, and converting the fixed-point data type operation result into a floating-point data type operation result through the k The basic processing circuit is transmitted to the main processing circuit; the main processing circuit is configured to process the operation result to obtain the The calculated data block and the instruction result of the operation instruction. The integrated circuit chip device according to item 2 of the scope of patent application, wherein the main processing circuit is specifically configured to broadcast the broadcast data block to the k basic processing circuits through one broadcast. The integrated circuit chip device according to item 2 of the scope of patent application, wherein the main processing circuit is configured to, when the operation result is the result of the inner product processing, accumulate the operation result to obtain an accumulation result, and the accumulation result Arrange to obtain the data block to be calculated and the instruction result of the operation instruction. The integrated circuit chip device according to item 2 of the scope of patent application, wherein the main processing circuit is specifically configured to divide the broadcast data block into multiple partial broadcast data blocks, and broadcast the multiple partial broadcast data blocks to the multiple times. k basic processing circuits. The integrated circuit chip device according to item 5 of the scope of patent application, wherein the plurality of basic processing circuits are specifically configured to convert the partial broadcast data block and the basic data block into a fixed-point data type, and perform an inner product with the fixed-point data type. After processing, the inner product processing result of a certain point data type is obtained. The inner product processing result of the fixed point data type is accumulated to obtain a partial operation result of a certain point data type. The k basic processing circuits are sent to the main processing circuit. The integrated circuit chip device according to item 6 of the scope of patent application, wherein the plurality of basic processing circuits are specifically configured to multiplex the partial broadcast data block n times to execute the partial broadcast data block and the n basic data in a fixed-point data type. The inner product operation of the block obtains n partial processing results of the fixed-point data type, and the n partial processing results of the fixed-point data type are accumulated respectively to obtain the n partial operation results of the fixed-point data type. The operation result is converted into a floating-point data type and n partial operation results are sent to the main processing circuit through the k basic processing circuits, where n is an integer greater than or equal to 2. The integrated circuit chip device according to any one of claims 1-7, wherein the main processing circuit includes: a main register or a main on-chip cache circuit; and the plurality of basic processing circuits includes: a basic register or a basic on-chip cache circuit. The integrated circuit chip device according to item 8 of the scope of patent application, wherein the main processing circuit includes a vector operator circuit, an arithmetic logic unit circuit, an accumulator circuit, a matrix transposition circuit, a direct memory access circuit, and a data type operation circuit. Or one or any combination of data rearrangement circuits. The integrated circuit chip device according to item 1 of the scope of patent application, wherein the data is one or any combination of a vector, a matrix, a three-dimensional data block, a four-dimensional data block, and an n-dimensional data block. The integrated circuit chip device according to item 2 of the scope of patent application, wherein if the operation instruction is a multiplication instruction, the main processing circuit determines that the multiplier data block is the broadcast data block and the multiplicand data block is the distribution data block; The operation instruction is a convolution instruction. The main processing circuit determines that the input data block is the broadcast data block and the convolution kernel is the distribution data block. A neural network computing device, wherein the neural network computing device includes one or more integrated circuit chip devices according to any one of claims 1-11 of the scope of patent application. A combined processing device, wherein the combined processing device includes a neural network computing device, a universal interconnection interface, and a universal processing device, such as the scope of application for a patent; the neural network computing device communicates with the universal processing device through the universal interconnection interface. Device connection. A chip in which the device is integrated with any one of items 1 to 13 of the scope of patent application. A smart device, wherein the smart device includes a chip such as the scope of patent application No. 14. A method for calculating a neural network, wherein the method is applied in an integrated circuit chip device, and the integrated circuit chip device includes the integrated circuit chip device such as any one of claims 1-11 in the scope of patent application, the integrated circuit chip device Used to perform operations on neural networks. The method according to item 16 of the scope of patent application, wherein the operation of the neural network includes: convolution operation, matrix multiplication matrix operation, matrix multiplication vector operation, paranoid operation, fully connected operation, GEMM operation, GEMV operation, and activation operation. One or any combination.