CN110197271B - Integrated circuit chip device and related product - Google Patents

Integrated circuit chip device and related product Download PDF

Info

Publication number
CN110197271B
CN110197271B CN201810164737.5A CN201810164737A CN110197271B CN 110197271 B CN110197271 B CN 110197271B CN 201810164737 A CN201810164737 A CN 201810164737A CN 110197271 B CN110197271 B CN 110197271B
Authority
CN
China
Prior art keywords
data block
processing circuit
data
basic
circuit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810164737.5A
Other languages
Chinese (zh)
Other versions
CN110197271A (en
Inventor
不公告发明人
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Cambricon Information Technology Co Ltd
Original Assignee
Shanghai Cambricon Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Cambricon Information Technology Co Ltd filed Critical Shanghai Cambricon Information Technology Co Ltd
Priority to CN201810164737.5A priority Critical patent/CN110197271B/en
Priority to PCT/CN2019/075979 priority patent/WO2019165940A1/en
Publication of CN110197271A publication Critical patent/CN110197271A/en
Application granted granted Critical
Publication of CN110197271B publication Critical patent/CN110197271B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means

Abstract

The present disclosure provides an integrated circuit chip device and related products, the integrated circuit chip device comprising: a main processing circuit and a plurality of basic processing circuits; the main processing circuit includes a first mapping circuit, at least one of the plurality of basic processing circuits includes a second mapping circuit, and the first mapping circuit and the second mapping circuit are each configured to perform compression processing of respective data in a neural network operation. The technical scheme provided by the disclosure has the advantages of small calculation amount and low power consumption.

Description

Integrated circuit chip device and related product
Technical Field
The present disclosure relates to the field of neural networks, and more particularly to an integrated circuit chip device and related products.
Background
Artificial Neural Networks (ANN) are a research hotspot in the field of Artificial intelligence since the 80 s of the 20 th century. The method abstracts the human brain neuron network from the information processing angle, establishes a certain simple model, and forms different networks according to different connection modes. It is also often directly referred to in engineering and academia as neural networks or neural-like networks. A neural network is an operational model, which is formed by connecting a large number of nodes (or neurons). The operation of the existing neural network is realized based on a Central Processing Unit (CPU) or a Graphics Processing Unit (GPU), and the operation has a large amount of calculation and high power consumption.
Disclosure of Invention
Embodiments of the present disclosure provide an integrated circuit chip device and related products, which can increase the processing speed and efficiency of a computing device.
In a first aspect, an integrated circuit chip device is provided, the integrated circuit chip device comprising: a main processing circuit and a plurality of basic processing circuits; the main processing circuit comprises a first mapping circuit, at least one circuit (namely, part or all of the basic processing circuits) in the plurality of basic processing circuits comprises a second mapping circuit, and the first mapping circuit and the second mapping circuit are used for executing compression processing of each data in the neural network operation;
the main processing circuit is used for acquiring an input data block, a convolution kernel data block and a convolution instruction, dividing the input data block into broadcast data blocks according to the convolution instruction, and dividing the convolution kernel data block into distribution data blocks; determining to start a first mapping circuit to process a first data block according to the operation control of the convolution instruction to obtain a processed first data block; the first data block comprises the distribution data block and/or the broadcast data block; sending the processed first data block to at least one basic processing circuit in basic processing circuits connected with the main processing circuit according to the convolution instruction;
the plurality of basic processing circuits are used for determining whether to start a second mapping circuit to process a second data block according to the operation control of the convolution instruction, executing operation in a neural network in a parallel mode according to the processed second data block to obtain an operation result, and transmitting the operation result to the main processing circuit through the basic processing circuit connected with the main processing circuit; the second data block is determined by the basic processing circuit to receive the data block sent by the main processing circuit, and the second data block is associated with the processed first data block;
and the main processing circuit is used for processing the operation result to obtain an instruction result of the convolution instruction.
In a second aspect, a neural network computing device is provided, which includes one or more integrated circuit chip devices provided in the first aspect.
In a third aspect, there is provided a combined processing apparatus comprising: the neural network arithmetic device, the universal interconnection interface and the universal processing device are provided by the second aspect;
the neural network operation device is connected with the general processing device through the general interconnection interface.
In a fourth aspect, a chip is provided that integrates the apparatus of the first aspect, the apparatus of the second aspect, or the apparatus of the third aspect.
In a fifth aspect, an electronic device is provided, which comprises the chip of the fourth aspect.
In a sixth aspect, a method for operating a neural network is provided, where the method is applied in an integrated circuit chip device, and the integrated circuit chip device includes: the integrated circuit chip apparatus of the first aspect, configured to perform a convolution operation of a neural network.
It can be seen that, according to the embodiment of the disclosure, the mapping circuit is provided to compress the data blocks and then perform the operation, so that the transmission resource and the calculation resource are saved, and therefore, the mapping circuit has the advantages of low power consumption and small calculation amount.
Drawings
FIG. 1a is a schematic diagram of an integrated circuit chip device.
FIG. 1b is a schematic diagram of another integrated circuit chip device.
FIG. 1c is a schematic diagram of a basic processing circuit.
FIG. 2a is a schematic diagram of a process of multiplying a matrix by a vector.
Fig. 2b is a schematic diagram of a matrix multiplied by a vector.
Fig. 2c is a schematic diagram of a process of multiplying a matrix by a matrix.
Fig. 2d is a schematic diagram of the matrix Ai multiplied by the vector B.
Fig. 2e is a schematic diagram of matrix a multiplied by matrix B.
Fig. 2f is a schematic diagram of matrix Ai multiplied by matrix B.
FIG. 3a is a schematic diagram of neural network training.
FIG. 3b is a schematic diagram of convolution operation.
Fig. 4 is a schematic structural diagram of a neural network chip according to an embodiment of the present disclosure;
fig. 5 a-5 b are schematic structural diagrams of two mapping circuits provided in the present embodiment.
Detailed Description
In order to make the technical solutions of the present disclosure better understood by those skilled in the art, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are only a part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments, which can be derived by one of ordinary skill in the art from the embodiments disclosed herein without making any creative effort, shall fall within the scope of protection of the present disclosure.
In the apparatus provided in the first aspect, the main processing circuit includes a first mapping circuit, at least one of the plurality of basic processing circuits includes a second mapping circuit, and the first mapping circuit and the second mapping circuit are each configured to perform compression processing of respective data in a neural network operation;
the main processing circuit is used for acquiring an input data block, a convolution kernel data block and a convolution instruction, dividing the input data block into vertical data blocks according to the convolution instruction, and dividing the convolution kernel data block into horizontal data blocks; determining to start a first mapping circuit to process a first data block according to the operation control of the convolution instruction to obtain a processed first data block; the first data block comprises the distribution data block and/or the broadcast data block; sending the processed first data block to at least one basic processing circuit in basic processing circuits connected with the main processing circuit according to the convolution instruction;
the plurality of basic processing circuits are used for determining whether to start a second mapping circuit to process a second data block according to the operation control of the convolution instruction, executing operation in a neural network in a parallel mode according to the processed second data block to obtain an operation result, and transmitting the operation result to the main processing circuit through the basic processing circuit connected with the main processing circuit; the second data block is determined by the basic processing circuit to receive the data block sent by the main processing circuit, and the second data block is associated with the processed first data block; and the main processing circuit is used for processing the operation result to obtain an instruction result of the convolution instruction.
In the apparatus provided in the first aspect, when the first data block includes a distribution data block and a broadcast data block, the main processing circuit is specifically configured to start the first mapping circuit to process the distribution data block and the broadcast data block to obtain a processed distribution data block, an identification data block associated with the distribution data block, a processed broadcast data block, and an identification data block associated with the broadcast data block; splitting the processed distribution data block and the identification data block associated with the distribution data block to obtain a plurality of basic data blocks and identification data blocks associated with the basic data blocks; distributing the plurality of basic data blocks and the identification data blocks respectively associated with the plurality of basic data blocks to a basic processing circuit connected with the basic processing circuit, and broadcasting the broadcast data blocks and the identification data blocks associated with the broadcast data blocks to the basic processing circuit connected with the basic processing circuit;
the basic processing circuit is used for starting the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the basic data block and the identification data block associated with the broadcast data block; and processing the basic data block and the broadcast data block according to the connection identification data block, performing convolution (namely inner product) operation on the processed basic data block and the processed broadcast data block to obtain an operation result, and sending the operation result to the main processing circuit.
The identification data block may be specifically represented by a direct index or a step index, and may also be represented by a List of Lists (List of Lists, LIL), a Coordinate List (COO), a Compressed Sparse Row (CSR), a Compressed Sparse Column (CSC), an ELLPack, ELL), a hybrid (Hybird, HYB), and the like.
Taking the identification data block represented by a direct index as an example, the identification data block may specifically be a data block composed of 0 and 1, where 0 represents that an absolute value of data (such as a weight or an input neuron) included in the data block is less than or equal to a first threshold, 1 represents that an absolute value of data (such as a weight or an input neuron) included in the data block is greater than a first threshold, and the first threshold is randomly set by a user side or a device side in a customized manner, for example, 0.05, 0, and so on.
In order to save data transmission amount and improve data transmission efficiency, in the process that the main processing circuit sends data to the basic processing circuit, target data in the plurality of basic data blocks and identification data blocks respectively associated with the plurality of basic data blocks can be specifically distributed to the basic processing circuit connected with the main processing circuit; optionally, the target data in the processed broadcast data block and the identification data block associated with the broadcast data block may also be broadcast to a basic processing circuit connected thereto. The target data refers to data with an absolute value greater than a first threshold in a data block, or refers to non-0 data in a data block (which may be specifically a processed distribution data block or a processed broadcast data block).
For example, distributing the data block to M1Line N1Matrix of columns, basic data block M2Line N2A matrix of columns, wherein M1>M2，N1>N2. Correspondingly, the identification data block associated with the distribution data block is also M1Line N1A matrix of columns, the identification data block associated with the basic data block being likewise M2Line N2A matrix of columns. Take the matrix with 2 x 2 as the basic data block as an example, set as
Figure GDA0001637487100000031
If the first threshold is 0.05, the identification data block associated with the basic data block is
Figure GDA0001637487100000032
The processing of the data blocks with respect to the first mapping circuit and the second mapping circuit will be described in detail later.
In the apparatus provided in the first aspect, when the first data block includes a distribution data block, the main processing circuit is specifically configured to start the first mapping circuit to process the distribution data block to obtain a processed distribution data block and an identification data block associated with the distribution data block, or start the first mapping circuit to process the distribution data block according to a pre-stored identification data block associated with the distribution data block to obtain a processed distribution data block; splitting the processed distribution data block and the identification data block associated with the distribution data block to obtain a plurality of basic data blocks and identification data blocks associated with the basic data blocks; distributing the plurality of elementary data blocks and the identification data blocks associated with each of the plurality of elementary data blocks to a base processing circuit connected thereto; broadcasting the broadcast data block to a base processing circuit connected thereto;
the basic processing circuit is used for starting the second mapping circuit to process the broadcast data block according to the identification data block associated with the basic data block, performing convolution operation on the processed broadcast data block and the basic data block to obtain an operation result, and sending the operation result to the main processing circuit.
In an optional embodiment, the main processing circuit is further specifically configured to split the broadcast data block or the processed broadcast data block and the identification data block associated with the broadcast data block to obtain a plurality of partial broadcast data blocks and identification data blocks associated with the plurality of partial broadcast data blocks; broadcasting the plurality of partial broadcast data blocks and the identification data blocks respectively associated with the plurality of partial broadcast data blocks to the basic processing circuit by one or more times; wherein the plurality of partial broadcast data blocks are combined to form the broadcast data block or the processed broadcast data block.
Correspondingly, the basic processing circuit is specifically configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the partial broadcast data block and the identification data block associated with the basic data block; processing the partial broadcast data block and the basic data block according to the connection identification data to obtain a processed partial broadcast data block and a processed basic data block; performing an inner product operation on the processed partial broadcast data block and the processed basic data block.
Wherein the connection identification data block is a data block obtained by performing an element-by-element AND operation on the identification data block associated with the basic data block and the identification data block associated with the partial broadcast data block. Optionally, the connection identification data block is used to indicate that the data in both data blocks (specifically, the basic data block and the broadcast data block) is larger than an absolute value. Details will be described later.
For example, a distribution data block associated matrix identifying data blocks as 2 x 3
Figure GDA0001637487100000041
Matrix with identification data block of 2 x 2 related to partial broadcast data block
Figure GDA0001637487100000042
The connection identification data block obtained correspondingly is
Figure GDA0001637487100000043
In an optional embodiment, the main processing circuit is further specifically configured to split the broadcast data block to obtain a plurality of partial broadcast data blocks; broadcasting the plurality of partial broadcast data blocks to the base processing circuitry one or more times; wherein the plurality of partial broadcast data blocks are combined to form the broadcast data block or the processed broadcast data block.
Correspondingly, the basic processing circuit is specifically configured to process the partial broadcast data block according to the identification data block associated with the basic data block to obtain a processed partial broadcast data block; performing an inner product operation on the basic data block and the processed partial broadcast data block.
In the apparatus provided in the first aspect, when the first data block includes a broadcast data block, the main processing circuit is specifically configured to start the first mapping circuit to process the broadcast data block to obtain a processed broadcast data block and an identifier data block associated with the broadcast data block, or start the first mapping circuit to process the broadcast data block according to a pre-stored identifier data block associated with the broadcast data block to obtain a processed broadcast data block; splitting the distribution data block to obtain a plurality of basic data blocks; distributing the plurality of base data to a base processing circuit connected thereto; broadcasting the processed broadcast data block and the identification data block related to the broadcast data block to a basic processing circuit connected with the broadcast data block;
the basic processing circuit is used for starting the second mapping circuit to process the basic data block according to the identification data block associated with the broadcast data block to obtain a processed basic data block; and performing convolution operation on the processed basic data block and the processed broadcast data block to obtain an operation result, and sending the operation result to the main processing circuit.
In an optional embodiment, the main processing circuit is further specifically configured to split the processed broadcast data block and the identification data block associated with the broadcast data block to obtain a plurality of partial broadcast data blocks and identification data blocks associated with the plurality of partial broadcast data blocks; broadcasting the plurality of partial broadcast data blocks and the identification data blocks respectively associated with the plurality of partial broadcast data blocks to the basic processing circuit by one or more times; wherein the plurality of partial broadcast data blocks are combined to form the broadcast data block or the processed broadcast data block.
Correspondingly, the basic processing circuit is specifically configured to process the basic data block according to the identification data block associated with the partial broadcast data block to obtain a processed basic data block; performing an inner product operation on the processed basic data block and the partial broadcast data block.
In the apparatus provided in the first aspect, the basic processing circuit is specifically configured to perform convolution operation on the basic data block and the broadcast data block to obtain a convolution result, accumulate the convolution result to obtain an operation result, and send the operation result to the main processing circuit; and the main processing circuit is used for obtaining accumulation results after accumulating the operation results and arranging the accumulation results to obtain the instruction results.
In the apparatus provided in the first aspect, the main processing circuit is specifically configured to send the broadcast data block (specifically, the broadcast data block or the processed broadcast data block) to the basic processing circuit connected thereto through one-time broadcast.
In the apparatus provided in the first aspect, the basic processing circuit is specifically configured to perform an inner product processing on the basic data block (which may be the basic data block or the processed basic data block) and the broadcast data block to obtain an inner product processing result, accumulate the inner product processing result to obtain an operation result, and send the operation result to the main processing circuit.
In the apparatus provided in the first aspect, the main processing circuit is configured to, when the operation result is a result of inner product processing, accumulate the operation result to obtain an accumulated result, and arrange the accumulated result to obtain the instruction result.
In the apparatus provided in the first aspect, the main processing circuit is specifically configured to divide the broadcast data block into a plurality of partial broadcast data blocks, and broadcast the plurality of partial broadcast data blocks to the base processing circuit by multiple times; the plurality of partial broadcast data blocks are combined to form the broadcast data block.
In the apparatus provided in the first aspect, the basic processing circuit is specifically configured to perform an inner product process on the partial broadcast data block (specifically, the partial broadcast data block or the processed partial broadcast data block) and the basic data block to obtain an inner product process result, accumulate the inner product process result to obtain a partial operation result, and send the partial operation result to the main processing circuit.
Here, the basic data block takes core 3 × 3 as an example, the partial broadcast data block takes 3 × 3 matrix as an example, and the 3 × 3 matrix and core 3 × 3 respectively execute corresponding position multiplication, so that the corresponding inner product result has 3 inner product processing results, and the 3 inner product processing results are accumulated to obtain a partial operation result. The 3 inner product processing results Out0 (inner product of row 0 of 3 × 3 matrix and row 0 of core 3 × 3), Out1 (inner product of row 1 of 3 × 3 matrix and row 1 of core 3 × 3), and Out2 (inner product of row 2 of 3 × 3 matrix and row 2 of core 3 × 3) may specifically be:
Out0＝r00*k0[0]+r01*k0[1]+r02*k0[2]
Out1＝r10*k1[0]+r11*k1[1]+r12*k1[2]
Out2＝r20*k2[0]+r21*k2[1]+r22*k2[2]
where r of r00 denotes a partial broadcast data block, and 00 denotes a 0 th column element of row 0.
k of k0[0], denotes an elementary data block, 0[0] denotes the 0 th column element of row 0;
the partial operation result is Out0+ Out1+ Out 2.
In the apparatus provided in the first aspect, the basic processing circuit is specifically configured to multiplex n times the partial broadcast data block to execute the partial broadcast data block and perform an inner product operation on the n basic data blocks to obtain n groups of inner product operation results, where the n groups of inner product operation results correspond to the n basic data blocks, each group of inner product operation results in the n groups of inner product operation results is accumulated to obtain n partial operation results, and the n partial operation results are sent to the main processing circuit, where n is an integer greater than or equal to 2.
Here, the basic data block takes n cores 3 × 3 as an example, the partial broadcast data block takes 3 × 3 matrix as an example, n times of 3 × 3 matrix multiplexing is respectively performed with the cores 3 × 3 for n times of corresponding position multiplication, each operation, that is, the corresponding inner product result, has 3 inner product results, the 3 inner product results form a group of inner product operation results, and the 3 inner product results of each group in n groups are accumulated to obtain n partial operation results.
The above example is only for explaining the convolution operation, and the kernel 3 × 3 may be replaced by another rule, and of course, the partial input data may be a data block having another rule.
In an apparatus provided in the first aspect, the main processing circuit includes: a master register or on-master cache circuit;
the base processing circuit includes: basic registers or basic on-chip cache circuits.
In an apparatus provided in the first aspect, the main processing circuit includes: one or any combination of vector arithmetic unit circuit, arithmetic logic unit circuit, accumulator circuit, matrix transposition circuit, direct memory access circuit or data rearrangement circuit.
In the apparatus provided in the first aspect, the input data block may be represented by a tensor, which may be specifically: one or any combination of vector, matrix, three-dimensional data block, four-dimensional data block and n-dimensional data block;
the convolution kernel data block is: one or any combination of a matrix, a three-dimensional data block, a four-dimensional data block, and an n-dimensional data block.
In a method provided in a sixth aspect, the operation of the neural network comprises: one or any combination of convolution operation, matrix multiplication matrix operation, matrix multiplication vector operation, partial execution operation, full connection operation, GEMM operation, GEMV operation and activation operation.
Referring to fig. 1a, fig. 1a is a schematic structural diagram of an integrated circuit chip device, as shown in fig. 1a, the chip device includes: main processing circuitry, basic processing circuitry, and branch processing circuitry (optional). Wherein the content of the first and second substances,
the main processing circuit may include a register and/or an on-chip cache circuit, and may further include a control circuit, a vector operator circuit, an ALU (arithmetic and logic unit) circuit, an accumulator circuit, a DMA (Direct Memory Access) circuit, and other circuits, such as a conversion circuit (e.g., a matrix transpose circuit), a data rearrangement circuit, an activation circuit, and the like;
optionally, the main processing circuit may include: the first mapping circuit may be configured to process received or transmitted data to obtain processed data and mask data associated with the data, where the mask data is used to indicate whether an absolute value of the data is greater than a preset threshold, and optionally, the mask data may be 0 or 1, where 0 indicates that the absolute value of the data is less than or equal to the preset threshold; conversely, a 1 indicates that the absolute value of the data is greater than the preset threshold. The preset threshold is set by the user side or the terminal device side in a self-defined manner, for example, 0.1 or 0.05, and the like. In practical applications, the first mapping circuit may be used to eliminate data that is 0 or not greater than a predetermined threshold (e.g., 0.1), or set these data to 0. The method has the advantages that the data volume transmitted from the main processing circuit to the basic processing circuit is reduced, the calculated amount of data operation in the basic processing circuit is reduced, and the data processing efficiency is improved. The present invention is not limited to the specific form of the first mapping circuit described above. Specific implementations for the first mapping circuit are set forth below.
For example, the input data of the main processing circuit is a matrix data block
Figure GDA0001637487100000061
After being processed by the first mapping circuit, the processed matrix data block can be obtained as
Figure GDA0001637487100000062
The identification data block associated with the matrix data block is
Figure GDA0001637487100000063
The specific processing of the first mapping circuit will be described in detail later.
Accordingly, when the main processing circuit distributes data to the basic processing circuit, only two data of 1 and 0.5 may be transmitted, instead of transmitting the processed matrix data block, 8 data; meanwhile, the identification data block associated with the matrix data block is also required to be sent to the basic processing circuit together, so that the basic processing circuit correspondingly knows the positions of the two data in the original matrix data block according to the received identification data block and the received two data (1 and 0.5). That is, the basic processing circuit may correspondingly restore the matrix data block processed in the main processing circuit according to the received identification data block and the received data.
The main processing circuit further includes a data transmitting circuit, a data receiving circuit or an interface, the data transmitting circuit may integrate the data distributing circuit and the data broadcasting circuit, and certainly in practical application, the data distributing circuit and the data broadcasting circuit may also be separately configured; in practical applications, the data transmitting circuit and the data receiving circuit may be integrated together to form a data transmitting/receiving circuit. For broadcast data, i.e. data that needs to be sent to each of the basic processing circuits. For the distribution data, i.e. the data that needs to be selectively sent to part of the basic processing circuits, the specific selection mode can be specifically determined by the main processing circuit according to the load and the calculation mode. For the broadcast transmission mode, broadcast data is transmitted to each base processing circuit in a broadcast form. (in practical applications, broadcast data is transmitted to each basic processing circuit by one-time broadcasting, or broadcast data is transmitted to each basic processing circuit by multiple-time broadcasting, and the specific embodiments of the present invention do not limit the number of times of broadcasting), the distribution transmission method is to selectively transmit the distribution data to a part of the basic processing circuits.
When data distribution is realized, the control circuit of the main processing circuit transmits data to part or all of the basic processing circuits (the data may be the same or different, specifically, if the data is transmitted in a distribution mode, the data received by each basic processing circuit receiving the data may be different, and certainly, the data received by some basic processing circuits may be the same;
specifically, when data is broadcast, the control circuit of the main processing circuit transmits data to part or all of the basic processing circuits, and each basic processing circuit receiving data can receive the same data.
Optionally, the vector operator circuit of the main processing circuit may perform vector operations, including but not limited to: two vectors are added, subtracted, multiplied, divided, the vectors are added, subtracted, multiplied, divided with a constant, or any operation is performed on each element in the vector. The continuous operation may be, for example, addition, subtraction, multiplication, division, activation, accumulation, and the like of the vector and the constant.
Each base processing circuit may include a base register and/or a base on-chip cache circuit; each base processing circuit may further include: an inner product operator circuit, a vector operator circuit, an accumulator circuit, or the like, in any combination. The inner product operator circuit, the vector operator circuit, and the accumulator circuit may be integrated circuits, or the inner product operator circuit, the vector operator circuit, and the accumulator circuit may be circuits provided separately.
The chip device may optionally further include one or more branch processing circuits, for example, when the branch processing circuit is provided, the main processing circuit is connected to the branch processing circuit, the branch processing circuit is connected to the basic processing circuit, the inner product operator circuit of the basic processing circuit is configured to perform inner product operation between data blocks, the control circuit of the main processing circuit controls the data receiving circuit or the data transmitting circuit to receive and transmit external data, and controls the data transmitting circuit to distribute the external data to the branch processing circuit, and the branch processing circuit is configured to receive and transmit data from the main processing circuit or the basic processing circuit. The structure shown in fig. 1a is suitable for the computation of complex data, because the number of units connected to the main processing circuit is limited, so that a branch processing circuit needs to be added between the main processing circuit and the basic processing circuit to realize the access of more basic processing circuits, thereby realizing the computation of complex data blocks. The connection structure of the branch processing circuit and the basic processing circuit may be arbitrary and is not limited to the H-type structure of fig. 1 a. Optionally, the main processing circuit to the basic processing circuit is a broadcast or distributed structure, and the basic processing circuit to the main processing circuit is a gather structure. Broadcast, distribution and collection are defined as follows, for a distribution or broadcast configuration, the number of basic processing circuits is greater than that of the main processing circuits, i.e. 1 main processing circuit corresponds to a plurality of basic processing circuits, i.e. a configuration for broadcasting or distribution from the main processing circuit to the plurality of basic processing circuits, whereas a configuration for collection from the plurality of basic processing circuits to the main processing circuit may be provided.
And the basic processing circuit receives data distributed or broadcasted by the main processing circuit, stores the data into an on-chip cache of the basic processing circuit, can perform operation to generate a result, and can send the data to the main processing circuit. Optionally, the basic processing circuit may further process the received data, store the processed data in an on-chip cache, perform an operation using the processed data to generate a result, and optionally send the processed data to other basic processing circuits or a main processing circuit, and the like, which is not limited in this application.
Optionally, each basic processing circuit may include a second mapping circuit, or a second mapping circuit may be configured in a part of the basic processing circuits; the second mapping circuit may be used to process (i.e., compress) the received or transmitted data. The present invention is not limited to the specific form of the second mapping circuit described above. The specific implementation of the second mapping circuit will be described in detail below.
Optionally, the vector operator circuit of the basic processing circuit may perform a vector operation on two vectors (any one or both of the two vectors may be processed vectors), and in practical applications, the inner product operator circuit of the basic processing circuit may perform an inner product operation on the two vectors, and the accumulator circuit may also accumulate results of the inner product operation.
In one alternative, the two vectors may be stored in on-chip caches and/or registers, and the underlying processing circuitry may fetch the two vectors to perform the operation as needed for the actual computation. This operation includes, but is not limited to: inner product operations, multiplication operations, addition operations, or other operations.
In one alternative, the result of the inner product operation may be accumulated onto an on-chip cache and/or register; the alternative scheme has the advantages of reducing the data transmission quantity between the basic processing circuit and the main processing circuit, improving the operation efficiency and reducing the data transmission power consumption.
In one alternative, the result of the inner product operation is not accumulated and is directly transmitted as a result; the technical scheme has the advantages that the internal operation amount of the basic processing circuit is reduced, and the operation efficiency of the basic processing circuit is improved.
In an alternative, each basic processing circuit can execute inner product operations of a plurality of groups of two vectors, and can also respectively accumulate the results of the inner product operations of the plurality of groups;
in one alternative, multiple sets of two vector data may be stored in on-chip caches and/or registers;
in one alternative, the results of multiple sets of inner product operations may be accumulated in an on-chip cache and/or a register, respectively;
in one alternative, the results of the inner product operations in each group can be directly transmitted as results without accumulation;
in one alternative, each base processing circuit may perform an inner product operation of the same vector with multiple vectors (a "one-to-many" inner product, i.e., one vector of two vectors of each group of inner products is shared), and accumulate the inner product results corresponding to each vector separately. According to the technical scheme, the same set of weight can be used for calculating different input data for multiple times, data multiplexing is increased, the data transmission quantity of data in a basic processing circuit is reduced, the calculation efficiency is improved, and the power consumption is reduced.
Specifically, in the data used to compute the inner product, the data sources of the vector shared by the groups and the other vector of each group (i.e., the vector that differs between each group) may differ:
in one alternative, the sets of shared vectors are broadcast or distributed from the main processing circuit or the branch processing circuit when calculating the inner product;
in one alternative, the sets of shared vectors come from an on-chip cache when computing the inner product;
in one alternative, the sets of shared vectors come from registers when computing the inner product;
in one alternative, in calculating the inner product, the other unshared vector of each group is broadcast or distributed from the main processing circuit or the branch processing circuit;
in one alternative, in computing the inner product, the other unshared vector of each group is from the slave on-chip cache;
in one alternative, the other unshared vector of each group comes from a register when calculating the inner product;
in one alternative, when performing inner product operation of multiple groups, each group of shared vectors keeps any number of parts in an on-chip cache and/or a register of the basic processing circuit;
in one alternative, the shared vector may be reserved one for each set of inner products;
in one alternative, the shared vector may be reserved only one copy;
specifically, the results of the multiple sets of inner product operations may be accumulated in an on-chip cache and/or a register, respectively;
specifically, the result of each group of inner product operations can be directly transmitted as a result without accumulation;
in an alternative, the vector or matrix involved in the base processing circuit may be a vector or matrix processed by the second mapping circuit, which will be described later.
Referring to FIG. 1a, the architecture includes a main processing circuit (which can perform vector operations) and multiple basic processing circuits (which can perform inner product operations). The benefits of such a combination are: the device can not only use the basic processing circuit to execute matrix and vector multiplication operation, but also use the main processing circuit to execute other arbitrary vector operation, so that the device can complete more operations more quickly under the configuration of limited hardware circuit, thereby reducing the times of data transmission with the outside of the device, improving the calculation efficiency and reducing the power consumption. In addition, the chip may be provided with a first mapping circuit in the main processing circuit to execute processing of data in the neural network, for example, first input data smaller than or equal to a preset threshold is removed, and at the same time, mark mask data associated with the first input data may be obtained, where the mask data is used to indicate whether an absolute value of the first input data is greater than the preset threshold. For details, reference may be made to the foregoing embodiments, which are not described herein again. The design has the advantages of reducing the data volume transmitted to the basic processing circuit, reducing the calculation amount of the basic processing circuit data, improving the data processing speed and reducing the power consumption.
A second mapping circuit may be provided in the base processing circuit to perform processing of data in the neural network, such as processing the second input data according to mask data associated with the first input data or selecting the first input data and the second input data having absolute values greater than a preset threshold according to mask data associated with the first input data and mask data associated with the second input data to perform corresponding arithmetic operations, and so on. For the specific processing of the data by the first mapping circuit and the second mapping circuit, reference may be made to the following detailed description.
Optionally, the first mapping circuit and the second mapping circuit are both used for processing data, and may be specifically designed into any one or more of the following circuits: a main processing circuit, a branch processing circuit, a base processing circuit, and the like. Therefore, the data amount of calculation can be reduced when the neural network calculation is carried out, and the chip can dynamically allocate which circuit carries out data compression processing according to the operation amount (namely the load amount) of each circuit (mainly a main processing circuit and a basic processing circuit), so that the complex program of data calculation can be reduced, the power consumption can be reduced, and the dynamic allocation data processing can be realized without influencing the calculation efficiency of the chip. The manner of this assignment includes, but is not limited to: load balancing, load minimum distribution, and the like.
Referring to the apparatus shown in FIG. 1b, the apparatus shown in FIG. 1b is a computing apparatus without branch processing circuit, such as the apparatus shown in FIG. 1b, which comprises: a main processing circuit and N basic processing circuits, where the main processing circuit (a specific structure is shown in fig. 1 c) and the N basic processing circuits may be directly or indirectly connected, for example, in an indirect connection manner, an optional scheme may include, as shown in fig. 1a, N/4 branch processing circuits, each branch processing circuit is connected to 4 basic processing circuits, and for the circuits included in the main processing circuit and the N basic processing circuits, reference may be made to the description shown in fig. 1a, which is not described herein again, where it is to be noted that the basic processing circuits may also be disposed in the branch processing circuits, and in addition, the number of the basic processing circuits connected to each branch processing circuit may also be not limited to 4, and a manufacturer may configure the basic processing circuits according to actual needs. The main processing circuit and the N basic processing circuits may be respectively designed with a first mapping circuit and a second mapping circuit, specifically, the main processing circuit may include the first mapping circuit, and the N basic processing circuits or a part thereof includes the second mapping circuit; the main processing circuit may include a first mapping circuit and a second mapping circuit, and the N basic processing circuits or a part thereof may include a first mapping circuit and a second mapping circuit. The main processing circuit may dynamically allocate an operation entity of a data compression processing step according to the neural network computation instruction, specifically, the main processing circuit may determine whether to execute the data compression processing step on the received data according to its own load, specifically, a value of the load may be set to a plurality of intervals, each interval corresponds to an execution subject of the data compression processing step, for example, 3 intervals are taken as an example, a load value of an interval 1 is lower, and N basic processing circuits may execute the data compression processing step. The main processing circuit performs the data compression processing step independently, the section 2 load value is located between the section 1 and the section 3, the main processing circuit can perform the data compression processing step independently, the section 3 load value is high, and the main processing circuit or the N basic processing circuits can perform the data compression processing step together. In this regard, the processing may be performed in an explicit manner, for example, the main processing circuit may be configured with a special indication or instruction, and when the basic processing circuit receives the special indication or instruction, the data compression processing step is determined to be performed, for example, when the basic processing circuit does not receive the special indication or instruction, the data compression processing step is determined not to be performed. As another example, the compression may be performed in an implied manner, for example, when the basic processing circuit receives sparse data (i.e. containing 0, or containing more than a preset number of data smaller than a preset threshold value) and determines that the inner product operation needs to be performed, the sparse data is compressed.
The data compression processing to which the present application relates is specifically performed in the first mapping circuit and the second mapping circuit described above. It should be understood that, since the neural network is an algorithm with high computation amount and high memory access, the more the weight is, the more the computation amount and the memory access amount are increased. Particularly, in the case of a small weight (e.g. 0, or a weight smaller than a set value), the data with a small weight needs to be compressed to increase the calculation rate and reduce the overhead. In practical application, the data compression processing is applied to the sparse neural network, and the effect is most obvious, such as reducing the workload of data calculation, reducing the data overhead, improving the data calculation rate and the like.
The specific embodiment related to the data compression processing is explained by taking input data as an example. The input data includes, but is not limited to, at least one input neuron and/or at least one weight.
In a first embodiment:
after the first mapping circuit receives first input data (specifically, a data block to be calculated, such as a distribution data block or a broadcast data block, sent by the main processing circuit), the first mapping circuit may process the first input data to obtain identification mask data associated with the processed first input data by the first input data, where the mask data is used to indicate whether an absolute value of the first input data is greater than a first threshold, such as 0.5, 0, or the like.
Specifically, when the absolute value of the first input data is greater than a first threshold, the input data is retained; otherwise, deleting the first input data or setting the first input data to be 0. For example, the input matrix data block is
Figure GDA0001637487100000101
The first threshold is 0.05, and the processed matrix data block can be obtained after the processing of the first mapping circuit
Figure GDA0001637487100000102
The identification data block (also called mask matrix) associated with the matrix data block is
Figure GDA0001637487100000103
Further, in order to reduce the data transmission amount, when the main processing circuit distributes data to the basic processing circuit connected to the main processing circuit, the target data (in this case, 1,0.06, and 0.5) in the processed matrix data block and the identification data block associated with the matrix data block may be sent. In a specific implementation, the main processing circuit may distribute the target data in the processed matrix data block to the basic processing circuit according to a set rule, for example, the target data is sequentially sent according to a row order or sequentially sent according to a column order, and the like, which is not limited in this application. Accordingly, theAfter receiving the target data and the identification data block associated with the target data, the basic processing circuit restores the target data and the identification data block to a processed matrix data block according to a set rule (for example, a row sequence). For example, in this example, the base processing circuit may be based on the received data (1, 0.06 and 0.5) and the identification data block
Figure GDA0001637487100000104
The matrix data block corresponding to the data (i.e. the matrix data block processed by the first mapping circuit in the main processing circuit) can be known as
Figure GDA0001637487100000105
In an embodiment of the present invention, the first input data may be a distribution data block and/or a broadcast data block.
Correspondingly, the second mapping circuit can process the second input data by using the identification data associated with the first input data, thereby obtaining the processed second input data; wherein the first input data is different from the second input data. For example, when the first input data is at least one weight, then the second input data may be at least one input neuron; alternatively, when the first input data is at least one input neuron, then the second input data may be at least one weight.
In an embodiment of the present invention, the second input data is different from the first input data, and the second input data may be any one of the following: distribution data blocks, basic data blocks, broadcast data blocks, and partial broadcast data blocks.
For example, when the first input data is a distribution data block, then the second input data is a partial broadcast data block. Assuming the second input data as a matrix data block
Figure GDA0001637487100000111
Using mask matrix in the above example accordingly
Figure GDA0001637487100000112
After treatment, obtainGet the processed partial broadcast data block as
Figure GDA0001637487100000113
Since the dimension of the matrix data block related to the input data is large in practical application, the present application is only illustrative and should not be construed as limiting.
In a second embodiment:
the first mapping circuit may be configured to process first input data and second input data to obtain processed first input data and first identification mask data associated with the first input data, processed second input data and second identification mask data associated with the second input data. Wherein, the first mask data or the second mask data is used to indicate whether the absolute value of the first or the second input data is greater than a second threshold, and the second threshold is set by the user side or the device side in a self-defined way, such as 0.05, 0, etc.
The processed first input data or the second input data may be processed input data or unprocessed input data. For example, the first input data is a distribution data block, such as a matrix data block as in the above example
Figure GDA0001637487100000114
The processed distribution data block can be obtained after being processed by the first mapping circuit, and the processed distribution data block can be an original matrix data block
Figure GDA0001637487100000115
Or the compressed matrix data block
Figure GDA0001637487100000116
It should be understood that, in order to reduce the transmission of data amount and the data processing efficiency in the basic processing circuit, it is preferable that the processed input data (such as the processed basic data block or the partial broadcast data block) should be the compressed data. Preferably, the data sent by the main processing circuit to the basic processing circuit may be specifically the processed input numberAccording to the target data, the target data may be specifically data with an absolute value greater than a preset threshold, or may be non-0 data, and so on.
Correspondingly, in the basic processing circuit, the second mapping circuit may obtain connection identification data according to the first identification data associated with the first input data and the second identification data associated with the second input data; the connection identification data is used to indicate data whose absolute value is greater than a third threshold in the first input data and the second input data, where the third threshold is set by a user or a device in a user-defined manner, such as 0.05, 0, or the like. Further, the second mapping circuit may process the received first input data and the second input data according to the connection identification data, respectively, so as to obtain processed first input data and processed second input data.
For example, the first input data is a matrix data block
Figure GDA0001637487100000117
The second input data block is likewise a matrix data block
Figure GDA0001637487100000118
After being processed by the first mapping circuit, the first identification data block related to the first input data can be obtained
Figure GDA0001637487100000121
And a processed first input data block
Figure GDA0001637487100000122
Correspondingly obtaining a second identification data block associated with the second input data
Figure GDA0001637487100000123
The second input data block after processing is
Figure GDA0001637487100000124
Accordingly, to increase the data transmission rate, only the purpose of the first processed input data block in the main processing circuit can be achievedThe marking data 1,0.06 and 0.5 and a first identification data block associated with the first input data block are sent to a basic processing circuit; and simultaneously, sending the target data 1,1.1,0.6,0.3 and 0.5 in the processed second input data block and a second identification data block associated with the second input data block to the basic processing circuit.
Correspondingly, after the basic processing circuit receives the data, the basic processing circuit can perform element-by-element and operation on the first identification data block and the second identification data block through the second mapping circuit to obtain a connection identification data block
Figure GDA0001637487100000125
Correspondingly, the second mapping circuit respectively processes the processed first input data block and the processed second input data block by using the connection identification data block, so as to obtain the processed first input data block as
Figure GDA0001637487100000126
The second input data block after processing is
Figure GDA0001637487100000127
The basic processing circuit can determine a first data block (i.e. the first data block processed by the first mapping circuit) corresponding to the target data according to the first identification data block and the received target data in the first data block; correspondingly, according to the second identification data block and the received target data in the second data block, determining a second data block (namely, the second data block processed by the first mapping circuit) corresponding to the target data; then, after the second mapping circuit learns the connection identification data block, the connection identification data block is used to perform element-by-element and operation with the determined first data block and the determined second data block respectively, so as to obtain the first data block processed by the second mapping circuit and the processed second data block.
In the third embodiment:
the first mapping circuit is not arranged in the main processing circuit, but the main processing circuit can send third input data and third identification data which is pre-stored and associated with the third input data to a basic processing circuit connected with the main processing circuit. A second mapping circuit is disposed in the base processing circuit. A specific example of the data compression process involved in the second mapping circuit is set forth below.
It should be understood that the third input data includes, but is not limited to, a basic data block, a partial broadcast data block, a broadcast data block, and the like. Similarly, in the neural network processor, the third input data may also be at least one weight, and/or at least one input nerve, which is not limited in this application.
In the second mapping circuit, the second mapping circuit may process the third input data according to third identification data associated with the received third input data, so as to obtain processed third input data, so as to subsequently perform a correlation operation, such as an inner product operation, on the processed third input data.
For example, the third input data received by the second mapping circuit is a matrix data block
Figure GDA0001637487100000128
A third identification data block (also referred to as a mask matrix data block) associated with the third input data, which is prestored correspondingly
Figure GDA0001637487100000129
Further, the second mapping circuit processes the third input data block according to the third identification data block to obtain a processed third input data block, which is specifically the processed third input data block
Figure GDA0001637487100000131
In addition, the input neurons and the output neurons mentioned in the embodiments of the present invention do not refer to neurons in an input layer and neurons in an output layer of the entire neural network, but for any two adjacent layers of neurons in the neural network, neurons in a lower layer of the network feedforward operation are input neurons, and neurons in an upper layer of the network feedforward operation are output neurons. Taking the convolutional neural network as an example, assuming that a convolutional neural network has L layers, where K is 1,2,3 … L-1, for the K-th layer and the K + 1-th layer, the K-th layer is referred to as an input layer, the neuron in the layer is the input neuron, the K + 1-th layer is referred to as an input layer, the neuron in the layer is the output neuron, that is, except for the top layer, each layer may be used as an input layer, and the next layer is a corresponding output layer.
In the fourth embodiment:
the main processing circuit is not provided with a mapping circuit, and the basic processing circuit is provided with a first mapping circuit and a second mapping circuit. For data processing of the first mapping circuit and the second mapping circuit, reference may be made to the foregoing first embodiment to the third embodiment, which are not described herein again.
Alternatively, a fifth embodiment is also present. In a fifth embodiment, a mapping circuit is not disposed in the basic processing circuit, and both the first mapping circuit and the second mapping circuit are disposed in the main processing circuit, and for data processing of the first mapping circuit and the second mapping circuit, reference may be specifically made to the foregoing first to third embodiments, and details are not repeated here. That is, the main processing circuit completes the compression processing of the data, and sends the processed input data to the basic processing circuit, so that the basic processing circuit performs the corresponding operation by using the processed input data (specifically, the processed neurons and the processed weights).
The following sets forth a specific structural schematic diagram of the present application relating to a mapping circuit. Two possible mapping circuits are shown in fig. 5a and 5 b. Wherein the mapping circuit as shown in fig. 5a comprises a comparator and a selector. The present application is not limited with respect to the number of comparators and selectors. Fig. 5a shows a comparator and two selectors, wherein the comparator is used to determine whether the input data meets the preset condition. The preset condition may be set by a user or a device, for example, an absolute value of the input data is greater than or equal to a preset threshold. If the preset condition is met, the comparator can determine that the input data is allowed to be output, and the input data corresponds to the associated identification data and is 1; otherwise, it may be determined not to output the input data, or to default the input data to 0. Accordingly, the input data is 0 corresponding to the associated identification data at this time. That is, after passing through the comparator, the identification data associated with the input data can be known.
Further, after the comparator determines the preset condition for the input data, the obtained identification data may be input to the selector, so that the selector uses the identification data to determine whether to output the corresponding input data, that is, to obtain the processed input data.
As shown in fig. 5a, taking the input data as a matrix data block as an example, each data in the matrix data block may be determined by a comparator according to a preset condition, so that an identification data block (mask matrix) associated with the matrix data block may be obtained. Further, the matrix data block can be screened by the identification data block in the first selector, data with an absolute value greater than or equal to a preset threshold (that is, meeting a preset condition) in the matrix data block is retained, and the rest of data is deleted to output the processed matrix data block. Optionally, the second selector may further process other input data (e.g., a second matrix data block) by using the identification data block, for example, perform an element-by-element and operation to reserve data whose absolute value is greater than or equal to a preset threshold in the second matrix data block, so as to output the processed second matrix data block.
It should be understood that, corresponding to the first and second embodiments described above, the specific structure of the first mapping circuit may include at least one comparator and at least one selector, such as the comparator and the first selector in fig. 5a in the above example; the specific result of the second mapping circuit may comprise one or more selectors, such as the second selector of fig. 5a in the example above.
Fig. 5b shows a schematic diagram of another mapping circuit. As shown in fig. 5b, the mapping circuit includes selectors, and the number of the selectors is not limited, and may be one or more. Specifically, the selector is configured to select the input data according to identification data associated with the input data, so as to output data, of which an absolute value is greater than or equal to a preset threshold, from the input data, and delete/not output the remaining data, thereby obtaining processed input data.
Taking the input data as a matrix data block as an example, the matrix data block and an identification data block associated with the matrix data block are input to the mapping circuit, the selector can select the matrix data block according to the identification data block, output data of which the absolute value is greater than or equal to 0, and output no other data, thereby outputting the processed matrix data block.
It will be appreciated that the structure shown in fig. 5b may be applied to the second mapping circuit in the third embodiment described above, i.e. the specific result of the second mapping circuit in the third embodiment described above may comprise at least one selector. Similarly, the first mapping circuit and the second mapping circuit designed in the main processing circuit and the basic processing circuit may be cross-combined or split according to the functional components shown in fig. 5a and 5b, and the present application is not limited thereto.
In practical applications, the forward operation may perform matrix multiplication, convolution, activation, transformation, and other operations according to different input data, and all the operations may be implemented by using the apparatus shown in fig. 1 a.
For example, the first mapping circuit of the main processing circuit can compress the data and then transmit the data to the basic processing circuit, which has the advantages of reducing the data transmission amount, reducing the total bit number for transmission, having higher efficiency of the basic processing circuit for performing data operation, and having lower power consumption.
The main processing circuit transmits data to be calculated to all or a part of basic processing circuits; taking the matrix multiplied by the vector calculation as an example, the control circuit of the main processing circuit may split each column of matrix data into one basic data, for example, an m × n matrix, and may split the matrix data into n vectors of m rows, and the control circuit of the main processing circuit distributes the split n vectors of m rows to a plurality of basic processing circuits. For vectors, the control circuitry of the main processing circuitry may broadcast the vector as a whole to each of the base processing circuitry. If the value of m is relatively large, the control circuit may first split the m × n matrix into x × n vectors, taking x as an example, 2, specifically, 2n vectors, each vector including m/2 rows, that is, each vector in n m rows is equally split into 2 vectors, taking the first row as an example, if the first vector of the n m rows is 1000 rows, then equally split into 2 vectors may be that the first 500 rows are combined into the first vector, the last 500 rows are combined into the second vector, and the control circuit broadcasts the 2 vectors to the plurality of basic processing circuits through 2 broadcasts.
The data transmission mode can be broadcasting or distribution, or any other possible transmission mode;
after the basic processing circuit receives the data, the data are processed through the second mapping circuit, and then operation is performed to obtain an operation result;
the basic processing circuit transmits the operation result back to the main processing circuit;
the operation result may be an intermediate operation result or a final operation result.
The operation of multiplying the tensor, which is the same as the data block described above, can be performed by using the apparatus shown in fig. 1a, and the tensor can be any one or a combination of a matrix, a vector, a three-dimensional data block, a four-bit data block and a high-dimensional data block; the following shows specific implementations of the matrix-by-vector and matrix-by-matrix operations as fig. 2a and 2c, respectively.
The operation of multiplying the vector by the matrix is completed by using the device shown in FIG. 1 a; (the matrix multiplication vector can be that each row in the matrix is respectively subjected to inner product operation with the vector, and the results are arranged into a vector according to the sequence of the corresponding rows.)
The following describes the operation of multiplying a matrix S of size M rows and L columns by a vector P of length L, as shown in fig. 2b below, (each row in the matrix S is the same length as the vector P, and the data in them are in one-to-one correspondence by position) the neural network computing device has K basic processing circuits:
referring to fig. 2a, fig. 2a provides a method for implementing matrix multiplication vector, which may specifically include:
step S201, a control circuit of a main processing circuit distributes each row of data in a matrix S to one of K basic processing circuits, and the basic processing circuits store the received distributed data in an on-chip cache and/or a register of the basic processing circuits;
in an optional scheme, the data of the matrix S is processed data. Specifically, the main processing circuit enables the first mapping circuit to process the matrix S, so as to obtain the processed matrix S and a first identifier (mask) matrix associated with the matrix S. Or the first mapping circuit of the main processing circuit processes the matrix S according to a first mask matrix associated with a pre-stored matrix S to obtain a processed matrix S. Further, each row of data in the processed matrix S and the identification data of the row of data corresponding to the first mask matrix are sent to one or more of the K basic processing circuits through the control circuit. When the main processing circuit sends data to the basic processing circuit, the processed data with the absolute value greater than the preset threshold value in the matrix S or the non-0 data can be sent to the basic processing circuit to reduce the data transmission amount. For example, the set of rows in the matrix S processed by the ith basic processing circuit is Ai, and there are Mi rows in total; correspondingly, an identification matrix Bi corresponding to Ai is distributed at the same time, wherein the Bi is a part of the first mask matrix and has a row number which is larger than or equal to Mi in total.
In an alternative, if the number M < ═ K of rows of the matrix S, the control circuit of the main processing circuit distributes one row of the matrix S to the K basic processing circuits, respectively; optionally, the identification data of the row in the first identification matrix corresponding to the row is also sent at the same time;
in an alternative, the control circuit of the main processing circuit distributes data of one or more rows of the S matrix to each of the elementary processing circuits, respectively, if the number of rows M > K of the matrix S. Optionally, the identification data of the row or rows corresponding to the rows in the first identification matrix is also sent at the same time;
the set of rows in S distributed to the ith basic processing circuit is Ai, and there are Mi rows in total, as shown in fig. 2d representing the calculations to be performed on the ith basic processing circuit.
In one alternative, in each base processing circuit, e.g., the ith base processing circuit, the received dispatch data, e.g., the matrix Ai, may be stored in a register and/or on-chip cache of the ith base processing circuit; the method has the advantages of reducing the data transmission quantity of the subsequent distribution data, improving the calculation efficiency and reducing the power consumption.
Step S202, the control circuit of the main processing circuit transmits each part in the vector P to K basic processing circuits in a broadcasting mode;
in one alternative, the data (parts) of the vector P may be processed data. Specifically, the main processing circuit enables the first mapping circuit to process the vector P, so as to obtain the processed vector P and a second identifier (mask) matrix associated with the vector P. Or the first mapping circuit of the main processing circuit processes the vector P according to a second mask matrix associated with the pre-stored vector P to obtain the processed vector P. Further, the data (i.e. each part) in the processed vector P and the identification data corresponding to the data and corresponding to the data in the second mask matrix are sent to one or more of the K basic processing circuits through the control circuit. When the main processing circuit sends data to the basic processing circuit, data with an absolute value larger than a preset threshold value or non-0 data in the processed vector P can be sent to the basic processing circuit to reduce the data transmission amount.
In an alternative, the control circuit of the main processing circuit may broadcast each part of the vector P only once to the register or on-chip buffer of each basic processing circuit, and the ith basic processing circuit may fully multiplex the data of the vector P obtained this time, and perform the inner product operation corresponding to each row in the matrix Ai. The method has the advantages of reducing the data transmission quantity of repeated transmission of the vector P from the main processing circuit to the basic processing circuit, improving the execution efficiency and reducing the transmission power consumption.
In an alternative, the control circuit of the main processing circuit may broadcast each part of the vector P to the register or on-chip cache of each basic processing circuit for multiple times, and the ith basic processing circuit does not multiplex the data of the vector P obtained each time, and completes the inner product operation corresponding to each row in the matrix Ai for multiple times; the method has the advantages of reducing the data transmission quantity of the vector P of single transmission in the basic processing circuit, reducing the capacity of the cache and/or the register of the basic processing circuit, improving the execution efficiency, reducing the transmission power consumption and reducing the cost.
In an alternative, the control circuit of the main processing circuit may broadcast each part of the vector P to the register or on-chip cache of each basic processing circuit for multiple times, and the ith basic processing circuit performs partial multiplexing on the data of the vector P obtained each time, and completes the inner product operation corresponding to each row in the matrix Ai; the method has the advantages of reducing the data transmission quantity from the main processing circuit to the basic processing circuit, reducing the data transmission quantity in the basic processing circuit, improving the execution efficiency and reducing the transmission power consumption.
Step S203, calculating the inner product of the matrix S and the data of the vector P by an inner product arithmetic circuit of K basic processing circuits, for example, the ith basic processing circuit, calculating the inner product of the data of the matrix Ai and the data of the vector P;
in a specific embodiment, the basic processing circuit receives the processed data in the matrix S and the identification data associated with the data in the first mask matrix; and simultaneously receiving the data in the processed vector P. Correspondingly, the basic processing circuit enables the second mapping circuit to process the received data of the vector P according to the received identification data in the first mask matrix, and the processed data of the vector P is obtained. Further, the basic processing circuit enables the inner product arithmetic circuit to execute the inner product operation on the received data in the processed matrix S and the data of the processed vector P, and the result of the inner product operation is obtained. For example, the ith basic processing circuit receives a matrix Ai, an identification matrix Bi associated with the Ai, and a vector P; at the moment, a second mapping circuit can be started to process the vector P by utilizing Bi to obtain a processed vector P; and starting an inner product arithmetic circuit to carry out inner product operation on the matrix Ai and the processed vector P.
In a specific embodiment, the basic processing circuit receives the data in the processed vector P and the identification data associated with the data in the second mask matrix; and simultaneously, receiving the processed data in the matrix S. Correspondingly, the basic processing circuit enables the second mapping circuit to process the received data of the matrix S according to the identification data in the received second mask matrix to obtain the processed data of the matrix S. Further, the basic processing circuit enables the inner product arithmetic circuit to execute inner product operation on the received data of the processed vector P and the data in the processed matrix S, and the result of the inner product operation is obtained. For example, the ith basic processing circuit receives the matrix Ai, the processed vector P and a second identification matrix associated with the vector P; at the moment, a second mapping circuit can be started to process Ai by using a second identification matrix to obtain a processed matrix Ai; and then starting an inner product arithmetic circuit to carry out inner product operation on the processed matrix Ai and the processed vector P.
In a specific embodiment, the basic processing circuit receives the processed data in the matrix S and the identification data associated with the data in the first mask matrix; and meanwhile, receiving the data in the processed vector P and the identification data which is associated in the second mask matrix and corresponds to the data. Correspondingly, the basic processing circuit starts a second mapping circuit to obtain a relation identification matrix according to the received identification data in the first mask matrix and the received identification data in the second mask matrix; and then, respectively processing the received data in the matrix S and the received data in the vector P by using the identification data in the relation identification matrix to obtain the processed data of the matrix S and the processed data of the vector P. Further, an inner product arithmetic circuit is started to execute inner product operation on the data in the processed matrix S and the data of the processed vector P, and the result of the inner product operation is obtained. For example, the ith base processing circuit receives a matrix Ai, an identification matrix Bi associated with the Ai, a vector P, and a second identification matrix associated with the vector P; at this time, the second mapping circuit can be started to obtain a relation identification matrix by using the Bi and the second identification matrix, and then the matrix Ai and the vector P are processed simultaneously or respectively by using the relation identification matrix to obtain a processed matrix Ai and a processed vector P. Next, the inner product operator circuit is enabled to perform an inner product operation on the processed matrix Ai and the processed vector P.
And S204, accumulating the results of the inner product operation by the accumulator circuits of the K basic processing circuits to obtain accumulated results, and transmitting the accumulated results back to the main processing circuit in a fixed-point type mode.
In an alternative, the partial sums (i.e., a portion of the accumulated result, e.g., F1G 1+ F2G 2+ F3G 3+ F4G 4+ F5G 5, then the partial sums may be the values of F1G 1+ F2G 2+ F3G 3) resulting from each inner product operation performed by the basic processing circuit may be transmitted back to the main processing circuit for accumulation; the method has the advantages of reducing the internal operation amount of the basic processing circuit and improving the operation efficiency of the basic processing circuit.
In an alternative, the partial sum obtained by the inner product operation executed by the basic processing circuit each time can be stored in a register and/or an on-chip cache of the basic processing circuit, and the partial sum is transmitted back to the main processing circuit after the accumulation is finished; the method has the advantages of reducing the data transmission quantity between the basic processing circuit and the main processing circuit, improving the operation efficiency and reducing the data transmission power consumption.
In an alternative, the partial sum obtained by the inner product operation executed by the basic processing circuit each time is stored in a register and/or an on-chip cache of the basic processing circuit for accumulation in partial cases, and is transmitted to the main processing circuit for accumulation in partial cases, and is transmitted back to the main processing circuit after the accumulation is finished; the method has the advantages of reducing the data transmission quantity between the basic processing circuit and the main processing circuit, improving the operation efficiency, reducing the data transmission power consumption, reducing the operation quantity in the basic processing circuit and improving the operation efficiency of the basic processing circuit.
Referring to FIG. 2c, the matrix multiplication operation is performed using the apparatus shown in FIG. 1 a;
the following describes the operation of calculating the multiplication of a matrix S of size M rows and L columns and a matrix P of size L rows and N columns, (each row in the matrix S being the same length as each column of the matrix P, as shown in fig. 2 e) the neural network computing device possesses K basic processing circuits:
step S201b, the control circuit of the main processing circuit distributes each line of data in the matrix S to one of the K basic processing circuits, and the basic processing circuits store the received data in the on-chip cache and/or the register;
in an optional scheme, the data of the matrix S is processed data. Specifically, the main processing circuit enables the first mapping circuit to process the matrix S, so as to obtain the processed matrix S and a first identifier (mask) matrix associated with the matrix S. Or the first mapping circuit of the main processing circuit processes the matrix S according to a first mask matrix associated with a pre-stored matrix S to obtain a processed matrix S. Further, each row of data in the processed matrix S and the identification data of the row of data corresponding to the first mask matrix are sent to one or more of the K basic processing circuits through the control circuit. When the main processing circuit sends data to the basic processing circuit, the processed data with the absolute value greater than the preset threshold value in the matrix S or the non-0 data can be sent to the basic processing circuit to reduce the data transmission amount.
In one alternative, if the number of rows M < ═ K of S, the control circuit of the main processing circuit distributes one row of the S matrix to the M basic processing circuits, respectively; optionally, the identification data of the row in the first identification matrix corresponding to the row is also sent at the same time;
in an alternative, the control circuit of the main processing circuit distributes data of one or more rows in the S matrix to each of the elementary processing circuits, respectively, if the number of rows M > K of S. Optionally, the identification data of the row or rows corresponding to the rows in the first identification matrix is also sent at the same time;
in S, Mi rows are distributed to the ith basic processing circuit, and the set of Mi rows is called Ai, as shown in fig. 2f, which represents the calculation to be performed on the ith basic processing circuit.
In one alternative, in each base processing circuit, for example, in the ith base processing circuit:
the received matrix Ai distributed by the main processing circuit stores the matrix Ai in an ith basic processing circuit register and/or an on-chip cache; the method has the advantages of reducing the subsequent data transmission quantity, improving the calculation efficiency and reducing the power consumption.
Step S202b, the control circuit of the main processing circuit transmits each part in the matrix P to each basic processing circuit in a broadcast mode;
in one alternative, the data (portions) of the matrix P may be processed data. Specifically, the main processing circuit enables the first mapping circuit to process the matrix P, so as to obtain the processed matrix P and a second identifier (mask) matrix associated with the matrix P. Or the first mapping circuit of the main processing circuit processes the matrix P according to a second mask matrix associated with the pre-stored matrix P to obtain the processed matrix P. Further, the processed data (i.e. each part) in the matrix P and the identification data corresponding to the data and associated in the second mask matrix are sent to one or more of the K basic processing circuits through the control circuit. When the main processing circuit sends data to the basic processing circuit, the processed data with the absolute value greater than the preset threshold value in the matrix P or the non-0 data can be sent to the basic processing circuit to reduce the data transmission amount.
In an alternative scheme, each part in the matrix P may be broadcasted to the register or on-chip cache of each basic processing circuit only once, and the ith basic processing circuit multiplexes the data of the matrix P obtained this time sufficiently to complete the inner product operation corresponding to each row in the matrix Ai; the multiplexing in this embodiment may be specifically that the basic processing circuit is repeatedly used in the calculation, for example, the multiplexing of the data of the matrix P may be that the data of the matrix P is used multiple times.
In an alternative, the control circuit of the main processing circuit may broadcast each part of the matrix P to the register or on-chip cache of each basic processing circuit for multiple times, and the ith basic processing circuit does not multiplex the data of the matrix P obtained each time, and completes the inner product operation corresponding to each row in the matrix Ai for multiple times;
in an alternative, the control circuit of the main processing circuit may broadcast each part of the matrix P to the register or on-chip cache of each basic processing circuit for multiple times, and the ith basic processing circuit performs partial multiplexing on the data of the matrix P obtained each time, and completes the inner product operation corresponding to each row in the matrix Ai;
in one alternative, each basic processing circuit, for example the ith basic processing circuit, calculates the inner product of the data of matrix Ai and the data of matrix P;
in step S203b, the accumulator circuit of each basic processing circuit accumulates the result of the inner product operation and transmits it back to the main processing circuit.
Optionally, before step S203b, the inner product operator of the basic processing circuit needs to calculate the inner product of the data of the matrix S and the matrix P, and the following embodiments exist in detail.
In a specific embodiment, the basic processing circuit receives the processed data in the matrix S and the identification data associated with the data in the first mask matrix; and meanwhile, receiving the processed data in the matrix P. Correspondingly, the basic processing circuit enables the second mapping circuit to process the received data of the matrix P according to the identification data in the received first mask matrix, and the processed data of the matrix P is obtained. Further, the basic processing circuit enables the inner product arithmetic circuit to execute the inner product operation on the received data in the processed matrix S and the data of the processed matrix P, and the result of the inner product operation is obtained.
In a specific embodiment, the basic processing circuit receives the processed data in the matrix P and the identification data associated with the data in the second mask matrix; and simultaneously, receiving the processed data in the matrix S. Correspondingly, the basic processing circuit enables the second mapping circuit to process the received data of the matrix S according to the identification data in the received second mask matrix to obtain the processed data of the matrix S. Further, the basic processing circuit enables the inner product arithmetic circuit to execute inner product operation on the received data of the processed matrix P and the data in the processed matrix S, and the result of the inner product operation is obtained.
In a specific embodiment, the basic processing circuit receives the processed data in the matrix S and the identification data associated with the data in the first mask matrix; and meanwhile, receiving the processed data in the matrix P and the identification data which is associated in the second mask matrix and corresponds to the data. Correspondingly, the basic processing circuit starts a second mapping circuit to obtain a relation identification matrix according to the received identification data in the first mask matrix and the received identification data in the second mask matrix; and then, the identification data in the relation identification matrix is used for respectively processing the received data in the matrix S and the received data in the matrix P to obtain the processed data of the matrix S and the processed data of the matrix P. Further, the inner product arithmetic circuit is started to execute inner product operation on the data in the processed matrix S and the data of the processed matrix P, and the result of the inner product operation is obtained. For example, the ith basic processing circuit receives a matrix Ai, an identification matrix Bi associated with the Ai, a matrix P and a second identification matrix associated with the matrix P; at this time, the second mapping circuit can be started to obtain a relation identification matrix by using the Bi and the second identification matrix, and then the matrix Ai and the matrix P are processed simultaneously or respectively by using the relation identification matrix to obtain a processed matrix Ai and a processed matrix P. Next, the inner product operator circuit is enabled to perform inner product operation on the processed matrix Ai and the processed matrix P.
In one alternative, the base processing circuit may transmit the partial sums obtained by performing the inner product operation each time back to the main processing circuit for accumulation;
in an alternative, the partial sum obtained by the inner product operation executed by the basic processing circuit each time can be stored in a register and/or an on-chip cache of the basic processing circuit, and the partial sum is transmitted back to the main processing circuit after the accumulation is finished;
in an alternative, the partial sum obtained by the inner product operation performed by the basic processing circuit each time may be stored in a register and/or an on-chip buffer of the basic processing circuit in some cases for accumulation, and transmitted to the main processing circuit for accumulation in some cases, and transmitted back to the main processing circuit after the accumulation is finished.
Referring to FIG. 3a, a full join operation is performed using the apparatus shown in FIG. 1 a:
if the input data of the fully-connected layer is a vector (i.e. the input of the neural network is the case of a single sample), taking the weight matrix of the fully-connected layer as a matrix S and the input vector as a vector P, and performing the operation of multiplying the matrix by the vector as shown in FIG. 2a according to the first using method of the device;
if the input data of the fully connected layer is a matrix (i.e. the input of the neural network is the case of multiple samples as the batch), then the weight matrix of the fully connected layer is used as the matrix S and the input vector is used as the matrix P, or the weight matrix of the fully connected layer is used as the matrix P and the input vector is used as the matrix S, and the execution operation of the matrix multiplication matrix shown in fig. 2d is performed according to the device;
referring to FIG. 3b, the convolution operation is performed using the apparatus shown in FIG. 1 a:
for a convolution layer, recording the number of convolution kernels as M;
step S301, the control circuit of the main processing circuit distributes the weight of each convolution kernel in the convolution layer weight to one of K basic processing circuits and stores the weight in an on-chip cache and/or a register of the basic processing circuits;
in one alternative, the weights of the convolution kernel are processed weight data. Specifically, the main processing circuit enables the first mapping circuit to process the weight of the convolution kernel, so as to obtain a processed convolution kernel and a first identifier (mask) matrix associated with the convolution kernel. Or, the first mapping circuit of the main processing circuit processes the convolution kernel according to a pre-stored first mask matrix associated with the convolution kernel to obtain a processed convolution kernel (i.e., a weight of the processed convolution kernel). Further, the weight value in the processed convolution kernel and the identification data which is correspondingly associated with the weight value in the first mask matrix are sent to one or more of the K basic processing circuits through the control circuit. When the main processing circuit sends data to the basic processing circuit, the processed data with the absolute value larger than the preset threshold value in the convolution kernel or the non-0 data can be sent to the basic processing circuit to reduce the data transmission quantity. For example, the set of Mi processed convolution kernels distributed to the ith base processing circuit is Ai; accordingly, an identification matrix Bi corresponding to Ai is also distributed at the same time.
In an alternative scheme, if the number M < ═ K of convolution kernels, the control circuit of the main processing circuit distributes the weight of one convolution kernel to each of the M basic processing circuits; optionally, the identification matrix (i.e. the identification data in the identification matrix) associated with the convolution kernel is also sent at the same time.
In one alternative, if the number M > K of the convolution kernels, the control circuit of the main processing circuit distributes the weight of one or more convolution kernels to each basic processing circuit respectively; optionally, the identification matrix (i.e. the identification data in the identification matrix) associated with each of the one or more convolution kernels is also sent at the same time.
There are a total of Mi convolution kernels distributed to the ith base processing circuit, and the set of these convolution kernel weights is called Ai.
In one alternative, in each base processing circuit, for example, in the ith base processing circuit:
storing the received convolution kernel weight Ai distributed by the main processing circuit in a register and/or an on-chip cache of the main processing circuit;
step S302, the control circuit of the main processing circuit transmits each part in the input data P to each basic processing circuit in a broadcasting mode;
in one alternative, the input data P is processed data. Specifically, the main processing circuit enables the first mapping circuit to process the input data P, so as to obtain the processed input data P and a second identifier (mask) matrix (i.e., identifier data) associated with the input data P. Or the first mapping circuit of the main processing circuit processes the input data P according to a second mask matrix associated with the pre-stored input data P to obtain the processed input data P. Further, the data (i.e. each part) in the processed input data P and the identification data corresponding to the data and corresponding to the data in the second mask matrix are sent to one or more of the K basic processing circuits through the control circuit. When the main processing circuit sends data to the basic processing circuit, data with an absolute value larger than a preset threshold value or non-0 data in the processed input data P can be sent to the basic processing circuit to reduce the data transmission amount.
In an alternative, the control circuit of the main processing circuit may broadcast each part of the input data P to the register or on-chip cache of each basic processing circuit only once, and the ith basic processing circuit fully multiplexes the data of the input data P obtained this time, and completes the inner product operation corresponding to each convolution kernel in Ai;
in an alternative, the control circuit of the main processing circuit may broadcast each part of the input data P to the register or on-chip cache of each basic processing circuit for multiple times, and the ith basic processing circuit does not multiplex the data of the input data P obtained each time, and completes the inner product operation corresponding to each convolution kernel in Ai in multiple times;
in an alternative, the control circuit of the main processing circuit may broadcast each part of the input data P to the register or on-chip cache of each basic processing circuit for multiple times, and the ith basic processing circuit performs partial multiplexing on the data of the input data P obtained each time, and completes the inner product operation corresponding to each convolution kernel in Ai;
step S303, each basic processing circuit calculates a data inner product of the convolution kernel and the input data P, for example, the ith basic processing circuit calculates an inner product of each convolution kernel of Ai and the data of the input data P;
in a specific embodiment, the basic processing circuit receives the processed data in the convolution kernel and the identification data associated with the data in the first mask matrix; while also receiving processed input data P. Correspondingly, the basic processing circuit enables the second mapping circuit to process the received input data P according to the identification data in the received first mask matrix, and the processed input data P is obtained. Further, the basic processing circuit enables an inner product arithmetic circuit to execute inner product operation on the received data in the processed convolution kernel and the processed input data P, and a result of the inner product operation is obtained. For example, the ith basic processing circuit receives the convolution kernel weights Ai, the identification matrix Bi associated with the Ai and the input data P; at the moment, a second mapping circuit can be started to process the input data P by utilizing Bi to obtain processed input data P; and then starting an inner product arithmetic circuit to carry out inner product operation on the convolution kernel weight Ai and the processed input data P.
In a specific embodiment, the basic processing circuit receives data in the processed input data P and identification data associated with the data in the second mask matrix; and receiving the processed data in the convolution kernel. Correspondingly, the basic processing circuit enables the second mapping circuit to process the received data of the convolution kernel according to the identification data in the received second mask matrix, and the processed data of the convolution kernel is obtained. Further, the basic processing circuit enables the inner product arithmetic circuit to execute inner product operation on the received processed input data P and the data in the processed convolution kernel, and a result of the inner product operation is obtained. For example, the ith basic processing circuit receives the convolution kernel weights Ai, the processed input data P and a second identification matrix associated with the input data P; at this time, a second mapping circuit can be started to process Ai by using a second identification matrix to obtain a processed convolution kernel weight Ai; and then starting an inner product arithmetic circuit to carry out inner product operation on the processed convolution kernel weight Ai and the processed input data P.
In a specific embodiment, the basic processing circuit receives the processed data in the convolution kernel and the identification data associated with the data in the first mask matrix; and meanwhile, receiving the data in the processed input data P and the identification data which is associated with the data in the second mask matrix. Correspondingly, the basic processing circuit starts a second mapping circuit to obtain a relation identification matrix according to the received identification data in the first mask matrix and the received identification data in the second mask matrix; and then, respectively processing the data in the received convolution kernel and the data in the input data P by using the identification data in the relation identification matrix to obtain the processed data of the convolution kernel and the processed input data P. Further, an inner product arithmetic circuit is started to execute inner product operation on the data in the processed convolution kernel and the processed input data P, and an inner product operation result is obtained. For example, the ith basic processing circuit receives the convolution kernel weight Ai, the identification matrix Bi associated with the Ai, the input data P and the second identification matrix associated with the input data P; at this time, the second mapping circuit can be started to obtain a relationship identifier matrix by using the Bi and the second identifier matrix, and then the relationship identifier matrix is used to process the convolution kernel weight Ai and the input data P simultaneously or respectively to obtain the processed convolution kernel weight Ai and the processed input data P. And then, starting an inner product arithmetic circuit to carry out inner product operation on the processed convolution kernel weight Ai and the processed input data P.
Step S304, the accumulator circuit of each basic processing circuit accumulates the result of the inner product operation and transmits it back to the main processing circuit:
in one alternative, the base processing circuitry may be configured to transmit the partial sum resulting from each inner product operation back to the main processing circuitry for accumulation;
in an alternative, the basic processing circuit may also store the partial sum obtained by the inner product operation performed each time in a register and/or an on-chip cache of the basic processing circuit, and transmit the partial sum back to the main processing circuit after the accumulation is finished;
in an alternative, the basic processing circuit may also store the partial sum obtained by the inner product operation performed each time in a register and/or an on-chip cache of the basic processing circuit for accumulation in some cases, transmit the partial sum to the main processing circuit for accumulation in some cases, and transmit the partial sum back to the main processing circuit after the accumulation is finished;
the method for updating the weight using the device shown in FIG. 1 a:
the weight updating function in the neural network training process is realized by utilizing a vector arithmetic unit circuit of the main processing circuit, and specifically, the weight updating refers to a method for updating the weight by using the gradient of the weight.
In an alternative scheme, a vector operator circuit of the main processing circuit is used for performing addition and subtraction operation on the two vectors of the weight and the weight gradient to obtain an operation result, and the operation result is the updated weight.
In an alternative scheme, a vector operator circuit of the main processing circuit multiplies or divides the weight and the gradient of the weight by a number to obtain a middle weight and a gradient value of the middle weight, and the vector operator circuit performs addition and subtraction operation on the middle weight and the gradient value of the middle weight to obtain an operation result, wherein the operation result is the updated weight.
In an alternative scheme, a group of momentum can be calculated by using the gradient of the weight, and then the updated weight is obtained by performing addition and subtraction calculation by using the momentum and the weight;
the invention also provides a chip comprising a computing device, the computing device comprising:
the data involved in the main processing circuit may be compressed data, and in an alternative embodiment, the compressed data includes at least one input neuron or at least one weight value, and each neuron in the at least one neuron is greater than a first threshold value or each weight value in the at least one weight value is greater than a second threshold value. The first threshold and the second threshold are set by a user side in a self-defined way, and can be the same or different.
In one alternative, the main processing circuit includes a first mapping circuit;
in one alternative, the main processing circuit includes an arithmetic unit such as a vector arithmetic unit or the like that performs data compression processing;
specifically, the system comprises a data input interface for receiving input data;
in one alternative, the source of the received data may be: part or all of a basic processing circuit outside the neural network operation circuit device or the neural network operation circuit device;
in one alternative, there may be a plurality of the data input interfaces; specifically, a data output interface that outputs data may be included;
in one alternative, the destination of the output data may be: a part or all of a basic processing circuit outside the neural network operation device or the neural network operation circuit device;
in one alternative, the number of the data output interfaces may be plural;
in one alternative, the main processing circuitry comprises on-chip caches and/or registers;
in an alternative, the main processing circuit comprises an arithmetic unit which can execute data arithmetic;
in one alternative, an arithmetic operation unit is included in the main processing circuit;
in an alternative, the main processing circuit comprises a vector operation unit which can simultaneously perform operation on a group of data; in particular, the arithmetic operations and/or vector operations may be any type of operations, including but not limited to: two numbers are added, subtracted, multiplied, divided, one number is added, subtracted, multiplied, divided with a constant, an exponential operation, a power operation, a logarithmic operation are performed on one number, and various nonlinear operations, a comparison operation, a logical operation, etc. are performed on two numbers. Two vectors are added, subtracted, multiplied, divided, each element in one vector is added, subtracted, multiplied, divided with a constant, exponential, logarithmic, and various nonlinear operations are performed on each element in one vector, comparison operations, logical operations, and the like are performed on each two corresponding elements in one vector.
In one alternative, the main processing circuit includes a data rearranging unit for transferring data to the base processing circuit in a certain order or rearranging data in place in a certain order;
in one alternative, the order in which the data is arranged includes: carrying out dimension sequence transformation on a multi-dimensional data block; the order of the data arrangement may further include: a block of data is partitioned for transmission to different underlying processing circuits.
The computing device also includes a plurality of basic processing circuits: each basic processing circuit is used for calculating the inner product of two vectors, and the calculation method is that the basic processing circuit receives two groups of numbers, correspondingly multiplies elements in the two groups of numbers, and accumulates the multiplication results; the result of the inner product is transmitted, where it is possible to transmit it to other basic processing circuits, depending on the position of the basic processing circuit, or directly to the main processing circuit.
The data involved in the base processing circuit may be compressed data, and in an alternative embodiment, the compressed data includes at least one input neuron or at least one weight value, each of the at least one neuron is greater than a first threshold value or each of the at least one weight value is greater than a second threshold value. The first threshold and the second threshold are set by a user side in a self-defined way, and can be the same or different.
In one alternative, the base processing circuit includes a second mapping circuit;
in one alternative, the base processing circuit includes a vector operation unit that performs data compression processing;
specifically, the memory unit comprises an on-chip cache and/or a register;
in particular, one or more data input interfaces to receive data;
in one alternative, two data input interfaces are included, one or more data being respectively available from the two data input interfaces at a time;
in one alternative, the base processing circuit may store the input data received from the data input interface in a register and/or an on-chip cache;
the data input interface may receive data from: other basic processing circuitry and/or main processing circuitry.
A main processing circuit of the neural network arithmetic circuit device;
other basic processing circuits of the neural network operation circuit device (the neural network operation circuit device has a plurality of basic processing circuits);
specifically, one or more data output interfaces for transmitting output data are included;
in one alternative, one or more data may be transmitted out of the data output interface;
specifically, the data transmitted through the data output interface may be: one or any combination of data received from the data input interface, data stored in an on-chip cache and/or register, a multiplier operation result, an accumulator operation result or an inner product operator operation result.
In one alternative, the system comprises three data output interfaces, wherein two of the three data output interfaces correspond to two data input interfaces respectively, a layer above each layer is used for outputting data received from the data input interfaces, and the third data output interface is used for outputting an operation result;
specifically, the destination of the data output interface to transmit data may be: the above data sources and the data destinations herein determine the connection relationships of the underlying processing circuitry in the device.
A main processing circuit of the neural network arithmetic circuit device;
a further basic processing circuit of the neural network arithmetic circuit device, the neural network arithmetic circuit device having a plurality of basic processing circuits;
specifically, an arithmetic operation circuit is included: the arithmetic operation circuit may specifically be: one or more multiplier circuits, one or more accumulator circuits, one or more circuits that perform two sets of inner product operations, or any combination thereof.
In an alternative, a multiplication operation of two numbers can be executed, and the result can be stored in an on-chip cache and/or a register or can be directly added into the register and/or the on-chip cache;
in an alternative, an inner product operation of two groups of data can be executed, and the result can be stored in an on-chip cache and/or a register or directly added into the register and/or the on-chip cache;
in one alternative, an accumulation operation of data may be performed, accumulating the data into an on-chip cache and or register;
specifically, the data accumulated by the accumulator circuit may be: one or any combination of data received from the data input interface, data stored in an on-chip cache and/or register, a multiplier operation result, an accumulator operation result, and an inner product operator operation result.
It should be noted that the "data input interface" and the "data output interface" used in the above description of the basic processing circuit refer to the data input and output interface of each basic processing circuit, not the data input and output interface of the whole device.
In one embodiment, the present invention discloses a neural network computing device, which includes functional units for executing all or part of the embodiments provided in the method embodiments described above.
In one embodiment, the present invention discloses a chip for performing all or part of the embodiments provided in the method embodiments described above.
In one embodiment, the invention discloses an electronic device comprising functional units for performing all or part of the embodiments of the method as described above.
Electronic devices include data processing devices, robots, computers, printers, scanners, tablets, smart terminals, cell phones, tachographs, navigators, sensors, cameras, servers, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, vehicles, home appliances, and/or medical devices.
The vehicle comprises an airplane, a ship and/or a vehicle; the household appliances comprise a television, an air conditioner, a microwave oven, a refrigerator, an electric cooker, a humidifier, a washing machine, an electric lamp, a gas stove and a range hood; the medical equipment comprises a nuclear magnetic resonance apparatus, a B-ultrasonic apparatus and/or an electrocardiograph.
The above-described embodiments, objects, technical solutions and advantages of the present disclosure are further described in detail, it should be understood that the above-described embodiments are only illustrative of the embodiments of the present disclosure, and are not intended to limit the present disclosure, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.

Claims (12)

1. An integrated circuit chip apparatus, comprising: a main processing circuit and a plurality of basic processing circuits; the main processing circuit comprises a first mapping circuit, at least one circuit of the plurality of basic processing circuits comprises a second mapping circuit, and the first mapping circuit and the second mapping circuit are used for executing compression processing of each data in neural network operation;
the main processing circuit is used for acquiring an input data block, a convolution kernel data block and a convolution instruction, dividing the input data block into broadcast data blocks according to the convolution instruction, and dividing the convolution kernel data block into distribution data blocks; determining to start a first mapping circuit to process a first data block according to the operation control of the convolution instruction, so as to obtain the processed first data block and an identification data block associated with the first data block; the first data block comprises the distribution data block and/or the broadcast data block; sending the processed first data block and the identification data block associated with the first data block to at least one basic processing circuit in basic processing circuits connected with the main processing circuit according to the convolution instruction;
the plurality of basic processing circuits are used for determining whether to start a second mapping circuit to process a second data block according to the identification data block of the second data block according to the operation control of the convolution instruction, executing operation in a neural network in a parallel mode according to the processed second data block to obtain an operation result, and transmitting the operation result to the main processing circuit through the basic processing circuit connected with the main processing circuit; the second data block is determined by the basic processing circuit to receive the data block sent by the main processing circuit, and the second data block is associated with the processed first data block;
and the main processing circuit is used for processing the operation result to obtain an instruction result of the convolution instruction.
2. The integrated circuit chip apparatus of claim 1, wherein when the first data block includes a distribution data block and a broadcast data block,
the main processing circuit is specifically configured to start the first mapping circuit to process the distribution data block and the broadcast data block to obtain a processed distribution data block, an identification data block associated with the distribution data block, a processed broadcast data block, and an identification data block associated with the broadcast data block; splitting the processed distribution data block and the identification data block associated with the distribution data block to obtain a plurality of basic data blocks and identification data blocks associated with the basic data blocks; distributing the plurality of basic data blocks and the identification data blocks respectively associated with the plurality of basic data blocks to a basic processing circuit connected with the basic processing circuit, and broadcasting the broadcast data blocks and the identification data blocks associated with the broadcast data blocks to the basic processing circuit connected with the basic processing circuit;
the basic processing circuit is used for starting the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the basic data block and the identification data block associated with the broadcast data block; and processing the basic data block and the broadcast data block according to the connection identification data block, performing convolution operation on the processed basic data block and the processed broadcast data block to obtain an operation result, and sending the operation result to the main processing circuit.
3. The integrated circuit chip apparatus of claim 1, wherein when the first data block comprises a distribution data block,
the main processing circuit is specifically configured to start the first mapping circuit to process the distribution data block to obtain a processed distribution data block and an identification data block associated with the distribution data block, or start the first mapping circuit to process the distribution data block according to a pre-stored identification data block associated with the distribution data block to obtain a processed distribution data block; splitting the processed distribution data block and the identification data block associated with the distribution data block to obtain a plurality of basic data blocks and identification data blocks associated with the basic data blocks; distributing the plurality of elementary data blocks and the identification data blocks associated with each of the plurality of elementary data blocks to a base processing circuit connected thereto; broadcasting the broadcast data block to a base processing circuit connected thereto;
the basic processing circuit is used for starting the second mapping circuit to process the broadcast data block according to the identification data block associated with the basic data block, performing convolution operation on the processed broadcast data block and the basic data block to obtain an operation result, and sending the operation result to the main processing circuit.
4. The integrated circuit chip apparatus of claim 1, wherein when the first data block comprises a broadcast data block,
the main processing circuit is specifically configured to start the first mapping circuit to process the broadcast data block to obtain a processed broadcast data block and an identifier data block associated with the broadcast data block, or start the first mapping circuit to process the broadcast data block according to a pre-stored identifier data block associated with the broadcast data block to obtain a processed broadcast data block; splitting the distribution data block to obtain a plurality of basic data blocks; distributing the plurality of base data to a base processing circuit connected thereto; broadcasting the processed broadcast data block and the identification data block related to the broadcast data block to a basic processing circuit connected with the broadcast data block;
the basic processing circuit is used for starting the second mapping circuit to process the basic data block according to the identification data block associated with the broadcast data block to obtain a processed basic data block; and performing convolution operation on the processed basic data block and the processed broadcast data block to obtain an operation result, and sending the operation result to the main processing circuit.
5. The integrated circuit chip apparatus of any of claims 2-4,
the basic processing circuit is specifically configured to perform convolution operation on the basic data block and the broadcast data block to obtain a convolution result, accumulate the convolution result to obtain an operation result, and send the operation result to the main processing circuit;
and the main processing circuit is used for obtaining accumulation results after accumulating the operation results and arranging the accumulation results to obtain the instruction results.
6. The integrated circuit chip apparatus of any of claims 2-4,
the main processing circuit is specifically configured to broadcast the broadcast data block or the processed broadcast data block to the plurality of basic processing circuits at a time; alternatively, the first and second electrodes may be,
the main processing circuit is specifically configured to divide the broadcast data block or the processed broadcast data block into a plurality of partial broadcast data blocks, and broadcast the plurality of partial broadcast data blocks to the plurality of basic processing circuits by multiple times.
7. The integrated circuit chip apparatus of any of claims 2-4,
the main processing circuit is specifically configured to split the processed broadcast data block and the identification data block associated with the broadcast data block to obtain a plurality of partial broadcast data blocks and identification data blocks associated with the plurality of partial broadcast data blocks; broadcasting the plurality of partial broadcast data blocks and the identification data blocks respectively associated with the plurality of partial broadcast data blocks to the basic processing circuit by one or more times; the plurality of partial broadcast data blocks are combined to form the processed broadcast data block;
the basic processing circuit is specifically configured to start the second mapping circuit to obtain a connection identification data block according to the identification data block associated with the partial broadcast data block and the identification data block associated with the basic data block; processing the partial broadcast data block and the basic data block according to the connection identification data block to obtain a processed broadcast data block and a processed basic data block; performing convolution operation on the processed broadcast data block and the processed basic data block;
or, the basic processing circuit is specifically configured to start the second mapping circuit to process the basic data block according to the identification data block associated with the partial broadcast data block to obtain a processed basic data block, and perform convolution operation on the processed basic data and the partial broadcast data block.
8. The integrated circuit chip apparatus of claim 7,
the basic processing circuit is specifically configured to perform a convolution operation on the partial broadcast data block and the basic data block to obtain a convolution result, accumulate the convolution result to obtain a partial operation result, and send the partial operation result to the main processing circuit; alternatively, the first and second electrodes may be,
the basic processing circuit is specifically configured to multiplex n times the partial broadcast data block to perform inner product operation between the partial broadcast data block and n basic data blocks to obtain n partial processing results, accumulate the n partial processing results respectively to obtain n partial operation results, and send the n partial operation results to the main processing circuit, where n is an integer greater than or equal to 2.
9. The integrated circuit chip apparatus of claim 1, further comprising: branch processing circuitry disposed between the main processing circuitry and the at least one base processing circuitry;
the branch processing circuit is used for forwarding data between the main processing circuit and at least one basic processing circuit.
10. The integrated circuit chip apparatus of claim 1,
the input data block is: one or any combination of a matrix, a three-dimensional data block, a four-dimensional data block and an n-dimensional data block;
the convolution kernel data block is: one or any combination of a matrix, a three-dimensional data block, a four-dimensional data block, and an n-dimensional data block.
11. A chip incorporating a device according to any one of claims 1 to 10.
12. A method of operation of a neural network, the method being implemented within an integrated circuit chip device, the integrated circuit chip device comprising: the integrated circuit chip apparatus of any of claims 1-10, the integrated circuit chip apparatus to perform convolution operations of a neural network.
CN201810164737.5A 2018-02-27 2018-02-27 Integrated circuit chip device and related product Active CN110197271B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201810164737.5A CN110197271B (en) 2018-02-27 2018-02-27 Integrated circuit chip device and related product
PCT/CN2019/075979 WO2019165940A1 (en) 2018-02-27 2019-02-23 Integrated circuit chip apparatus, board card and related product

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810164737.5A CN110197271B (en) 2018-02-27 2018-02-27 Integrated circuit chip device and related product

Publications (2)

Publication Number Publication Date
CN110197271A CN110197271A (en) 2019-09-03
CN110197271B true CN110197271B (en) 2020-10-27

Family

ID=67750999

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810164737.5A Active CN110197271B (en) 2018-02-27 2018-02-27 Integrated circuit chip device and related product

Country Status (1)

Country Link
CN (1) CN110197271B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111737977B (en) * 2020-06-24 2022-05-17 平安科技（深圳）有限公司 Data dictionary generation method, data query method, device, equipment and medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104463324A (en) * 2014-11-21 2015-03-25 长沙马沙电子科技有限公司 Convolution neural network parallel processing method based on large-scale high-performance cluster
CN106447034A (en) * 2016-10-27 2017-02-22 中国科学院计算技术研究所 Neutral network processor based on data compression, design method and chip
CN106650923A (en) * 2015-10-08 2017-05-10 上海兆芯集成电路有限公司 Neural network elements with neural memory and neural processing unit array and sequencer
CN107301456A (en) * 2017-05-26 2017-10-27 中国人民解放军国防科学技术大学 Deep neural network multinuclear based on vector processor speeds up to method
CN107316078A (en) * 2016-04-27 2017-11-03 北京中科寒武纪科技有限公司 Apparatus and method for performing artificial neural network self study computing
CN107609641A (en) * 2017-08-30 2018-01-19 清华大学 Sparse neural network framework and its implementation

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104463324A (en) * 2014-11-21 2015-03-25 长沙马沙电子科技有限公司 Convolution neural network parallel processing method based on large-scale high-performance cluster
CN106650923A (en) * 2015-10-08 2017-05-10 上海兆芯集成电路有限公司 Neural network elements with neural memory and neural processing unit array and sequencer
CN107316078A (en) * 2016-04-27 2017-11-03 北京中科寒武纪科技有限公司 Apparatus and method for performing artificial neural network self study computing
CN106447034A (en) * 2016-10-27 2017-02-22 中国科学院计算技术研究所 Neutral network processor based on data compression, design method and chip
CN107301456A (en) * 2017-05-26 2017-10-27 中国人民解放军国防科学技术大学 Deep neural network multinuclear based on vector processor speeds up to method
CN107609641A (en) * 2017-08-30 2018-01-19 清华大学 Sparse neural network framework and its implementation

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"DaDianNao:A Machine-Learning Supercomputer";Yunji Chen et al.;《IEEE》;20150119;第609-622页 *

Also Published As

Publication number Publication date
CN110197271A (en) 2019-09-03

Similar Documents

Publication Publication Date Title
CN110197270B (en) Integrated circuit chip device and related product
CN109993301B (en) Neural network training device and related product
CN109993291B (en) Integrated circuit chip device and related product
CN111160542B (en) Integrated circuit chip device and related products
CN109993292B (en) Integrated circuit chip device and related product
CN110197271B (en) Integrated circuit chip device and related product
CN110197272B (en) Integrated circuit chip device and related product
CN111160541A (en) Integrated circuit chip device and related product
CN110197268B (en) Integrated circuit chip device and related product
US20200125937A1 (en) Integrated circuit chip device and related product
CN110197266B (en) Integrated circuit chip device and related product
CN110197274B (en) Integrated circuit chip device and related product
CN110197275B (en) Integrated circuit chip device and related product
US11704544B2 (en) Integrated circuit chip device and related product
CN110197265B (en) Integrated circuit chip device and related product
CN110197263B (en) Integrated circuit chip device and related product
CN111091189A (en) Integrated circuit chip device and related product
CN110197273B (en) Integrated circuit chip device and related product
CN110197269B (en) Integrated circuit chip device and related product
CN110197267B (en) Neural network processor board card and related product
US11734548B2 (en) Integrated circuit chip device and related product
WO2019165940A1 (en) Integrated circuit chip apparatus, board card and related product

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant