CN106203622B - Neural network computing device - Google Patents
Neural network computing device Download PDFInfo
- Publication number
- CN106203622B CN106203622B CN201610556380.6A CN201610556380A CN106203622B CN 106203622 B CN106203622 B CN 106203622B CN 201610556380 A CN201610556380 A CN 201610556380A CN 106203622 B CN106203622 B CN 106203622B
- Authority
- CN
- China
- Prior art keywords
- neural network
- vector
- layer
- arithmetic processor
- correction value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
Abstract
The invention discloses a kind of neural network computing devices, belong to electronic chip field.Neural network computing device includes: the first arithmetic processor and the second arithmetic processor: the first arithmetic processor, for in the training operation of neural network, carry out the first operation or the second operation, first operation includes dot product operation, and the second operation includes dot product operation and nonlinear transformation operation；Second arithmetic processor, for being modified according at least one weight vector of at least one correction value to neural network, obtain at least one amendment weight vector, wherein, each correction value is corresponding with a vector element in the weight vector of neural network, at least one correction value is obtained according to the operation result of the first arithmetic processor.Neural network computing device provided by the invention can promote the efficiency of neural network computing.
Description
Technical field
The present invention relates to electronic chip field, in particular to a kind of neural network computing device.
Background technique
Neural network is a kind of common sorting algorithm, more in speech recognition, image recognition, natural language processing etc.
A field has been achieved for biggish success, but also during rapid development.Neural network can substantially be regarded as
A kind of digraph, as shown in Figure 1 comprising multiple nodes and the directed arc for connecting these nodes.Wherein, more in neural network
A node is divided into multiple layers, and neural network basecoat node is known as input layer, and the most upper node layer of neural network is known as exporting
Layer, the layer between input layer and output layer are known as hidden layer.Each node layer in neural network can pass through directed arc and phase
The node of adjacent bed connects, but does not have directed arc connection between same node layer, wherein each directed arc corresponds to a power
Value.The output that the output valve of a certain node can be made of the output valve of lower layer of all nodes of a certain node in neural network
The weight vector of all directed arcs that vector sum next all nodes of layer are directed toward a certain node corresponding weight composition pass through to
Amount dot product is calculated.In the case that the corresponding weight of each directed arc has determined in neural network, neural network can be into
Row sort operation, and the corresponding weight of each directed arc, neural network then need to be trained operation in neural network in order to determine.
In the related technology, neural network computing device includes arithmetic processor, which can execute nerve net
The training operation of network, while the sort operation of neural network can also be performed in the arithmetic processor.
In the implementation of the present invention, the inventor finds that the existing technology has at least the following problems:
The arithmetic processor of existing neural network computing device carry out neural network training operation during,
The processor active task of a variety of different modes is needed to be implemented, this causes its hardware circuit complex, so that its operation efficiency is difficult
To get a promotion.
Summary of the invention
In order to solve problems in the prior art, the embodiment of the invention provides a kind of neural network computing devices.The mind
Include the first arithmetic processor and the second arithmetic processor through network operations device:
First arithmetic processor, for carrying out the first operation or the second operation in the training operation of neural network,
First operation includes dot product operation, and second operation includes dot product operation and nonlinear transformation operation；
Second arithmetic processor, for according at least one correction value at least one power of the neural network to
Amount is modified, and obtains at least one amendment weight vector, wherein in the weight vector of each correction value and the neural network
One vector element is corresponding, at least one described correction value is obtained according to the operation result of first arithmetic processor.
In the first possible embodiment, first arithmetic processor, be also used to according to first operation or
The operation result of second operation calculates at least one described correction value；
Accordingly, second arithmetic processor, for according to first arithmetic processor be calculated described in extremely
A few correction value is modified at least one described weight vector, obtains at least one described amendment weight vector.
In second of possible embodiment, second arithmetic processor, be also used to according to first operation or
The operation result that second operation obtains calculates at least one described correction value, and according at least one described correction value to institute
It states at least one weight vector to be modified, obtains at least one described amendment weight vector.
In the third possible embodiment, second arithmetic processor is also used to according to target correction value to institute
It states at least one weight vector to be modified, obtains at least one described amendment weight vector, the target correction value is greater than first
The correction value of preset threshold.
In the 4th kind of possible embodiment, first arithmetic processor, for by the power of the neural network to
It measures and each layer of the neural network is obtained by the sample vector progress forward operation of the input layer input of the neural network
Positive output vector, the forward operation includes first operation or second operation；
First arithmetic processor is also used to by the weight vector of the neural network and by the defeated of the neural network
The error vector of layer input carries out reversed operation and obtains each layer of the neural network of reversed output vector out, the error to
Amount is the difference vector of the positive output vector target output vector corresponding with the sample vector of the output layer, described reversed
Operation includes first operation.
In the 5th kind of possible embodiment, first arithmetic processor is also used to every according to the neural network
One layer of positive output vector and each layer of the neural network of reversed output vector calculate at least one described correction value.
In the 6th kind of possible embodiment, first arithmetic processor is also used to push away in the forward direction of neural network
It manages in calculating process, carries out first operation or second operation.
In the 7th kind of possible embodiment, second arithmetic processor is also used in the reversed output vector
In at least one reversed output vector be greater than the second preset threshold when, according at least one described correction value to described at least one
A weight vector is modified, and obtains at least one amendment weight vector.
In the 8th kind of possible embodiment, first arithmetic processor, for the training operation in neural network
In, the positive output vector of i-th layer of the neural network is obtained, and by described i-th layer of positive output vector and by described i-th
At least one corresponding weight vector of directed arc that layer is directed toward i+1 layer carries out dot product operation respectively and obtains primary vector, root
Obtain the positive output vector of the i+1 layer according to the primary vector, when i is 1 described in i-th layer of positive output vector be
The sample vector, i are the positive integer more than or equal to 1.
In the 9th kind of possible embodiment, first arithmetic processor, for by the primary vector to
Secondary element carries out nonlinear transformation operation and obtains the positive output vector of the i+1 layer.
In the tenth kind of possible embodiment, first arithmetic processor, for the training operation in neural network
In, the reversed output vector of the neural network jth layer is obtained, and by the reversed output vector of the jth layer and by the jth
At least one corresponding weight vector of directed arc that layer is directed toward -1 layer of jth carries out dot product operation respectively, obtains -1 layer of the jth
Reversed output vector, when the jth layer be the neural network the last layer when, the reversed output vector of the jth layer
For the error vector, j is the positive integer greater than 1.
Technical solution provided in an embodiment of the present invention has the benefit that
By centering the first arithmetic processor and the second arithmetic processor, and the first operation in neural network computing dress
Processor is only used for carrying out dot product operation in the training operation of neural network, alternatively, the first arithmetic processor is only used for
Dot product operation and nonlinear transformation operation are carried out in the training operation of neural network, so that the first arithmetic processor can be with
The processor active task of one mode is only handled, can only include special circuit designed by the operation for the mode therefore, from
And achieved the purpose that simplify the first arithmetic processor hardware circuit so that the operation efficiency of the first arithmetic processor obtained compared with
Big promotion.Meanwhile in neural metwork training operation different from dot product operation other modes operation, such as amendment power to
The operation of amount can be executed by the second arithmetic processor, since operation most in the training operation of neural network is all
Dot product operation is all dot product operation and nonlinear transformation operation, therefore, the first arithmetic processor operation efficiency
Promotion can greatly promote neural metwork training operation efficiency promotion.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment
Attached drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the invention, for
For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other
Attached drawing.
Fig. 1 is the schematic diagram of neural network provided in an embodiment of the present invention.
Fig. 2 is a kind of structural schematic diagram of neural network computing device provided in an embodiment of the present invention.
Fig. 3 is a kind of partial schematic diagram of neural network provided in an embodiment of the present invention.
Fig. 4 is a kind of partial schematic diagram of neural network provided in an embodiment of the present invention.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with attached drawing to embodiment party of the present invention
Formula is described in further detail.
Referring to FIG. 2, it illustrates the structural representations of neural network computing device 200 provided by one embodiment of the present invention
Figure.As shown in Fig. 2, the apparatus may include: the first arithmetic processor 210 and the second arithmetic processor 220.
First arithmetic processor 210, in the training operation of neural network, carrying out the first operation or the second fortune
It calculates, which includes dot product operation, which includes dot product operation and nonlinear transformation operation.
Second arithmetic processor 220, for according at least one correction value at least one power of the neural network to
Amount is modified, and obtains at least one amendment weight vector, wherein one in the weight vector of each correction value and the neural network
A vector element is corresponding, which obtains according to the operation result of first arithmetic processor.
In conclusion neural network computing device provided in this embodiment, by being centered in neural network computing dress
First arithmetic processor and the second arithmetic processor, and the first arithmetic processor be only used in the training operation of neural network into
Row vector dot-product operation, alternatively, the first arithmetic processor is only used for carrying out dot product fortune in the training operation of neural network
It calculates and nonlinear transformation operation, allowing the first arithmetic processor only to handle the processor active task of one mode therefore can be with
Only comprising special circuit designed by the operation for the mode, simplify the first arithmetic processor hardware circuit to reach
Purpose, so that the operation efficiency of the first arithmetic processor has obtained biggish promotion.Meanwhile it is different in neural metwork training operation
In the operation of other modes of dot product operation, the operation for such as correcting weight vector can be executed by the second arithmetic processor,
Due to operation most in the training operation of neural network be all dot product operation or all be dot product operation and
Nonlinear transformation operation, therefore, the promotion of the first arithmetic processor operation efficiency can greatly promote neural metwork training operation
Efficiency promotion.
With continued reference to FIG. 2, it illustrates the structures of neural network computing device 200 provided by one embodiment of the present invention
Schematic diagram.In order to be conducive to reader understand the present invention, below the present embodiment will in conjunction with a upper embodiment to neural network computing device
200 are specifically described.
The purpose of neural metwork training operation is to obtain one group of weight vector of neural network, so that the neural network after training
One can be obtained close to ideal output to any input based on this group of weight vector.In order to reach this purpose, nerve net
The training operation of network needs repeatedly to correct the weight vector of neural network, until revised weight vector can satisfy it is above-mentioned
It is required that.And above-mentioned amendment needs are carried out based on the state of the neural network in using, and that is to say that needs are neural network based
The operation result of forward operation carries out, wherein the forward operation of neural network refers to input of the input vector from neural network
Layer input and operation always to neural network output layer, to obtain the calculating process of the output vector of each layer, the operation
Journey generally comprises dot product operation and nonlinear transformation operation.
According to above-mentioned analysis it is found that the training operation of neural network generally may include two parts, first part be to
Measure dot-product operation and nonlinear transformation operation, the i.e. operation of the state of neural network in reflection use, second part be power to
Amount amendment operation, i.e., the state based on the neural network in using carry out the modified operation of weight vector.Wherein, in neural network
In training operation, most operations is all the operation of first part, i.e. dot product operation and nonlinear transformation operation, because
This, the operation efficiency for promoting dot product operation and nonlinear transformation operation will be significant promote neural metwork training operation
Operation efficiency.
And inventors have found that dot product operation and nonlinear transformation operation are a kind of special operation mode, processor
The operation mode can be executed based on specific hardware circuit, then above-mentioned first part's operation can be individually by the first calculation process
Device 210 executes, i.e. the first arithmetic processor 210, in the training operation of neural network, carrying out the first operation or the second fortune
It calculates, which includes dot product operation, which includes dot product operation and nonlinear transformation operation.Due to
First arithmetic processor 210 only executes the operation of above-mentioned special operation mode, so that the hardware circuit of the first arithmetic processor 210
It greatly simplifies, so that the operation efficiency of the first arithmetic processor 210 greatly promotes.Certainly, in practical applications, the first fortune
Dot product operation can only be executed by calculating processor 210, and nonlinear transformation operation can by neural network computing device its
His module executes, alternatively, the first arithmetic processor 210 can not only execute dot product operation but also execute nonlinear transformation fortune
It calculates, the present invention is not specifically limited in this embodiment.
Above-mentioned second part operation can then be executed by the second arithmetic processor 220, i.e. the second arithmetic processor 220, be used
Be modified according at least one weight vector of at least one correction value to neural network, obtain at least one amendment power to
Amount, wherein each correction value is corresponding with a vector element in the weight vector of neural network, at least one correction value according to
The operation result of first arithmetic processor obtains.For example, a certain weight vector is (a, b, c) in neural network, wherein vector element
The corresponding correction value of a is Δ a, and the corresponding correction value of vector element b is Δ b, and the corresponding correction value of vector element c is Δ c, then passes through
Crossing revised vector element is respectively a+ Δ a, b+ Δ b, c+ Δ c, and amendment weight vector is (a+ Δ a, b+ Δ b, c+ Δ c).Its
In, " correction value is obtained according to the operation result of the first arithmetic processor 210 " refers to 210 basis of the first arithmetic processor
The correction value is calculated in the result of first operation or the second operation, alternatively, the second arithmetic processor 220 according to the first operation at
The correction value is calculated in the result for managing the first operation or the second operation of device 210, in this regard, the present invention will hereafter carry out specifically
Bright, details are not described herein.
It should be noted that the calculation of above-mentioned correction value is different for different neural network BP training algorithms,
However, being which kind of neural network BP training algorithm, the state computation that above-mentioned correction value is all based on the neural network in use is obtained
, it that is to say what the first operation for being all based on the progress of the first arithmetic processor 210 or the second operation obtained.The present invention is hereafter
The calculating of above-mentioned correction value will be illustrated by taking relatively conventional backpropagation training algorithm as an example, details are not described herein.
In practical applications, the training algorithm of neural network is varied, but also in continuous development, difference training
The difference of algorithm is that weight vector correction strategy is different, and therefore, the second arithmetic processor 220 can be according to different nerve nets
Network training algorithm is configured, so as to meet the demand of different neural network BP training algorithms, so that neural network computing
Device is more flexible.
Further, the second part of above-mentioned neural metwork training operation, i.e. weight vector correct operation, can specifically include
Following two steps: one, according to the result and correction strategy of dot product operation and nonlinear transformation operation, at least one is calculated
Correction value；Two, at least one vector element at least one weight vector of neural network is enabled to be added with corresponding correction value
At least one modification vector element is obtained, and at least one amendment weight vector is obtained based at least one modification vector element.
Wherein, the operation of above-mentioned second step is executed by the second arithmetic processor 220, and the operation of the above-mentioned first step can be by
First arithmetic processor 210 executes, and can also be executed by the second arithmetic processor 220, the present invention is not specifically limited in this embodiment.
When the operation of the above-mentioned first step is executed by the first arithmetic processor 210, which is also used
In calculating at least one correction value according to the operation result of above-mentioned first operation or the second operation.Accordingly, at the second operation
Manage device 220, at least one correction value for being calculated according to the first arithmetic processor 210 to it is above-mentioned at least one weigh to
Amount is modified, and obtains at least one amendment weight vector.
When the operation of the above-mentioned first step is executed by the second arithmetic processor 220, which is also used
Calculate at least one correction value in the operation result obtained according to above-mentioned first operation or the second operation, and according to this at least one
A correction value is modified at least one above-mentioned weight vector, obtains at least one amendment weight vector.
Further, in practical applications, the weight vector quantity in neural network may be very huge, and, each power
The quantity of vector element may also be very huge in vector, if be all modified to each vector element in each weight vector,
Then the operation strength of the second arithmetic processor 220 will be very big, and spent operation time will also be grown very much.Simultaneously as the
Two arithmetic processors 220 need to correct weight vector and are transmitted in the first arithmetic processor 210 after obtaining amendment weight vector, so that
First arithmetic processor 210 can be carried out dot product operation based on amendment weight vector or carry out dot product operation and non-linear
Transform operation, and based on operation result reflection use in neural network state so that the second arithmetic processor 220 after
It is continuous that the amendment weight vector is modified, therefore, if the second arithmetic processor 220 in each weight vector of neural network it is each to
Secondary element is all modified, then the quantity for correcting weight vector will be very huge, so that the second arithmetic processor 220 and first
Very big communication bandwidth is needed between arithmetic processor 210.
As a result, in order to mitigate the operation strength of the second arithmetic processor 220, reduce the operation of the second arithmetic processor 220
Time, also for the communication bandwidth needed for reducing between the second arithmetic processor 220 and the first arithmetic processor 210, the second fortune
Can only the part vector element in neural network weight vector be modified by calculating processor 220.Specifically, at second operation
Reason device 220 can be modified at least one weight vector according to target correction value, obtain at least one amendment weight vector,
In, target correction value is the correction value greater than the first preset threshold.It that is to say, second arithmetic processor 220 is available should
At least one correction value, and obtain the correction value for being greater than the first preset threshold at least one correction value, i.e., above-mentioned target is repaired
Positive value, then, the second arithmetic processor 220 can use the target correction value to the corresponding vector element of target correction value into
Row amendment.In brief, the second arithmetic processor 220 can only correct that portion that corresponding correction value is greater than the first preset threshold
Divide vector element.
Further, in practical applications, in addition to the training operation of neural network needs to carry out dot product operation and non-
Outside linear transformation operation, the forward reasoning operation of neural network is also required to carry out dot product operation and nonlinear transformation operation.
Wherein, the forward reasoning operation of neural network refers to operation when being over trained neural network normal use, i.e., will
Input vector is inputted by the input layer of neural network, and obtained after training based on above-mentioned input vector and neural network
Multiple weight vectors obtain the calculating process of the output vector exported by neural network output layer after carrying out operation.
It is since the forward reasoning operation of neural network is also required to carry out dot product operation and nonlinear transformation operation, then refreshing
Forward reasoning operation through network can be multiplexed above-mentioned first arithmetic processor 210, that is to say first arithmetic processor 210 also
Can be executed during the forward reasoning operation of neural network dot product operation (above-mentioned first operation) or execute to
Measure dot-product operation and nonlinear transformation operation (above-mentioned second operation).
First arithmetic processor 210 can answer in the forward reasoning operation of neural network and the training operation of neural network
With, to save the hardware spending of neural network computing, while but also neural network computing device provided by the invention both
The training operation for being able to carry out neural network is able to carry out the forward reasoning operation of neural network again, to enhance neural network
The applicability of arithmetic unit.
Further, in practical applications, a kind of relatively conventional neural network BP training algorithm is that backpropagation training is calculated
Method, backpropagation training algorithm include propagated forward operation, backpropagation operation and weight vector amendment operation, so-called propagated forward
Operation refers to the input layer input sample vector by neural network, and the direction from input layer to output layer successively calculates mind
Output vector through network hidden layer and output layer, so-called backpropagation operation refer to inputting by the output layer of neural network
Error vector, and the direction from output layer to input layer successively calculates the output vector of neural network hidden layer, so-called weight vector
Amendment operation refers to being modified according to weight vector of the result of propagated forward operation and back-propagating operation to neural network,
Wherein, error vector refers to the difference in forward-propagating operation between the vector value and ideal value of output layer output.It is reversed to pass
The purpose of broadcasting the backpropagation operation in training algorithm is to obtain each layer of neural network of evaluated error, i.e., by one layer below of mistake
Difference derives the error of one layer of front, the layer-by-layer anti-pass of error so can be obtained to each layer of neural network of evaluated error, then
Weight vector is modified according to the result of each layer of evaluated error and propagated forward operation.
For the backpropagation training algorithm, the first arithmetic processor 210, for by the weight vector of neural network and
The sample vector inputted by the input layer of neural network carry out forward operation obtain each layer of neural network of forward direction export to
Amount, forward operation include the first operation or the second operation.
What so-called forward operation referred to is exactly the operation that this direction carries out from the input layer of neural network to output layer, institute
Each layer of the neural network of meaning of positive output vector refers to the output of each layer be calculated during forward operation
Vector.Since the propagated forward operation in backpropagation training algorithm only includes dot product operation and nonlinear transformation operation,
Then the propagated forward operation can be executed by the first arithmetic processor 210.In other words, the first arithmetic processor described above
210 as propagated forward operations executed.It should be noted that executing propagated forward fortune in the first arithmetic processor 210
During calculation, which can not only execute dot product operation but also execute nonlinear transformation operation, at this
In one embodiment of invention, which can only execute dot product operation, and by neural network computing
Other modules of device 200 execute nonlinear transformation operation.
In the following, the present invention says the particular technique process for executing propagated forward operation to the first arithmetic processor 210
It is bright, specifically:
First arithmetic processor 210, for obtaining i-th layer of neural network of forward direction in the training operation of neural network
Output vector, and by i-th layer of positive output vector with by i-th layer of direction directed arc of i+1 layer it is corresponding at least one power
Vector carries out dot product operation respectively and obtains primary vector, and obtains the positive output vector of i+1 layer according to primary vector,
When i is 1, i-th layer of positive output vector is sample vector, and i is the positive integer more than or equal to 1.Wherein, first operation
Processor 210 obtains the forward direction of the i+1 layer for the vector element in above-mentioned primary vector to be carried out nonlinear transformation operation
Output vector.
It is illustrated in figure 3 the partial schematic diagram of a certain neural network, includes i-th layer and i+1 of neural network in Fig. 3
Layer.Wherein, i-th layer includes two nodes, and i-th layer of positive output vector is (ai, bi), i+1 layer include three nodes, i-th
The corresponding weight of directed arc of layer direction i+1 layer is non-Wei not w11, w21, w12, w22, w13, w23, wherein wpqRefer to i-th layer
P-th of node is directed toward the corresponding weight of directed arc of q-th of node of i+1 layer, then the corresponding weight group of above-mentioned each directed arc
At weight vector be respectively (w11, w21), (w12, w22), (w13, w23).In propagated forward operation, the first arithmetic processor 210
It can be by (ai, bi) and (w11, w21), (w12, w22), (w13, w23) dot product operation: c is carried out respectivelyi+1=ai×w11+bi×
w21, di+1=ai×w12+bi×w22, ei+1=ai×w13+bi×w23, to obtain primary vector, which is (ci+1,
di+1, ei+1), each of primary vector vector element can be carried out nonlinear transformation fortune by the first arithmetic processor 210
It calculates: c 'i+1=f (ci+1), d 'i+1=f (di+1), e 'i+1=f (ei+1), the output vector (c ' of i+1 layer can be obtainedi+1, d
′i+1, e 'i+1)。
It should be noted that the partial schematic diagram of above-mentioned neural network shown in Fig. 3 is only exemplary, actually answering
In, each layer of neural network may include any number of node, and the present invention is not especially limited this.
It should also be noted that, the example above in, from the first arithmetic processor 210 by each of the primary vector to
Secondary element carries out nonlinear transformation operation, that is to say that the first arithmetic processor 210 had not only executed dot product operation but also executed non-thread
Property transform operation, but in practical applications, the first arithmetic processor 210 can only execute dot product operation, and non-linear change
Changing operation can be executed by other modules of neural network computing device, in this regard, the present invention is not especially limited.
It should also be noted that, above-mentioned i is positive integer more than or equal to 1, when i is equal to 1, i-th layer of forward direction export to
What amount referred to is exactly the sample vector inputted by neural network input layer.
In addition, for the backpropagation training algorithm, the first arithmetic processor 210 is also used to neural network
It weight vector and reversed operation is carried out by the error vector that the output layer of neural network inputs obtains each layer of neural network anti-
To output vector, error vector be the positive output vector target output vector corresponding with sample vector of output layer it is poor to
Amount, reversed operation include the first operation.
What so-called reversed operation referred to is exactly the operation that this direction carries out from the output layer of neural network to input layer, institute
Each layer defeated that each layer of the neural network of meaning of reversed output vector refers to being calculated in reversed calculating process
Outgoing vector.It, then can be by the first fortune since the backpropagation operation in backpropagation training algorithm only includes dot product operation
It calculates processor 210 and executes the backpropagation operation.In other words, what the first arithmetic processor 210 described above executed is to be somebody's turn to do
Backpropagation operation.
In the following, the present invention says the particular technique process for executing backpropagation operation to the first arithmetic processor 210
It is bright, specifically:
First arithmetic processor 210, in the training operation of neural network, obtaining the neural network jth layer
Reversed output vector, and the reversed output vector of the jth layer and the directed arc for being directed toward -1 layer of jth by jth layer is corresponding at least
One weight vector carries out dot product operation respectively, obtains the reversed output vector of -1 layer of the jth, when the jth layer is the nerve
When the last layer of network, the reversed output vector of the jth layer is the error vector, and j is the positive integer greater than 1.
It is illustrated in figure 4 the partial schematic diagram of a certain neural network, includes the jth layer and jth -1 of neural network in Fig. 4
Layer.Wherein, jth layer includes two nodes, and the reversed output vector of jth layer is (aj, bj), -1 layer of jth includes three nodes, jth
The corresponding weight of directed arc of -1 layer of jth of layer direction is non-Wei not h11, h21, h12, h22, h13, h23, wherein hpqRefer to jth layer
P-th of node is directed toward the corresponding weight of directed arc of -1 layer of jth of q-th of node, then the corresponding weight group of above-mentioned each directed arc
At weight vector be respectively (h11, h21), (h12, h22), (h13, h23).In backpropagation operation, the first arithmetic processor 210
It can be by (aj, bj) and (h11, h21), (h12, h22), (h13, h23) dot product operation: c is carried out respectivelyj-1=aj×h11+bj×
h21, dj-1=aj×h12+bj×h22, ej-1=aj×h13+bj×h23, so that the reversed output vector for obtaining -1 layer of jth is (cj-1,
dj-1, ej-1)。
It should be noted that the partial schematic diagram of above-mentioned neural network shown in Fig. 4 is only exemplary, actually answering
In, each layer of neural network may include any number of node, and the present invention is not especially limited this.
It should also be noted that, above-mentioned j is the positive integer greater than 1, when the last layer that the jth layer is the neural network
When, the reversed output vector of the jth layer is the error vector.
Since the purpose of neural metwork training operation is to obtain one group of weight vector of neural network, so that the nerve after training
Network can obtain one close to ideal output, therefore, if the first calculation process to any input based on this group of weight vector
Device 210 obtains reversed output vector in backpropagation operation, that is to say that the error vector of each layer is smaller, then illustrates Current neural
The weight vector of network has been able to the purpose for meeting neural metwork training operation, at this point, the second arithmetic processor 220 does not need to hold
The modified operation of row weight vector.Specifically, second arithmetic processor 220 is also used to calculate in the first arithmetic processor 210
When at least one reversed output vector is greater than the second preset threshold in obtained reversed output vector, according to this, at least one is repaired
Positive value is modified at least one weight vector, obtains at least one amendment weight vector.
In addition, neural network computing device also needs to carry out weight vector amendment for the backpropagation training algorithm
Operation, that is to say and be modified according to weight vector of the result of propagated forward operation and back-propagating operation to neural network,
The modified operation of the weight vector includes following two step: one, according to propagated forward operation and back-propagating operation as a result, meter
Calculate at least one correction value；Two, enable at least one vector element at least one weight vector of neural network with it is respectively corresponding
Correction value be added to obtain at least one modification vector element, and at least one is obtained based at least one modification vector element
Correct weight vector.
Wherein, the above-mentioned first step can be executed by the first arithmetic processor 210, can also be by the second arithmetic processor 220
It executes, and above-mentioned second step can be executed by the second arithmetic processor 220.It is exemplary, it is executed with the first arithmetic processor 210
For stating the first step, then first arithmetic processor 210, be also used to according to each layer of the neural network of positive output vector and
Each layer of the neural network of reversed output vector calculates at least one correction value.
Specifically, with the activation primitive of neural network for Sigmoid function, and the neural network includes input layer, implies
For layer and output layer, which can use following formula and calculates above-mentioned correction value:
Δwkj=ε1δkaj
Wherein, δk=(tk-ak)ak(1-ak), Δ wkjIt is directed toward for hidden layer in the corresponding weight vector of directed arc of output layer
The correction value of vector element, Δ wijThe amendment of vector element in the corresponding weight vector of directed arc of hidden layer is directed toward for input layer
Value, ε1For the reversed output vector of output layer, ε2For the reversed output vector of hidden layer, wkjHaving for output layer is directed toward for hidden layer
To the corresponding weight vector of arc, ajFor the positive output vector of hidden layer, aiFor sample input vector, tkFor target output vector, ak
For the positive output vector of output layer.
It should be noted that in practical applications, which can be ASIC (Application
Specific Integrated Circuit, specific integrated circuit) operation chip, FPGA (Field-Programmable
Gate Array, field programmable gate array) operation chip etc., which can be CPU (Central
Processing Unit, central processing unit) operation chip, GPU (Graphics Processing Unit, graphics processor)
Operation chip etc., the present invention is not specifically limited in this embodiment.
In conclusion neural network computing device provided in this embodiment, by being centered in neural network computing dress
First arithmetic processor and the second arithmetic processor, and the first arithmetic processor be only used in the training operation of neural network into
Row vector dot-product operation, alternatively, the first arithmetic processor is only used for carrying out dot product fortune in the training operation of neural network
It calculates and nonlinear transformation operation, allowing the first arithmetic processor only to handle the processor active task of one mode therefore can be with
Only comprising special circuit designed by the operation for the mode, simplify the first arithmetic processor hardware circuit to reach
Purpose, so that the operation efficiency of the first arithmetic processor has obtained biggish promotion.Meanwhile it is different in neural metwork training operation
In the operation of other modes of dot product operation, the operation for such as correcting weight vector can be executed by the second arithmetic processor,
Due to operation most in the training operation of neural network be all dot product operation or all be dot product operation and
Nonlinear transformation operation, therefore, the promotion of the first arithmetic processor operation efficiency can greatly promote neural metwork training operation
Efficiency promotion.
Those of ordinary skill in the art will appreciate that realizing that all or part of the steps of above-described embodiment can pass through hardware
It completes, relevant hardware can also be instructed to complete by program, the program can store in a kind of computer-readable
In storage medium, storage medium mentioned above can be read-only memory, disk or CD etc..
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all in spirit of the invention and
Within principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.
Claims (10)
1. a kind of neural network computing device, which is characterized in that the neural network computing device includes the first arithmetic processor
With the second arithmetic processor:
First arithmetic processor, for by the weight vector of the neural network and leading in the training operation of neural network
The sample vector for crossing the input layer input of the neural network carries out forward operation and obtains each layer of the neural network of forward direction
Output vector, the forward operation include the first operation or the second operation, and first operation includes dot product operation, described
Second operation includes dot product operation and nonlinear transformation operation；
First arithmetic processor is also used to by the weight vector of the neural network and by the output layer of the neural network
The error vector of input carries out reversed operation and obtains each layer of the neural network of reversed output vector, and the error vector is
The difference vector of the positive output vector of output layer target output vector corresponding with the sample vector, the reversed operation
Including first operation；
Second arithmetic processor, for according at least one correction value at least one weight vector of the neural network into
Row amendment obtains at least one amendment weight vector, wherein one in the weight vector of each correction value and the neural network
Vector element is corresponding, at least one described correction value is obtained according to the operation result of first arithmetic processor.
2. neural network computing device according to claim 1, which is characterized in that first arithmetic processor is also used
At least one correction value described in being calculated according to the operation result of first operation or second operation；
Accordingly, second arithmetic processor, for according to first arithmetic processor be calculated described at least one
A correction value is modified at least one described weight vector, obtains at least one described amendment weight vector.
3. neural network computing device according to claim 1, which is characterized in that second arithmetic processor is also used
In calculating at least one described correction value according to the operation result of first operation or second operation, and according to it is described extremely
A few correction value is modified at least one described weight vector, obtains at least one described amendment weight vector.
4. neural network computing device according to claim 1, which is characterized in that second arithmetic processor is also used
In being modified according to target correction value at least one described weight vector, at least one described amendment weight vector is obtained, it is described
Target correction value is the correction value greater than the first preset threshold.
5. neural network computing device according to claim 1 or 2, which is characterized in that first arithmetic processor, also
Based on according to each layer of the neural network of positive output vector and each layer of the neural network of reversed output vector
Calculate at least one described correction value.
6. neural network computing device according to claim 1, which is characterized in that first arithmetic processor is also used
In the forward reasoning calculating process in neural network, first operation or second operation are carried out.
7. neural network computing device according to claim 1, which is characterized in that second arithmetic processor is also used
When at least one reversed output vector is greater than the second preset threshold in the reversed output vector, according to described at least one
A correction value is modified at least one described weight vector, obtains at least one amendment weight vector.
8. neural network computing device according to claim 4, which is characterized in that first arithmetic processor is used for
In the training operation of neural network, the positive output vector of i-th layer of the neural network is obtained, and just by described i-th layer
Vector point is carried out respectively with by corresponding at least one weight vector of the described i-th layer direction directed arc of i+1 layer to output vector
Product operation obtains primary vector, and obtains the positive output vector of the i+1 layer according to the primary vector, when i is 1 when institute
The positive output vector for stating i-th layer is the sample vector, and i is the positive integer more than or equal to 1.
9. neural network computing device according to claim 8, which is characterized in that first arithmetic processor is used for
Vector element in the primary vector is subjected to nonlinear transformation operation and obtains the positive output vector of the i+1 layer.
10. neural network computing device according to claim 1, which is characterized in that first arithmetic processor is used for
In the training operation of neural network, the reversed output vector of the neural network jth layer is obtained, and by the anti-of the jth layer
Vector point is carried out respectively to output vector at least one weight vector corresponding with the directed arc for being directed toward -1 layer of jth by the jth layer
Product operation, obtains the reversed output vector of -1 layer of the jth, when the jth layer is the last layer of the neural network, institute
The reversed output vector for stating jth layer is the error vector, and j is the positive integer greater than 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610556380.6A CN106203622B (en) | 2016-07-14 | 2016-07-14 | Neural network computing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610556380.6A CN106203622B (en) | 2016-07-14 | 2016-07-14 | Neural network computing device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106203622A CN106203622A (en) | 2016-12-07 |
| CN106203622B true CN106203622B (en) | 2018-12-25 |
Family
ID=57475341
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610556380.6A Active CN106203622B (en) | 2016-07-14 | 2016-07-14 | Neural network computing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106203622B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10565494B2 (en) * | 2016-12-31 | 2020-02-18 | Via Alliance Semiconductor Co., Ltd. | Neural network unit with segmentable array width rotator |
| IT201700008949A1 (en) * | 2017-01-27 | 2018-07-27 | St Microelectronics Srl | OPERATING PROCEDURE FOR NEURAL NETWORKS, NETWORK, EQUIPMENT AND CORRESPONDENT COMPUTER PRODUCT |
| CN109389209B (en) * | 2017-08-09 | 2022-03-15 | 上海寒武纪信息科技有限公司 | Processing apparatus and processing method |
| CN107729990B (en) * | 2017-07-20 | 2021-06-08 | 上海寒武纪信息科技有限公司 | Apparatus and method for performing forward operations in support of discrete data representations |
| CN109426859B (en) * | 2017-08-22 | 2021-03-05 | 华为技术有限公司 | Neural network training system, method and computer readable storage medium |
| CN108364061B (en) * | 2018-02-13 | 2020-05-05 | 北京旷视科技有限公司 | Arithmetic device, arithmetic execution apparatus, and arithmetic execution method |
| US20200005125A1 (en) * | 2018-06-27 | 2020-01-02 | International Business Machines Corporation | Low precision deep neural network enabled by compensation instructions |
| CN111147603A (en) * | 2019-09-30 | 2020-05-12 | 华为技术有限公司 | Method and device for networking reasoning service |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103544528A (en) * | 2013-11-15 | 2014-01-29 | 南京大学 | BP neural-network classification method based on Hadoop |
| CN103854055A (en) * | 2014-03-31 | 2014-06-11 | 南京工程学院 | Electric transmission line icing prediction model based on neural network and fuzzy logic algorithm |
| CN105162738A (en) * | 2015-07-30 | 2015-12-16 | 南京信息工程大学 | Satellite channel complex-valued neural polynomial network blind equalization system and method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100820723B1 (en) * | 2006-05-19 | 2008-04-10 | 인하대학교 산학협력단 | Separately trained system and method using two-layered neural network with target values of hidden nodes |
-
2016
- 2016-07-14 CN CN201610556380.6A patent/CN106203622B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103544528A (en) * | 2013-11-15 | 2014-01-29 | 南京大学 | BP neural-network classification method based on Hadoop |
| CN103854055A (en) * | 2014-03-31 | 2014-06-11 | 南京工程学院 | Electric transmission line icing prediction model based on neural network and fuzzy logic algorithm |
| CN105162738A (en) * | 2015-07-30 | 2015-12-16 | 南京信息工程大学 | Satellite channel complex-valued neural polynomial network blind equalization system and method |
Non-Patent Citations (1)
| Title |
|---|
| 图像识别中BP神经网络的硬件实现;张雄;《中国优秀硕士学位论文全文数据库 信息科技辑》;20150315(第3期);第2.1节、图2.1 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106203622A (en) | 2016-12-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106203622B (en) | Neural network computing device | |
| US11461626B2 (en) | Brain-like computing chip and computing device | |
| KR102592721B1 (en) | Convolutional neural network system having binary parameter and operation method thereof | |
| CN107578095B (en) | Neural computing device and processor comprising the computing device | |
| EP3407266B1 (en) | Artificial neural network calculating device and method for sparse connection | |
| CN108009640A (en) | The training device and its training method of neutral net based on memristor | |
| CN107918794A (en) | Neural network processor based on computing array | |
| CN108416437A (en) | The processing system and method for artificial neural network for multiply-add operation | |
| CN107578098A (en) | Neural network processor based on systolic arrays | |
| CN108416327A (en) | A kind of object detection method, device, computer equipment and readable storage medium storing program for executing | |
| CN109472356A (en) | A kind of accelerator and method of restructural neural network algorithm | |
| CN107341541A (en) | A kind of apparatus and method for performing full articulamentum neural metwork training | |
| CN108629411A (en) | A kind of convolution algorithm hardware realization apparatus and method | |
| US20220147877A1 (en) | System and method for automatic building of learning machines using learning machines | |
| CN107392308A (en) | A kind of convolutional neural networks accelerated method and system based on programming device | |
| CN107766935B (en) | Multilayer artificial neural network | |
| Guo et al. | Novel recurrent neural network for time-varying problems solving [research frontier] | |
| Han et al. | Nonlinear system modeling using a self-organizing recurrent radial basis function neural network | |
| KR20200111948A (en) | A method for processing artificial neural network and electronic device therefor | |
| CN108320018A (en) | A kind of device and method of artificial neural network operation | |
| Beer et al. | Quantum machine learning of graph-structured data | |
| Wu et al. | A deep learning approach for solving linear programming problems | |
| CN109426859A (en) | Neural metwork training system, method and computer readable storage medium | |
| Poli et al. | TorchDyn: implicit models and neural numerical methods in PyTorch | |
| JPH07200512A (en) | 1optimization problems solving device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200423 Address after: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Patentee after: HUAWEI TECHNOLOGIES Co.,Ltd. Address before: 301, A building, room 3, building 301, foreshore Road, No. 310052, Binjiang District, Zhejiang, Hangzhou Patentee before: Huawei Technologies Co.,Ltd. |
|
| TR01 | Transfer of patent right |