WO2017177446A1 - 支持离散数据表示的人工神经网络反向训练装置和方法 - Google Patents

支持离散数据表示的人工神经网络反向训练装置和方法 Download PDF

Info

Publication number
WO2017177446A1
WO2017177446A1 PCT/CN2016/079443 CN2016079443W WO2017177446A1 WO 2017177446 A1 WO2017177446 A1 WO 2017177446A1 CN 2016079443 W CN2016079443 W CN 2016079443W WO 2017177446 A1 WO2017177446 A1 WO 2017177446A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
discrete
module
unit
continuous
Prior art date
Application number
PCT/CN2016/079443
Other languages
English (en)
French (fr)
Inventor
郭崎
于涌
陈天石
陈云霁
Original Assignee
北京中科寒武纪科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京中科寒武纪科技有限公司 filed Critical 北京中科寒武纪科技有限公司
Priority to US16/093,958 priority Critical patent/US20190130274A1/en
Priority to PCT/CN2016/079443 priority patent/WO2017177446A1/zh
Priority to EP16898264.3A priority patent/EP3444758B1/en
Publication of WO2017177446A1 publication Critical patent/WO2017177446A1/zh
Priority to US16/182,439 priority patent/US20190080241A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/01—Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound

Definitions

  • the present invention generally relates to artificial neural networks, and in particular to an apparatus and method for performing artificial neural network reverse training.
  • the data in the present invention supports discrete form representation and replaces operations such as multiplication of continuous data by bit operations of discrete data.
  • Multi-layer artificial neural networks are widely used in the fields of pattern recognition, image processing, function approximation and optimization calculation.
  • Multi-layer artificial networks have been accepted by Kirin, image processing, function approximation and optimization calculation.
  • Multi-layer artificial networks have been accepted by Kirin, image processing, function approximation and optimization calculation.
  • Multi-layer artificial networks have been accepted by Kirin, image processing, function approximation and optimization calculation.
  • Multi-layer artificial networks have been accepted by Kir in recent years due to their high recognition accuracy and good parallelism. The industry is getting more and more attention.
  • One known method of supporting multi-layer artificial neural network reverse training is to use a general purpose processor.
  • the method supports the above algorithm by executing general purpose instructions using a general purpose register file and generic functions.
  • Another known method of supporting multi-layer artificial neural network reverse training is to use a graphics processing unit (GPU).
  • the method supports the above algorithm by executing a generic SIMD instruction using a general purpose register file and a generic stream processing unit.
  • Discrete data representation refers to the way in which non-contiguous real data is stored by a specific number.
  • the four numbers 0, 1, 2, and 3 can represent the real numbers -1, -1/8, 1/8, and 1 respectively.
  • index-like representation it is possible to replace the discontinuously discretized real data with a formally continuous number.
  • the current known method of multi-layer artificial neural networks in data representation is represented by continuous data such as floating point numbers or fixed point numbers. Because of the high precision and large number of multi-layer neural network weights, the representation of continuous data leads to greater storage and power consumption overhead. In the way of discrete data representation, it is possible to replace the multiplication of continuous data by data bitwise XOR, shift, and the like. This greatly reduces the number of multiplier components.
  • the device for reverse training includes an instruction cache unit, a controller unit, a data access unit, an interconnection module, a main operation module, and a plurality of slave operation modules, wherein:
  • An instruction cache unit is used to cache instructions
  • the controller unit is configured to read an instruction from the instruction cache unit and decode the instruction into a micro-instruction that controls the interconnect module, the main operation module, and the slave operation module;
  • the data access unit is configured to write discrete representations or consecutively represented data from the memory to the main data processing unit of the main operation module and the respective slave operation modules or read the discrete representation or the continuously represented data from the data buffer unit to the memory;
  • the main operation module transmits the input gradient vector of the layer to all the slave arithmetic modules through the interconnection module.
  • the interconnection module will step by step. The output gradient vector portion of the operation module and the two pairs are added to obtain an output gradient vector of the layer;
  • the main operation module is used to perform subsequent calculations by using the output gradient vector of the layer in the calculation process of each layer.
  • the calculation module takes pre-set for different discrete data. Corresponding calculation method;
  • Each slave arithmetic module calculates the corresponding output gradient vector portion sum in parallel by using the same input gradient vector and respective discrete or continuous weight data, and when the input data is mixed data of discrete data and continuous data, the slave operation
  • the module takes a preset calculation method for different discrete data.
  • discrete data representation refers to the representation of replacing real continuous data with a particular discrete number.
  • the plurality of slave computing modules calculate the gradients of the respective weights in parallel using the same input gradient vector and update the respective weight data using the calculated gradients of the respective weights.
  • the main operation module multiplies the output gradient vector of each layer by the alignment of the activation function of the next layer as the input gradient vector of the next layer.
  • the interconnect module constitutes a continuous or discretized data path between the main operation module and the plurality of slave operation modules, and can be implemented into different interconnection topologies.
  • the interconnection module has an H-tree structure, and the H-tree is a binary tree path composed of a plurality of nodes, and each node sends the upstream data to the downstream two nodes in the same manner, and the downstream two The continuous or discretized data returned by the node is added and returned to the upstream node.
  • the main operation module includes an operation unit, a data dependency determination unit, and a neuron cache unit supporting the discrete data representation, wherein:
  • a neuron cache unit supporting discrete data representation is used to buffer discrete or continuous input data and output data used by the main operation module in the calculation process;
  • the operation unit completes various calculation functions of the main operation module.
  • the input data is mixed data of discrete data and continuous data, a corresponding calculation manner preset for different discrete data is adopted;
  • the data dependency judgment unit is a port for the operation unit to read and write the neuron cache unit, and ensures that there is no consistency conflict between the discrete or continuous data read and write in the neuron cache unit, and is responsible for reading the input gradient vector from the neuron cache unit.
  • the interconnect module is sent to the slave arithmetic module;
  • the output gradient vector from the interconnect module is sent to the arithmetic unit.
  • each slave computing module includes an arithmetic unit, a data dependency determining unit, a neuron buffer unit supporting discrete data representation, a weight buffer unit supporting discrete data representation, and a weight gradient buffer unit supporting discrete data representation, among them:
  • the arithmetic unit receives the micro-instruction issued by the controller unit and performs arithmetic logic operation.
  • the input data is mixed data of discrete data and continuous data, a corresponding calculation manner preset for different discrete data is adopted;
  • the data dependency judgment unit is responsible for reading and writing the neuron cache unit supporting the discrete data representation, the weight buffer unit supporting the discrete data representation, and the weight gradient buffer unit supporting the discrete data representation in the calculation process, and ensuring support for the discrete data. There is no consistency conflict between the represented neuron cache unit, the weight buffer unit supporting the discrete data representation, and the weight gradient buffer unit supporting the discrete data representation;
  • the neuron buffer unit buffer input supporting the discrete data representation supports the scalar data corresponding to the slave operation module in the gradient vector data of the discrete representation and the output gradient vector portion sum calculated by the slave operation module;
  • the weight buffer unit supporting the discrete data representation buffers the weighted data of the discrete or continuous representation required by the slave arithmetic module in the calculation process, and for each slave arithmetic module, only the weight matrix is stored and stored by the slave arithmetic module. Corresponding columns of scalar data;
  • the weight gradient buffer unit supporting the discrete data representation buffers the weight gradient data required by the operation module in the process of updating the weight, and the weight gradient data stored by each slave operation module and the stored discrete Or the weight data corresponding to the continuous representation corresponds.
  • micro-instruction it is ensured that there is no consistency conflict between the read and write by determining whether there is a dependency between the micro-instruction that has not been executed and the data of the micro-instruction being executed, and if not, allowing the micro-instruction to immediately transmit Otherwise, it is necessary to wait until all micro-instructions on which the micro-instruction depends are executed, and the micro-instruction is allowed to be transmitted.
  • the operation unit includes an operation determination unit and a mixed data operation unit.
  • the operation determination unit determines, according to the discrete data therein, what operation is to be performed on the mixed data, and then the mixed data operation unit performs the operation according to the operation. Determine the decision result of the unit and perform the corresponding operation.
  • the operation unit further includes at least one of a discrete data operation unit and a continuous data operation unit, and a data type determination unit.
  • the discrete data operation unit passes the table according to the input discrete data. The corresponding operation is performed, and when the input data is all continuous data, the corresponding operation is performed by the continuous data operation unit.
  • the data corresponds to the M values in the predetermined interval [-zone,zone], where:
  • the distance calculation module calculates a distance between the preprocessed data y and each of the above values
  • the decision module 82 calculates and outputs discrete data based on the distance.
  • the predetermined interval [-zone,zone] is [-1,1] or [-2,2]; and/or the absolute value of the M values is a reciprocal of the power of 2; and/or the determining module executes: And/or output discrete data corresponding to the value closest to the preprocessed data y, if there are two values equal to the preprocessed data, output discrete data corresponding to any one of the two; and/or calculate Preprocessing the data y to the normalized probability of any of the two nearest values, between the normalized probability corresponding to either of the two values and the (0, 1) generated by the random number generating module The random number z comparison, if the z is less than the probability, the discrete data is output, otherwise another discrete data is output.
  • Another aspect of the present invention provides a method of performing a single layer artificial neural network reverse training using the above apparatus.
  • the controller controls the input neurons and activation letters during the read operation.
  • the numerical value and the input gradient are read, and then the weight data and the weight gradient data are read, and then the constants such as the training precision and the learning rate of the neural network are read.
  • These data can be represented by discrete data or not.
  • the master-slave computing module and the interconnect module complete the weight update operation.
  • the multiplication operation is replaced by the bit operation on the related data according to the value of the discrete data.
  • the weight data is represented by 1 bit of discrete data, 0 represents +1, and 1 represents -1, and the multiplication of the weight is realized by XORing the sign bit of the data multiplied by the weight.
  • Another aspect of the present invention provides a method of supporting artificial neural network batch normalization using the above apparatus.
  • the controller controls the data access unit to read the input data, and then controls the master-slave operation module to find the mean and variance of the respective positions according to the batch size or use the set mean variance.
  • the controller then controls the input data for the corresponding location minus the mean divided by the variance.
  • the controller controls multiplying the processed data by the learning parameters and adding another learning parameter.
  • Another aspect of the present invention provides a method of performing multi-layer artificial neural network reverse training using the above apparatus.
  • the implementation process is similar to the single-layer neural network.
  • the next-level operation instruction will use the output gradient vector calculated in the main operation module as the input gradient vector of the next layer of training.
  • the weight address and the weight gradient address in the instruction are also changed to the address corresponding to this layer.
  • FIG. 1 shows an example block diagram of the overall structure of an apparatus for performing artificial neural network reverse training supporting discrete data representations in accordance with an embodiment of the present invention.
  • FIG. 2 schematically illustrates the structure of an H-tree module (an embodiment of an interconnect module) in an apparatus for performing artificial neural network reverse training supporting discrete data representations in accordance with an embodiment of the present invention.
  • FIG. 3 illustrates an example block diagram of a main operational module structure in an apparatus for performing artificial neural network reverse training supporting discrete data representations in accordance with an embodiment of the present invention.
  • FIG. 4 illustrates an example block diagram of a slave arithmetic module structure in an apparatus for performing artificial neural network reverse training supporting discrete data representations in accordance with an embodiment of the present invention.
  • FIG. 5 illustrates an example block diagram of a neural network reverse training process in accordance with an embodiment of the present invention.
  • FIG. 6 shows an example block diagram of a neural network reverse training process for discrete data representations in accordance with an embodiment of the present invention.
  • Figure 7 shows a flow chart of a single layer artificial neural network operation in accordance with an embodiment of the present invention.
  • FIG. 8 shows an example structure of an arithmetic unit for discrete data.
  • Figure 9 shows an example structure of a continuous discrete conversion module for continuous data and discrete data conversion.
  • the reverse training of a multi-layer artificial neural network supporting discrete data representation includes two or more layers of multiple neurons.
  • the input gradient vector is first weighted and summed to calculate the output gradient vector of the layer.
  • the output gradient vector is multiplied by the derivative value of the activation function of the following layer in the forward operation to obtain the input gradient vector of the next layer.
  • the input gradient vector is multiplied by the input neuron in the forward operation to obtain the gradient of the layer weight, and then the weight of the layer can be updated according to the obtained gradient of the layer weight.
  • the data in this process can be either discretely represented or continuous.
  • the device supports the conversion of the dot product operation into a shift, a non-exclusive, an exclusive OR, or an equal bit operation of the data.
  • the device supports discrete representation or continuous representation of data, and the user can customize which layer Which data is expressed in discrete representation or continuous, and the number of bits of discrete data can be customized according to specific needs, instead of the number of consecutive data represented, for example, discrete bits of 1 bit, 2 bits, 3 bits, etc. Data, which can represent 2, 4, and 8 consecutive data, respectively.
  • the apparatus includes an instruction cache unit 1, a controller unit 2, a data access unit 3, an interconnection module 4, a main operation module 5, and a plurality of slave operation modules 6.
  • the apparatus may further comprise a continuous discrete conversion module 7.
  • the instruction cache unit 1, the controller unit 2, the data access unit 3, the H-tree module 4, the main operation module 5, the slave operation module 6, and the continuous discrete conversion module 7 can all pass through hardware circuits (including but not limited to FPGA, CGRA, dedicated Implementation of integrated circuit ASICs, analog circuits, and memristors.
  • the device supports the conversion of continuous data into discrete data as well as the storage and operation of discrete data.
  • the instruction cache unit 1 reads in an instruction through the data access unit 3 and caches the read instruction.
  • the controller unit 2 reads the instructions from the instruction cache unit 1, translates the instructions into micro-instructions that control the behavior of other modules, and sends them to other modules such as the data access unit 3, the main operation module 5, and the slave operation module 6.
  • the data access unit 3 can access an external address space (such as a memory or other storage device), directly read and write data to each cache unit inside the device, and complete data loading and storage.
  • the data is either discretely represented or non-discretely represented. This unit is used to design data that can read discrete representations
  • the interconnection module 4 is used for connecting the main operation module and the slave operation module, and can be implemented into different interconnection topologies (such as a tree structure, a ring structure, a grid structure, a hierarchical interconnection, a bus structure, etc.).
  • FIG. 2 schematically shows an embodiment of the interconnection module 4: an H-tree structure.
  • the H-tree module 4 constitutes a data path between the main arithmetic module 5 and the plurality of slave arithmetic modules 6, and has an H-tree structure.
  • the H-tree is a binary tree path composed of multiple nodes. Each node sends the upstream data to the downstream two nodes in the same way, and the data returned by the two downstream nodes are combined and returned to the upstream node. For example, in the inverse operation of the neural network, the vectors returned by the two downstream nodes are added to a vector at the current node and returned to the upstream node.
  • each slave operation module 6 outputs The sum of the output gradient vectors will be summed two-by-two in the H-tree module 4, ie, summing and summing all the output gradient vectors as the final output gradient vector.
  • the main operation module 5 includes an operation unit 51, a data dependency determination unit 52, and a neuron buffer unit 53 that supports discrete data representation.
  • the neuron buffer unit 53 supporting the discrete data representation is used to buffer the input data and the output data used by the main operation module 5 in the calculation process.
  • the arithmetic unit 51 performs various arithmetic functions of the main arithmetic module.
  • the addition, subtraction, multiplication and division of discrete data and discrete data can be realized by looking up the table.
  • 2-bit discrete data can represent 4 consecutive data values.
  • There are 4*4 16 combinations for 4 consecutive data.
  • the 4*4 index table can be created and maintained, and the corresponding calculated value is found through the index table.
  • a total of four 4*4 index tables are required for the four operations.
  • the corresponding bit operations may be preset for the addition, subtraction, multiplication, and division operations for different discrete data.
  • a dot product operation of discrete data and continuous data may be replaced by a method of cumulative summation after bitwise exclusive OR and multiplied by 2 corresponding powers.
  • a multiplication operation if the multiplicative factor data is discretely represented, it may be replaced by a discrete data index corresponding operation (eg, bitwise XOR, NAND, shift, etc. of the corresponding data) and the discrete data representation. Multiplication of continuous data, which reduces the number of multiplier parts.
  • the function of the arithmetic unit can be replaced by a method of finding a switch such as a search index. For example, it can be specified that the discrete data representation method of -1/2 is 01. If an operation factor is -1/2, the discrete data received by the arithmetic unit 51 is 01. The arithmetic unit 51 uses the operation corresponding to the discrete data 01.
  • the decimal representation is -8.
  • 16 is divided by -2. 16 is continuous data and -2 is discrete data. If the discrete data-2 binary is specified as 10.
  • the arithmetic unit uses the division operation corresponding to the discrete data 10. The result is obtained by inverting the sign bit by subtracting 1000 bits from the 8-bit fixed-point number of 16 to 10001000 and obtaining 10001000 in decimal.
  • the addition and subtraction operations are similar to the above process. According to the binary of the discrete data as an index, the index is shifted to the left, right, or XOR. After this operation, the addition or subtraction operation with the real data represented by the discrete data is realized.
  • the data dependency judging unit 52 is a port for the arithmetic unit 51 to read and write the neuron buffer unit 53, and at the same time, can ensure that there is no consistency conflict with the reading and writing of data in the neuron buffer unit 53. specifically, The data dependency determining unit 52 determines whether there is a dependency between the microinstruction that has not been executed and the data of the microinstruction in the process of execution. If it does not exist, the microinstruction is allowed to be transmitted immediately, otherwise it is necessary to wait until the microinstruction depends on The microinstruction is allowed to be transmitted after all the microinstructions have been executed.
  • all microinstructions sent to the data dependency unit 52 are stored in an instruction queue internal to the data dependency unit 52, in which the read data of the read instruction ranges from the write command to the queue position. If the range of write data conflicts, the instruction must wait until the write instruction it depends on is executed.
  • the data dependency determining unit 52 is also responsible for reading the input gradient vector from the neuron buffer unit 53 and transmitting it to the slave computing module 6 through the interconnecting module 4, and the output data from the computing module 6 is directly sent to the computing unit 51 through the interconnecting module 4. .
  • the command output from the controller unit 2 is sent to the arithmetic unit 51 and the dependency determining unit 52 to control its behavior.
  • each slave operation module 6 includes an operation unit 61, a data dependency determination unit 62, a neuron buffer unit 63 supporting discrete data representation, a weight buffer unit 64 supporting discrete data representation, and supporting discrete data representation.
  • Weight gradient buffer unit 65 is shown in FIG. 4, and is not limited to, but not limited to, but not limited to, but not limited to, but not limited to, but not limited to, but not limited to, but not limited to, but not limited to a weight buffer unit 64.
  • weight gradient buffer unit 65 is an example block diagram of the structure of a slave arithmetic module 6 in an apparatus for performing artificial neural network reverse training supporting discrete data representations in accordance with an embodiment of the present invention.
  • each slave operation module 6 includes an operation unit 61, a data dependency determination unit 62, a neuron buffer unit 63 supporting discrete data representation, a weight buffer unit 64 supporting discrete data representation, and supporting discrete data representation.
  • the arithmetic unit 61 receives the microinstructions issued by the controller unit 2 and performs an arithmetic logic operation.
  • the addition, subtraction, multiplication and division of discrete data and discrete data can be realized by looking up the table.
  • 2-bit discrete data can represent 4 consecutive data values.
  • There are 4*4 16 combinations for 4 consecutive data.
  • the 4*4 index table can be created and maintained, and the corresponding calculated value is found through the index table. A total of four 4*4 index tables are required for the four operations.
  • the corresponding bit operations may be preset for the addition, subtraction, multiplication, and division operations for different discrete data.
  • a dot product operation of discrete data and continuous data may be replaced by a method of cumulative summation after bitwise exclusive OR and multiplied by 2 corresponding powers.
  • a multiplication operation if the multiplicative factor data is discretely represented, it may be replaced by a discrete data index corresponding operation (eg, bitwise XOR, NAND, shift, etc. of the corresponding data) and the discrete data representation. Multiplication of continuous data, which reduces the number of multiplier parts.
  • arithmetic unit 51 For example, for multiplication operations of continuous data and discrete data, -1/2 is multiplied by 16. Traditional multiplier components will multiply -1/2 and 16 directly.
  • the function of the arithmetic unit can be replaced by a method of finding a switch such as a search index.
  • a discrete data representation method of -1/2 can be specified Is 01. If an operation factor is -1/2, the discrete data received by the arithmetic unit 51 is 01. The arithmetic unit 51 uses the operation corresponding to the discrete data 01.
  • the decimal representation is -8.
  • 16 is divided by -2. 16 is continuous data and -2 is discrete data. If the discrete data-2 binary is specified as 10.
  • the arithmetic unit uses the division operation corresponding to the discrete data 10. The result is obtained by inverting the sign bit by subtracting 1000 bits from the 8-bit fixed-point number of 16 to 10001000 and obtaining 10001000 in decimal.
  • the addition and subtraction operations are similar to the above process. According to the binary of the discrete data as an index, the index is shifted to the left, right, or XOR. After this operation, the addition or subtraction operation with the real data represented by the discrete data is realized.
  • the data dependency judging unit 62 is responsible for reading and writing operations on the cache unit in the calculation process.
  • the data dependency judging unit 62 ensures that there is no consistency conflict between the reading and writing of the cache unit.
  • the data dependency determining unit 62 determines whether there is a dependency relationship between the microinstruction that has not been executed and the data of the microinstruction that is being executed, and if not, allows the microinstruction to be immediately transmitted, otherwise it is necessary to wait until the micro The microinstruction is allowed to be transmitted after all the microinstructions on which the instruction depends are executed.
  • all microinstructions sent to the data dependency unit 62 are stored in an instruction queue inside the data dependency unit 62, in which the range of read data of the read instruction is a write command ahead of the queue position. If the range of write data conflicts, the instruction must wait until the write instruction it depends on is executed.
  • the neuron buffer unit 63 supporting the discrete data representation buffers the scalar data corresponding to the slave arithmetic module 6 in the input gradient vector data and the output gradient vector partial sum calculated by the slave arithmetic module 6.
  • the weight buffer unit 64 supporting the discrete data representation buffers the weight data required by the slave arithmetic module 6 in the calculation process.
  • the data can be discrete or discrete based on user definition. For each slave arithmetic module, only the columns in the weight matrix corresponding to the scalar data stored by the slave arithmetic module 6 are stored.
  • the weight gradient buffer unit 65 buffers the weight gradient data required by the corresponding slave module in updating the weights.
  • Each of the weight gradient data stored from the arithmetic module 6 corresponds to its stored weight data.
  • the first half of the parallel and the update of the weights in the process of calculating the output gradient vector for each layer of artificial neural network reverse training are implemented.
  • MLP artificial neural network full connection layer
  • Each of the slave arithmetic modules 6 calculates a portion and sum of the output gradient vectors and performs a summation operation in the interconnect module 4 to obtain a final output gradient vector.
  • Each slave arithmetic module 6 simultaneously multiplies the input gradient vector and the output value of each layer in the forward operation to calculate a gradient of the weights to update the weight stored by the slave arithmetic module 6.
  • Forward and reverse training are the two main processes of neural network algorithms. To train (update) the weights in the network, the neural network needs to calculate the forward output of the input vector in the network composed of the current weights. This is positive.
  • the weight of each layer is trained (updated) layer by layer according to the difference between the output value and the label value of the input vector itself.
  • the output vector of each layer and the derivative value of the activation function are saved. These data are required for the reverse training process, so these data are guaranteed to exist at the beginning of the reverse training.
  • the output value of each layer in the forward operation is the data existing at the beginning of the reverse operation, and can be buffered in the main operation module by the data access unit and sent to the slave operation module through the interconnection module.
  • the main operation module 5 performs subsequent calculation based on the output gradient vector, for example, multiplying the output gradient vector by the derivative of the activation function in the forward operation to obtain the input gradient value of the next layer.
  • the derivative of the activation function in the forward operation is the data existing at the beginning of the reverse operation, and can be cached in the main arithmetic module by the data access unit.
  • FIG. 8 shows a structural block diagram of an arithmetic unit which can be used for the arithmetic unit 51 in the main arithmetic module or the arithmetic unit 61 in the slave arithmetic module.
  • the input data during the operation can be discrete data or continuous data.
  • the data type judging unit 71 judges that the input data is all continuous data, all discrete data, or mixed data including both continuous data and discrete data.
  • the continuous data operation unit 72 performs a corresponding operation.
  • the discrete data operation unit 73 When the input data is all discrete data, the discrete data operation unit 73 performs a corresponding operation.
  • the addition, subtraction, multiplication and division of discrete data and discrete data can be realized by looking up the table.
  • 2-bit discrete data can represent 4 consecutive data values.
  • There are 4*4 16 combinations for 4 consecutive data.
  • the operation decision unit 74 decides which operation to perform according to the discrete data therein.
  • the corresponding operations can be preset for different discrete data.
  • the mixed data operation unit 75 performs a corresponding operation based on the determination result of the arithmetic decision unit 74.
  • the corresponding bit operations may be preset for the addition, subtraction, multiplication, and division operations for different discrete data.
  • a dot product operation of discrete data and continuous data may be replaced by a method of cumulative summation after bitwise exclusive OR and multiplied by 2 corresponding powers.
  • multiplicative factor data for a multiplication operation, if the multiplicative factor data is discretely represented, it may be replaced by a discrete data index corresponding operation (eg, bitwise XOR, NAND, shift, etc. of the corresponding data) and the discrete data representation.
  • a discrete data index corresponding operation eg, bitwise XOR, NAND, shift, etc. of the corresponding data
  • Multiplication of continuous data which reduces the number of multiplier parts. For example, for multiplication operations of continuous data and discrete data, -1/2 is multiplied by 16. Traditional multiplier components will multiply -1/2 and 16 directly.
  • the function of the arithmetic unit can be replaced by a method of finding a switch such as a search index. For example, it can be specified that the discrete data representation method of -1/2 is 01.
  • the discrete data received by the arithmetic unit 51 is 01.
  • the arithmetic unit 51 uses the operation corresponding to the discrete data 01.
  • the decimal representation is -8.
  • 16 is divided by -2.
  • 16 is continuous data and -2 is discrete data.
  • the discrete data-2 binary is specified as 10.
  • the arithmetic unit uses the division operation corresponding to the discrete data 10. The result is obtained by inverting the sign bit by subtracting 1000 bits from the 8-bit fixed-point number of 16 to 10001000 and obtaining 10001000 in decimal.
  • the addition and subtraction operations are similar to the above process. According to the binary of the discrete data as an index, the index is shifted to the left, right, or XOR. After this operation, the addition or subtraction operation with the real data represented by the discrete data is realized.
  • FIG. 9 shows a schematic structural view of the continuous discrete conversion module 7. Users can define the use of this module to convert continuous data to discrete data or not. Continuous data is input to the continuous discrete conversion module 7 to output discrete data.
  • the absolute value of the M values may be a reciprocal of a power of two.
  • the distance calculation module 82 calculates the distance between the preprocessed data y and each of the above values.
  • the decision module 82 calculates and outputs discrete data based on the distance. For example, the determination module 82 can output discrete data corresponding to the value closest to the preprocessed data.
  • the determining module 82 may calculate a normalized probability of the pre-processed data y to any one of the nearest two values, and generate a normalized probability corresponding to any one of the two values and the random number generating module 82.
  • the random number z comparison between (0, 1), if the z is less than the probability, the discrete data is output, otherwise another discrete data is output. For example, if the user definition produces binary discrete data, then M 2.
  • the pre-processing module 81 performs an operation clip(-1, 1) on the input arbitrary continuous data x to obtain pre-processed data y.
  • the distance calculation module 82 calculates the distances D1, D2 between the pre-processed data y and the values -1, 1, assuming D1 > D2.
  • the judging module 83 can output the discrete data-1 corresponding to D1.
  • the judging module 83 can calculate the normalized probability of any one of y to the nearest two values (since there are only two values in this example, so the nearest two values are -1 and 1).
  • the normalized probability P2 is compared with the random number z, and if z is less than the probability P2, the discrete data 1 is output, otherwise another discrete data-1 is output.
  • the judging module outputs the discrete data in the form of binary data, and the relationship between the discrete data and the binary data can be preset, for example, by a lookup table setting. For example, 0 can be used to represent 1, and 1 is -1. Store the resulting discrete data back into memory. Waiting for the operation unit in the master-slave operation module to generate the corresponding operation.
  • the weight data in the reverse training process can be represented by discrete data.
  • the multiplication operation for the weights is replaced by the exclusive OR, the non-displacement, and the displacement according to the storage method of the discrete data.
  • 0 represents +1 and 1 represents -1
  • the multiplication of the weights is achieved by XORing the sign bits of the data multiplied by the weights.
  • an instruction set for performing an artificial neural network inverse operation on the aforementioned apparatus includes a CONFIG instruction, a COMPUTE instruction, an IO instruction, a NOP instruction, a JUMP instruction, and a MOVE instruction, among which:
  • the CONFIG command configures the various layers required for the current layer calculation before each layer of artificial neural network calculation begins. number;
  • the COMPUTE instruction completes the arithmetic logic calculation of each layer of artificial neural network
  • the IO instruction implements the input data required for the calculation from the external address space and stores the data back to the external space after the calculation is completed.
  • the data supports the discretization representation
  • the NOP instruction is responsible for clearing all the microinstructions in the microinstruction buffer queue of the current device, and ensuring that all the instructions before the NOP instruction are all completed.
  • the NOP instruction itself does not contain any operations;
  • the JUMP instruction is responsible for the jump of the next instruction address that the controller will read from the instruction cache unit to implement the control flow jump;
  • the MOVE instruction is responsible for carrying data of an address in the internal address space of the device to another address in the internal address space of the device.
  • the process is independent of the operation unit and does not occupy the resources of the operation unit during execution.
  • FIG. 5 illustrates an example block diagram of a neural network reverse training process in accordance with an embodiment of the present invention.
  • the output gradient vector input gradient of the previous layer is multiplied by the corresponding activation function derivative to obtain the input data of this layer, and then multiplied by the weight matrix to obtain the output gradient vector.
  • the operation module 6 multiplies the input gradient and the input neuron in the forward operation to calculate the weight update gradient dw, and then uses w, dw, and the weight used to update the weight when updating the gradient dw'. Rate update weight w.
  • the input gradient ([input gradient0,..., input gradient3] in FIG. 5) is the output gradient vector of the n+1th layer, which is first to be the nth layer in the forward operation process.
  • the derivative value ([f'(out0),...,f'(out3)]) in Fig. 5 is multiplied to obtain the input gradient vector of the nth layer, which is completed in the main operation module 5 by the interconnection module 4 is sent to the slave arithmetic module 6, and temporarily stored in the neuron buffer unit 63 of the slave arithmetic module 6. Then, the input gradient vector is multiplied by the weight matrix to obtain an output gradient vector of the nth layer.
  • the i-th slave computing module calculates the product of the i-th scalar in the input gradient vector and the column vector [w_i0,...,w_iN] in the weight matrix, and the resulting output vector is stepwise in the interconnect module 4. The two pairs are added to obtain the final output gradient vector output gradient ([output gradient0,...,outp ut gradient3] in Fig. 5).
  • the jth element of the vector, in_gradient_i is the i-th element of the inverse input n-th layer input gradient vector (ie, the product of input gradient and derivative f' in Figure 5).
  • the input of the nth layer in the forward operation is the data existing at the beginning of the reverse training, and is sent to the slave arithmetic module 6 through the interconnect module 4 and temporarily stored in the neuron buffer unit 63.
  • the slave operation module 6 after completing the calculation of the sum of the output gradient vectors, the i-th scalar of the input gradient vector and the input vector of the n-th layer of the forward operation are multiplied to obtain a gradient vector dw of the updated weight. This update weight.
  • FIG. 6 is an implementation of an artificial neural network reverse training that supports a single layer supporting discrete data representations, according to one embodiment.
  • the flowchart depicts an artificial neural network reverse training process that implements a single layer discrete data representation as shown in FIG. 5 using the apparatus and instruction set of the present invention.
  • Step S1.1 storing the initial instruction into the instruction storage unit 1;
  • Step S1.2 reading an instruction from the instruction storage unit 1;
  • Step S1.3 decoding the above instruction
  • Step S1.4 performing corresponding operations according to the decoded control signal
  • step S1.5 the operation result is written back to the corresponding storage.
  • step S1.1 an initialization IO instruction may be stored for carrying subsequent instructions.
  • the readable instructions include, but are not limited to, a CONFIG instruction, a COMPUTE instruction, an IO instruction, a NOP instruction, a JUMP instruction, and a MOVE instruction.
  • step S1.3 the control signal of the corresponding module is obtained according to the operation type of the instruction (CONFIG, COMPUTE, IO, NOP, JUMP, MOVE, etc.).
  • the decoding obtains the configuration information of the remaining modules.
  • the control signal of the master-slave operation module is obtained by decoding, and the corresponding operations taken by different discrete data are controlled.
  • the control signal of the data access module is decoded.
  • the NOP instruction the actual control signal is not generated, and only the control signal in all the control signal buffer queues of the current device is cleared, and all the instructions before the NOP instruction are all executed.
  • the JUMP instruction a control signal for the jump instruction stream is obtained.
  • the MOVE command a control signal for carrying data inside the device is obtained.
  • step S1.4 the above module 2-6 performs a corresponding operation in accordance with the control signal.
  • the interconnect interconnect module will input the gradient vector. [in0,...,inN] is sent to all slave arithmetic modules, temporarily stored in the neuron cache unit that supports discrete data representation.
  • a corresponding operation is performed according to the binary of the discrete data to calculate a dot product of the corresponding weight vector [w_i0, . . . , w_iN] and the gradient vector.
  • the result obtained is multiplied by the derivative of the weight and the activation function to obtain the final output gradient vector [out0, out1, out2, ..., outN]. Update the weight, offset and other data with the weight gradient vector of the intermediate result.
  • each module writes the result of the operation back to the corresponding cache.
  • the output gradient vector obtained by the main operation module is written back to the storage unit.
  • the updated weights, offsets, etc. are also written back to the storage unit.
  • FIG. 7 is another, more detailed implementation of a single layer artificial neural network reverse training in accordance with one embodiment.
  • the flowchart depicts the process of implementing a single layer neural network reverse training as shown in FIG. 5 using the apparatus and instruction set of the present invention.
  • step S1 an IO instruction is pre-stored at the first address of the instruction cache unit 1.
  • step S2 the operation starts, the controller unit 2 reads the IO instruction from the first address of the instruction cache unit 1, and according to the translated microinstruction, the data access unit 3 reads from the external address space and the single-layer artificial neural network. All instructions related to the training are reversed and cached in the instruction cache unit 1.
  • step S3 the controller unit 2 then reads in the next IO instruction from the instruction cache unit, and according to the translated microinstruction, the data access unit 3 reads all the data required by the main operation module 5 from the external address space to the main operation module 5.
  • the neuron buffer unit 53 the data includes input neuron and activation function derivative values and input gradient vectors in the previous forward operation. This data supports discrete representations, which can be all discrete or partially discrete.
  • step S4 the controller unit 2 then reads in the next IO instruction from the instruction cache unit, and according to the translated microinstruction, the data access unit 3 reads the ownership value data and the weight gradient required from the operation module 6 from the external address space.
  • the data is stored in the weight buffer unit 64 and the weight gradient buffer unit 65 of the corresponding slave arithmetic module 6, respectively. This data supports discrete representations, which can be all discrete or partially discrete.
  • step S5 the controller unit 2 then reads the next CONFIG instruction from the instruction cache unit, and the operation unit configures the value of the internal unit register of the operation unit according to the parameters in the translated microinstruction, including various constants required for the calculation of the layer neural network. , the accuracy of the calculation of this layer, the learning rate when updating the weight, and so on.
  • step S6 the controller unit 2 then reads the next COMPUTE instruction from the instruction cache unit, and according to the translated microinstruction, the main operation module 5 inputs the gradient vector and the input in the forward operation through the interconnection module 4.
  • the incoming neurons are sent to the respective slave arithmetic modules 6, and the input gradient vectors and the input neurons at the time of the forward operation are stored in the neuron buffer unit 63 of the slave arithmetic module 6.
  • step S7 according to the micro-instruction decoded by the COMPUTE instruction, the weight vector (i.e., the partial column of the weight matrix stored by the slave module) is read from the weight buffer unit 64 from the arithmetic unit 61 of the arithmetic module 6 to complete the right.
  • the value vector and the vector multiplication scalar operation of the input gradient vector return the output vector portion and through the interconnection; at the same time, the input gradient vector is multiplied from the input neuron by the operation module 6, and the weight gradient is obtained and stored in the weight gradient buffer unit 65.
  • the custom uses an exclusive OR equivalent to replace the dot product or not. For example, for a 1-bit discrete data representation, 0 represents +1 and 1 represents -1, and the multiplication of the weights is achieved by XORing the sign bits of the data multiplied by the weights.
  • step S8 in the interconnect module 4, the output gradient portions returned from each of the arithmetic modules 6 are added step by step to obtain a complete output gradient vector.
  • step S9 the main operation module 5 obtains the return value of the interconnection module 4, and reads the activation function derivative value in the forward operation from the neuron buffer unit 53 according to the micro-instruction decoded by the COMPUTE instruction, and multiplies the derivative value by the returned value.
  • the vector is output to obtain the input gradient vector of the next layer of reverse training, which is written back to the neuron buffer unit 53.
  • the data of the process can be customized to discretize the data or not.
  • step S10 the controller unit 2 then reads the next COMPUTE instruction from the instruction cache unit, and reads the weight w from the weight buffer unit 64 from the operation module 6 according to the translated microinstruction, and reads from the weight gradient buffer unit.
  • the weight gradient dw of this time and the weight gradient dw' used by the last update weight are used to update the weight w.
  • step S11 the controller unit then reads the next IO instruction from the instruction cache unit, and according to the translated microinstruction, the data access unit 3 stores the output gradient vector in the neuron buffer unit 53 to the external address space specified address, and operates. End.
  • the artificial neural network batch normalization operation steps are similar to the above process.
  • the controller completes the following process.
  • the controller controls the data access unit 3 to read the input gradient data, and then controls the master-slave operation module to determine the gradient mean and the gradient variance of the respective positions according to the batch size or use the set mean variance.
  • the controller then controls the gradient data of the corresponding position minus the gradient mean divided by the gradient variance.
  • the controller controls multiplying the processed data by the learning parameters and adding another learning parameter.
  • the implementation process is similar to that of a single-layer neural network.
  • the next-level operation instruction will calculate the output gradient vector calculated in the main operation module.
  • the above calculation process is performed, and the weight address and the weight gradient address in the instruction are also changed to the address corresponding to the layer.
  • the storage energy consumption and the like of the device and the number of multiplier components are greatly reduced. It can optimize the structure layout on a limited area and improve the calculation speed or performance energy consumption ratio.
  • the invention can be applied to the following (including but not limited to) scenarios: data processing, robots, computers, printers, scanners, telephones, tablets, smart terminals, mobile phones, driving recorders, navigators, sensors, cameras, cloud servers , cameras, camcorders, projectors, watches, earphones, mobile storage, wearable devices and other electronic products; aircraft, ships, vehicles and other types of transportation; televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, Electric lights, gas stoves, range hoods and other household appliances; and including nuclear magnetic resonance instruments, B-ultrasound, electrocardiograph and other medical equipment.
  • the processes or methods depicted in the preceding figures may include hardware (eg, circuitry, dedicated logic, etc.), firmware, software (eg, software embodied on a non-transitory computer readable medium), or both
  • the combined processing logic is executed.
  • the processes or methods have been described above in some order, it should be understood that certain operations described can be performed in a different order. Moreover, some operations may be performed in parallel rather than sequentially.
  • the representation of discrete data it should be understood which data discretization representations and which non-discrete representations can be selected. The spirit of whether the data is discretely represented throughout the entire process.

Abstract

一种支持离散数据表示的用于执行人工神经网络反向训练的装置，包括指令缓存单元（1）、控制器单元（2）、数据访问单元（3）、互联模块（4）、主运算模块（5）、以及多个从运算模块（6）、连续离散转换模块（7）。使用该装置可以实现多层人工神经网络的反向训练。本装置着重体现在对于离散数据的支持上。包括离散数据的存储、运算、连续数据向离散数据的转换上。使用本装置进行人工神经网络反向计算的过程中的权值、神经元值等数据均支持离散形式表示或连续形式表示的。离散数据表示指通过特定的数字来表示数据的存储方式。例如，可以通过00、01、10、11四个数字分别代表数据-1、-1/8、1/8、1四个数字。这种存储方式不同于连续存储方式用二进制的00/01/10/11代表0/1/2/3四个数字。连续离散转换模块提供了将连续数据转换为离散数据的模块。

Description

支持离散数据表示的人工神经网络反向训练装置和方法 技术领域
本发明总体上涉及人工神经网络，具体地涉及一种用于执行人工神经网络反向训练的装置和方法。本发明中的数据支持离散形式表示，并通过离散数据的位操作代替了连续数据的乘法等操作。
背景技术
多层人工神经网络被广泛应用于模式识别、图像处理、函数逼近和优化计算等领域，多层人工网络在近年来由于其较高的识别准确度和较好的可并行性，受到学术界和工业界越来越广泛的关注。
一种支持多层人工神经网络反向训练的已知方法是使用通用处理器。该方法通过使用通用寄存器堆和通用功能部件执行通用指令来支持上述算法。另一种支持多层人工神经网络反向训练的已知方法是使用图形处理器(GPU)。该方法通过使用通用寄存器堆和通用流处理单元执行通用SIMD指令来支持上述算法。
这两种装置在数据存储和运算上都是使用的连续数据。连续数据的存储需要较多的资源，例如一个32位的浮点数据，就需要32个比特位来存储该数据。在连续数据的运算上，所需要的加法器、乘法器等功能部件的实现也较为复杂。
离散数据表示指通过特定的数字来代替不连续的真实数据的存储方式。例如，可以通过0、1、2、3四个数字分别代表真实数字-1、-1/8、1/8、1四个数字。通过这种类似于索引的表示方式，可以用形式上连续的数字代替不连续离散化的真实数据。目前的多层人工神经网络在数据表示上的已知方法是用浮点数或者定点数这样的连续数据表示。因为多层神经网络权值的精度较高和数量较大，连续数据的表示方式带来更大的存储和功耗开销。而通过离散数据表示的方式，可以通过数据按位的异或、移位等运算代替连续数据的乘法运算。从而大大减少乘法器部件的数量。
发明内容
本发明的一个方面提供了一种支持离散数据表示的用于执行人工神经网络 反向训练的装置，包括指令缓存单元、控制器单元、数据访问单元、互联模块、主运算模块、多个从运算模块，其中：
指令缓存单元用于缓存指令；
控制器单元用于从指令缓存单元读取指令，并将该指令译码成控制互联模块、主运算模块、以及从运算模块行为的微指令；
数据访问单元用于从内存向主运算模块和各从运算模块的相应数据缓存单元中写离散表示或连续表示的数据或从所述数据缓存单元向内存读离散表示或连续表示的数据；
在每层神经网络反向训练开始计算的阶段，主运算模块通过互联模块向所有的从运算模块传输本层的输入梯度向量，在从运算模块的计算过程完成后，互联模块逐级将各从运算模块的输出梯度向量部分和两两相加得到本层的输出梯度向量；
主运算模块用于在每一层的计算过程中，利用本层的输出梯度向量完成后续计算，当输入数据是离散数据与连续数据的混合数据时，从运算模块针对不同离散数据采取预先设置的相应计算方式；以及
每个从运算模块利用相同的输入梯度向量和各自的离散或连续的权值数据，并行地计算出相应的输出梯度向量部分和，当输入数据是离散数据与连续数据的混合数据时，从运算模块针对不同离散数据采取预先设置的相应计算方式。
可选地，离散数据表示指用特定的离散数字代替真实的连续数据的表示方式。
可选地，多个从运算模块利用相同的输入梯度向量并行地计算出各自权值的梯度并使用计算得到的各自权值的梯度来更新各自的权值数据。
可选地，主运算模块将每一层的输出梯度向量与下一层的激活函数求导值对位相乘，作为下一层的输入梯度向量。
可选地，互联模块构成主运算模块和所述多个从运算模块之间的连续或离散化的数据通路，可以实现成不同的互联拓扑。在一种实施方式中，互联模块具有H树型的结构，H树是由多个节点构成的二叉树通路，每个节点将上游的数据同样地发给下游的两个节点，将下游的两个节点返回的连续或离散化的数据相加，并返回给上游的节点。
可选地，主运算模块包括运算单元、数据依赖关系判断单元和支持离散数据表示的神经元缓存单元，其中：
支持离散数据表示的神经元缓存单元用于缓存主运算模块在计算过程中用到的离散或连续的输入数据和输出数据；
运算单元完成主运算模块的各种运算功能，当输入数据是离散数据与连续数据的混合数据时，针对不同离散数据采取预先设置的相应计算方式；
数据依赖关系判断单元是运算单元读写神经元缓存单元的端口，保证对神经元缓存单元中离散或连续的数据读写不存在一致性冲突，并且负责从神经元缓存单元读取输入梯度向量通过互联模块发送给从运算模块；以及
来自互联模块的输出梯度向量被发送到运算单元。
可选地，每个从运算模块包括运算单元、数据依赖关系判定单元、支持离散数据表示的神经元缓存单元、支持离散数据表示的权值缓存单元和支持离散数据表示的权值梯度缓存单元，其中：
运算单元接收控制器单元发出的微指令并进行算数逻辑运算，当输入数据是离散数据与连续数据的混合数据时，针对不同离散数据采取预先设置的相应计算方式；
数据依赖关系判断单元负责计算过程中对支持离散数据表示的神经元缓存单元、支持离散数据表示的权值缓存单元和支持离散数据表示的权值梯度缓存单元的读写操作，保证对支持离散数据表示的神经元缓存单元、支持离散数据表示的权值缓存单元和支持离散数据表示的权值梯度缓存单元的读写不存在一致性冲突；
支持离散数据表示的神经元缓存单元缓存输入支持离散表示的梯度向量数据中与该从运算模块相对应的标量数据以及该从运算模块计算得到的输出梯度向量部分和；
支持离散数据表示的权值缓存单元缓存该从运算模块在计算过程中需要的离散或连续表示的权值数据，对于每一个从运算模块，都只存储权值矩阵中与该从运算模块所存储的标量数据相对应的列；以及
支持离散数据表示的权值梯度缓存单元缓存相应从运算模块在更新权值过程中需要的权值梯度数据，每个从运算模块存储的权值梯度数据与其存储的离散 或连续表示的权值数据相对应。
可选地，通过以下方式保证读写不存在一致性冲突：判断尚未执行的微指令与正在执行过程中的微指令的数据之间是否存在依赖关系，如果不存在，允许该条微指令立即发射，否则需要等到该条微指令所依赖的所有微指令全部执行完成后该条微指令才允许被发射。
可选地，运算单元包括运算决定单元和混合数据运算单元，当输入数据是混合数据时，运算决定单元根据其中的离散数据决定应对该混合数据执行何种操作，然后，混合数据运算单元根据运算决定单元的决定结果，执行相应操作。
可选地，运算单元还包括离散数据运算单元和连续数据运算单元中的至少一个，以及数据类型判断单元，当输入数据全是离散数据时，由离散数据运算单元根据输入的离散数据通过查表执行相应操作，当输入数据全是连续数据时，由连续数据运算单元执行相应操作。
可选地，该装置还包括连续离散转换单元，连续离散转换单元包括预处理模块、距离计算模块、和判断模块，假设使用M(M＝2m，m≥1)个离散数据，令这些离散数据分别对应于预定区间[-zone，zone]内的M个数值，其中：
预处理模块对于输入的连续数据x使用clip(-zone，zone)运算进行预处理，得到区间[-zone，zone]内的预处理数据y，其中，如果x≤-zone则y＝-zone，如果x≥zone则y＝zone，如果-zone＜x＜zone，则预处理数据y＝x；
距离计算模块计算预处理数据y与上述各数值之间的距离；以及
判断模块82基于该距离计算并输出离散数据。
可选地，预定区间[-zone，zone]是[-1，1]或[-2，2]；并且/或者M个数值的绝对值是2的幂的倒数；并且/或者判断模块执行：并且/或者输出与该预处理数据y距离最近的数值所对应的离散数据，如果有两个数值与该预处理数据距离相等，则输出二者中任一个所对应的离散数据；并且/或者计算预处理数据y分别到距离最近的两个数值中任一个的归一化概率，将这两个数值中任一个所对应的归一化概率与随机数生成模块生成的(0，1)之间的随机数z比较，如果该z小于该概率则输出该离散数据，否则输出另一离散数据。
本发明的另一个方面提供了一种使用上述装置执行单层人工神经网络反向训练的方法。通过提供的指令集，控制器控制读入运算时的输入神经元和激活函 数导数值以及输入梯度，之后读入权值数据和权值梯度数据，之后读入神经网络训练精度、学习率等常数。这些数据可以采用离散数据表示或不采用。之后主从运算模块以及互联模块完成权值更新操作。特别对于离散数据表示的数据，在进行乘法操作时，根据离散数据的数值，通过对相关数据的位操作代替了乘法运算。例如权值数据用1比特的离散数据表示，0代表+1，1代表-1，通过对与权值相乘数据的符号位异或，实现了对权值的乘法运算。
本发明的另一个方面提供了一种使用上述装置支持人工神经网络批归一化运算(Batch Normalization)的方法。通过提供的指令集，控制器控制数据访问单元读入输入的数据，之后控制主从运算模块根据batch大小求出各自位置的均值以及方差或使用设定好的均值方差。之后控制器控制对应位置的输入数据减去均值除以方差。最后控制器控制用处理后的数据与学习参数相乘后加上另一个学习参数。
本发明的另一方面提供了一种使用上述装置执行多层人工神经网络反向训练的方法。其实现过程与单层神经网络类似，当上一层人工神经网络执行完毕后，下一层的运算指令会将主运算模块中计算出的输出梯度向量作为下一层训练的输入梯度向量进行如上的计算过程，指令中的权值地址和权值梯度地址也会变更至本层对应的地址。
附图说明
为了更完整地理解本发明及其优势，现在将参考结合附图的以下描述，其中：
图1示出了根据本发明实施例的用于执行支持离散数据表示的人工神经网络反向训练的装置的整体结构的示例框图。
图2示意性示出了根据本发明实施例的用于执行支持离散数据表示的人工神经网络反向训练的装置中H树模块(互联模块的一种实施方式)的结构。
图3示出了根据本发明实施例的用于执行支持离散数据表示的人工神经网络反向训练的装置中主运算模块结构的示例框图。
图4示出了根据本发明实施例的用于执行支持离散数据表示的人工神经网络反向训练的装置中从运算模块结构的示例框图。
图5示出了根据本发明实施例的神经网络反向训练过程的示例框图。
图6示出了根据本发明实施例的离散数据表示的神经网络反向训练过程的示例框图
图7示出了根据本发明实施例的单层人工神经网络运算的流程图。
图8示出了针对离散数据的运算单元示例结构。
图9示出了连续数据和离散数据转化的连续离散转化模块的示例结构。
在所有附图中，相同的装置、部件、单元等使用相同的附图标记来表示。
具体实施方式
根据结合附图对本发明示例性实施例的以下详细描述，本发明的其它方面、优势和突出特征对于本领域技术人员将变得显而易见。
在本发明中，术语“包括”和“含有”及其派生词意为包括而非限制；术语“或”是包含性的，意为和/或。
在本说明书中，下述用于描述本发明原理的各种实施例只是说明，不应该以任何方式解释为限制发明的范围。参照附图的下述描述用于帮助全面理解由权利要求及其等同物限定的本发明的示例性实施例。下述描述包括多种具体细节来帮助理解，但这些细节应认为仅仅是示例性的。因此，本领域普通技术人员应认识到，在不背离本发明的范围和精神的情况下，可以对本文中描述的实施例进行多种改变和修改。此外，为了清楚和简洁起见，省略了公知功能和结构的描述。此外，贯穿附图，相同参考数字用于相似功能和操作。
根据本发明实施例的支持离散数据表示的多层人工神经网络的反向训练，包括两层或者两层以上的多个神经元。对于每一层来说，首先对输入梯度向量进行加权求和计算出本层的输出梯度向量。该输出梯度向量乘以下一层在正向运算时的激活函数的导数值可以得到下一层的输入梯度向量。将输入梯度向量与正向运算时的输入神经元对位相乘得到本层权值的梯度，然后可以根据所得到的本层权值的梯度来更新本层的权值。此过程中的数据可以采用离散表示的数据也可以采用连续数据。
对于离散数据表示的输入神经元向量或离散数据表示的权值向量的点积运算，本装置支持将点积运算转换为数据的移位、取非、异或等位运算。对于数据的表示方式，本装置支持数据离散表示或连续表示，用户可以自定义哪一个层的 哪些数据采用离散表示形式或连续表示，并且可以根据具体需要自定义离散数据的位数，从而代替表示的连续数据的个数，例如设定为1比特、2比特、3比特等位数的离散数据，分别可以表示2个、4个、8个连续数据。
图1示出了根据本发明实施例的用于执行支持离散数据表示的人工神经网络反向训练的装置的整体结构的示例框图。如图1所示，该装置包括指令缓存单元1、控制器单元2、数据访问单元3、互联模块4、主运算模块5、多个从运算模块6。根据本发明的实施例，该装置还可以包括连续离散转换模块7。指令缓存单元1、控制器单元2、数据访问单元3、H树模块4、主运算模块5和从运算模块6、连续离散转换模块7均可以通过硬件电路(包括但不限于FPGA、CGRA、专用集成电路ASIC、模拟电路和忆阻器等)实现。本装置支持将连续数据转换为离散数据以及离散数据的存储和运算。
指令缓存单元1通过数据访问单元3读入指令并缓存读入的指令。
控制器单元2从指令缓存单元1中读取指令，将指令译成控制其他模块行为的微指令并发送给其他模块如数据访问单元3、主运算模块5和从运算模块6等。
数据访问单元3能够访存外部地址空间(例如内存或其他存储设备)，直接向装置内部的各个缓存单元读写数据，完成数据的加载和存储。该数据是离散表示的或非离散表示的。该单元用来设计可以读取离散表示的数据
互联模块4用于连接主运算模块和从运算模块，可以实现成不同的互联拓扑(如树状结构、环状结构、网格状结构、分级互联、总线结构等)。
图2示意性示出了互联模块4的一种实施方式：H树结构。H树模块4构成主运算模块5和多个从运算模块6之间的数据通路，并具有H树型的结构。H树是由多个节点构成的二叉树通路，每个节点将上游的数据同样地发给下游的两个节点，将下游的两个节点返回的数据进行合并，并返回给上游的节点。例如，在神经网络反向运算过程中，下游两个节点返回的向量会在当前节点相加成一个向量并返回给上游节点。在每层人工神经网络开始计算的阶段，主运算模块5内的输入梯度通过H树模块4发送给各个从运算模块6；当从运算模块6的计算过程完成后，每个从运算模块6输出的输出梯度向量部分和会在H树模块4中逐级两两相加，即对所有输出梯度向量部分和求和，作为最终的输出梯度向量。
图3示出了根据本发明实施例的用于执行人工神经网络反向训练的装置中主 运算模块5的结构的示例框图。如图3所示，主运算模块5包括运算单元51、数据依赖关系判断单元52和支持离散数据表示的神经元缓存单元53。
支持离散数据表示的神经元缓存单元53用于缓存主运算模块5在计算过程中用到的输入数据和输出数据。
运算单元51完成主运算模块的各种运算功能。对于运算因子全是离散数据的情况，可以通过查表实现离散数据与离散数据的加减乘除运算。例如2位的离散数据，可以表示4个连续数据值。对于4个连续数据共有4*4＝16种组合。对于每种加减乘除运算的操作，可以制作并维护该4*4的索引表，通过索引表找到对应的计算值。4种运算共需要4张4*4的索引表。
对于运算因子包含离散数据和连续数据的情况，可以针对不同离散数据，为加、减、乘、除运算预先设定相应的位操作。例如，可以采取按位异或后乘2的相应位次幂之后累加求和的方式代替离散数据与连续数据的点积运算。例如，对于乘法操作，乘法因子数据如果存在离散表示的，可以通过离散数据索引相应的操作(例如，对相应数据的按位异或、取非、移位等操作)代替和该离散数据表示的连续数据的乘法操作，从而减少了乘法器部件数量。例如对于连续数据与离散数据的乘法操作，-1/2乘以16。传统的乘法器部件会将-1/2与16直接做乘法。在运算单元51中，由于离散数据的可能性较少，可以通过查找索引这样一种开关判断的方法代替了运算单元的功能。例如，可以规定-1/2的离散数据表示方法为01。如果一个运算因子是-1/2，则运算单元51接收到的离散数据为01。运算单元51便采用离散数据01对应的操作。通过对于16的8位定点数表示00010000符号位取反，向右移1位得到10001000，十进制表示为-8。对于除法操作，16除以-2。其中16是连续数据，-2是离散数据。如果规定离散数据-2二进制表示为10。运算单元便采用离散数据10对应的除法操作。通过对16的8位定点数表示0001000右移1位之后符号位取反得到10001000，十进制表示为-8得到结果。加法和减法操作与上述过程类似。根据离散数据的二进制作为一个索引，索引到按位左移、右移、异或等操作。经过该操作后实现了与离散数据表示的真实数据的相加或者相减操作。
数据依赖关系判断单元52是运算单元51读写神经元缓存单元53的端口，同时能够保证对神经元缓存单元53中数据的读写不存在一致性冲突。具体地， 数据依赖关系判断单元52判断尚未执行的微指令与正在执行过程中的微指令的数据之间是否存在依赖关系，如果不存在，允许该条微指令立即发射，否则需要等到该条微指令所依赖的所有微指令全部执行完成后该条微指令才允许被发射。例如，所有发往数据依赖关系单元52的微指令都会被存入数据依赖关系单元52内部的指令队列里，在该队列中，读指令的读取数据的范围如果与队列位置靠前的写指令写数据的范围发生冲突，则该指令必须等到所依赖的写指令被执行后才能够执行。同时，数据依赖关系判断单元52也负责从神经元缓存单元53读取输入梯度向量通过互联模块4发送给从运算模块6，而从运算模块6的输出数据通过互联模块4直接发送给运算单元51。控制器单元2输出的指令发送给运算单元51和依赖关系判断单元52，来控制其行为。
图4示出了根据本发明实施例的用于执行支持离散数据表示的人工神经网络反向训练的装置中从运算模块6的结构的示例框图。如图4所示，每个从运算模块6包括运算单元61、数据依赖关系判定单元62、支持离散数据表示的神经元缓存单元63、支持离散数据表示的权值缓存单元64和支持离散数据表示的权值梯度缓存单元65。
运算单元61接收控制器单元2发出的微指令并进行算数逻辑运算。对于运算因子全是离散数据的情况，可以通过查表实现离散数据与离散数据的加减乘除运算。例如2位的离散数据，可以表示4个连续数据值。对于4个连续数据共有4*4＝16种组合。对于每种加减乘除运算的操作，可以制作并维护该4*4的索引表，通过索引表找到对应的计算值。4种运算共需要4张4*4的索引表。
对于运算因子包含离散数据和连续数据的情况，可以针对不同离散数据，为加、减、乘、除运算预先设定相应的位操作。例如，可以采取按位异或后乘2的相应位次幂之后累加求和的方式代替离散数据与连续数据的点积运算。例如，对于乘法操作，乘法因子数据如果存在离散表示的，可以通过离散数据索引相应的操作(例如，对相应数据的按位异或、取非、移位等操作)代替和该离散数据表示的连续数据的乘法操作，从而减少了乘法器部件数量。例如对于连续数据与离散数据的乘法操作，-1/2乘以16。传统的乘法器部件会将-1/2与16直接做乘法。在运算单元51中，由于离散数据的可能性较少，可以通过查找索引这样一种开关判断的方法代替了运算单元的功能。例如，可以规定-1/2的离散数据表示方法 为01。如果一个运算因子是-1/2，则运算单元51接收到的离散数据为01。运算单元51便采用离散数据01对应的操作。通过对于16的8位定点数表示00010000符号位取反，向右移1位得到10001000，十进制表示为-8。对于除法操作，16除以-2。其中16是连续数据，-2是离散数据。如果规定离散数据-2二进制表示为10。运算单元便采用离散数据10对应的除法操作。通过对16的8位定点数表示0001000右移1位之后符号位取反得到10001000，十进制表示为-8得到结果。加法和减法操作与上述过程类似。根据离散数据的二进制作为一个索引，索引到按位左移、右移、异或等操作。经过该操作后实现了与离散数据表示的真实数据的相加或者相减操作。
数据依赖关系判断单元62负责计算过程中对缓存单元的读写操作。数据依赖关系判断单元62保证对缓存单元的读写不存在一致性冲突。具体地，数据依赖关系判断单元62判断尚未执行的微指令与正在执行过程中的微指令的数据之间是否存在依赖关系，如果不存在，允许该条微指令立即发射，否则需要等到该条微指令所依赖的所有微指令全部执行完成后该条微指令才允许被发射。例如，所有发往数据依赖关系单元62的微指令都会被存入数据依赖关系单元62内部的指令队列里，在该队列中，读指令的读取数据的范围如果与队列位置靠前的写指令写数据的范围发生冲突，则该指令必须等到所依赖的写指令被执行后才能够执行。
支持离散数据表示的神经元缓存单元63缓存输入梯度向量数据中与该从运算模块6相对应的标量数据以及该从运算模块6计算得到的输出梯度向量部分和。
支持离散数据表示的权值缓存单元64缓存该从运算模块6在计算过程中需要的权值数据。该数据根据用户定义可以是离散数据或离散数据。对于每一个从运算模块，都只会存储权值矩阵中与该从运算模块6所存储的标量数据相对应的列。
权值梯度缓存单元65缓存相应从运算模块在更新权值过程中需要的权值梯度数据。每一个从运算模块6存储的权值梯度数据与其存储的权值数据相对应。
从运算模块6实现每层人工神经网络反向训练计算输出梯度向量的过程中可以并行的前半部分以及权值的更新。以人工神经网络全连接层(MLP)为例，过 程为out_gradient＝w*in_gradient，其中以离散数据表示的权值矩阵w和输入梯度向量in_gradient的乘法可以划分为不相关的并行计算子任务，out_gradient与in_gradient是列向量，每个从运算模块只计算in_gradient中相应的部分标量元素与权值矩阵w对应的列的乘积，得到的每个输出向量都是最终结果的一个待累加的部分和，这些部分和在互联模块中逐级两两相加得到最后的结果。所以计算过程变成了并行的计算部分和的过程和后面的累加的过程。每个从运算模块6计算出输出梯度向量的部分和，所有的部分和在互联模块4中完成求和运算得到最后的输出梯度向量。每个从运算模块6同时将输入梯度向量和正向运算时每层的输出值相乘，计算出权值的梯度，以更新本从运算模块6存储的权值。正向运算和反向训练是神经网络算法的两个主要过程，神经网络要训练(更新)网络中的权值，首先需要计算输入向量在当前权值构成的网络中的正向输出，这是正向过程，然后根据输出值与输入向量本身的标注值之间的差值，反向逐层训练(更新)每层的权值。在正向计算过程中会保存每一层的输出向量以及激活函数的导数值，这些数据是反向训练过程所需要的，所以在反向训练开始时，这些数据已经保证存在。正向运算中每层的输出值是反向运算开始时已有的数据，可以通过数据访问单元缓存在主运算模块中并通过互联模块发送给从运算模块。主运算模块5基于输出梯度向量进行后续计算，例如将输出梯度向量乘以正向运算时的激活函数的导数得到下一层的输入梯度值。正向运算时的激活函数的导数是在反向运算开始时已有的数据，可以通过数据访问单元缓存在主运算模块中。
图8示出了运算单元的结构框图，其可用于主运算模块中的运算单元51或从运算模块中的运算单元61。运算过程中输入数据可以是离散数据或连续数据。数据类型判断单元71判断输入数据全是连续数据、全是离散数据或是既包含连续数据又包含离散数据的混合数据。当输入数据全是连续数据时，连续数据运算单元72执行相应运算。
当输入数据全是离散数据时，离散数据运算单元73执行相应运算。对于运算因子全是离散数据的情况，可以通过查表实现离散数据与离散数据的加减乘除运算。例如2位的离散数据，可以表示4个连续数据值。对于4个连续数据共有4*4＝16种组合。对于每种加减乘除运算的操作，我们制作并维护该4*4的索引表，通过索引表找到对应的计算值。4种运算共需要4张4*4的索引表。
当输入数据是混合数据时，运算决定单元74根据其中的离散数据决定应对其执行何种操作。可以针对不同的离散数据分别预先设置相应操作。然后，混合数据运算单元75根据运算决定单元74的决定结果，执行相应操作。对于运算因子包含离散数据和连续数据的情况，可以针对不同离散数据，为加、减、乘、除运算预先设定相应的位操作。例如，可以采取按位异或后乘2的相应位次幂之后累加求和的方式代替离散数据与连续数据的点积运算。例如，对于乘法操作，乘法因子数据如果存在离散表示的，可以通过离散数据索引相应的操作(例如，对相应数据的按位异或、取非、移位等操作)代替和该离散数据表示的连续数据的乘法操作，从而减少了乘法器部件数量。例如对于连续数据与离散数据的乘法操作，-1/2乘以16。传统的乘法器部件会将-1/2与16直接做乘法。在运算单元51中，由于离散数据的可能性较少，可以通过查找索引这样一种开关判断的方法代替了运算单元的功能。例如，可以规定-1/2的离散数据表示方法为01。如果一个运算因子是-1/2，则运算单元51接收到的离散数据为01。运算单元51便采用离散数据01对应的操作。通过对于16的8位定点数表示00010000符号位取反，向右移1位得到10001000，十进制表示为-8。对于除法操作，16除以-2。其中16是连续数据，-2是离散数据。如果规定离散数据-2二进制表示为10。运算单元便采用离散数据10对应的除法操作。通过对16的8位定点数表示0001000右移1位之后符号位取反得到10001000，十进制表示为-8得到结果。加法和减法操作与上述过程类似。根据离散数据的二进制作为一个索引，索引到按位左移、右移、异或等操作。经过该操作后实现了与离散数据表示的真实数据的相加或者相减操作。
图9示出了连续离散转换模块7的结构示意图。用户可以定义采用该模块将连续数据转换为离散数据或不采用。将连续数据输入连续离散转换模块7，输出离散数据。该连续离散转换模块7包括预处理模块81、距离计算模块82、和判断模块82。假设使用M(M＝2m，m≥1)个离散数据，令这些离散数据分别对应于预定区间内的M个数值。根据本发明的实施例，该区间可以是关于零对称的区间[-zone，zone]，例如[-1，1]或[-2，2]。根据本发明的实施例，该M个数值的绝对值可以是2的幂的倒数。以预处理模块81对于输入的连续数据x使用clip(-zone，zone)运算进行预处理，得到区间[-zone，zone]内的预处理数据y。其中， 如果x≤-zone则y＝-zone，如果x≥zone则y＝zone，如果-zone＜x＜zone，则预处理数据y＝x。距离计算模块82计算预处理数据y与上述各数值之间的距离。判断模块82基于该距离计算并输出离散数据。例如，判断模块82可以输出与该预处理数据距离最近的数值所对应的离散数据。如果有两个数值与该预处理数据y距离相等，则输出二者中任一个所对应的离散数据。或者，判断模块82可以计算预处理数据y分别到距离最近的两个数值中任一个的归一化概率，将这两个数值中任一个所对应的归一化概率与随机数生成模块82生成的(0，1)之间的随机数z比较，如果该z小于该概率则输出该离散数据，否则输出另一离散数据。例如，用户定义产生二元离散数据，则M＝2。预处理模块81对于输入的任意连续数据x执行运算clip(-1，1)得到预处理数据y。然后，距离计算模块82计算预处理数据y与数值-1、1之间的距离D1、D2，假设D1＞D2。判断模块83可以输出D1对应的离散数据-1。或者，判断模块83可以计算y分别到距离最近的两个数值(由于本例中仅有两个数值，所以距离最近的两个数值即为-1和1)中任一个的归一化概率，其中y到-1的概率为P1＝D2/(D1+D2)，y到-2的概率为P2＝D1/(D1+D2)，将归一化概率P1与随机数生成模块82生成的(0，1)之间的随机数z比较，如果z小于概率P1，则输出离散数据-1，否则输出另一离散数据1。或者，将归一化概率P2与随机数z比较，如果z小于概率P2，则输出离散数据1，否则输出另一离散数据-1。根据本发明的实施例，判断模块以二进制数据的形式输出离散数据，离散数据与二进制数据之间的关系可预先设定，例如通过查找表设定。例如，可以用0代表1，1代表-1。将得到的离散数据存储回内存中。等待主从运算模块中的运算单元使用，产生相应的操作。
反向训练过程中的权值数据可以用离散数据表示的，上述过程中对于权值的乘法运算根据离散数据的存储方式采用异或、取非、位移等方式代替。例如对于1比特的离散数据表示，0代表+1，1代表-1，通过对与权值相乘数据的符号位异或，实现了对权值的乘法运算。
根据本发明实施例，还提供了在前述装置上执行人工神经网络反向运算的指令集。指令集中包括CONFIG指令、COMPUTE指令、IO指令、NOP指令、JUMP指令和MOVE指令等，其中：
CONFIG指令在每层人工神经网络计算开始前配置当前层计算需要的各种常 数；
COMPUTE指令完成每层人工神经网络的算术逻辑计算；
IO指令实现从外部地址空间读入计算需要的输入数据以及在计算完成后将数据存回至外部空间该数据支持离散化表示的；
NOP指令负责清空当前装置内部所有微指令缓存队列中的微指令，保证NOP指令之前的所有指令全部指令完毕。NOP指令本身不包含任何操作；
JUMP指令负责控制器将要从指令缓存单元读取的下一条指令地址的跳转，用来实现控制流的跳转；
MOVE指令负责将装置内部地址空间某一地址的数据搬运至装置内部地址空间的另一地址，该过程独立于运算单元，在执行过程中不占用运算单元的资源。
图5示出了根据本发明实施例的神经网络反向训练过程的示例框图。计算输出梯度向量的过程为out_gradient＝w*in_gradient，其中权值矩阵w和输入梯度向量in_gradient的矩阵向量乘法可以划分为不相关的并行计算子任务，每个从运算模块6计算出输出梯度向量的部分和，所有的部分和在互联模块4中完成求和运算得到最后的输出梯度向量。图5中上一层的输出梯度向量input gradient乘以对应的激活函数导数得到本层的输入数据，再与权值矩阵相乘得到输出梯度向量。计算权值更新梯度的过程为dw＝x*in_gradient，其中每个从运算模块6计算本模块对应部分的权值的更新梯度。从运算模块6将输入梯度和正向运算时的输入神经元相乘计算出权值更新梯度dw，然后使用w、dw和上一次更新权值时使用的权值更新梯度dw’根据指令设置的学习率更新权值w。
参考图5所示，input gradient(图5中的[input gradient0，...，input gradient3])是第n+1层的输出梯度向量，该向量首先要与正向运算过程中第n层的导数值(图5中的[f’(out0)，...，f’(out3)])相乘，得到第n层的输入梯度向量，该过程在主运算模块5中完成，由互联模块4发往从运算模块6，暂存在从运算模块6的神经元缓存单元63中。然后，输入梯度向量与权值矩阵相乘得到第n层的输出梯度向量。在这个过程中，第i个从运算模块计算输入梯度向量中第i个标量和权值矩阵中列向量[w_i0，...，w_iN]的乘积，得到的输出向量在互联模块4中逐级两两相加得到最后的输出梯度向量output gradient(图5中的[output gradient0，...，outp ut gradient3])。
同时，从运算模块6还需要更新本模块中存储的权值，计算权值更新梯度的过程为dw_ij＝x_j*in_gradient_i，其中x_j是正向运算时第n层的输入(即第n-1层的输出)向量的第j个元素，in_gradient_i是反向运算第n层的输入梯度向量(即图5中input gradient与导数f’的乘积)的第i个元素。正向运算时第n层的输入是在反向训练开始时就存在的数据，通过互联模块4送往从运算模块6并暂存在神经元缓存单元63中。则，在从运算模块6中，在完成输出梯度向量部分和的计算后，将输入梯度向量第i个标量和正向运算第n层的输入向量相乘，得到更新权值的梯度向量dw并据此更新权值。
图6是示出根据一个实施例的单层支持离散数据表示的人工神经网络反向训练的一种实施方法。该流程图描述利用本发明的装置和指令集实现图5所示的一种单层离散数据表示的人工神经网络反向训练过程。
步骤S1.1，将初始指令存放到指令存储单元1中；
步骤S1.2，从指令存储单元1中读取一条指令；
步骤S1.3，对上述指令进行译码；
步骤S1.4，根据译码得到的控制信号，进行相应操作；
步骤S1.5，将操作结果写回到相应存储中。
在步骤S1.1中，可以存入初始化IO指令，用于搬运后续指令。
在步骤S1.2中，可读取的指令包括但不限于CONFIG指令、COMPUTE指令、IO指令、NOP指令、JUMP指令和MOVE指令等。
在步骤S1.3中，根据指令的操作类型(CONFIG，COMPUTE，IO，NOP，JUMP，MOVE等)译码得到相应模块的控制信号。对于CONFIG指令，译码得到配置其余模块的配置信息。对于COMPUTE指令，译码得到主从运算模块的控制信号，控制不同离散数据采取的对应操作。对于IO指令，译码得到数据访问模块的控制信号。对于NOP指令，不产生实际控制信号，只用于清空当前装置内部所有控制信号缓存队列中的控制信号，保证NOP指令之前的所有指令全部执行完毕。对于JUMP指令，得到跳转指令流的控制信号。对于MOVE指令，得到在装置内部搬运数据的控制信号。
在步骤S1.4中，上述模块2-6根据控制信号执行相应操作。以执行支持离散数据表示的神经网络反向的COMPUTE指令为例，互连互联模块将输入梯度向量 [in0，…，inN]发送给所有的从运算模块，暂存在支持离散数据表示的神经元缓存单元中。对于第i个从运算模块，根据离散数据的二进制采取相应的操作计算相应相应的权值向量[w_i0，…，w_iN]与梯度向量的点积。得到的结果与权值和激活函数的导数相乘，得到最后的输出梯度向量[out0，out1，out2，…，outN]。用中间结果的权值梯度向量更新权值、偏置等数据。
在步骤S1.5中，各个模块将操作结果写回到相应缓存中。以执行离散数据表示的神经网络反向的运算为例，主运算模块得到的输出梯度向量被写回到存储单元。更新后的权值、偏置等数据也被写回到存储单元。
图7是示出根据一个实施例的单层人工神经网络反向训练的另一种更详细的实施方法。该流程图描述利用本发明的装置和指令集实现图5所示的一种单层神经网络反向训练的过程。
在步骤S1，在指令缓存单元1的首地址处预先存入一条IO指令。
在步骤S2，运算开始，控制器单元2从指令缓存单元1的首地址读取该条IO指令，根据译出的微指令，数据访问单元3从外部地址空间读取与该单层人工神经网络反向训练有关的所有指令，并将其缓存在指令缓存单元1中。
在步骤S3，控制器单元2接着从指令缓存单元读入下一条IO指令，根据译出的微指令，数据访问单元3从外部地址空间读取主运算模块5需要的所有数据至主运算模块5的神经元缓存单元53，所述数据包括之前正向运算时的输入神经元和激活函数导数值以及输入梯度向量。该数据支持离散表示，可以是全部离散或部分离散。
在步骤S4，控制器单元2接着从指令缓存单元读入下一条IO指令，根据译出的微指令，数据访问单元3从外部地址空间读取从运算模块6需要的所有权值数据和权值梯度数据，并分别存储到相应的从运算模块6的权值缓存单元64和权值梯度缓存单元65。该数据支持离散表示，可以是全部离散或部分离散。
在步骤S5，控制器单元2接着从指令缓存单元读入下一条CONFIG指令，运算单元根据译出的微指令里的参数配置运算单元内部寄存器的值，包括该层神经网络计算需要的各种常数，本层计算的精度设置、更新权值时的学习率等。
在步骤S6，控制器单元2接着从指令缓存单元读入下一条COMPUTE指令，根据译出的微指令，主运算模块5通过互联模块4将输入梯度向量和正向运算时的输 入神经元发给各从运算模块6，所述输入梯度向量和正向运算时的输入神经元存至从运算模块6的神经元缓存单元63。
在步骤S7，根据COMPUTE指令译出的微指令，从运算模块6的运算单元61从权值缓存单元64读取权值向量(即该从运算模块存储的权值矩阵的部分列)，完成权值向量和输入梯度向量的向量乘标量运算，将输出向量部分和通过互联返回；同时从运算模块6将输入梯度向量与输入神经元相乘，得到权值梯度存至权值梯度缓存单元65。对于离散数据表示的，自定义采用异或等位运算代替点积运算或不采用。例如对于1比特的离散数据表示，0代表+1，1代表-1，通过对与权值相乘数据的符号位异或，实现了对权值的乘法运算。
在步骤S8，在互联模块4中，各从运算模块6返回的输出梯度部分和被逐级两两相加得到完整的输出梯度向量。
在步骤S9，主运算模块5得到互联模块4的返回值，根据COMPUTE指令译出的微指令，从神经元缓存单元53读取正向运算时的激活函数导数值，将导数值乘以返回的输出向量，得到下一层反向训练的输入梯度向量，将其写回至神经元缓存单元53。该过程的数据可以自定义采用离散化表示数据或不采用。
在步骤S10，控制器单元2接着从指令缓存单元读入下一条COMPUTE指令，根据译出的微指令，从运算模块6从权值缓存单元64读取权值w，从权值梯度缓存单元读取本次的权值梯度dw和上一次更新权值使用的权值梯度dw’，更新权值w。
在步骤S11，控制器单元接着从指令缓存单元读入下一条IO指令，根据译出的微指令，数据访问单元3将神经元缓存单元53中的输出梯度向量存至外部地址空间指定地址，运算结束。
对于人工神经网络批归一化运算(Batch Normalization)运算步骤与上述过程相仿。通过提供的指令集，控制器完成以下过程。控制器控制数据访问单元3读入输入的梯度数据，之后控制主从运算模块根据batch大小求出各自位置的梯度均值以及梯度方差或使用设定好的均值方差。之后控制器控制对应位置的梯度数据减去梯度均值除以梯度方差。最后控制器控制用处理后的数据与学习参数相乘后加上另一个学习参数。
对于多层人工神经网络，其实现过程与单层神经网络类似，当上一层人工神经网络执行完毕后，下一层的运算指令会将主运算模块中计算出的输出梯度向量 作为下一层训练的输入梯度向量进行如上的计算过程，指令中的权值地址和权值梯度地址也会变更至本层对应的地址。
通过采用用于执行人工神经网络反向训练的装置和指令集，解决了CPU和GPU运算性能不足，前端译码开销大的问题。有效提高了对多层人工神经网络正向运算的支持。
通过采用针对多层人工神经网络反向训练的专用片上缓存，充分挖掘了输入神经元和权值数据的重用性，避免了反复向内存读取这些数据，降低了内存访问带宽，避免了内存带宽成为多层人工神经网络正向运算性能瓶颈的问题。
通过采用离散数据表示的方法，相较于浮点数、定点数等连续数据表示方法，大大较少了装置的存储能耗等开销、以及乘法器部件数量。可以在有限的面积上优化结构布局，提高运算速度或性能能耗比等指标。
该发明可以应用于以下(包括但不限于)场景中：数据处理、机器人、电脑、打印机、扫描仪、电话、平板电脑、智能终端、手机、行车记录仪、导航仪、传感器、摄像头、云端服务器、相机、摄像机、投影仪、手表、耳机、移动存储、可穿戴设备等各类电子产品；飞机、轮船、车辆等各类交通工具；电视、空调、微波炉、冰箱、电饭煲、加湿器、洗衣机、电灯、燃气灶、油烟机等各类家用电器；以及包括核磁共振仪、B超、心电图仪等各类医疗设备。
前面的附图中所描绘的进程或方法可通过包括硬件(例如，电路、专用逻辑等)、固件、软件(例如，被具体化在非瞬态计算机可读介质上的软件)，或两者的组合的处理逻辑来执行。虽然上文按照某些顺序操作描述了进程或方法，但是，应该理解，所描述的某些操作能以不同顺序来执行。此外，可并行地而非顺序地执行一些操作。并且对于离散数据的表示问题，应该理解可以选择哪些数据离散化表示、哪些不离散表示。数据是否离散表示的精神贯穿于整个运算过程中。
在前述的说明书中，参考其特定示例性实施例描述了本发明的各实施例。显然，可对各实施例做出各种修改，而不背离所附权利要求所述的本发明的更广泛的精神和范围。相应地，说明书和附图应当被认为是说明性的，而不是限制性的。

Claims (15)

  1. 一种支持离散数据表示的用于执行人工神经网络反向训练的装置，包括指令缓存单元、控制器单元、数据访问单元、互联模块、主运算模块、多个从运算模块，其中：
    指令缓存单元用于缓存指令；
    控制器单元用于从指令缓存单元读取指令，并将该指令译码成控制互联模块、主运算模块、以及从运算模块行为的微指令；
    数据访问单元用于从内存向主运算模块和各从运算模块的相应数据缓存单元中写离散表示或连续表示的数据或从所述数据缓存单元向内存读离散表示或连续表示的数据；
    在每层神经网络反向训练开始计算的阶段，主运算模块通过互联模块向所有的从运算模块传输本层的输入梯度向量，在从运算模块的计算过程完成后，互联模块逐级将各从运算模块的输出梯度向量部分和两两相加得到本层的输出梯度向量；
    主运算模块用于在每一层的计算过程中，利用本层的输出梯度向量完成后续计算，当输入数据是离散数据与连续数据的混合数据时，从运算模块针对不同离散数据采取预先设置的相应计算方式；以及
    每个从运算模块利用相同的输入梯度向量和各自的离散或连续的权值数据，并行地计算出相应的输出梯度向量部分和，当输入数据是离散数据与连续数据的混合数据时，从运算模块针对不同离散数据采取预先设置的相应计算方式。
  2. 根据权利要求1所述的装置，其中，离散数据表示指用特定的离散数字代替真实的连续数据的表示方式。
  3. 根据权利要求1所述的装置，其中，多个从运算模块利用相同的输入梯度向量并行地计算出各自权值的梯度并使用计算得到的各自权值的梯度来更新各自的权值数据。
  4. 根据权利要求1所述的装置，其中，主运算模块将每一层的输出梯度向量与下一层的激活函数求导值对位相乘，作为下一层的输入梯度向量。
  5. 根据权利要求1所述的装置，其中，互联模块构成主运算模块和所述多个从运算模块之间的连续或离散化的数据通路。
  6. 根据权利要求1所述的装置，其中，主运算模块包括运算单元、数据依赖关系判断单元和支持离散数据表示的神经元缓存单元，其中：
    支持离散数据表示的神经元缓存单元用于缓存主运算模块在计算过程中用到的离散或连续的输入数据和输出数据；
    运算单元完成主运算模块的各种运算功能，当输入数据是离散数据与连续数据的混合数据时，针对不同离散数据采取预先设置的相应计算方式；
    数据依赖关系判断单元是运算单元读写神经元缓存单元的端口，保证对神经元缓存单元中离散或连续的数据读写不存在一致性冲突，并且负责从神经元缓存单元读取输入梯度向量通过互联模块发送给从运算模块；以及
    来自互联模块的输出梯度向量被发送到运算单元。
  7. 根据权利要求1所述的装置，其中，每个从运算模块包括运算单元、数据依赖关系判定单元、支持离散数据表示的神经元缓存单元、支持离散数据表示的权值缓存单元和支持离散数据表示的权值梯度缓存单元，其中：
    运算单元接收控制器单元发出的微指令并进行算数逻辑运算，当输入数据是离散数据与连续数据的混合数据时，针对不同离散数据采取预先设置的相应计算方式；
    数据依赖关系判断单元负责计算过程中对支持离散数据表示的神经元缓存单元、支持离散数据表示的权值缓存单元和支持离散数据表示的权值梯度缓存单元的读写操作，保证对支持离散数据表示的神经元缓存单元、支持离散数据表示的权值缓存单元和支持离散数据表示的权值梯度缓存单元的读写不存在一致性冲突；
    支持离散数据表示的神经元缓存单元缓存输入支持离散表示的梯度向量数据中与该从运算模块相对应的标量数据以及该从运算模块计算得到的输出梯度向量部分和；
    支持离散数据表示的权值缓存单元缓存该从运算模块在计算过程中需要的离散或连续表示的权值数据，对于每一个从运算模块，都只存储权值矩阵中与该从运算模块所存储的标量数据相对应的列；以及
    支持离散数据表示的权值梯度缓存单元缓存相应从运算模块在更新权值过程中需要的权值梯度数据，每个从运算模块存储的权值梯度数据与其存储的离散 或连续表示的权值数据相对应。
  8. 根据权利要求6或7所述的装置，其中，通过以下方式保证读写不存在一致性冲突：判断尚未执行的微指令与正在执行过程中的微指令的数据之间是否存在依赖关系，如果不存在，允许该条微指令立即发射，否则需要等到该条微指令所依赖的所有微指令全部执行完成后该条微指令才允许被发射。
  9. 根据权利要求6或7所述的装置，其中运算单元包括运算决定单元和混合数据运算单元，当输入数据是混合数据时，运算决定单元根据其中的离散数据决定应对该混合数据执行何种操作，然后，混合数据运算单元根据运算决定单元的决定结果，执行相应操作。
  10. 根据权利要求9所述的装置，其中运算单元还包括离散数据运算单元和连续数据运算单元中的至少一个，以及数据类型判断单元，当输入数据全是离散数据时，由离散数据运算单元根据输入的离散数据通过查表执行相应操作，当输入数据全是连续数据时，由连续数据运算单元执行相应操作。
  11. 根据权利要求1所述的装置，还包括连续离散转换单元，连续离散转换单元包括预处理模块、距离计算模块、和判断模块，假设使用M(M＝2m，m≥1)个离散数据，令这些离散数据分别对应于预定区间[-zone，zone]内的M个数值，其中：
    预处理模块对于输入的连续数据x使用clip(-zone，zone)运算进行预处理，得到区间[-zone，zone]内的预处理数据y，其中，如果x≤-zone则y＝-zone，如果x≥zone则y＝zone，如果-zone＜x＜zone，则预处理数据y＝x；
    距离计算模块计算预处理数据y与上述各数值之间的距离；以及
    判断模块82基于该距离计算并输出离散数据。
  12. 根据权利要求11所述的装置，其特征在于以下任意一项或多项：
    预定区间[-zone，zone]是[-1，1]或[-2，2]；
    M个数值的绝对值是2的幂的倒数；或者
    判断模块执行：
    输出与该预处理数据y距离最近的数值所对应的离散数据，如果有两个数值与该预处理数据距离相等，则输出二者中任一个所对应的离散数据；或者
    计算预处理数据y分别到距离最近的两个数值中任一个的归一化概率，将这两个数值中任一个所对应的归一化概率与随机数生成模块生成的(0，1)之间的随机数z比较，如果该z小于该概率则输出该离散数据，否则输出另一离散数据。
  13. 一种使用根据权利要求1-12中的任一项的装置执行单层人工神经网络反向训练的方法，包括：
    数据访问单元从外部地址空间读取与该单层人工神经网络反向训练有关的所有人工神经网络运算指令，并将其缓存在指令缓存单元中；
    连续离散转换模块从外部地址空间读取该层神经网络需要转换的连续数据转换为离散数据后存储回外部地址空间；
    数据访问单元从外部地址空间读取主运算模块需要的所有离散或连续表示的数据至主运算模块的神经元缓存单元，连续或离散表示的数据包括：输入梯度向量、以及之前正向运算时的激活函数导数值和输入神经元；
    数据访问单元从外部地址空间读取从运算模块需要的离散或连续表示的所有权值数据和权值梯度数据，并分别存储到相应的从运算模块的支持离散数据表示的权值缓存单元和权值梯度缓存单元；
    主运算模块和从运算模块各自中的运算单元根据译出的微指令里的参数配置该运算单元内部寄存器的值，所述参数包括该层神经网络计算需要的离散或连续表示的各种常数、本层计算的精度设置参数、和更新权值时的学习率；
    主运算模块通过互联模块将输入梯度向量和正向运算时的离散或连续表示的输入神经元发给各从运算模块，所述输入梯度向量和正向运算时的离散或连续表示的输入神经元被存至从运算模块的神经元缓存单元；
    从运算模块的运算单元从支持离散数据表示的权值缓存单元读取离散或连续表示的权值向量，完成离散或连续表示的权值向量和输入梯度向量的向量乘标量运算，用离散数据相应的操作代替连续数据的标量乘标量运算；将输出向量部分和通过互联模块返回；同时从运算模块将离散或连续表示的输入梯度向量与输入神经元相乘，得到权值梯度存至权值梯度缓存单元，其中，权值向量是该从运算模块存储的权值矩阵的部分列；
    在互联模块中，各从运算模块返回的输出梯度部分和被逐级两两相加得到完 整的输出梯度向量；
    主运算模块得到互联模块的返回值，从神经元缓存单元读取正向运算时的离散或连续表示的激活函数导数值，将导数值乘以返回的输出梯度向量，得到下一层反向训练的输入梯度向量，将其写回至神经元缓存单元；
    从运算模块从支持离散数据表示的权值缓存单元读取离散或连续表示的权值w，从权值梯度缓存单元读取本次的权值梯度dw和上一次更新权值使用的权值梯度dw’，更新权值w；
    数据访问单元将神经元缓存单元中的输出梯度向量存至外部地址空间指定地址。
  14. 一种使用权利要求1-12中的任一项装置执行批归一化反向运算的方法：
    数据访问单元从外部地址空间读取与该批归一化反向训练有关的所有人工神经网络运算指令，并将其缓存在指令缓存单元中；
    连续离散转换模块从外部地址空间读取该层神经网络需要转换的连续数据转换为离散数据后存储回外部地址空间；
    数据访问单元从外部地址空间读取主运算模块需要的所有离散或连续表示的数据至主运算模块的神经元缓存单元，连续或离散表示的数据包括：输入梯度向量、以及之前正向运算时的激活函数导数值和输入神经元；
    数据访问单元从外部地址空间读取从运算模块需要的离散或连续表示的所有权值数据和权值梯度数据，并分别存储到相应的从运算模块的支持离散数据表示的权值缓存单元和权值梯度缓存单元；
    主运算模块和从运算模块各自中的运算单元根据译出的微指令里的参数配置该运算单元内部寄存器的值，所述参数包括该层神经网络计算需要的离散或连续表示的各种常数、本层计算的精度设置参数、和更新权值时的学习率；
    主运算模块通过互联模块将输入梯度向量和正向运算时的离散或连续表示的输入神经元发给各从运算模块，所述输入梯度向量和正向运算时的离散或连续表示的输入神经元被存至从运算模块的神经元缓存单元；
    从运算模块将离散或连续表示的输入梯度向量求出相应的梯度的和值，传送入互联模块；
    在互联模块中，各从运算模块返回的输出梯度部分和被逐级两两相加得到完 整的输出梯度和值向量；
    主运算模块得到互联模块的返回值，求出相应梯度位置的均值和标准差数据，之后用梯度数据减去均值除以标准差得到下一层反向训练的输入梯度向量，将其写回至神经元缓存单元；
    数据访问单元将神经元缓存单元中的输出梯度向量存至外部地址空间指定地址。
  15. 一种执行多层人工神经网络反向训练的方法，包括：
    针对每一层，执行根据权利要求13或14中的一项所述的方法，其中：
    当上一层人工神经网络执行完毕后，使用主运算模块中计算出的下一层训练的输入梯度向量，针对所述下一层再次执行根据权利要求13或14中的所述一项所述的方法。
PCT/CN2016/079443 2016-04-15 2016-04-15 支持离散数据表示的人工神经网络反向训练装置和方法 WO2017177446A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US16/093,958 US20190130274A1 (en) 2016-04-15 2016-04-15 Apparatus and methods for backward propagation in neural networks supporting discrete data
PCT/CN2016/079443 WO2017177446A1 (zh) 2016-04-15 2016-04-15 支持离散数据表示的人工神经网络反向训练装置和方法
EP16898264.3A EP3444758B1 (en) 2016-04-15 2016-04-15 Discrete data representation-supporting apparatus and method for back-training of artificial neural network
US16/182,439 US20190080241A1 (en) 2016-04-15 2018-11-06 Apparatus and methods for backward propagation in neural networks supporting discrete data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2016/079443 WO2017177446A1 (zh) 2016-04-15 2016-04-15 支持离散数据表示的人工神经网络反向训练装置和方法

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US16/093,958 A-371-Of-International US20190130274A1 (en) 2016-04-15 2016-04-15 Apparatus and methods for backward propagation in neural networks supporting discrete data
US16/182,439 Continuation-In-Part US20190080241A1 (en) 2016-04-15 2018-11-06 Apparatus and methods for backward propagation in neural networks supporting discrete data

Publications (1)

Publication Number Publication Date
WO2017177446A1 true WO2017177446A1 (zh) 2017-10-19

Family

ID=60041320

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/079443 WO2017177446A1 (zh) 2016-04-15 2016-04-15 支持离散数据表示的人工神经网络反向训练装置和方法

Country Status (3)

Country Link
US (1) US20190130274A1 (zh)
EP (1) EP3444758B1 (zh)
WO (1) WO2017177446A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110689109A (zh) * 2018-07-04 2020-01-14 三星电子株式会社 神经网络方法和装置
CN111160543A (zh) * 2017-12-14 2020-05-15 中科寒武纪科技股份有限公司 集成电路芯片装置及相关产品
CN111291874A (zh) * 2018-12-06 2020-06-16 黄朝宗 卷积神经网络处理器及其数据处理方法

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107086910B (zh) * 2017-03-24 2018-08-10 中国科学院计算技术研究所 一种针对神经网络处理的权重加解密方法和系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5631469A (en) * 1996-04-15 1997-05-20 The United States Of America As Represented By The Secretary Of The Army Neural network computing system for pattern recognition of thermoluminescence signature spectra and chemical defense
CN101625735A (zh) * 2009-08-13 2010-01-13 西安理工大学 基于ls-svm分类和回归学习递归神经网络的fpga实现方法
CN101833691A (zh) * 2010-03-30 2010-09-15 西安理工大学 一种基于fpga的最小二乘支持向量机串行结构实现方法
CN103150596A (zh) * 2013-02-22 2013-06-12 百度在线网络技术（北京）有限公司 一种反向传播神经网络dnn的训练系统
CN105144203A (zh) * 2013-03-15 2015-12-09 谷歌公司 信号处理系统

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110992935B (zh) * 2014-09-12 2023-08-11 微软技术许可有限责任公司 用于训练神经网络的计算系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5631469A (en) * 1996-04-15 1997-05-20 The United States Of America As Represented By The Secretary Of The Army Neural network computing system for pattern recognition of thermoluminescence signature spectra and chemical defense
CN101625735A (zh) * 2009-08-13 2010-01-13 西安理工大学 基于ls-svm分类和回归学习递归神经网络的fpga实现方法
CN101833691A (zh) * 2010-03-30 2010-09-15 西安理工大学 一种基于fpga的最小二乘支持向量机串行结构实现方法
CN103150596A (zh) * 2013-02-22 2013-06-12 百度在线网络技术（北京）有限公司 一种反向传播神经网络dnn的训练系统
CN105144203A (zh) * 2013-03-15 2015-12-09 谷歌公司 信号处理系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3444758A4 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111160543A (zh) * 2017-12-14 2020-05-15 中科寒武纪科技股份有限公司 集成电路芯片装置及相关产品
CN111160543B (zh) * 2017-12-14 2023-08-29 中科寒武纪科技股份有限公司 集成电路芯片装置及相关产品
CN110689109A (zh) * 2018-07-04 2020-01-14 三星电子株式会社 神经网络方法和装置
CN110689109B (zh) * 2018-07-04 2024-04-12 三星电子株式会社 神经网络方法和装置
CN111291874A (zh) * 2018-12-06 2020-06-16 黄朝宗 卷积神经网络处理器及其数据处理方法
CN111291874B (zh) * 2018-12-06 2023-12-01 神盾股份有限公司 卷积神经网络处理器及其数据处理方法

Also Published As

Publication number Publication date
EP3444758B1 (en) 2021-11-17
US20190130274A1 (en) 2019-05-02
EP3444758A1 (en) 2019-02-20
EP3444758A4 (en) 2019-12-25

Similar Documents

Publication Publication Date Title
CN109358900B (zh) 支持离散数据表示的人工神经网络正向运算装置和方法
CN107807819B (zh) 一种支持离散数据表示的用于执行人工神经网络正向运算的装置及方法
CN107301454B (zh) 支持离散数据表示的人工神经网络反向训练装置和方法
CN109219821B (zh) 运算装置和方法
WO2017185387A1 (zh) 一种用于执行全连接层神经网络正向运算的装置和方法
WO2017185391A1 (zh) 一种用于执行卷积神经网络训练的装置和方法
WO2018192500A1 (zh) 处理装置和处理方法
KR102470264B1 (ko) 완전연결층 신경망 역방향 트레이닝 실행용 장치와 방법
WO2017177442A1 (zh) 支持离散数据表示的人工神经网络正向运算装置和方法
US20190332945A1 (en) Apparatus and method for compression coding for artificial neural network
WO2017124641A1 (zh) 用于执行人工神经网络反向训练的装置和方法
WO2017185386A1 (zh) 一种用于执行卷积神经网络正向运算的装置和方法
CN111260025B (zh) 用于执行lstm神经网络运算的装置和运算方法
WO2017124642A1 (zh) 用于执行人工神经网络正向运算的装置和方法
WO2017185347A1 (zh) 用于执行循环神经网络和lstm运算的装置和方法
CN110163357B (zh) 一种计算装置及方法
TWI827432B (zh) 計算裝置、機器學習運算裝置、組合處理裝置、神經網絡芯片、電子設備、板卡、及計算方法
WO2018120016A1 (zh) 用于执行lstm神经网络运算的装置和运算方法
WO2017177446A1 (zh) 支持离散数据表示的人工神经网络反向训练装置和方法
WO2018113790A1 (zh) 一种人工神经网络运算的装置及方法
WO2017185248A1 (zh) 用于执行人工神经网络自学习运算的装置和方法
CN111178492B (zh) 计算装置及相关产品、执行人工神经网络模型的计算方法
CN109993276B (zh) 用于执行人工神经网络反向训练的装置和方法
CN113570053A (zh) 一种神经网络模型的训练方法、装置以及计算设备
CN111198714A (zh) 重训练方法及相关产品

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16898264

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2016898264

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2016898264

Country of ref document: EP

Effective date: 20181115