JPH04237388A - Neuro processor - Google Patents

Abstract

PURPOSE:To offer a neuro processor capable of rapidly executing the forward propagation and learning of a hierarchical structure neural network. CONSTITUTION:The neuro processor is provided with two pairs of input signal memories 1A, 1B and teaching signal memories 2A, 2B. In parallel with calculation using input signal data stored in the memory 1A and teaching signal data stored in the memory 2A, input signal data and teaching signal data to be used for the succeeding calculation are inputted and respectively written in the memories 1B, 2B. In the succeeding step, the succeeding data are written in the memories 1A, 2A during the calculation of the data of the memories 1B, 2B. Since data input work is executed in parallel with calculation, the processing time can be shortened.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neuroprocessor for performing forward propagation and learning processing of a hierarchical neural network at high speed.

0002

[Prior Art] Neural networks model and imitate the functions of neurons in the human brain.
The aim is to create a new computer that is good at recognition, association, optimization problems, speech synthesis, etc., which conventional so-called Neumann computers were weak at.

[0003] Neural networks have various structures, such as those with a hierarchical structure in which neurons are arranged in layers, and those with an interconnected structure in which all neurons are interconnected. Among them, hierarchical networks can be easily trained using a learning algorithm called a backpropagation algorithm, and are thought to be widely applicable to control, character recognition, image recognition, image processing, etc. .

FIG. 4 shows an example of a network with a hierarchical structure. In FIG. 4, 101 is a neuron arranged in each layer, 102 is a connection between neurons called a synapse, 103 is an input layer, 104 is an intermediate layer, and 105 is an output layer. Although FIG. 4 shows an example of a three-layer network, it is also possible to create a network with a hierarchical structure of four or more layers by providing a plurality of intermediate layers. Hereinafter, the explanation will be limited to three-layer networks, but networks with four or more layers will be treated in exactly the same way. Also, Figure 4
Although the number of neurons in each layer is illustrated as 3, 2, and 3, the same effect can be obtained even if the number of neurons in each layer is increased or decreased. Input signal data to the neural network is provided to each neuron in the input layer 103. Then, the signal propagates in the order of the input layer, intermediate layer, and output layer, and the signal of the neuron in the output layer 105 becomes the output of the network. This normal propagation that propagates in the order of input layer, intermediate layer, and output layer is called forward propagation.

FIG. 5 is a diagram showing the function of neurons. In FIG. 5, 101, 107, 108, and 109 are neurons, 102 is a synapse, and 106 is a characteristic function f of the neuron. Each neuron receives the output of the neuron in the previous layer via synapses. Each synapse has a value called a connection weight, and the output value of the neuron in the previous layer is multiplied by that connection weight value, and the result is given to the next neuron. The weights of synaptic connections have different values for each synapse. For example, Figure 5
The i-th neuron 108 of the l-th layer and j of the l+1-th layer
The synapse with the i-th neuron 101 has a connection weight wlji, and the synapse with the i-th neuron 1 of the l-th layer has a connection weight wlji.
The result of multiplying the output Oli of 08 by the value of the connection weight is wlji×Oli, which is the jth neuron 1 of the l+1st layer.
Give to 01. Then, the neuron 101 adds all the inputs given from all the synapses connected to it, applies the characteristic function 106 of the neuron to the addition result, and converts the function value into the output Ol+1 of the neuron 101.
, j. This can be expressed as the following formula.

[0006]

[Math 1]

[0007] Therefore, forward propagation in the case of a three-layer network is calculated using the following procedure. First, the output of all hidden layer neurons is

0
008]

[Math 2]

Calculate according to: The output O1i of a neuron in the input layer is input signal data given to that neuron. Next, use that result to calculate the output for all output layer neurons,

0010

[Math 3]

Calculate according to the following. Next, a learning method using a backpropagation algorithm for the three-layer hierarchical network shown in FIG. 4 will be described. In the backpropagation algorithm, a pair of input and its ideal output is prepared, and the weights of synaptic connections are modified so that the difference between the actual output and the ideal output for that input is reduced. This ideal output is usually called a teacher signal. Below, a specific calculation method for the three-layer network will be explained step by step.

1. Compute the actual output by forward propagation. 2. A value (hereinafter simply referred to as delta) corresponding to the error of each neuron in the output layer is calculated according to the following equation.

[0013]

[Math 4]

δlj is the delta for the j-th neuron in the l-th layer, tj is the teacher signal for the j-th neuron in the output layer, and g is the delta for the j-th neuron in the l-th layer.

[0015]

[Math 5]

The characteristic function f of the neuron is expressed as
is the differential coefficient of Since a monotonically non-decreasing function is used as the characteristic function f of the neuron, the differential coefficient of the characteristic function can be expressed as a function of the function value of the characteristic function, as shown in Equation 5.

3. The amount of modification of the weight of the synaptic connection between the intermediate layer and the output layer is calculated according to the following equation, and the weight is modified.

[0018]

[Math 6]

η is a correction coefficient. 4. Compute the delta for the neurons in the hidden layer according to the following equation:

[0020]

[Math 7]

5. The amount of modification of the weight of the synaptic connection between the intermediate layer and the input layer is calculated according to the following formula, and the weight is modified.

[0022]

[Math. 8]

In actual learning, the above-described procedure is performed for a plurality of pairs of inputs and teacher signals, and is repeated many times. That is, if the steps 1 to 5 above are called one learning, the total number of learning times is (number of input and teacher signal pairs used for learning) x (number of repetitions).

The above are the steps for forward propagation and learning calculations of a hierarchical neural network. However, in actual processing, input signal data must be provided in forward propagation calculations before starting the calculation. Furthermore, in learning calculations, input signal data and teacher signal data must be given. Therefore, conventionally, when forward propagation and learning processing is performed, the following steps are taken.

[0025] First, the neuroprocessor receives input signal data and teacher signal data from the outside, and once writes these data into the memory. The process of receiving input signal data and teacher signal data from the outside and writing those data into memory will be referred to as data input hereinafter. Next, actual calculations are performed using the input signal data and teacher signal data on the memory. Such a two-step processing method is normally used. FIG. 3A is a diagram showing the processing flow of this conventional method. First, data is input, and then calculations are performed. Normally, different data are processed in succession, so after the calculation, the next data is input, as shown in FIG. 3(a). Repeat this operation many times.

[0026]

[Problems to be Solved by the Invention] Generally, neural network calculations involve a very large amount of calculation, and therefore require a large amount of processing time. In particular, since learning needs to be repeated many times, the processing time becomes enormous. therefore,
A neuroprocessor that can perform high-speed processing is desired.

However, if you input data and then perform calculations as in the conventional neuroprocessor, 1
It takes (time required to input data) + (time required for calculation) to complete the process. That is, in addition to the time required for actual calculation, additional time is required for inputting data.

The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a neuroprocessor that can perform forward propagation and learning processing of a hierarchical neural network at high speed.

[0029]

[Means for Solving the Problems] In order to solve the above-mentioned problems, the neuroprocessor of the present invention is provided with two sets of input signal memory and teacher signal memory. While a calculation is being performed using the input signal data and the teacher signal data in the teacher signal memory, input the input signal data and teacher signal data to be used in the next calculation in parallel, and calculate the other set. writes to the input signal memory and teacher signal memory.

[0030]

[Operation] With the above configuration, the data input operation is performed in parallel with the calculation, so that the time required for data input is seemingly unnecessary. That is, the overall processing time is reduced by the time conventionally required for inputting data. Therefore, processing speed can be increased.

[0031]

[Embodiments] Hereinafter, embodiments of the present invention will be explained with reference to the drawings.

FIG. 1 is a block diagram of the neuroprocessor of the present invention. 1A and 1B in Fig. 1 are input signal memories, 2
A and 2B are teacher signal memories, 1A and 2A are one set, and 1B and 2B are another set. Hereinafter, the set of 1A and 2A will be called memory A, and the set of 1B and 2B will be called memory B. 3 in FIG. 1 is a neural network arithmetic processing unit where forward propagation and learning calculations are performed. 4 in FIG. 1 is an input section, and input signal data and teacher signal data are input through this input section. 5 in Figure 1
and 6 are switches, and switch 5 switches whether data input from input section 4 is written to memory A or memory B. The switch 6 switches whether the data used by the neural network calculation processing section 3 is read from the memory A or the memory B.

FIG. 2 is a diagram showing how the switches and memory are used in this embodiment. The operation of the neuroprocessor of this embodiment will be described below with reference to FIG.

First, in the first step a, the switch 5
By switching to A, the data input from the input section 4 is written into the memory A. In the next step b, the switch 6 is switched to A, the data written in the memory A in step a is read out, and the neural network arithmetic processing unit 3 performs calculation. At the same time, the switch 5 is switched to B, and the data input from the input section 4 is written into the memory B. In the next step c, the switch 6 is switched to B, the data written in the memory B in step b is read out, and the neural network arithmetic processing unit 3 performs calculations. At the same time, turn switch 5 to
The data input from the input section 4 is written into the memory A. From now on, repeat these operations.

FIG. 3A is a diagram showing the flow of this process. Data input and calculations are performed in parallel, and calculations are performed using the data input in the previous step. As is clear from the comparison with the conventional method shown in FIG. 3(a), one process is completed in two steps in the conventional method shown in FIG. 3(a), whereas the method of the present invention ,
One step completes one process.

As can be seen from the above explanation of the operation, this neuroprocessor is provided with two sets of input signal memory and teacher signal memory, and the input signal data and the teacher signal on one set of input signal memory are While performing calculations using the teacher signal data on the memory, in parallel,
Input signal data and teacher signal data to be used in the next calculation are input and written to the other set of input signal memory and teacher signal memory. As a result, the data input work is performed in parallel with the calculation, so that the time required for data input is seemingly unnecessary. That is, the overall processing time is reduced by the time conventionally required for inputting data.

Note that the present invention is independent of the structure of the neural network arithmetic processing section, and is effective for any neural network arithmetic processing section.

[0038]

As is clear from the above embodiments, according to the present invention, two sets of input signal memory and teacher signal memory are provided, and data input and calculation are performed in parallel. It is possible to provide a neuroprocessor that apparently requires no time for inputting data and can perform forward propagation and learning processing of a hierarchical neural network at high speed.

[Brief explanation of the drawing]

FIG. 1 is a configuration diagram of a neuroprocessor according to an embodiment of the present invention.

[Figure 2] Diagram showing how to use switches and memory

[Figure 3
] Diagram showing the flow of processing

[Figure 4] Diagram of hierarchical neural network

[Figure 5]
Diagram showing how neurons work

[Explanation of symbols]

1A, 1B Input signal memory 2A, 2B Memory for teacher signal 3 Neural network calculation processing section 4 Input section 5, 6 Switch 101 Neuron 102 Synapse 103 Input layer 104 Middle class 105 Output layer 106 Characteristic function of neuron 107-109 Neuron

Claims (2)

[Claims] Claim 1: A neural network calculation processing unit; 2.
one input signal memory, and while forward propagation calculations are being performed using the input signal data on one input signal memory, the input signal data used in the next forward propagation calculation is provided in parallel. A neuroprocessor whose job is to input data and write it to memory for another input signal. 2. A neural network arithmetic processing unit, two sets of an input signal memory and a teacher signal memory, the input signal data on one set of the input signal memory and the teacher signal on the teacher signal memory. While performing learning calculations using data, in parallel, input signal data and teacher signal data to be used in the next learning calculation, and store them in the memory for the other set of input signals. and a neuroprocessor that writes to the memory for the teacher signal.