JPH0784975A - Information processor and learning arithmetic processing method - Google Patents

Abstract

PURPOSE:To learn fast learning with high precision by preventing the learning from being delayed since digit omissions are caused as the learning advances because of the small number of bits of a weight value and an inverse propagation error as to a neuro computer which performs fixed point and single- precision arithmetic. CONSTITUTION:For the learning of the neuro computer, the weight value is stored with double precision; and single-precision arithmetic is performed by the output value and inverse propagation error of a neuron 1 are calculated and double-precision arithmetic is performed at the time of addition for correcting the weight value. As the learning advances, the decimal point position of the inverse propagation error is shifted if certain conditions are met. In another way, plural microprograms having inverse propagation errors differing in decimal point position are prepared and a proper program is selected and executed according to a problem to be learnt.

Description

Detailed Description of the Invention

[0001]

The present invention relates to voice recognition, image processing,
In a neuro computer used for control, etc., the present invention relates to arithmetic processing in a neuron in a neuro computer that performs neural network operation by fixed point and single precision arithmetic processing. More specifically, by using each arithmetic unit used in the single precision arithmetic processing The present invention relates to an information processing device and a high-accuracy learning calculation processing method that enable higher-precision learning calculation processing.

[0002]

2. Description of the Related Art Neural networks are applied to information processing such as recognition and knowledge processing. Neural networks can self-organize by learning if they are given inputs and expected output values. Therefore, it does not require a program and its application is expected.

As one of learning algorithms, "back propagation" introduced on pages 115 to 121 of Nikkei Electronics 1987. 8.10 (NO.427) is known. The back propagation will be described below.

FIG. 2 shows a model of a neuron that constitutes a neural network. A neuron inputs the input x _i from another neuron. Then, each input x _i is weighted by the weight value w _ij , and the internal energy u _i which is the sum of the weights is calculated. This value is converted by, for example, a sigmoid function f, and the output x _i is output. here,
The weight value w _ij is a weight value for the input neuron i in the neuron j.

[0005]

[Equation 1]

[0006]

[Equation 2]

A neural net can be constructed by connecting a plurality of neurons. Then, the weight value w
It is possible to perform various information processing by changing _ij .

FIG. 3 shows a hierarchical neural network. Although this figure is composed of three layers of an input layer, an intermediate layer, and an output layer, it may be configured to have a plurality of intermediate layers. A mode in which information is processed and output in the forward direction with respect to the input, and an expected output value for the given input is given from the outside, and the weight value w _ij is corrected from the output layer side to the input layer side. It has a mode to operate in the opposite direction. The modified value Δw _ij of the weight value w _ij at that time is

[0009]

[Equation 3]

[0010] Here, n is the number of learnings, η, α
Are constants, which are called learning coefficient and moment coefficient, respectively. Also, the function δ is determined as follows.

In the neuron k in the output layer, if the expected output value is t _k ,

[0012]

[Equation 4]

Given by, and in the neuron j of the intermediate layer,

[0014]

[Equation 5]

And decide. Here, the function f'is a derivative of the function f.

The correction value Δw is determined as described above, and the weight value w is corrected. Then, the above learning is repeated until the output value of the output layer reaches the expected value.

In general, the research on the elucidation of the neural network and its application are carried out by software simulation on a sequential computer. However, in order to learn a large number of large-scale neural nets each composed of, for example, thousands or tens of thousands of neurons, enormous calculation time is required. In addition, learning often requires a large number of iterations. For this reason, the capabilities of large-scale neural networks are not well understood.

On the other hand, there is an attempt to calculate a neural network at high speed by using dedicated hardware. One of them, "Basic design of fully digital neuro WSI",
1989 Proceedings of IEICE Spring National Convention 7
Figure 4 shows the method announced on pages 291 to 292 of the separate volume SD-1-5. In FIG. 4, 8 is a neuron calculation unit, 9
Is a bus. Each neuron calculation unit 8 is connected by a bus.

One of the neuron calculation units 8 is selected and its output value is output to the bus 9 one after another. Each neuron calculation unit 8 reads the weight value for the output neuron from the memory, weights it, and cumulatively adds the products. When outputting to the bus 9, the value is converted by the sigmoid function of the equation 1 and output. The weighting circuit is operated in time division. If each neuron calculation unit 8 outputs the same to the bus 9, all can perform the calculation of equation 2. Hereinafter, this is referred to as the time division bus coupling method.

FIG. 5 shows a configuration diagram of a general-purpose learning type neurocomputer using such a time division bus. The neuro computer 10 is composed of a plurality of neuro boards 12 that perform neural network operations and a control board 11. The control board 11 is composed of a neural network control device 13, a control storage 15, and a global memory 16.

The neural network control device 13 is
The neuro computer 10 is controlled by decoding and executing the micro program code stored in the control storage 15. In addition, the neural network control device 13 is a control storage control circuit 1
4a, an external bus is provided as a communication means between the workstation 18 and the neurocomputer 10, and a neuron control circuit 14c for controlling this external bus and a global memory control circuit 14d for storing the values of the input layer neurons and learning coefficients are constructed. It In addition, the external bus is a memory,
The CPU is connected to a SCSI board, which in turn is connected to a workstation 18 having a microgenerator that generates microprogram code.

The neurochip 17 shown in FIG. 5 is composed of a plurality of neurons corresponding to the brain cells of an organism. A block diagram of one neuron in the neurochip 17 is shown in FIG. Input / output of data to / from the neuron 1 is performed via the input / output bus. Further, an instruction for the neuron 1 is input to the neuron 1 via the instruction bus and transmitted to each logic circuit via the control circuit. As a storage device for each data, there are register file 2, RAM3a, RAM3
There is b.

Flip-flop (FF) groups for performing multi-stage pipeline processing are FF4a, FF4b, F
It is composed of five F4c, FF4d, and FF4e. The buses in the neuron 1 are 7 of A, B, C, D, E, F and G.
It consists of a book bus.

In the dedicated hardware of the neural network, in addition to the time-division bus connection method, a fixed-point operation method is adopted for high-speed operation. In addition, as described above, in hardware, cumulative addition of the output of another neuron and the weight value is important. Among the arithmetic units involved in this, the multiplier 5 is one of the circuits occupying a large area in the chip. Therefore, in order to reduce the cost of the chip and increase the parallelism by inserting many neuron calculation units in one chip, it is necessary to limit the number of input bits of the multiplier 5 and reduce the chip area. Accordingly, the hardware limits the input of the multiplier 5 to 16 bits and 10 bits, respectively. The ALU 6 is set to 32 bits because the chip area does not become larger than that of the multiplier even if the number of input bits is increased.

The number of bits of each variable in the hardware is 16 bits for the weight value according to the input of the multiplier 5, the output value of the broadcast neuron, the back propagation error, the learning coefficient, etc. is 10 bits. Weight value, output value,
The format of the back propagation error is as shown in FIG.

Next, the operation of the back propagation learning using the neuro computer 10 will be described with reference to FIG. 1 and in correspondence with the processing flow of FIG.

In step 30, the output value of the neuron in each layer is calculated. FIG. 8 shows details of step 30. In step 33, the output value of each neuron in the previous layer is multiplied by the weight value of the synapse connected to that neuron, and cumulative addition is performed to calculate the internal energy. That is, each neuron in the previous layer broadcasts the output value every machine cycle. Each neuron in the layer takes in the output value broadcast by the pipeline processing, reads out the weight value stored in the RAM 3a, RAM 3b, and performs cumulative addition with the output value. Step 34
Then, the internal energy is converted by the sigmoid transformation to calculate the output value. Here, the sigmoid transform is calculated by polynomial approximation such as Chebyshev approximation. The calculated output value is stored in the register file 2 of each neuron. As a result of the processing in step 30 above, the output value of the neuron in each layer is calculated.

In step 31, the back propagation error of the neuron in each layer is calculated. FIG. 9 shows a detailed flow of step 31. In step 35, the back propagation error of the output layer neuron is calculated. FIG. 10 shows a detailed flow of step 35. In step 37, the output value is read from the register file 2 and the differential value of the output value of the output layer neuron is calculated. In step 38, the output value is read from the register file 2, the teacher signal is fetched from the input bus, and the error between the output value of each output layer neuron and the teacher signal is calculated. In step 39, the back propagation error is calculated by multiplying the differential value of the output value calculated in step 37 and the error calculated in step 38. In Step 40, Step 3
The back propagation error calculated in 9 is stored in the register file 2. As a result of the processing in step 35, the back propagation error of the output layer is calculated and stored in the register file 2.

In step 36, the back propagation error of the hidden layer neuron is calculated. FIG. 11 shows the detailed flow of step 36. In step 41, the output value is read from the register file 2 and the differential value of the output value of the hidden layer neuron is calculated. In step 42, the back propagation error of each neuron on the output layer side is multiplied by the weight value of the synapse connected to that neuron to perform cumulative addition. That is,
Each neuron on the output layer side broadcasts a backpropagation error every machine cycle. Each neuron in the layer is
The backpropagation error broadcast by the pipeline processing is fetched, the weight value stored in the RAMs 3a and 3b is read out, and cumulative addition with the backpropagation error is performed. In step 43, the differential value of the output value calculated in step 41 is multiplied by the weight value calculated in step 42 and the cumulative addition of the back propagation error to calculate the back propagation error. In step 44, the back propagation error calculated in step 43 is stored in the register file 2. As a result of the processing in step 36, the back propagation error of the intermediate layer is calculated and stored in the register file 2. When there are a plurality of intermediate layers, the process of step 36 is repeated for the number of intermediate layers.

As a result of the above processing in step 31, the back propagation error of each neuron is calculated.

In step 32, the output value calculated in step 30 and the back propagation error calculated in step 31 are used to correct the weight value stored in the RAM 3a, RAM 3b of each neuron. For the sake of simplicity, it is assumed that the moment coefficient is 0. FIG. 12 shows the detailed flow of step 32. In step 45, the back propagation error is first read from the register file 2 and the output value of the neuron in the layer before the input bus is fetched. Then, they are multiplied to calculate the correction value of the weight value based on the equation 3. In step 46,
The weight value stored in the RAM 3a, RAM 3b is read out, and the weight value is corrected by adding the correction value calculated in step 45. In step 47, the weight values corrected in step 46 are stored again in the RAMs 3a and 3b. As a result of the processing in step 32 above, the RAM 3a, RAM 3b
The weight value stored in is corrected. Repeat step 32 for each layer to modify the weight values for all layers.

The learning process will be described by dividing it into operations for each machine cycle using the neuron block diagram of FIG.
The back propagation error calculation of the output layer and the intermediate layer will be described in detail with reference to the PADs of FIGS. 10 and 11, and the weight value correction will be described with reference to the PAD of FIG.
The flow of data will be described with reference to FIG. In addition, the sentences described below correspond to the micro program code, and the number at the beginning of the sentence represents the number of machine cycles from the start.

First, the calculation of the back propagation error in the output layer of FIG. 10 will be described. In step 37, the differential value of the output value is calculated. This is performed by the processing of machine cycles 0 to 3. 0. Store the output value from register file 2 in FF4b.
Store the output value from register file 2 in FF4b. 1. NOP (NO OPERATION: Neuron 1
State to give nothing to). 2. Store the output value from register file 2 in FF4b.
The shifter 7 shifts the value of FF4e to the left by 0 bit. Store the shift result in FF4d. 3. ALU6 subtracts the FF4d value from the FF4b value. Store the subtraction result in register file 2. As a result of the above processing of machine cycles 0 to 3, the differential value of the output value is calculated.

In step 38, the error between the output value and the teacher signal is calculated. This is performed by the processing of machine cycles 4 to 5. 4. Store the teacher signal in FF4b. Read the output value from register file 2 and store it in FF4c. 5. ALU6 subtracts the FF4c value from the FF4b value. Store the subtraction result in the register file 30. As a result of the above processing of machine cycles 4 to 5, the differential value of the output value is calculated.

In step 39, the back propagation error is calculated by multiplying the differential value calculated in step 37 and the error calculated in step 38. However, in order to match the bit format of the back propagation error in FIG. 6, the back propagation error is calculated and then shifted by 3 bits. The above is performed by the processing of machine cycles 6 to 9. 6. The differential value of the output value from the register file 2 is FF4b
Stored in. The error of the output value from the register file 2 is FF4
Stored in c. 7. NOP. 8. The shifter 7 shifts the value of FF4e left by 3 bits.
Store the shift result in FF4c. Store a 10-bit mask in FF4b. As a result of the processing of the above machine cycles 6 to 9, the back propagation error is calculated.

In step 40, the back propagation error calculated in step 39 is stored. However, since the input / output bus is 10 bits when calculating the back propagation error of the middle layer,
A 10-bit mask is applied and stored. This is performed by the processing of the machine cycle 9. 9. The AND of the FF4b value and the FF4c value is executed by the ALU6. Store AND result in register file 2. As a result of the above machine cycles 0 to 9, the back propagation error in the output layer is calculated and stored.

Next, the calculation of the back propagation error of the intermediate layer in FIG. 11 will be described. In step 41, the differential value of the output value is calculated. This is performed by the processing of machine cycles 0 to 3. 0. Store the output value from register file 2 in FF4b.
Store the output value from register file 2 in FF4b. 1. NOP. 2. Store the output value from register file 2 in FF4b.
The shifter 7 shifts the value of FF4e to the left by 0 bit. Store the shift result in FF4d. 3. ALU6 subtracts the FF4d value from the FF4b value. Store the subtraction result in register file 2. As a result of the above processing of machine cycles 0 to 3, the differential value of the output value is calculated.

At step 42, the product sum of the back propagation error of the output layer side neuron and the weight value is calculated. 4. Store the back propagation error in FF4b. RAM3a, RAM
16-bit weight value is read from 3b and stored in FF4c. 5. Reset FF4d. 6. Add FF4d and FF4e in ALU6. Store the addition result in FF4d. As a result of the above machine cycles 4 to 6, the product sum of the back propagation error of the output layer side neuron and the weight value is calculated. When there are multiple output layer side neurons, the processing of machine cycles 4 and 6 is repeated by the pipeline.

In step 43, the differential value of the output value calculated in step 41 is multiplied by the product sum of the back propagation error calculated in step 42 and the weight value to calculate the back propagation error of the intermediate layer. However, in the calculation of the sum of products, the input from the FF 4e to the ALU 6 is 26 bits lower justified, so that 7 bits are shifted to match the position of the decimal point. Further, in order to match the bit format of the back-propagation error in FIG. 6, after the back-propagation error is calculated, it is shifted by 3 bits. The above is performed by the processing of machine cycles 4 to 6. 7. The shifter 7 shifts the value of FF4d to the left by 7 bits.
Store the shift result in FF4b. Read the differential value of the output value from the register file 2 and store it in FF4c. 8. NOP. 9. The shifter 7 shifts the value of FF4e left by 3 bits.
Store the shift result in FF4c. Store a 10-bit mask in FF4b. The back-propagation error is calculated by the processing of the above machine cycles 7 to 9.

In step 44, the back propagation error calculated in step 43 is stored. However, since the input / output bus is 10 bits when calculating the back propagation error of the middle layer,
A 10-bit mask is applied and stored. This is performed by the processing of the machine cycle 10. 10. AND of FF4b value and FF4c value in ALU6
Run. Store AND result in register file 2. As a result of the above machine cycles 0 to 10, the back propagation error of the intermediate force layer is calculated and stored.

Next, the correction of the weight value shown in FIG. 12 will be described. In step 45, the correction value of the weight value is calculated. This is performed by the processing of machine cycles 0 to 4. 0. The output value of the neuron in the previous layer is stored in FF4b, FF
Store backpropagation error in 4c. 1. NOP. 2. The learning coefficient is stored in FF4b. FF4e with shifter 7
Shift the value of 0 to the left. Store the shift result in FF4c. 3. NOP. 4. Shift the result of FF4e by 0 bits. Store the shift result in register file 2. The weight value correction value is calculated by the above processing from machine cycle 0 to 4.

In step 46, the 16-bit weight value is read out and corrected. This is performed by the processing of machine cycles 5 to 6. 5. 16-bit weight value is read from RAM3a and RAM3b and stored in FF4c. The weight value correction value is read from register file 2 and stored in FF4b. 6. Add the values of FF4b and FF4c in ALU6. The result is stored in FF4a. The weight value correction is performed by the above processing of the machine cycles 5 to 6.

In step 47, the corrected weight value is stored. This is performed by the processing of machine cycle 7. 7. Store the value of FF4a in RAM3a, RAM3b. Through the above-described processing of machine cycles 0 to 7, the 16-bit weight value is read, its correction is performed, and the corrected weight value is stored. A neurocomputer that performs fixed-point and pipeline processing in this way enables high-speed learning.

[0044]

In a neuro computer realized by hardware, the input bit width of the multiplier is set to 16 bits in order to prevent the circuit scale of the multiplier occupying a large area in the chip from expanding. It is limited to 10 bits. Accordingly, the number of bits of the weight value and the back propagation error is limited to 16 bits and 10 bits, respectively. As a result of this bit number limitation, there arises a problem that it is difficult to perform highly accurate learning.

That is, in the learning of the neural network, the back propagation error gradually becomes smaller as the learning progresses. As the learning progresses, the output value of the output layer approaches the teacher signal, so the error between the output value and the teacher signal becomes smaller. Furthermore, as learning progresses, the number of neurons whose output value approaches 0 or 1 generally increases, so the differential value of the output value tends to decrease. As a result, the back propagation error of the output layer shown in Formula 4 and the back propagation error of the intermediate layer shown in Formula 5 are reduced.

When the back-propagation error becomes very small, the correction value of the weight value shown in equation 3 also becomes very small. Therefore, even if the correction value takes a non-zero value, when the correction value is added to the 16-bit weight value shown in FIG. An unchanging situation arises.

The minimum value of the back propagation error shown in FIG. 6 is 2 minus 12 and the minimum output value is 2 minus 9.
It is the square. The value obtained by multiplying the propagation error and the output value is 2 to the 21st power, while the minimum weight value is 2 to the 11th power. This is a difference of 2 to the 10th power, that is, there is a difference in size of about 1000 times. Even if the correction value is calculated by multiplying the learning coefficient, the maximum value of the learning coefficient is about 9 at most, and therefore, it becomes smaller than the minimum value of the weight values. As a result, even if the correction value is added to the weight value, there is a situation in which after the rounding to 16 bits, it is the same as before the correction.

On the other hand, the back-propagation error itself becomes smaller than 2 minus the 12th power as the learning progresses for the above-mentioned reason, and the digit cancellation occurs.

If it is attempted to increase the number of bits for each operation in order to prevent the cancellation of the weight value and the cancellation of the back-propagation error, the circuit scale of the multiplier or the like is expanded and the chip area is expanded. An increase in chip cost can be considered. Alternatively, when an operation such as multiplication of a weight value with an increased number of bits or a back propagation error is performed using the circuit shown in FIG. 1, for example, one multiplication of the weight value and the back propagation error requires a plurality of multiplications. It is necessary to divide into two. For this reason, the high speed of hardware based on pipeline processing cannot be utilized.

The present invention provides a high-accuracy learning arithmetic processing method capable of performing arithmetic processing for high-speed and high-accuracy learning without expanding the circuit scale as described above, and information processing that can be realized by this arithmetic processing method. It is to provide a device.

[0051]

When the weight value is increased from 16 bits of single precision to 32 bits of double precision, the arithmetic unit that requires a very long time for the operation in the circuit of FIG. 1 is a multiplier. . On the other hand, when the learning progresses, the weight value does not change even if the weight value is corrected due to cancellation of digits because the weight value is rounded to 16 bits after the correction.

Considering these things, one weight value is 3
It is stored in 2 bits, and the weight value in the calculation of the output value of the neuron and the cumulative addition of the output value of other neurons, and the weight value in the calculation of the back propagation error and the cumulative addition of the back propagation error of other neurons are calculated. In this case, the top 16 weight values
Perform operations using bits. On the other hand, when correcting the weight value, the entire 32-bit double precision is read out and corrected.
That is, as a first invention, a multiplier for performing fixed-point multiplication using a predetermined number of bits and an adder are provided,
In a neurocomputer that performs a desired neural network operation using a weight value of a predetermined number of bits, an output value, and backpropagation error, a weight value of a number of bits larger than the predetermined number of bits is stored when learning the weight value. Every time a neuron output value is calculated and a backpropagation error is calculated, a step of reading a predetermined number of bits from the stored weight value and using it in the calculation, and a number of bits larger than the predetermined number of bits when correcting the weight value Can be realized by the steps of reading out and using for calculation.

Furthermore, in the state where learning has progressed to some extent,
The back-propagation error becomes very small, and digit cancellation occurs. Therefore, as a second invention, a multiplier for performing fixed-point multiplication using a predetermined number of bits and an adder are provided, and a desired neural network is obtained by using a weight value of a predetermined number of bits, an output value, and a back propagation error. In a neurocomputer that performs net operation, a step of storing a plurality of learning microprograms having different decimal point positions, a step of storing a condition for switching a microprogram used for learning in the learning process, and the above switching in the learning process When the condition is satisfied, it can be realized by selecting one from a plurality of microprograms and using it for learning.

Further, as a third invention, a multiplier for performing fixed point multiplication using a predetermined number of bits, and an adder are provided, and a weight value of a predetermined number of bits, an output value, and a back propagation error are used. In a neurocomputer that performs a desired neural network operation, a step of preparing a plurality of learning microprograms having different decimal point positions, and, before starting learning, according to the problem to be learned by the user from the plurality of microprograms This can be realized by the step of selecting one and the step of learning by storing the selected program in a neuro computer.

[0055]

According to the first invention, there are the following four actions.
First of all, since the weight value is set to double precision 32 bits,
It is possible to prevent digit cancellation in the modification, and the weight values before and after modification will not be the same value. Second
In addition, it is possible to prevent the learning from stopping due to the cancellation of the weight value. That is, in the case of the cumulative addition of the weight value in the calculation of the output value of the neuron and the output value of the other neuron, and the cumulative addition of the weight value in the calculation of the back propagation error and the back propagation error of the other neuron, the weight value Calculation is performed using the upper 16 bits of. However, since the weight value is modified many times during the learning process, one modification is the top 1
Even if it is smaller than 6 bits, it will eventually be accumulated and the result will also affect the upper 16 bits. As a result, it is possible to prevent the learning from stopping due to the cancellation of the weight value. Thirdly, high-speed calculation is possible even though highly accurate weight value correction learning is performed. In the cumulative addition of the weight value using the multiplier and the output value of the other neuron, and the cumulative addition of the back propagation error of the other neuron, since the weight value uses only 16 bits of single precision as in the conventional case, one multiplication is required. There is no need to divide it into multiple times. As a result, it is possible to perform high-speed calculation similar to the conventional one. Fourth, it is possible to prevent an increase in the chip area of the multiplier, which increases the circuit scale. As in the past, since the multiplication is performed in the single precision operation, it is not necessary to increase the number of input bits, and the circuit scale of the multiplier occupying a large area in the chip can be reduced. As a result, it becomes possible to prevent an increase in chip cost.

The second invention has the following effects. The back propagation error becomes smaller as the learning progresses. However, since the decimal point position of the back-propagation error is shifted according to the progress of learning, it is possible to prevent the back-propagation error from being canceled even though the number of bits is limited. As a result, high-speed learning can be performed without slowing the progress of learning.

According to the third invention, there are the following effects. The user can select a microprogram with a different back-propagation error decimal point position depending on the problem to be learned. Therefore, when highly accurate learning is required, a microprogram that uses back propagation error with a small decimal point position can be used. Therefore, learning does not become stagnant on the way, and it is possible to achieve both high-speed learning and highly accurate learning.

[0058]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment of the present invention is shown in FIGS.
2, it demonstrates using FIG. 1 shows a block diagram of a neuron, FIG. 7 shows a PAD for backpropagation learning, and FIG. 12 shows step 32 of FIG.
13 is a PAD showing the details of FIG. 13, and FIG. 13 is a format of a bit of a weight value and a back propagation error according to the embodiment of the present invention.

The operation of the neuron shown in FIG. 1 in the first embodiment of the present invention will be described with reference to FIG. In this embodiment, the bit format of the weight value is 32 bits as shown in FIG.
The data is divided into bits and stored in the RAM 3a and RAM 3b.
In FIG. 7, step 32 is a feature of the present invention.

In step 30, the upper 16 bits of the weight value are read from the RAM 3a, RAM 3b and the output value of each neuron is calculated by the same processing as described in the prior art. In step 31, the upper 16 bits of the weight value are read from the RAM 3a and RAM 3b and the back propagation error of each neuron is calculated by the same processing as described in the conventional technique. In step 32, step 30
Using the output value calculated in step 3 and the back propagation error calculated in step 31, the weight value stored in the RAM 3a, RAM 3b of each neuron is corrected. For the sake of simplicity, it is assumed that the moment coefficient is 0.

FIG. 12 shows the details of step 32. In step 45, the back propagation error is first read from the register file 2 and the output value of the neuron in the previous layer is fetched from the input bus. Then, they are multiplied to calculate the weight value correction value of Equation 3. In step 46, RAM3a, RAM3
The weight value stored in b is read in 16-bit units, and the correction value calculated in step 45 is added to correct the weight value. In step 47, the 32-bit weight value corrected in step 46 is stored in the RAM 3a and RAM 3b in 16 bits.
Store bit by bit again. By the processing of step 32, the 32-bit weight value stored in the RAM 3a, RAM 3b is modified. By repeating step 32 for each layer, the weight value is corrected for all layers.

Next, using the block diagram of the neuron of FIG. 1, the process of double-precision weight value correction in step 47 is divided into operations for each machine cycle, and the weight value correction P of FIG. 12 is performed.
It will be described in detail in association with AD. The flow of data will be described with reference to FIG. Also, the sentences written below correspond to the micro program code, and the numbers at the beginning of the sentence are from the time when the high-accuracy learning process starts,
Indicates the number of machine cycles.

At step 45, the weight value correction value is calculated. This is performed by the processing of machine cycles 0 to 4. 0. The output value of the neuron in the previous layer is stored in FF4b, FF
Store backpropagation error in 4c.

1. NOP (NO OPERATION:
(No state is given to the neuron 20). 2. Shifter 7 shifts the result of FF4e by 0 bits. Store the shift result in FF4c. 3. NOP. 4. Shifter 7 shifts the result of FF4e by 0 bits. Store the shift result in register file 2. The weight value correction value is calculated by the above processing from machine cycle 0 to 4.

In step 46, the 32-bit weight value is read out and corrected. This is performed by the processing of machine cycles 5 to 9. 5. The lower word of the weight value is read from RAM3a and RAM3b and stored in FF4c. 6. Throughput the value of FF4c in ALU6. The result is stored in FF4b.

The upper word of the weight value is read from the RAM 3a, RAM 3b and stored in the FF 4c. 7. The shifter 7 shifts the value of FF4b to the right by 16 bits. Store the shift result in FF4d. FF4 with ALU6
The value of c is the throughput. The result is stored in FF4b. 8. ALU6 executes the OR of the value of FF4b and the value of FF4d. Store the OR result in FF4c. The modified value of the weight value is read from the register file 2 and stored in FF4b. 9. ALU6 adds the value of FF4b and the value of FF4c.
Store the addition result in FF4d. 32 by the above processing of machine cycles 5 to 9
The bit weight value is read and the correction is performed.

In step 47, the corrected weight value is stored. This is performed by the processing of machine cycles 10 to 12. 10. Shifter 7 shifts the value of FF4d by 0 bits. Store the shift result in FF4a. 11. Shifter 7 shifts the value of FF4d by 16 bits.
Store the shift result in FF4a. The value of FF4a is stored in RAM3
a, stored in RAM3b. 12. Store the value of FF4a in RAM3a, RAM3b. By the above processing from machine cycle 10 to 12,
The modified 32-bit weight value is stored. In a neurocomputer that performs fixed-point and single-precision arithmetic processing in this way, high-accuracy learning becomes possible by using the neuron block diagram shown in FIG. 1 and the above-described weight value correction method.

Next, a second embodiment of the present invention will be described with reference to FIGS. 1, 5, 13 and 14. FIG. 5 is a block diagram of a neuro computer. Neuro Computer 1
Reference numeral 0 is composed of a plurality of neuro boards 12 and a control board 11 which perform a neural network operation. The control board 11 is composed of a neural network control device 13, a control storage 15, and a global memory 16. The neural network control device 13 controls the neuro computer 10 by decoding and executing the micro program code stored in the control storage 15. The neural network control device 13 includes a control storage control circuit 14
a, workstation 18 and neuro computer 1
An external bus is provided as a communication means for 0, and it comprises a neuron control circuit 14c for controlling this external bus and a global memory control circuit 14d for storing the values of the input layer neurons and the like. Furthermore, the external bus is memory, CPU, SC
Connected to the SI board, the SCSI board is connected to a workstation 18 having a microgenerator that generates microprogram code.

FIG. 14 is a diagram showing the PAD of the processing of the second embodiment. In the second embodiment, in addition to the modification of the 32-bit weight value in the first embodiment, changing the decimal point position of the back propagation error in the learning process will be described.

The operation of the neuron shown in FIG. 1 in the second embodiment will be described with reference to FIG. In step 50,
The RAM 3 is processed by the same processing as that described in the related art.
a, the upper 16 bits of the weight value are read from the RAM 3b, and the output value of each neuron is calculated. In step 51, RA processing is performed by the same processing as that described in the related art.
The upper 16 bits of the weight value are read from M3a and RAM3b, and the back propagation error of each neuron is calculated. The format of the bit of the back propagation error is shown in FIG. Step 52
Then, by the same processing as that described in the first embodiment, the output value calculated in step 50 and the back-propagation error calculated in step 51 are used, and the RAM 3 of each neuron is
a, the weight value stored in the RAM 3b is corrected. In step 53, steps 50 to 52 are repeated a fixed number of times Nc. In step 54, the mean square error is calculated, and if it is smaller than the reference value, step 55 is executed. In step 55, the position of the decimal point differs from that of the currently used microprogram.
Change to a microprogram that uses the back-propagation error of the format. In step 56, steps 53 and 54 are repeated Mc times a fixed number of times.

Next, the processing of step 51 after execution of step 55 will be described with reference to FIGS. 1, 9, 10, and 11. FIG. 9 shows details of step 51 in FIG.
10 is a PAD showing the details of step 35 in FIG. 9, and FIG. 11 is a PAD showing the details of step 36 in FIG.

The processing of step 51 will be described by dividing it into operations for each machine cycle using the neuron block diagram of FIG. The back propagation error calculation of the output layer and the intermediate layer will be described in detail with reference to the PADs of FIGS. 10 and 11, and the weight value correction will be described with reference to the PAD of FIG. The flow of data will be described with reference to FIG. In addition, the sentences described below correspond to the micro program code, and the number at the beginning of the sentence represents the number of machine cycles from the start.

First, step 35 in FIG. 9, that is, FIG.
The calculation of the back propagation error in the output layer of 0 will be described. In step 37, the differential value of the output value is calculated. This is performed by the processing of machine cycles 0 to 3. 0. Store the output value from register file 2 in FF4b.
Store the output value from register file 2 in FF4b. 1. NOP (NO OPERATION: Neuron 1
State to give nothing to). 2. Store the output value from register file 2 in FF4b.
The shifter 7 shifts the value of FF4e to the left by 0 bit. Store the shift result in FF4d. 3. ALU6 subtracts the FF4d value from the FF4b value. Store the subtraction result in register file 2. As a result of the above processing of machine cycles 0 to 3, the differential value of the output value is calculated.

In step 38, the error between the output value and the teacher signal is calculated. This is performed by the processing of machine cycles 4 to 5. 4. Store the teacher signal in FF4b. Read the output value from register file 2 and store it in FF4c. 5. ALU6 subtracts the FF4c value from the FF4b value. Store the subtraction result in the register file 30. As a result of the above processing of machine cycles 4 to 5, the differential value of the output value is calculated.

In step 39, the back propagation error is calculated by multiplying the differential value calculated in step 37 and the error calculated in step 38. However, in order to match the bit format of the back propagation error in FIG. 14, 6 bits are shifted after the back propagation error is calculated. Machine cycle 6
The process from 9 to 9 is performed. 6. The differential value of the output value from the register file 2 is FF4b
Stored in. The error of the output value from the register file 2 is FF4
Stored in c. 7. NOP. 8. The shifter 7 shifts the value of FF4e to the left by 6 bits.
Store the shift result in FF4c. Store a 10-bit mask in FF4b. As a result of the processing of the above machine cycles 6 to 9, the back propagation error is calculated.

In step 40, the back propagation error calculated in step 39 is stored. However, since the input / output bus is 10 bits when calculating the back propagation error of the middle layer,
A 10-bit mask is applied and stored. This is performed by the processing of the machine cycle 9. 9. The AND of the FF4b value and the FF4c value is executed by the ALU6. Store AND result in register file 2. As a result of the above machine cycles 0 to 9, the back propagation error in the output layer is calculated and stored.

Next, step 36 in FIG. 9, that is, FIG.
The calculation of the backpropagation error of the intermediate layer 1 will be described. In step 41, the differential value of the output value is calculated. This is performed by the processing of machine cycles 0 to 3. 0. Store the output value from register file 2 in FF4b.
Store the output value from register file 2 in FF4b. 1. NOP. 2. Store the output value from register file 2 in FF4b.
The shifter 7 shifts the value of FF4e to the left by 0 bit. Store the shift result in FF4d. 3. ALU6 subtracts the FF4d value from the FF4b value. Store the subtraction result in register file 2. As a result of the above processing of machine cycles 0 to 3, the differential value of the output value is calculated.

In step 42, the product sum of the back propagation error of the output layer side neuron and the weight value is calculated. 4. Store the back propagation error in FF4b. RAM3a, RAM
16-bit weight value is read from 3b and stored in FF4c. 5. Reset FF4d. 6. Add FF4d and FF4e in ALU6. Store the addition result in FF4d. As a result of the above machine cycles 4 to 6, the product sum of the back propagation error of the output layer side neuron and the weight value is calculated. When there are multiple output layer side neurons, the processing of machine cycles 4 and 6 is repeated by the pipeline.

In step 43, the differential value of the output value calculated in step 41 is multiplied by the product sum of the back propagation error and the weight value calculated in step 42 to calculate the back propagation error of the intermediate layer. However, in the calculation of the sum of products, the input from the FF 4e to the ALU 6 is 26 bits lower justified, so that 7 bits are shifted to match the position of the decimal point. Further, in order to match the bit format of the back-propagation error in FIG. 14, after the back-propagation error is calculated, it is shifted by 6 bits. The above is performed by the processing of machine cycles 4 to 6. 7. The shifter 7 shifts the value of FF4d to the left by 7 bits.
Store the shift result in FF4b. Read the differential value of the output value from the register file 2 and store it in FF4c. 8. NOP. 9. The shifter 7 shifts the value of FF4e to the left by 6 bits.
Store the shift result in FF4c. Store a 10-bit mask in FF4b.

The back-propagation error is calculated by the processing in the above machine cycles 7 to 9.

In step 44, the back propagation error calculated in step 43 is stored. However, since the input / output bus is 10 bits when calculating the back propagation error of the middle layer,
A 10-bit mask is applied and stored. This is performed by the processing of the machine cycle 10. 10. AND of FF4b value and FF4c value in ALU6
Run. Store AND result in register file 2. As a result of the above machine cycles 0 to 10, the back propagation error of the intermediate force layer is calculated and stored.

Next, the correction of the weight value shown in FIG. 12 will be described. In step 45, the weight value correction value is calculated. This is performed by the processing of machine cycles 0 to 4.
However, since the back-propagation error has the bit format shown in FIG. 14, the decimal point position is shifted to the left by 3 bits as compared with the case described in the first embodiment. Therefore, after the correction value of the weight value is calculated, the correction value is shifted to the right by 3 bits in order to match the decimal point position of the weight value. 0. The output value of the neuron in the previous layer is stored in FF4b, FF
Store backpropagation error in 4c. 1. NOP (NO OPERATION: Neuron 2
(A state where no command is given to 0). 2. Shifter 7 shifts the result of FF4e by 0 bits. Store the shift result in FF4c. The learning coefficient is stored in FF4b. 3. NOP. 4. The shifter 7 shifts the result of FF4e by 3 bits. Store the shift result in register file 2. The weight value correction value is calculated by the above processing from machine cycle 0 to 4.

In step 46, the 32-bit weight value is read out and corrected. This is performed by the processing of machine cycles 5 to 9. 5. The lower word of the weight value is read from RAM3a and RAM3b and stored in FF4c. 6. Throughput the value of FF4c in ALU6. The result is stored in FF4b.

The weight value upper word is read from the RAM 3a and RAM 3b and stored in the FF 4c.

7. Shifter 7 shifts the value of FF4b to the right by 16
Bit shift. Store the shift result in FF4d. ALU6
The value of FF4c is throughput. The result is FF4b
Stored in. 8. ALU6 executes the OR of the value of FF4b and the value of FF4d. Store the OR result in FF4c. The modified value of the weight value is read from the register file 2 and stored in FF4b. 9. ALU6 adds the value of FF4b and the value of FF4c.
Store the addition result in FF4d. 32 by the above processing of machine cycles 5 to 9
The bit weight value is read and the correction is performed.

In step 47, the corrected weight value is stored. This is performed by the processing of machine cycles 10 to 12. 10. Shifter 7 shifts the value of FF4d by 0 bits. Store the shift result in FF4a. 11. Shifter 7 shifts the value of FF4d by 16 bits.
Store the shift result in FF4a. The value of FF4a is stored in RAM3
a, stored in RAM3b. 12. Store the value of FF4a in RAM3a, RAM3b. By the above processing from machine cycle 10 to 12,
The modified 32-bit weight value is stored.

In the neurocomputer that performs fixed-point and single-precision arithmetic processing in this way, high-precision learning is possible by shifting the neuron block diagram shown in FIG. 1 and the decimal point position of the above-mentioned back-propagation error in the learning process. become.

The condition in step 54 of FIG. 14 is not limited to the mean squared error, and the decimal point position of the back propagation error may be shifted when a certain number of learning times is reached. Alternatively, the value of the back-propagation error may be checked, and the decimal point may be shifted if, for example, the ratio of the back-propagation error of a certain value or less among the back-propagation error is a certain value or more.

Next, a third embodiment will be described with reference to FIG. The workstation 18 stores a micro-generator that generates a plurality of micro-programs including back-propagation errors each having a different decimal point position. The user uses a microgenerator to generate a microprogram including a back propagation error at an appropriate decimal point position according to the problem to be learned. Then, it is stored in the control storage of the neuro computer through SCSI and learning is performed.

As a result, it is possible to realize accuracy learning according to the problem to be learned. That is, the user selects a microprogram corresponding to the learning that requires high-precision calculation that normally causes the back propagation error to be canceled and the learning does not proceed. be able to.

[0091]

The first invention has the following three effects. First of all, since the weight value is set to double-precision 32 bits, it is possible to prevent the digit loss in the correction, and to avoid the delay of the learning progress due to the digit loss. That is, in the case of the cumulative addition of the weight value in the calculation of the output value of the neuron and the output value of the other neuron, and the cumulative addition of the weight value in the calculation of the back propagation error and the back propagation error of the other neuron, the weight value Calculation is performed using the upper 16 bits of. However, since the weight value is modified many times during the learning process, even if one modification is smaller than the upper 16 bits, it is eventually accumulated and the result also affects the upper 16 bits. As a result, it is possible to prevent the learning from stopping due to the cancellation of the weight value. Second
In addition, although high-precision weight value correction learning is performed, the same high-speed calculation as in the past becomes possible. In the cumulative addition of the weight value using the multiplier and the output value of the other neuron, and the cumulative addition of the back propagation error of the other neuron, since the weight value uses only 16 bits of single precision as in the conventional case, one multiplication is required. There is no need to divide it into multiple times. As a result, it is possible to perform high-speed calculation similar to the conventional one. Third, it is possible to prevent an increase in the chip area of the multiplier, which increases the circuit scale. As in the past, since the multiplication is performed in the single precision operation, it is not necessary to increase the number of input bits, and the circuit scale of the multiplier occupying a large area in the chip can be reduced. As a result, it becomes possible to prevent an increase in chip cost. FIG. 15 shows a graph of a learning curve when the weight value is 16 bits and when the weight value is 32 bits. 16
It can be seen that the error does not decrease at all when the number of times of learning exceeds 1000 in the case of bits, but gradually decreases and the learning progresses even after 1000 times in the case of 32 bits.

The second invention has the following effects. The back propagation error becomes smaller as the learning progresses. However, since the decimal point position of the back-propagation error is shifted according to the progress of learning, it is possible to prevent the back-propagation error from being canceled even though the number of bits is limited. As a result, stagnation of learning can be prevented without increasing the number of bits of back propagation error, and high-speed learning can be performed.

The third invention has the following effects. The user can select a microprogram with a different back-propagation error decimal point position depending on the problem to be learned. Therefore, when highly accurate learning is required, a microprogram that uses back propagation error with a small decimal point position can be used. Therefore, learning does not become stagnant on the way, and it is possible to achieve both high-speed learning and highly accurate learning.

[Brief description of drawings]

FIG. 1 is a diagram showing a neuron block diagram.

FIG. 2 is a diagram showing a neuron model forming a neural net.

FIG. 3 is a diagram showing a three-layer hierarchical neural network.

FIG. 4 is a diagram showing the principle of dedicated hardware that calculates a neural network at high speed.

FIG. 5 is a diagram showing a configuration of a neuro computer.

FIG. 6 is a diagram showing a bit format of each variable used in the operation of the neurocomputer.

FIG. 7 is a diagram showing a PAD of processing by the back propagation method.

8 is a diagram showing details of step 30 in FIG. 7. FIG.

9 is a diagram showing details of step 31 in FIG. 7. FIG.

FIG. 10 is a diagram showing details of step 35 in FIG. 9;

FIG. 11 is a diagram showing details of step 36 in FIG. 9;

12 is a diagram showing details of step 32 in FIG.

FIG. 13 is a diagram showing a bit format of a weight value and a back propagation error in the embodiment of the present invention.

FIG. 14 is a diagram showing a processing flow of the second embodiment of the present invention.

FIG. 15 is a diagram showing comparison of learning curves when the number of weight value bits is 16 bits and 32 bits.

[Explanation of symbols]

1 ... Neuron, 2 ... Register file, 3a ... R
AM, 3b ... RAM, 4a ... Flip-flop, 4b ...
Flip-flop, 4c ... Flip-flop, 4d ... Flip-flop, 4e ... Flip-flop, 5 ... Multiplier, 6 ... ALU, 7 ... Shifter, 8 ... Neuron calculation unit, 9 ... Bus, 10 ... Neurocomputer , 11 ... Control board, 12 ... Neuro board, 13 ...
Neural network controller, 15 ... Control storage, 16 ... Global memory, 17 ... Neurochip.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Takahiro Sakaguchi 5-20-1 Kamimizuhoncho, Kodaira-shi, Tokyo Inside Hiritsu Cho-LS Engineering Co., Ltd. (72) Inventor Yuji Toda Kokubunji, Tokyo 1-280 Higashi-Kengokubo, Higashi-Kita Central Research Laboratory, Hitachi, Ltd. (72) Yoshihiro Kuwahara, 5-22-1 Kamisuihoncho, Kodaira-shi, Tokyo Hitachi Microcomputer System Co., Ltd. (72) Keiji Mogi, Keiji Mogi, Tokyo 5-22-1 Kamimizuhonmachi, Hitachi, Ltd. Inside Hitachi Microcomputer System Co., Ltd. (72) Inventor Tatsuo Ochiai 5-22-1 Kamuimizuhoncho, Kodaira-shi, Tokyo Inside Hitachi Microcomputer System Co., Ltd.

Claims (8)

[Claims] 1. A desired neural network operation comprising a multiplier for performing fixed-point multiplication using a predetermined number of bits, and an adder, and using a weight value of a predetermined number of bits, an output value, and back propagation error. In the neuro computer that performs the above, when the weight value of the neural network is learned, a means for storing the weight value of the number of bits larger than the predetermined number of bits, and a neuron output value calculation in the learning and a back propagation error calculation are performed. Configured as a neurocomputer having means for reading out a predetermined number of bits from the stored weight value and using it for calculation, and means for reading out a number of bits larger than the predetermined number of bits and using it for calculation when correcting the weight An information processing device characterized by the above. 2. A multiplier for performing fixed-point multiplication using a predetermined number of bits, and an adder, wherein a microprogram is controlled by using a weight value of a predetermined number of bits, an output value, and a back propagation error. In a neurocomputer that performs a desired neural network operation, a means for storing a plurality of microprograms for performing learning including back propagation errors with different decimal point positions, and a condition for switching between the microprograms used for learning It is configured as a neurocomputer having means for storing and means for selecting another one from a plurality of microprograms and using it for learning when the above-mentioned switching condition is satisfied in the course of learning. A characteristic information processing device. 3. A desired neural network operation using a multiplier for performing fixed point multiplication using a predetermined number of bits, and an adder, and using a weight value of a predetermined number of bits, an output value, and back propagation error. In the neurocomputer that performs, the means for storing the condition for changing the decimal point position of the back-propagation error in the process of learning the weight value, and the back-propagation when the condition for changing is satisfied in the process of learning. An information processing device comprising a neuro computer having means for changing a decimal point position of an error and means for continuing learning by using the back propagation error of the changed decimal point position. 4. A microprogram control, comprising: a multiplier for performing fixed point multiplication using a predetermined number of bits; and an adder, and using a weight value of a predetermined number of bits, an output value, and a back propagation error. In a neurocomputer for performing a desired neural network operation by means of means for storing a plurality of microprograms including back propagation errors each having a different decimal point position, and before starting the learning of weight values, the microprograms used for learning are An information processing apparatus configured as a neurocomputer having means for selecting from among the above and means for learning a weight value using the selected microprogram. 5. A desired neural network operation comprising a multiplier for performing fixed point multiplication using a predetermined number of bits, and an adder, and using a weight value of a predetermined number of bits, an output value, and back propagation error. In the neurocomputer that performs, the means for changing the decimal point position of the back-propagation error, the means for selecting one decimal point position from a plurality of decimal point positions by using the changing means before starting the learning, and the means for selecting the selected decimal point position An information processing apparatus configured as a neurocomputer having means for learning a weight value using a back propagation error. 6. A multiplier for performing fixed-point multiplication using a predetermined number of bits, and an adder, and a desired neural network operation using a weight value, an output value, and a back propagation error of a predetermined number of bits. A method for performing a learning calculation processing of a neuro computer, comprising storing a weight value of a number of bits larger than the predetermined number of bits at the time of learning a weight value of a neural network, and calculating a neuron output value at the time of learning, And a step of reading a predetermined number of bits from the stored weight value for calculation in the back propagation error calculation, and a step of reading a number of bits larger than the predetermined number of bits for calculation in correcting the weight,
A learning calculation processing method comprising: 7. A multiplier provided for performing fixed-point multiplication using a predetermined number of bits, and an adder, using a weight value of a predetermined number of bits, an output value, and a back-propagation error to control a microprogram. A learning operation processing method for a neurocomputer that performs a desired neural network operation, the method comprising: storing a plurality of microprograms for performing learning including back-propagation errors each having a different decimal point position;
The step of memorizing the condition for switching the microprogram used for learning and the condition for switching in the learning process are selected, and another one is selected from a plurality of microprograms and A learning operation processing method comprising: a step used for learning. 8. A multiplier provided for performing fixed-point multiplication using a predetermined number of bits, and an adder, using a weight value of a predetermined number of bits, an output value, and a back-propagation error to control a microprogram. A learning operation processing method of a neurocomputer for performing a desired neural network operation, which comprises a step of storing a plurality of microprograms including back propagation errors each having a different decimal point position, and a microprogram used for learning before starting weight value learning. A learning operation processing method comprising: a step of selecting a program from the plurality of programs; and a step of learning a weight value using the selected microprogram.