JPH0281161A - Signal processor - Google Patents

Abstract

PURPOSE:To attain the learning at a high speed and with high stability by normalizing the learning constant with use of the learning variable which is shown in a reciprocal of the value obtained by adding 1 to the square sum of the input value as a threshold. CONSTITUTION:A signal processing part 10 obtains the output value Opj out of an input signal pattern P while a learning process part 20 performs a learning action to obtain the output value Opj most approximate to the desired output value tpj from the pattern P. Then each of units UH1 - UHy and UO1 - UOy of the part 10 normalizes the learning constant with use of the learning variable shown in a reciprocal of the value obtained by adding 1 to the square sum of the input value as a threshold. Thus the learning rate is dynamically changed according to the input value and the connecting intensity coefficient is learned. As a result, the learning frequency is decreased and the learning is attained at a high speed and with high stability.

Description

DETAILED DESCRIPTION OF THE INVENTION A. Field of Industrial Application The present invention relates to a signal processing device having a learning function, and in particular to a signal processing device having a learning function.
The present invention relates to a signal processing device having a learning function according to a backpropagation (backpropagation) learning rule using a neural network.

B. Summary of the Invention The present invention provides a signal processing device having a learning function according to the pack propagation learning rule using a neural network, by dynamically changing the learning rate according to the input value. It is designed to enable learning.

C Conventional technology: Bank propagation learning rule, which is a neural network learning algorithm r rParallelDi
distributedProcessingJVol,
I The MIT Press 1986 and Nikkei Electronics August 10, 1987 issue No. 427. pρ
Reference J, such as 115-124, is applied to a multilayer neural network having an intermediate layer (32) between an input layer (31) and an output layer (33), as shown in Figure 3, and is used for high-speed image processing and Applications to various signal processing such as pattern recognition are being attempted.

That is, as shown in FIG. 3, each unit (u,) constituting this neural network is a unit Cu.
t) to the unit (uj) W j ! The output value of the unit (Ut) connected by 0. The sum of neLJ
is converted by a predetermined function r such as a sigmoid function, and a value 04 is output. That is, when the value of the pattern p is supplied as a human value to each unit (u,) of the input layer, each unit (uJ,) of the intermediate layer and the output layer
) is expressed by the first equation: oDi=f j (neL@J) 'r·(Σ"ji, Ol1ji) "".

Then, from the input layer (31) to the output layer (33), the output value of the unit (U,) corresponding to each neuron is calculated in sequence (thereby, the unit (u7) with the above output N (33) is calculated. The output flop; is obtained.

In the backpropagation learning algorithm,
Actual output value 0□ and desired output value L of each unit (uJ) of output Ji (33) when pattern p is given
eJ, that is, the sum E of squared errors with respect to the teacher signal.

By sequentially performing a learning process that changes the coupling coefficient wji from the output layer (33) to the input layer (31) so as to minimize the value, the output value Opj closest to the value Le= of the teacher signal is It comes to be output from the unit ) (u=) of layer (33).

Then, the coupling coefficient wj that reduces the sum of squared errors E
The amount of change Δ of i, wji, is ΔwJr CC-9Ep / 0W=i ++...+
If we decide on the third equation, the above third equation becomes 6w5. −η・δ2.・O,i・・・・・・・・・
It can be transformed into Equation 4 (see the above-mentioned document for this process).

Here, η is a learning rate (constant), which is determined empirically from the number of units, the number of layers, input/output values, etc. Further, δpj is an error value of the unit (U,).

Therefore, in order to determine the amount of change Δwji, the error values δ,j may be found in the opposite direction from the output layer to the input layer of the network. output layer unit) (u
j) error value δ9. is δpj=(jp,-opj)
The error value δ, , of the unit (u,) in the intermediate layer is given by the fifth equation f 'j (netJ) - Equation 5, and the error value δ, , , of the unit (u,) of the intermediate layer is determined by each unit (uh
) (In this example, the coupling coefficient W□ of each unino day in the output layer
and error value δ, δpj = f') (netJ)Σδ, to −w,
, ... Calculated using the recursive function of Equation 6 (see the above-mentioned literature for the process of obtaining Equations 5 and 6 above)
.

Note that the above r' j (net=) is the output function f 4
It is the differential value of (net=).

Then, the amount of change ΔW j i is determined by the fifth equation and the sixth equation above.
It is obtained by the above-mentioned fourth equation using the result of the equation,
Using the previous learning results, Δ Wjl(, order 1) 8 η ・δpjyu 0 ,
, +α, ΔWatlml... A more stable result can be obtained by calculating using the seventh equation. Note that α is a stabilization constant for reducing error oscillations and speeding up convergence.

Then, this learning is repeated, and the learning is completed when the sum E of the squared errors between the output value 02° and the value t0 of the teacher signal becomes sufficiently small.

Problems to be Solved by the Invention By the way, in the conventional signal processing apparatus that employs the above-mentioned backpropagation learning agent in the neural network, the unit corresponding to the neuron is The learning constant η was determined empirically from the number of layers, the input/output values, etc., and learning was performed at a constant learning rate using the seventh equation above, so the output value OpJ and the teacher The problem is that the sum of squared errors Ep with the signal value Lpj becomes sufficiently small, and the number of learning repetitions n required to complete learning becomes an enormous value, making it impossible to perform efficient learning. there were.

Therefore, in view of the conventional situation as described above, the present invention aims to adopt a backpropagation learning agent to a neural network to enable efficient learning, and to change the learning rate to the input value. It is possible to perform fast and stable learning by dynamically changing it according to the situation.

EL! Means for Solving the Problems In order to achieve the above-mentioned object, the present invention provides a signal processing section comprising an input layer, an intermediate layer, and an output layer, each of which is composed of a plurality of units that perform signal processing corresponding to neurons. The strength of the coupling between each unit is determined based on the error information δ, mu between the output value of the output layer and the desired output value given as a teacher signal for the input signal pattern manually input to the input layer. A signal processing device comprising: a learning processing unit that sequentially repeatedly calculates coefficients wji from the output layer side to the input layer side and performs learning processing of the coupling strength coefficient Wjl, the learning processing unit comprising: , the amount of change Δwji in the above coupling strength coefficient wji is expressed as η・β(δpj
where η is a learning constant, β is a learning variable, α is a stabilization constant), wji=wji+
Δwji ......The coupling strength coefficient wji shown in the 8th equation is given to each unit of the signal processing section, and the input value Op in each unit is
The above learning constant η is normalized using the above learning variable β, which is expressed as the reciprocal of the sum of squares of i plus 1, as shown in Equation 9. It is characterized by performing learning processing.

F Effect In the signal processing device according to the present invention, by normalizing the learning constant η with the learning variable β, which is the reciprocal of the value obtained by adding 1 as a threshold value to the sum of squares of the input values Opi in each unit, By dynamically changing the learning rate according to the input value Opl, the coefficient W a of the strength of the connection between each unit is calculated.
Perform the learning process for i.

G. Example Hereinafter, an example of the present invention will be described in detail with reference to the drawings.

The signal processing device according to the present invention, as conceptually shown in the block diagram of FIG. The signal processing section (1o) is composed of a learning processing section (2o) that performs learning to obtain an output value 0□ closest to a desired output value Lpj from an input signal pattern p.

The signal processing unit (10) is composed of a neural network, and includes at least an input layer (L) and a middle layer (LM).
) and output layer (Lo), each layer (L
l), (LM), and (LO) are arbitrary numbers of X+y1Z units (u11 to u
lll) + (uNl””ully) + (uol~uos
).

Each of the above units (u11 to u1.) 1 (u□ to u, y)
+(uo+-uos) is n6J mΣ” j i O□ ・・・・・・・・・
For the total sum netJ of inputs expressed in Equation 10, an output value Opj expressed by a sigmoid function expressed in Equation 11 as 1+e with θ as a threshold value is given.

The learning processing section (20) also includes the signal processing section (1).
0) so that the output value O6J of the output layer (LO) for the input signal pattern p input to the second
In the procedure shown in the flowchart in the figure, output Jl
! 1 (Lo) side toward the input layer (L, ) side, each of the units (u++~u1.), (u~U□L (
Coefficient of coupling strength between u ol"'-u oJ W5.
are sequentially and repeatedly calculated, and the learning process of the coupling coefficient wji is performed so that the sum E9 of the squared errors between the desired output value O2 and the output value O84 is made sufficiently small.

That is, first, in step 1, the learning processing section (20) calculates each of the units 1-(ull, ~U)+,
) (Giving the coupling coefficient W4. to u01 to uoJ, the signal processing unit (10) calculates the output value O8 of the output Fff(LO) for the input signal pattern p, and in the next step 2 , for the above output 4fiOo=, the above desired output (l!!) given as a teacher signal.
L, , and the above output value 0. The convergence condition determination operation is performed based on the sum E of the squared errors with j.

In the determination operation of step 2, the signal processing unit (lO
) is the closest value to the desired output value Lpj. The result of the judgment operation in step 2 above is r YES
j In other words, the sum E of the squared errors becomes sufficiently small, and the output (11!oOJ becomes the desired output value Lp).
If the value is closest to j, the learning process is completed, and if the determination result is "NO", the calculation processes of steps 3 to 6 are performed in order.

In the calculation process of step 3, the signal processing unit (lO
) each unit y (u+u~ulIy)+ (uo+~
The error value δ, J of uog) is calculated. In the calculation process of step 3, each unit (
Error value δ of u o+ to u oj. , is δoa=(tp
JOoJ) OoJ (10oJ) 1st
The error value δ9 of each unit (U□~uny) of the intermediate layer (LH) is given by the 12th equation consisting of two equations. is δII=
= o, IJ(1onj)Σδ0. -Wkj・・・・・・
... is given by the 13th equation, which is the 13th equation.

Next, in the calculation process of step 4, each unit (u
, I+-u, the coefficient W of the strength of the connection from the i-th unit to the j-th unit for yL (uot ~ uog)
, the learning variable β of i is set as a threshold value of 1 to the sum of squares of all human power.
It is calculated using Equation 14, which is the reciprocal of the sum of .

Furthermore, in the calculation process of step 5, each unit (
The amount of change Δwji in the coupling coefficient wji from the i-th unit to the j-th unit with respect to u
β(δ11J011.)+α・ΔWj+in+・・・・
...Calculated using the 15th equation.

Then, in the calculation process of step 6, the amount of change Δwji of the coupling coefficient WJl calculated in step 5 is
Based on , each of the above units (
Coupling coefficient W j of u□~uny)+(uo+~uot)
i to w,, = wji + Δwji ...... 1st
Change to type 6.

Then, returning to step 1 above, the signal processing section (1
The above output layer (
Calculation processing for the output value 0°J of Lo) is performed.

This learning processing unit (20) repeatedly performs the operations of steps 1 to 6 described above, and calculates the sum E of the squared errors between the desired output value jp= given as the teacher signal and the output value 0゜j. When the value becomes sufficiently small and the output value 0°J obtained at the output layer (Lo) of the signal processing unit (lO) becomes the value closest to the desired output (aLp=), the learning is performed by the determination operation in step 2. Complete the process.

Like the signal processing device of this embodiment, each of the above units (
Normalize the learning constant η with the learning variable β, which is the reciprocal of the value obtained by adding 1 as a threshold value to the sum of squares of the input value o1 in uH+~u,ly)+(uo+~u,,). The learning rate is set to the above input value 02. By dynamically changing the coupling strength coefficient wji according to the learning process, the number of learning times n can be significantly reduced to 174 to 1/10 of the conventional learning process, resulting in a fast and stable process. I was able to learn.

In addition, the learning constant η and stabilization constant α in Equation 15
is expressed as the 17th and 18th equations as a function of the maximum error value EIlax for the total input cover turn, η=aEmax ・φ...17th equation α=
bEsax + c , , , , Equation 18 (
(a, b, c are constants) By dynamically changing these, faster learning processing can be performed.

H Effects of the Invention In the signal processing device according to the present invention, the learning constant η is normalized by the learning variable β, which is expressed as the reciprocal of the sum of squares of the input (1f!Opt) in each unit plus l as a threshold value. , the learning rate is dynamically changed according to the input value Opi, and the coefficient wji of the strength of the connection is
Since the learning process is performed, high-speed and stable learning can be performed.

[Brief explanation of the drawing]

FIG. 1 is a block diagram conceptually showing the configuration of a signal processing device according to the present invention, and FIG. 2 is a flowchart showing a learning processing process in a learning processing section constituting the signal processing device. FIG. 3 is a schematic diagram showing the most suitable configuration of a neural network to which the backprosivagation learning rule is applied. (10)・・・・・・・・・・・・Signal processing section (
20)・・・・・・・・・・・・Learning processing unit (L
l)・・・・・・・・・・・・・Input layer (LH)・・・・
・・・・・・・・・Middle class (L,)・・・・・・
...... Output layer (uz ~ u + z) + (u, + ~
uwz)+(uox~uom)・・・・・・Unit

Claims (1)

[Scope of Claims] A signal processing unit comprising an input layer, an intermediate layer, and an output layer each configured of a plurality of units that perform signal processing corresponding to a neuron; Based on the error information δ_j_i between the output value of the output layer and the desired output value given as a teacher signal, the coefficient w_j_i of the strength of the coupling between each unit is calculated sequentially from the output layer side to the input layer side. and a learning processing unit that repeatedly calculates the coupling strength coefficient w_j_i and performs learning processing of the coupling strength coefficient w_j_i, wherein the learning processing unit calculates the amount of change Δw_j_i of the coupling strength coefficient w_j_i by η・β(δ_p_j・o_p_i
) (where η is a learning constant and β is a learning variable), the coupling strength coefficient w_j_i of w_j_i = w_j_i + Δw_j_i is given to each unit of the signal processing section, and the input value o_p in each unit is calculated based on
The above learning variable β is expressed as the reciprocal of the sum of the squares of ___i plus 1 ▲ There are mathematical formulas, chemical formulas, tables, etc. ▼ The above learning constant η is normalized at the learning rate A signal processing device characterized by performing learning processing of coefficient w_j_i.