JPH0683794A - Non-linear network equipment in pulse frequency coding network, pulse frequency type neuron device and method for generating non-linear transmission function - Google Patents

Abstract

PURPOSE:To improve a non-linear transmission function so as to have saturated non-linear characteristics capable of being smoothly saturated by a large signal by generating an output signal obtained by combining a pulse frequency coding input signal and its delayed input signal and non-linearly converting the combined signal. CONSTITUTION:A non-linear network equipment is constituted of an input means 12 for receiving a pulse frequency coding input signal, a means 17 for delaying the input signal and an OR means 18 for generating a pulse frequency coded output signal by combining the input signal and the delayed input signal and non-linearly converting the combined signal. Since the OR means 18 is used as pool algebraic logic type structure and a time delay method based upon the delay means 17 is used, a non-linear transmission function having saturated non-linear characteristics to be smoothly saturated by a large signal can be improved. Thus an S-shaped transmission function can be improved by combining the pulse frequency coding network with a pulse frequency type neuron as a non-linear transmission means.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a nonlinear network device in a pulse frequency coding network, a pulse frequency type artificial neuron device and a method of generating a nonlinear transfer function.

[0002]

2. Description of the Related Art Neurons, which are the basic unit of information processing in living organisms
The so-called neural network aims to perform parallel processing of information by imitating the function of, and making this neural cell mimicking element a network.

[0003] Here, Rumerhart et al.
The back-propagation algorithm by elhart et al) is one of the most popular and successful neural network learning or fitting algorithms in neural networks. The normal neuron model is assumed to be used in the McCarraf-Pit format, in which the input signal vector is applied to a linearly weighted network, and the output value and vector representing the vector dot product of the input signal vector and the weighting vector are Is generated. The output of the dot product network is usually applied to an output or working network that has a non-linear memoryless transfer function. The most desired non-linearity for a wide range of applications is in the shape of an S-shaped transfer function.

The backpropagation algorithm provides a method for adjusting weights in a network by tracing back, from output to input, various sources of global error created by each weighting element of the weighting vector network. There is. This includes the shape of the transfer function, eg S
Knowledge of the glyph transfer function is required (where
"Sigmoid" is a descriptor for a class of nonlinearities that has two saturation (limit) states and a smooth transition from one to the other). Analytical expressions that describe the sigmoid or alternative "in-vivo" variation of the input signal can determine the small signal gain between the input and the output of the nonlinearity. The small signal variation gain corresponds to the derivative of the output relative to the input signal. Knowledge of the gain combined with the input signal and the current weight value can predict each weighting factor for error.

More importantly, the adaptation process is concerned with minimizing the error between the observed response and the desired or original response for a given input vector. Knowledge of the derivative of the function, the weights and the input values is required to predict the error component of each unadjusted weight.

In order to make a reasonable prediction of each error factor, the activation function should be known, stable and specific. In addition, practice has shown that an S-shaped characteristic is desirable.

The present invention relates to the generation of an S-shaped transfer function in an artificial neuron using pulse frequency type data coding, where a neuron using this type of data coding is deterministic pulse frequency and stochastic. Sometimes classified as a pulse frequency coding neuron.

The deterministic pulse frequency neuron is described by Tomlinson, Jr., January 1990, 9
This is detailed in U.S. Pat. No. 4,893,255 dated. Tomlinson discloses a neuron in which the vector dot product of the input and the reference weighting vector is performed by a pulse width modulation technique. Two dot products are generated, the first product for excitatory inputs and the second product for inhibitory inputs. The elements of each pulse width modulated dot product are ORed separately to form a non-linear dot product, and
Decoded as analog voltage level. Each voltage level is applied individually to the voltage frequency converter. The output of each converter is a deterministic pulse train with a pulse rate corresponding to the input voltage. At this point, two pulse trains (one representing the excitatory dot product and the other representing the inhibitory dot product) are obtained from which the activation function, or Tomlinson "pushes"
What is called a function will be generated. The two pulse trains are merged so that "when both excitatory and inhibitory spikes occur at the same time, the inhibitory spikes completely kill any output spikes." This final pulse train is an approximation representing the desired output function, i.e. the pulse rate corresponding to the reference dot vector modified by the vector dot product of the input vector and the S-shaped transfer function.

A. "Neural network LSI chip capable of on-chip learning" in the minutes of the International Neural Network Conference held in Seattle, Washington, July 8, 1991. The paper by Eguchi, A. et al, describes the state of the art in stochastic coding neural networks. This form of neuron was used for the simulation data presented in this paper. The stochastic coded pulse-frequency neuron used a pulse-width technique and Tomlinson's deterministic voltage-frequency generator instead of a Boolean algebraic logic method operating on a deterministic Bernoulli sequence. However, the indentation or activation functions obtained from either method are equivalent and have the same limitations and disadvantages as will become apparent in the detailed discussion below. Therefore, the present invention is applicable to both stochastic and deterministic neuron output signals for the purpose of producing more original S-shaped transfer functions.

The original S-shaped transfer function is approximated by the equation (1), and the shape as shown in FIG. 8 can be represented as a typical example.

[0011]

[Equation 1]

The derivative of this function is as shown in FIG. 9 and equation (2).

[0013]

[Equation 2]

A representative stochastic coded pulse frequency neuron is shown in FIG. Each neuron 1 has both a set of excitatory and inhibitory inputs, shown as composite elements 2 and 3, respectively. The input signals are weighted using the AND gates 4, 5 and the set of input weighting signals on the weight vector input line 6. The outputs of the two sets of AND gates 4 and 5 are applied to OR gates 7 and 8. The output of the excitatory OR gate 7 becomes a pulse train representing the excitatory input activation function, which is expressed by equation (3).

[0015]

[Equation 3]

Here, net ₊ is the total number of pulses (spike waves) generated at the input of the OR gate 7. f (net ₊ ) is the probability that all the outputs of the OR gate 7 are 1, and represents the upper half of the starting function. Similarly, the output of the OR gate 8 represents the lower half of the starting function, which is expressed by the equation (4).

[0017]

[Equation 4]

This suppressive half-start function signal is sent to the inverter 9
AND gate 10 combines it with the excitatory half-start function from OR gate 7. Pulse rate output of the AND gate 10 is f (net) = f (net +) (1-f (net -)) ............ the (5).

Here, f (net ₊ ) corresponds to the probability that all the output pulses from the OR gate 7 become 1, and (1
-F (net _-)) is the probability of the output of the OR gate 8 becomes zero. Thus, the inverter 9 allows the excitatory pulse from the OR gate 7 to pass through the AND gate 10 only if there is no inhibitory pulse at the output of the OR gate 8.
In this way the complete activation function f (net) is available at the output 11.

To take a closer look at the properties of f (net),
When equations (3) and (4) are applied to equation (5), equation (6) is derived.

[0021]

[Equation 5]

[0022] In addition, net is because they represent a linear sum, net = (net ₊₎ - a _- (net). This is because the pulse frequency coding separately codes the negative and positive values as a pulse rate having a positive value. As is clear from this, it indicates that there are multiple methods of expressing a certain number. For example, “0” can be expressed by (net ₊ ) = (net ₋ ) = q with q ≧ 0. This feature will be shown to have significant significance to the properties of the nonlinear device shown in FIG.

For example, consider a simple example having _two input variables x ₁ and x ₂ , and −1 ≦ x ₁ ≦ + 1 (7) and 0 ≦ x ₂ ≦ 1. ……………… (8).

Since the variable x ₁ is in the negative range, x ₁₊ and x
_It should be possible to express it by the magnitude of two absolute values of _1- , so x ₁ = x ₁₊ − x _1- …………………… (9). Here, it is assumed that 0 ≦ x ₁₊ ≦ 1 0 <x ₁₋ ≦ 1.

Further, in order to maintain consistency, x ₂ = x ₂ ′ ... (10) Here, 0 ≦ x ₂₊ ≦ 1.

Therefore, the sum of x ₁ and x ₂ or net can be expressed as net = x ₁₊ + x ₂₊ −x ₁ -... (11). _{Similarly, (net +) = x 1+} + x 2+ ............ (12) - a _{= x 1- .................. (13) (} net).

Referring to FIG. 10, the variables x ₁₊ and x ₂₊ are the excitatory signals applied to input line 2, while x _1- is the input line 3.
Will be an inhibitory signal applied to.

Applying equations (12) and (13) to equation (6),
Equation (14) is obtained.

[0029]

[Equation 6]

Since the equivalent x ₂ input can be represented by different combinations of x ₂₊ and x _1- , the value of f (net) is x
_It should be noted that ₁ and x ₂ are not determined alone. This is due to the failure of linear extension resulting from the non-linearity of the neurons in FIG.

FIG. 11 shows the evaluation of f (net) in three examples in which the suppressive signal x _1- is kept constant so that the value of net changes. The range of net is -1 to +2. The solid line is the locus of f (net) at x ₁ = 0, 1/2, 1. The broken line curve is f (net) for 0 ≦ x ₁ ≦ 1
It is drawn to suggest the envelope of the extreme range of.

The feature of FIG. 11 is that net and f (ne are not forced to impose unrealistic suppression on x ₁₊ , x ₂₊ , and x ₁₋ inputs.
That is, a single stable transfer characteristic cannot be constructed during t). Therefore, from this simple example, Tomlinson's "indentation" function expressed by Eq. (6) does not necessarily derive the S-shaped characteristic, but rather it is not true for all net values except the limit value ± 1. It can be seen that a uniformly defined function is obtained.

Preferably the derivative, eg df (net)
A better S-shaped transfer characteristic is desired for the purpose of effectively utilizing the back propagation method based on the determination of / d (net).

The above two input examples do not represent neural networks with large numbers of excitatory and inhibitory input signals. To illustrate this, reference is made to FIG. 12 which summarizes a number of simulation results using different values for the excitatory and inhibitory inputs for values of net ranging from -1 to +2. The two numbers attached to each curve represent the excitatory and inhibitory input values. That is, the lower the ₍₊ 6, 3 _-) and the curve appended excitability 6, inhibitory represents the input 3.

Also, unlike FIG. 11, each transfer characteristic is an average over all uniformly distributed combinations of excitatory and inhibitory inputs for a given value of any net.
FIG. 11 also shows that the average transfer characteristic can be substantially derived from the S-shaped characteristic. The number of positive and negative numbers is (1 ₊ ,
1 _-) from (6 _+, 3 _-) to increase until the average characteristic S
It gradually changes from a letter shape to a hyperbolic shape. This is due to the asymmetry of the function between excitatory and inhibitory pulses in the network, and at any particular instant the total number of excitable pulses to which a single inhibitory pulse is applied simultaneously at the input To disable. As the number of inputs increases, the possible increase in inputs adds the given value of net, and also at least 1
One inhibitory pulse will be included at a given moment. Therefore, if this part increases, for a given value of net,
The expected (average) value of the activation function decreases with increasing number of invalidations occurring in the pulse train. One object of the invention is to correct this erasing effect.

In addition to the fact that FIG. 12 is a set of average numbers, the actual simulation used to obtain this data is based on equation (6) and the theoretical process of inducing f (net) in FIG. It is not compliant and has the maximum value f (net) = 1 at the maximum value net = 2. However, the theoretical model for deriving Eq. (6) assumes an infinite number at the input, and if net increases to infinity, f (net) =
It becomes 1.

It is therefore an object of the present invention to provide a method and apparatus for synthesizing an S-shaped activation transfer function in a pulse frequency coded neuron by overcoming the aforementioned drawbacks of the prior art, Moreover,
S compatible with pulse frequency coding neuron currently realized by using Boolean logic and time delay method
It is an object of the present invention to provide a method and an apparatus for generating a sigmoid transfer function, and additionally to provide a method and an apparatus capable of complicating complexity and performance in synthesizing an S-shaped transfer function. .

[0038]

As a nonlinear network device in a pulse frequency coded network, an input means for receiving a pulse frequency coded input signal, a delay means for delaying the input signal, and the input are provided. Signal and the delayed input signal are combined to generate a non-linearly transformed pulse frequency encoded output signal. The pulse frequency type artificial neuron device includes a pulse frequency type neuron having a neuron output signal, at least one excitatory input, an inhibitory input, and a weighted input, and a pulse frequency type neuron coupled to the output of the pulse frequency type neuron. And a non-linear transmission means for forming a neuron network that cooperates with the neuron to improve the S-shaped transfer function.

[0039]

The non-linear transfer function is improved so as to have a saturation non-linearity that is smoothly saturated with a large signal because the OR means is used as the Boolean algebraic logic structure and the time delay method by the delay means is used. Will be things.

Since the Boolean algebraic structure is compatible with the pulse frequency type neuron, by combining such a pulse frequency coding network with the pulse frequency type neuron as the nonlinearity transmission means, the network operation of the network can be improved. Therefore, a large improvement of the S-shaped transfer function is efficiently made.

[0041]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT An embodiment of the present invention will be described with reference to FIGS. The same parts as those shown in FIGS. 8 to 12 are indicated by the same reference numerals. First, as an example, the example (6
₊ , 3 ₋ ), it is assumed that the neuron 1 of FIG. 10 is coupled to the nonlinear network having the smoothly saturated transfer characteristic shown in FIG. 1 in order to change the hyperbolic characteristic of f (net) shown in FIG. To do. Output f (ne of neuron 1
t) is the x (k) input on line 12. Output signal y
The output 13 of (k) becomes g (net). The non-linear network (non-linearity transmission means) 14 needs to be a Boolean algebraic type so as to maintain compatibility with the pulse frequency type processing used in the neuron structure 1. The integrated structure consists of the new neuron 15 shown in FIG.

FIG. 1 illustrates a preferred embodiment of network 14, which has the desirable characteristics described above. Input line 1
Assume that the input x (k) on 2 is a pulse frequency coded signal and is a Bernoulli sequence with time index k. Input signal x (k) and signal x (k delayed by one unit by delay element (delay means) 17 on line 16
-1) is applied to the two inputs of the OR gate 18. The output y (k) appears on the output line 13. The delay element 17 is, for example, a flip-flop.

The pulse frequency coded signal x (k) is a signal for coding the signal level by probability (a stochastic and deterministic averaging processor [Stochastic and Determinist].
icAveraging Processors], P. Mars, W. J. Poppel Baume [P. Mars, and W.M. J. Poppelbaum]
Written by I. E. E. , London, and New York, Peter Peregrinus Publishing [Peter Peregrinus, Lt
d.,], Stevenage, UK and New York, 1981.
Year, ISBN 0-90648-44-3, page 20-31), output y (k) is y (k) = x (k) + x (k-1) -x (k) * x (k-1) ... It can be expressed as (15).

Therefore, the output y (k) is generated at the time indexes k and k-1 by adding the probability of the pulse generated at the time index k to the input line 12 to the probability of the pulse generated at the time index k-1. Equal to the pulse probability. The latter terms are x (k) and x (km)
Is a product of probabilities since does not depend on the Bernoulli sequence for all m ≧ 1. This is the delay element 17
May have a delay of interval m (m ≧ 1).

The average or expected value of y (k) under the assumption that "all pulses represent statistically the same value" is y (k) = 2x (k) -x ² (k ) …………… (16). This is shown in FIGS. 3 and 4. When the value of the output x (k) is small, that is, when the pulse density is low, the output is almost linear with a gain of 2 as shown in FIGS. 3 (a) to 3 (c), and FIG. As shown in f), it saturates at a high pulse density. Squared factor x ² (k)
Can be ignored and the pulse rate is almost doubled. x
When (k) approaches the total amount and maximum pulse rate, the product term becomes important because a large proportion of pulses occur simultaneously. The result is a smoothly saturated transfer characteristic.

[0046] Figure 12 (1 ₊ 1 _-) and (6 _+, 3 _-) if the f (net Non) function for applied here to the network 14, as the results shown approximately in FIG. 5 become.
Example (1 ₊ 1 _-) is it leave S-shaped, kind hyperbolic Example (6 _+, 3 _-) is apparently changed so as to approach the S-shape. Since the curve of FIG. 12 is an average, it cannot be said that applying the average to the transfer characteristic of FIG. 4 is strictly correct. The averaging is done at the output of the non-linear network 14. However, this procedure is predictive of results.

[0047] Figure 6 is inhibitory 1 type excitatory 5 input (5 ₊ 1 _-) shows the results of a computer simulation for example using. The lower curve is the average transfer characteristic f
(Net) is obtained from the output of the circuit of FIG. The upper curve is the tandem network 1 shown in Fig. 7.
G (net) obtained from No. 4.

It should be noted that other possible network structures can be considered as an extension of the above disclosed concept. These include the use of one or more networks of the type shown in FIG. 1 coupled in tandem at the outputs of the neurons of FIG. Each successive tandem network has an increased tendency towards the S-shaped transfer characteristic g (net).
Therefore, the output nonlinearity of FIG. 2 can be interpreted as a tandem combination of networks of the type shown in FIG. Also, the delay interval can change for each tandem coupling network.

FIG. 7 shows a branch delay element 21 and a multi-input OR gate 2 for executing the non-linear conversion of the input signal f (net).
A non-linear structure consisting of two is shown. Single delay element 17 of FIG.
, Various delays (τ ₁ , τ ₂ , ...,
τ _m ) gives almost linear characteristics with low density pulse rate input. The gain is almost equal to the number of branches. As the rate increases or the number of branches increases, the maximum pulse rate approaches a smooth saturated transfer characteristic.

In summary, a method and apparatus has been disclosed for generating saturation nonlinearity using a combination of delay and OR gates. It has been shown to have the desired effect of producing new neuron structures with improved sigmoidal transfer characteristics for network actuation when combined with pulse frequency neurons. Also, unlike the prior art, the non-linear network used to produce this useful result introduces a memory (delay element) into the output stage of a new neuron network that has previously used only memoryless non-linearity. .

[0051]

As described above, the present invention provides, as a nonlinear network device in a pulse frequency coding network, an input means for receiving a pulse frequency coding input signal and a delay for delaying the input signal. And a logical sum means for generating a pulse frequency coded output signal that is nonlinearly converted by combining the input signal and the delayed input signal, and thus has a saturation nonlinearity that is smoothly saturated with a large signal. Thus the non-linear transfer function can be improved.

The pulse frequency type artificial neuron device includes a pulse frequency type neuron having a neuron output signal and at least one excitatory input, an inhibitory input and a weighted input, and an output of this pulse frequency type neuron. Coupled to this neuron in cooperation with S
Since the Boolean algebraic logical structure such as the logical sum means is compatible with the pulse frequency type neuron, the pulse frequency code By combining the optimization network with the pulse frequency type neuron as the non-linearity transmission means, the S-shaped transfer function for network operation can be significantly improved efficiently.

[Brief description of drawings]

FIG. 1 is a block diagram of a non-linear Boolean algebra network showing an embodiment of the present invention.

FIG. 2 is a block diagram showing an improvement in neuron structure.

FIG. 3 is a time series characteristic diagram showing an input signal to a saturated nonlinear network.

FIG. 4 is a transfer characteristic diagram thereof.

FIG. 5 is a waveform diagram showing transfer characteristics for two cases.

FIG. 6 is a characteristic diagram showing a comparison of simulation results depending on the presence / absence of a tandem output non-linear network for five excitatory inputs and one inhibitory input.

FIG. 7 is a block diagram of a delay line structure with branch lines for generating saturated output nonlinearity.

FIG. 8 is a characteristic diagram showing an S-shaped transfer function.

FIG. 9 is a characteristic diagram showing its derivative.

FIG. 10 is a block diagram showing a conventional stochastic pulse frequency neuron configuration with excitatory and inhibitory inputs.

FIG. 11 is a characteristic diagram showing evaluation of a unique starting function f (net) for two input variables and a constant suppression value.

FIG. 12 is a characteristic diagram showing average activation functions for different numbers of excitatory and inhibitory inputs obtained by simulation.

[Description of Reference Signs] 1 pulse frequency neuron 12 input means 13 pulse frequency coded output signal 14 nonlinear network device = nonlinearity transfer means 16 delay input signal 17 delay means = delay element 18 OR gate 21 branch line delay line 22 logic Japanese means

Claims (13)

[Claims] 1. An input means for receiving a pulse frequency encoded input signal, a delay means for delaying the input signal, and a pulse frequency code obtained by combining the input signal and the delayed input signal and performing non-linear conversion. A non-linear network device in a pulse frequency coding network, comprising: a logical sum means for generating a digitized output signal. 2. The nonlinear network device in a pulse frequency coding network according to claim 1, wherein the delay means includes a branch delay line having at least one branch line. 3. The nonlinear network device in a pulse frequency coding network according to claim 1, wherein the delay means includes at least one delay element. 4. A nonlinear network device in a pulse frequency coding network according to claim 3, wherein the delay element is a flip-flop. 5. A pulse frequency type neuron each having a neuron output signal and at least one excitatory input, inhibitory input and weighted input, and a neuron output signal coupled to an output of the pulse frequency type neuron and cooperating with the neuron. And a non-linear transfer means for forming a neuron network for improving the S-shaped transfer function. 6. Non-linear conversion means comprising: a non-linearity transmission means, an input means for receiving an output signal of a neuron, a delay means for delaying the output signal, and a combination of the output signal and the delayed output signal. 6. The pulse frequency type artificial neuron device according to claim 5, further comprising a logical sum means for generating the pulse frequency encoded output signal. 7. The pulse frequency type artificial neuron device according to claim 6, wherein the delay means includes a branch delay line having at least one branch line. 8. The pulse frequency type artificial neuron device according to claim 6, wherein the delay means includes at least one delay element. 9. The pulse frequency type artificial neuron device according to claim 8, wherein the delay element is a flip-flop. 10. The non-linearity transmission means is formed by the non-linearity network device according to claim 1, and includes at least one non-linearity transmission means connected in series to a neuron output. Item 5. A pulse frequency type artificial neuron device according to item 5. 11. A receiving step of receiving a pulse frequency encoded input signal, a delay step of delaying the input signal to generate a delayed input signal, and a logical sum of the input signal and the delayed input signal A method of generating a non-linear transfer function, comprising: a conversion step for combining an input signal and the delayed input signal to generate a converted output signal. 12. The method for generating a non-linear transfer function according to claim 11, wherein a large number of various delayed input signals are generated in the delay stage. 13. The method for generating a non-linear transfer function according to claim 11, wherein after applying the processing of each step to the input signal, it is applied at least once to the output signal.