JP2518007B2 - Dynamic neural network with learning mechanism - Google Patents

Description

TECHNICAL FIELD The present invention relates to a dynamic neural network having a pattern learning mechanism used for recognizing a time-series pattern of speech or the like.

(Prior Art) A neural network is an information processing system devised with reference to the fact that the cranial nerve system of a living body is an information processing system composed of nerve cells having relatively simple motion characteristics and a large number of connections between them. In the model, a processing unit corresponding to a nerve cell (hereinafter abbreviated as a unit) and an inter-unit connection connecting the units are provided. The system performs various information processing operations by changing the coefficient of the coupling between the units.

It is expected that this neural network model will be particularly effective as an information processing system for pattern recognition processing of images and voices. For details, see Nikkei Electronics, No. 427, page 115 (Showa 62). (August 10, 2010) "Neural network pattern recognition,
It is used for signal processing and knowledge processing ”. (Hereinafter, referred to as Document 1.) According to Document 1 above, the neural network is the second
As shown in the figure, it has a hierarchical structure called an input layer, an intermediate layer, and an output layer, and each layer is composed of a plurality of units. In addition, inter-unit bonding is allowed only between adjacent layers, and inter-unit bonding within a layer is prohibited. At the time of recognition, the network is given input data as the activity of each unit of the input layer, transmits the information to the adjacent intermediate layers through the unit coupling, and finally reaches the output layer. In this way, the response result of the network to the input data is obtained as the pattern of the activity of the units in the output layer.

A method called supervised learning is used to determine the connection between units so that the network performs the specified operation. That is, present the pattern to be learned to the input layer,
The teacher signal to be output is correspondingly presented to the output layer, and the coupling coefficient is determined so as to reduce the difference between the teacher signal in the output layer and the actual output value. In the case of the neural network having the above-mentioned configuration, this output error minimization learning is called back-fropagation learning, and its detailed algorithm is detailed in Reference 1.

(Problems to be Solved by the Invention) If such a neural network can be used for voice recognition, it is possible to absorb the diversity of voice patterns by learning and realize good recognition characteristics. In order to actually use the above neural network for speech recognition, there are some problems to be solved.

First, even if patterns of the same category (for example, words) have different durations for each utterance or for each speaker, voice patterns of different lengths are presented to the input layer of the same neural network. It is necessary to devise it.

Second, we must establish a learning method that determines the unit-to-unit coupling so that when the speech patterns of different lengths can be presented to the input of the neural network, the network will perform the expected recognition operation.

The present invention presents a speech pattern of different lengths to a neural network having an input layer capable of inputting a time series of characteristic parameters having a fixed time length, so that the time of the input layer is maximized during recognition so that the output of the output layer is maximized. Corresponds the axes to the input speech time series, determines the inter-unit coupling coefficient, and normalizes the pattern to be presented during learning to present it to the network by presenting it to the network to minimize the error in the output layer. It is intended to provide a dynamic neural network having a learning mechanism.

(Means for Solving Problems) The present invention is a neural network for recognizing a time-series pattern such as voice, which has a hierarchical structure including an input / output layer and a plurality of intermediate layers, and further includes an input layer and The middle layer has a time-series structure corresponding to the time axis, and at the time of recognition, the time axis of the input time-series pattern is
In a dynamic neural network that associates with the time axis of the input layer so that the output of the network is maximized, and the output of the output layer at that time is the recognition result, the unit coupling coefficient between the layers When learning, the learning time-series pattern normalized to a constant duration that is the same as the length of the input layer time axis is presented to the input layer, and the teacher signal to be output is presented to the output layer. And a mechanism for performing supervised learning for determining the coupling coefficient so as to reduce the difference between the teacher signal and the actual output value in the output layer.

(Operation) For simplicity of explanation of the principle of the present invention, a model having a three-layer structure with one intermediate layer is used. It goes without saying that the same applies to the case where the number of intermediate layers is two or more.

The input layer of the model is composed of J × P units so that it can receive a time series (length J) of P-dimensional feature vectors. The output value of this input unit is y
⁽¹⁾ _j (p) (j = 1 to J, p = 1 to P). Generally, the length J of the time axis of the input layer and the length I of the input time series pattern a _i (p) (i = 1 to I, p = 1 to P) input at the time of recognition
Are different from each other, it is necessary to perform some expansion / conversion on the time axis of the input time series to align the length J. The correspondence relation on the plane (i, j) constituted by the time axis j of the input layer and the time axis i of the input time series pattern is represented by the following equation.

c (k) = (i (k), j (k)), (k = 1 to K) ...
(1) However, Using this relationship, the output value of the input unit y ⁽¹⁾ _{j (k)} (p)
(J = 1 to J, p = ^{1 to} P) is expressed as y ⁽¹⁾ _{j (k)} (p) = _{ai (k)} (p) (3). That is, the input unit transmits the input data as it is to the next layer while matching the time axis.

The middle layer is J × M units (called hidden units)
Input value x ⁽²⁾ _j (m) (j
= 1 to J, m = 1 to M) is the output value y of the input unit
^{(1) Using} _j (p) and the coupling coefficients β ⁰ _j (m, p) and β ¹ _j (m, p) between the input unit and the hidden unit, it is given by the following equation.

Thus, the j (k) th hidden unit is i
The structure of the neural network in which the unit-to-unit coupling is limited so that the information is received only from the (k) th and i (k-1) th units will be referred to as a time series structure.
When the data itself has a time-series structure such as a voice pattern, the structure of such a network is
Since the model can be constructed with less inter-unit coupling than full coupling (connecting all input units and all hidden units), the amount of calculation at the time of recognition / learning can be significantly reduced. The response of the hidden unit to the input given by equation 4 is:

y ⁽²⁾ _j (m) = f (x ⁽²⁾ _j (m) -θ ⁽²⁾ _j (m)) ...
(5) f (x) = 1 / (1 + e- ^x ) (6) where θ ⁽²⁾ _j (m) is a threshold value of the hidden unit (j, m). As is clear from the equation (6), the hidden unit functions as a kind of threshold logic.

The output layer has N corresponding to N categories to be recognized.
It is composed of individual units. The input value x ⁽³⁾ (n) (n = 1 to N) to the nth output unit is the output value y ⁽²⁾ _j (m) of the hidden unit and the coupling coefficient α ⁿ between the hidden unit and the output unit. It is given by the following equation using (j, m).

The input / output response relationship of the output unit is the same as that in Equation 2.

y ⁽³⁾ (n) = f (x ⁽³⁾ (n) −θ ₍₃₎ (n)) ...
(8) where θ ⁽³⁾ (n) is the threshold value of the output unit n.

The output value y ⁽³⁾ (n) of the network thus obtained depends on the correspondence {c (k)} between the time axis of the input time series and the time axis of the input unit layer, which is given by the equation (1).
The final recognition result by the network of category n is obtained as an optimized (maximized) output value o _{n with} respect to {c (k)}.

Since equation (8) is a monotone function, equation (9) is It is the same even if replaced with. Here, the sum of the feature vector components p in f () is omitted. When the expression in {} of the expression (10) is defined as γ (c (k), c (k-1)), the expression (10) becomes It can be seen that this optimization can be solved using the well-known dynamic programming method. That is, γ (c (k), c
The (k-1) cumulative sum of) as g (k), may be obtained and by calculating the following recurrence formula o _n = g (K).

Next, the unit coupling coefficients {β ⁰ _j (m, p), β ¹ _j (m, p), which are the parameters of the neural network model,
A learning method for determining α ⁿ (j, m)} and the threshold value {θ ⁽²⁾ _j (m), θ ⁽³⁾ (n)} will be described.

A set of time series of feature vectors used for learning of category n is A ⁽ⁿ⁾ _q = {a ⁿ _{q, i} (p)}. Here, q is a subscript that distinguishes a plurality of time series patterns in the same category, i is a subscript that represents the time axis of the time series, and p is a subscript that represents the component of the feature vector at each time. The range of each subscript is n = 1 to N, q = 1 to Q ⁿ , i = 1 to I ^q , p = 1 to P (14) In ^order to present this data A ⁽ⁿ⁾ _q to the network, The length ^{Iq of the} sequence must be normalized to the length J of the time axis of the input layer of the network. It is difficult to use dynamic programming as in recognition because the parameters of the model are not optimized during learning.

Therefore, for learning, a set of data of category n A ⁽ⁿ⁾ _q
(Q = 1~Q ⁿ⁾ time series pattern A as a representative from among the ⁽ⁿ⁾
I picked out _q0, associating the time axis of the other data _{^{A (n) q (q ≠}} q 0) to the time axis of the representative pattern by DP matching. The method is shown below. Representative pattern A ⁽ⁿ⁾
The time axis of _q0 j (j = _1~J), correspondence of the time axis data want to do (normalized) A ⁽ⁿ⁾ _q time axis _{(q = q 0) i (} i = 1~I) And DP at this time two patterns
By matching, the correspondence (distortion function) i = i (j) between the time axes of the two patterns is obtained. Nikkei Electronics about DP matching and distortion function
The magazine, No. 329, page 171 (published on November 7, 1983) explains in detail (hereinafter referred to as reference 2). With this distortion function i (j), the frame vector a ⁿ _{q, i (j)} of the learning data time axis i = i (j) is represented on the time axis j of the representative pattern.
It is understood that it is sufficient to associate This distortion function is j depending on how to limit the local path used for DP matching.
= J (i), and there may be a plurality of frame vectors corresponding to a certain j. In such a case, by averaging the corresponding frame vectors, a similar time axis is obtained. Can be associated.

As a result, the time length I ^q that has varied for each data is normalized to a fixed length I ^q0 . The length J of the time axis of the input layer of the network is set equal to this I ^q0 .

Here, various methods can be considered for selecting the representative pattern of category n. For example, in the pattern set of category n, the cumulative distance d (A _q0 , A _q ) by DP matching between patterns is set as the inter-pattern distance. The quantity Δ given by Let q ₀ that minimizes. This q ₀ is all q = 1
It can be easily obtained by the brute force method of calculating Δ assuming that Q ⁿ is q ₀ . In addition to this, it is also possible to represent an arbitrary one pattern.

In this way, the input learning data obtained by normalizing the length of the time axis to the length J is set as A ⁽ⁿ⁾ _q = {a ⁿ _{q, i} (p)} (i = 1 to J). Further, the learning data of other categories normalized to the same length J is set to B ^(m) _r = {b ^m _{r, i} (p)} (r = 1 to R) (hereinafter, this B is the anti-learning). Called data). At this time, the output value of the network for the qth learning data is y ⁽³⁾ _q
(N), the desired output value is z _q (n) (= 1.0), and the output value of the nth unit for the r-th anti-learning data is y ⁽³⁾ _r
(N), if the desired output value is z _r (n) (= 0.0), the error E of the output value in the output unit layer is Given in. This error amount E must be determined by learning. The unit coupling coefficient {β ⁰ _j (m, p), β ¹ _j
(M, p), α ⁿ (j, m)} and threshold {θ ⁽²⁾ _j (m), θ
⁽³⁾ Since these are considered to be functions of (n)}, these parameters may be determined so as to minimize E as an evaluation function. Further, the threshold value of the unit can be learned in the same manner as the inter-unit coupling by considering a unit that always outputs 1 virtually and considering it as a coupling coefficient with the unit. Therefore two adjacent layers, when the inter-unit coupling coefficient linking the unit i and unit j of the n + 1 layer of the n-layer and omega ⁿ _ij, using a derivative of E about the _{^{ω n ij ω n ij (t}} + 1) = Ω ⁿ _ij (t) -ε (δE / δω ⁿ _ij ) t
If (17), then E (t + 1) ≦ E (t) (18). Here, t is an integer value that represents the step of iterative learning, and ε is a constant that determines the degree of correction. After all, the parameter learning is to repeatedly correct ω ⁿ _ij so as to reduce E. Here, ω ⁿ _ij and the coupling coefficient between units of the model {β ⁰ _j (m, p), β ¹ _j (m, p),
For example, α ⁿ (j, m), θ ⁽²⁾ j (m), θ ⁽³⁾ (n)} may be associated as follows.

It can be seen that the differential coefficient of E is as follows as a result of analytical calculation.

Here, δ ^{(n + 1)} _{i, q} is an error converted into the input value of the unit i of the n + 1-th layer when the q-th learning (or anti-learning) data is presented to the input layer, and y ⁽ⁿ⁾ _{j and q} are output values of the unit j of the nth layer for the qth learning data. δ
⁽ⁿ⁾ _{i, q} can be calculated using the following recurrence formula.

Where f (x) is the input / output response function of the unit given by Equation 6, x ⁽ⁿ⁾ _i is the input value to the unit i of the nth layer, and z _i
Is a value that the unit i of the Nth layer (output layer) should take, and is 1.0 for learning and 0.0 for anti-learning. This learning method is called the back-propagation learning method (back propagation learning method) because the calculation of the error amount δ converted to each unit proceeds from the output layer to the input layer based on this equation 21. (Refer to Reference 1 for details).

After all, starting from a model in which an arbitrary initial value is given to the inter-unit coupling coefficient, if multiple learning / anti-learning data are presented and the above iterative correction learning is performed for each inter-unit coupling, the output layer It is possible to obtain a set of inter-unit couplings that minimize the error.

(Embodiment) The third time is shown below as a time axis correspondence rule (relative positional relationship between c (k) and c (k-1)) on the (i, j) plane for the recurrence formula calculation of Expression 13. An embodiment of the present invention using the rules shown in the figure will be described. In the case of FIG. 3, c (k) = (i,
j), c (k-1) is (i-1, j), (i-
Only three points of 1, j-1) and (i-1, j-2) are possible.
Thus, in the case of the association rule, the equations (12) and (13) that determine the output of the neural network can be written as follows.

g ⁿ (i, j) = γ ⁿ (i, j) + max [g ⁿ (i−1, j), g ⁿ (i
−1, j−1), g ⁿ (i−1, j−2)] (24) FIG. 1 shows an embodiment in which the present invention is realized based on formulas (22) to (24). It is a block diagram. The analysis unit 10 analyzes the input voice waveform data, converts it into a time series of feature vectors, and stores it in the pattern buffer unit 20. The pattern buffer unit 20 stores the learning time series data during the learning operation, and stores the analysis data of the unknown utterance during the recognition operation. The learning operation and the recognition operation are switched by the subsequent changeover switch.

The time-axis matching unit 30 determines the representative pattern of each category in the learning data group based on Equation 15, and performs DP-matching of the other learning data to the representative pattern to perform time-axis matching and all learning data. The length of the time axis of is standardized to the length J. The correction amount calculation unit 40 uses the learning data sent from the time axis matching unit 30 and the coupling coefficient stored in the inter-unit coupling coefficient storage unit 50 to calculate the coupling coefficient ω ⁿ _ij based on Equations 17, 20, and 21. The correction amount Δω ⁿ _ij is calculated, and the coupling coefficient correction unit
Send to 60. The coupling coefficient modification unit 60 adds the modification amount Δω ⁿ _ij to the coupling coefficient stored in the inter-unit coupling coefficient storage unit 50 and writes it back. The correction amount calculation unit 40 repeats this correction operation until the correction amount Δω ⁿ _ij for all coupling coefficients becomes smaller than a predetermined threshold value or the number of correction times exceeds a predetermined number.

The lattice point calculation unit 70 uses the unknown utterance data sent from the pattern buffer unit 20 and the coupling coefficient stored in the inter-unit coupling coefficient storage unit 50 to calculate the lattice point data γ based on Equation 23.
Calculate ⁿ (i, j) (i = 1 to I, j = 1 to J, n = 1 to N). The calculated grid point data is stored in the grid point storage unit 80. The recurrence formula calculation unit 90 uses the grid point data stored in the grid point storage unit 80 to perform the recurrence formula calculation based on Formula 24 and store the cumulative value g ⁿ (I, J) in the working storage unit 100. Store. The work storage unit 100 is used for storing g ⁿ (i, j) even during the recurrence formula calculation. The recognition determination unit 110 outputs, as a recognition result, the value of n that gives the maximum cumulative value among the cumulative values g ⁿ (I, J) stored in the work storage unit 100.

(Effect of the Invention) As described above, according to the present invention, the fluctuation of the utterance time length of unknown speech data during the recognition operation is normalized by the dynamic programming method and can be input to the neural network. It is possible to provide a neural network having the capability of computerization. As described above, since the neural network of the present invention has the ability to normalize the time axis during the recognition operation, the variation of the characteristic parameter for each utterance of the voice data during the learning operation is reduced by a small number of learning data (variance due to the variation of the utterance time length is It is possible to provide a good recognition device by learning by using (you do not have to have).

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of the present invention, FIG. 2 is a diagram showing a hierarchical structure of a neural network, and FIG.
The figure is a diagram showing an example of a time axis association rule on the (i, j) plane for recurrence formula calculation. In the figure, 10 is an analysis unit, 20 is a pattern buffer unit, 30 is a time axis matching unit, 40 is a correction amount calculation unit, 50 is an inter-unit coupling coefficient storage unit, 60 is a coupling coefficient correction unit, and 70 is a grid point. Calculator,
Reference numeral 80 is a grid point storage unit, 90 is a recurrence formula calculation unit, 100 is a working storage unit, and 110 is a recognition determination unit.

Claims (3)

(57) [Claims] 1. A neural network for recognizing a time-series pattern such as voice, having a hierarchical structure composed of an input / output layer and a plurality of intermediate layers, and the input layer and the intermediate layer correspond to a time axis. It has a time-series structure, and at the time of recognition, the time axis of the input time-series pattern is associated with the time axis of the input layer so that the output of the neural network is maximized by dynamic programming, and the output layer at that time is associated. When learning the unit coupling coefficient between each layer in a dynamic neural network that uses the output of the above as the recognition result, the learning time is normalized to a constant duration that is the same as the length of the time axis of the input layer. The sequence pattern is presented to the input layer, the teacher signal to be output correspondingly is presented to the output layer, and the coupling coefficient is determined so as to reduce the difference between the teacher signal in the output layer and the actual output value. Dynamic neural network having a mechanism for supervised learning that. 2. The dynamic neural network according to claim 1, wherein the variation of the duration of the learning time series pattern is normalized by DP matching with the representative pattern. 3. The dynamic neural network according to claim 1, wherein the supervised learning of the inter-unit coupling coefficient is realized by a backpropagation learning method.