JP2736361B2 - Neural network configuration method - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to a method of constructing a neural network used for pattern recognition and learning by back propagation.

2. Description of the Related Art Conventionally, feature amounts (vectors) to be identified are input to a neural network, and learning is performed by back propagation to identify a category (pattern) to which the input belongs.

FIG. 1 shows an example of the configuration of a neural network that performs learning by back propagation. The units are connected to each other between the input layer, the hidden layer, and the output layer, and each connection has a weight. In the back propagation, the weight of the connection is learned so as to output a desired value based on an error from a desired output value.

First, the weight value of each connection is set at random, and an input and a desired output for the input are given. Next, the input layer,
The output for the input is determined in the direction of the hidden layer and the output layer. The units in the input layer use the input values as output values as they are. For the units of the hidden layer and the output layer, the input values are the weighted sum of the output values of the plurality of units of the lower layer and the weight of the connection between the units. Each unit of the hidden layer and the output layer converts an input value by a non-linear function (such as a sigmoid function) and outputs an output value. Next, the error between the output of the unit in the output layer and the desired output is evaluated. The threshold value of each unit and the weight of the connection are updated so as to reduce the value of this evaluation function.

Learning is repeated by the above procedure until the evaluation value becomes sufficiently small. (For information on back propagation, see Para
llel Distributed Processing ", Rumelhart, McClelland,
and PDP Research Group, The MIT Press, 1986. The amount of computation in one learning is the number of connections (when the number of units in the input, hidden, and output layers is I, J, and K, respectively,
I × J + J × K). Therefore, if the number of units I, J, K of each layer increases, the amount of calculation also increases.

When applying this conventional neural network configuration to an actual identification problem with a large number of categories to be identified,
The number of units in the output layer increases according to the number of categories to be identified. When the number of categories is increased, the number of units in the hidden layer must be increased in order to obtain a high identification rate. The number of units in the input layer varies depending on the feature quantity to be handled. However, if the number of categories is increased, more feature quantities are required, so the number of units in the input layer also needs to be increased. Therefore, the number of bonds increases in the order of two or three to the increase in the number of lizards to be identified. As the number of connections increases, the number of times of learning until the evaluation value converges to a sufficiently small value tends to increase.

As described above, when the number of categories to be identified is large, the amount of calculation required until the evaluation values converge, that is, the learning time becomes enormous, which is not realistic. Further, if the number of units in the hidden layer and the number of units in the input layer are suppressed to small values in order to make the calculation time realistic, there is a disadvantage that sufficient discrimination performance cannot be obtained.

An object of the present invention is to reduce the learning time and improve the discrimination performance when a neural network is applied to discrimination with many discrimination targets.

"Means for Solving the Problems" The present invention divides identification targets into several groups in advance,
A network for identifying the group and a small-scale network for identifying individual objects to be identified in the group are provided. After learning independently, identification is performed for each network. The most important feature is to determine the discrimination result from both the output result of the net and the output result of the neural net for discriminating the group. The difference from the conventional method is that learning and identification are performed by a plurality of independent small-scale networks, and the results are integrated to determine an identification result.

FIG. 2 is an explanatory view of the present invention. A is a network for identifying a group, B _{1 to} B _n are networks for each group, and each layer has a hidden layer in FIG. A,
Each of B _{1 to} B _n has the number of input units corresponding to the feature amount of the input data. In this figure, the input units A, B _{1 to} B _n are all common, but the input units are simply terminals that do not perform nonlinear conversion. No need to be. In each network, the number of units in the hidden layer is appropriately determined by preliminary examination and the like. In A, the network has n output units corresponding to each group, and B _{1 to} B _n have the number of output units corresponding to the elements in each group. A, B ₁ 〜
The output value of the output unit of _Bn is input to the determination circuit.

In this network, at the time of learning, A first learns by back propagation while setting a desired output pattern in the output unit of A so as to identify a group using all learning data. B _{1 to} B _n
Learns by back propagation similarly so as to identify the category in the group using only the learning data belonging to the category in each group. The judgment circuit does not operate during learning. Next, at the time of identification, test data is input to A, B _{1 to} B _n for which learning has been completed, and the identification result is determined by the determination circuit from the output results of A and the output results of B _{1 to} B _n. . The determination circuit selects a sub-network (B _{1 to} B _n ) corresponding to a unit having the maximum active value (output value) among the output units of A, and selects the maximum active value among the output units of the sub-network. B ₁ is a category corresponding to the unit indicating
_{Bn and} the output value of A corresponding to each subnetwork are evaluated, and the identification result is determined by a determination method such as determining the identification result based on the maximum value.

An example in which the neural network construction method of the present invention is applied to consonant recognition of speech will be described below.

Example 1 A monosyllable 14 uttered by a specific speaker and followed by a vowel / a /
Recognition experiments were performed using consonants / b, d, g, p, t, k, z, s, h, m, n, w, y, r /. The input data consists of 16th-order LPC cepstrum coefficients (12 kHz sampling, 16
ms hamming window, frame period 8 ms).

FIG. 3 shows the configuration of the network applied to this experiment. The network identifies the group from the A network and the sub-networks of B _{1 to} B _n (n: the number of groups having a plurality of elements. The number of elements in the group may be 1) which identifies each group. Become. In the experiment, the number of groups was set to seven, and A, B _{1 to} B _n were all three-layer networks. The number of units in the hidden layer of the network A was 30. The number of units in the hidden layers B _{1 to} B _n was selected so that the identification rate within the group was maximized. In this experiment, based on conventional phonetic knowledge, the same articulatory style,
/ z /, / s, h /, / b, d, g /, / p, t, k /, / w, y /, / m, n /, / r / were used as a group.

For each network, the weight of the connection and the threshold of the unit were learned by back propagation using the sigmoid function as a nonlinear function. A's network is 14
Using the consonant data, the sub-networks B _{1 to} B _n were learned using only the data in each group.

At this time, the number of times of learning of each network until the error between the desired output value and the actual output value converges below a predetermined threshold value is about 9000 × 14 times for the network of A (14 is the number of categories, The same applies to the following), and the subnetwork is / s, h /
Is 150 × 2 times, / b, d, g / is 2400 × 3 times, / p, t, k / is 1200 ×
Three times, / w, y / was 140 × 2 times, and / m, n / was 2300 × 2 times. When this is converted into the sum of the product of the number of connections and the number of times of learning, which is the evaluation value of the calculation amount, it is about 514.85 million, which is about 65604 when the previous syllable is learned by one neural network without grouping.
Less than 10,000 and less learning amount, that is, less learning time.

In the discrimination, the one having the largest activity value among the output units was defined as the discrimination result. When given input data, a group to which the input data belongs is determined based on the output of A, and a sub-network corresponding to the group is selected from B _{1 to} B _n . Then, the final identification result was determined based on the output of the selected sub-network. The selection of the sub-network was realized by gating the outputs of B _{1 to} B _{n in} the output result of A. When the group having only one element showed the maximum activity value, the output of A was directly used as the identification result.

The discrimination rate was as shown in FIG. 4. The discrimination rate was 95.7%, higher than 92.5% when all syllables were discriminated by one neural network without grouping.

Example 2 An experiment similar to that of Example 1 was performed using a method of setting a group based on the firing state of a hidden layer of a neural network as a group setting method. In other words, in this method, first, 14
Learning and constructing a neural network for identifying consonants without dividing them into groups. The number of units in the hidden layer at that time was set to 30. Next, the learning data is input to the neural network as input data, the output value is calculated, and the firing state of the hidden layer when the one showing the maximum activity value matches the category of the input data is a 30-dimensional vector. Then, the Euclidean distance between the lattice sound obtained in this way and the corresponding vector was obtained, and a group was set by performing clustering so that the consonants having a short Euclidean distance were put into the same group. This group was divided as shown in FIG.
For this grouping, learning was performed on the network A and the sub-networks B _{1 to} B _{7 in} the same manner as in the first embodiment to form a neural network. The discrimination rate using this neural network is also higher than when discriminating using one neural network without grouping.
5.0%.

In setting this group, when there are a plurality of hidden layers, a matrix of (the number of hidden layers) × (the number of units of each layer) is considered. Divide.

[Effects of the Invention] As described above, learning and identification are performed by a network that divides identification targets into groups and identifies groups, and a plurality of independent small-scale networks that perform identification within each group. By determining the identification result by integrating the above, there is an advantage that the identification performance can be improved as compared with the case where all the objects are identified simultaneously by one neural network. In addition, there is an advantage that learning of a large-scale network is not required and the amount of learning can be reduced.

In addition, all of the objects to be identified are learned by one neural network, and the firing state of the hidden layer unit when it is identified is considered as a vector or matrix, and those with similar firing states of the hidden layer are grouped as the same group by clustering the vector or matrix. By performing grouping, there is an advantage that grouping can be performed statistically without using a priori knowledge.

[Brief description of the drawings]

FIG. 1 is a diagram showing the configuration of a conventional neural network, and FIG.
The figure shows the configuration of the neural network of the present invention.
The figure shows the configuration of the neural network of the embodiment.
FIG. 5 and FIG. 5 are diagrams each showing the result of the example.

Claims (2)

(57) [Claims] 1. A neural network which learns to output a desired value by an algorithm for changing a weight of a connection between an input layer, a hidden layer, and an output layer according to an error between a desired output value and an actual output value. The method comprises: dividing an identification target into several groups, using a first network for identifying the group, and a second network for identifying individual identification targets in the group; Each network is individually trained. At the time of identification, identification is first performed for each network, and then the identification result is obtained from both the output result of the neural network that identifies the group and the output result of the neural network that identifies the group. Is determined. (2) All the objects to be identified are learned and identified by one neural network, and the firing state of the units of the hidden layer at that time is expressed by a vector of the dimension of the number of units of the hidden layer, or when there are a plurality of hidden layers (hidden). 2. The method according to claim 1, wherein a matrix of (number of layers) .times. (Number of units of each layer) is used, and the groups of vectors or matrices having similar firing states are classified into the same group by clustering the vectors or the matrix. Neural net construction method.