JPH01233579A - Neural net type pattern recognition device - Google Patents

Abstract

PURPOSE:To efficiently learn a new item as maintaining the identification capacity of a learned neural net by attaching new units on an intermediate layer part and an output layer part and providing a connection newly. CONSTITUTION:A unit 33 is provided on the intermediate layer part 30 of a neural net part 10 consisting of an input layer part 20, the intermediate layer part 30, and the output layer part 40, and a unit 42 on the layer part 40 newly, and furthermore, the connections (61-67) are provided newly corresponding to the extension of the units. Also, a comparator 90 compares and decides the output value of an existing unit 41 in the layer 40 with that of an attached unit 42, and issues output corresponding to each unit. In such a case, as a concrete comparison method, for example, a logical equation such as a method in which the output value of the unit with the maximum output value is set as '1' (true) and other as '0' (false), or a method in which the output value of the unit 42 as '1' when the output values of the units 41 and 42 exceed the prescribed value and the other as '0', etc., is used. In such a way, it is possible to learn the new item efficiently as maintaining the identification capacity of the learned net 10.

Description

[Detailed description of the invention] [Field of use in pregnancy].

The present invention relates to a neural network pattern identification device,
In particular, the present invention relates to a neural network pattern recognition device that can efficiently learn new items while maintaining the ability to identify neural patterns that have already been learned.

[Conventional technology]

Conventional devices of this type include, for example, the one described by Inaba
A device disclosed in "Using Neural Nets for Pattern Recognition, Signal Processing, and Knowledge Processing" (Nikkei Electronics, August 10, 1'187 issue (427), p. 115) is known. This device uses a neural network composed of units having the structure shown in FIG.

The unit shown in FIG. 6 consists of a part that receives input from other units, a part that converts the input according to certain rules, and a part that outputs the result. In addition, each unit that connects to another unit has a function of attaching a variable weight WiJ.

When a unit receives input from multiple units,
The output and the sum of the weights are used as the input value. That is,
The sum of inputs “net”′ is neti:ΣWLJoj
−・−(1). Each unit transforms the input summation net by applying it to a nonlinear function f. A typical function here is the sigmoid function, ie.

It is. In addition, adding the threshold O, You may also use

FIG. 7 shows an example of the configuration when the above unit is implemented electronically. The aforementioned connection weights are realized by variable resistors, and the nonlinearity of the unit utilizes the analog input/output characteristics of the inverter. Note that it is also possible to use an operational amplifier instead of an inverter.

FIG. 8 shows an example of the configuration of a conventional neural network pattern recognition device that combines the above-mentioned units. The circles in the figure indicate the above-mentioned units. , an intermediate layer section and an output layer section, which are connected in this direction by the connections.

A typical "backpropagation method" will be described below as an algorithm for learning the weights of each connection.

When input cover turns are given to each unit in the input layer,
Each input cover turn is output by each unit of the input layer section to all or some units of the middle layer section, and each unit after the middle layer section outputs the weighted sum of all input cover turns,
That is, the sum of the products of each output value of the previous stage and the weight of the connection is transformed by a nonlinear function, and finally output to the output layer section.

This output value is compared with a desired output value, that is, training data or an expected value, and the weight of the connection is changed to reduce the difference between the two.

When a certain pattern p is given, the difference between the actual output value (0□) and the desired output value (tPJ) of a certain unit in the output layer section is expressed as EP= (tPJ-○Pj)''...
Evaluate using 3). During learning, the weight W is changed so that this difference becomes smaller.

When p is given, the amount of change in the weight WiJ of the connection connecting unit i and unit j is.

Δ,wiJ::ηδ,,0. , ...(4)
It is. Here, ○Pi is the output value of unit i in the previous stage to unit j, and δ and J differ depending on whether unit j is an output unit or an intermediate unit. If the second knit j is the outgoing kani knit, δP J = (j P J−○PJ) f' = (n
etr=)””(5) Also, if unit j is an intermediate unit, δP J ” f ′ J (netp =
) Σδ, wk, (6).

Calculation of ΔW according to equation (4) starts from the unit in the output layer, moves to the unit in the intermediate layer, and finally goes back to the input layer. In addition, in this case, the intermediate layer portion may have multiple stages.

When performing learning, a learning pattern is input, a result is output, and, as described above, connection weights are changed to reduce errors in the result. By repeating this, the weights converge so that the correct output cover turn is produced for the input cover turn. The above is the configuration and basic operation of the conventional neural network pattern identification device.

[Problem to be solved by the invention]

In this way, you can change the connection weights, i.e.
In a neural network pattern identification device that extracts features and regularities from input information through learning and identifies appropriate information, the identification ability is defined by the number of units in the intermediate layer. Therefore, if a large number of patterns are learned while keeping the number of units in the intermediate layer constant, the discrimination ability will deteriorate.

However, in a pattern recognition device using a neural network, it is often necessary to improve the recognition ability or learn new patterns without changing the number of units in the input layer section. In such a case, you can increase the number of units in the middle layer and output layer, create arbitrary (usually all) connections between the units between each store, and re-learn from the beginning as described above. However, there was a problem in that the parts of the neural network that had already been learned could not be utilized, and learning required a great deal of effort.

The present invention has been made in view of the above circumstances, and its purpose is to solve the above-mentioned problems in conventional neural network pattern identification devices, maintain the already learned identification ability of the neural network, and An object of the present invention is to provide a neural network pattern recognition device that can efficiently learn new items.

[Means to solve the problem]

The above-mentioned object of the present invention is to provide an input layer section, an intermediate layer section, and an output layer section composed of a plurality of units, arranged in this order, and to have a directional connection in the direction of the output layer section connecting the units of each section. learning to change the weight of the connection and output the correct value using the difference between the output at the output layer section and the desired output value for the input value input to each unit of the input layer section. In the neural network pattern recognition device, when learning has progressed, a new unit is added to the intermediate layer section and the output layer section, and the new unit of the intermediate layer section and any unit of the input layer section are added. In learning, a connection is established between the new unit of the output layer section and the additional unit of the intermediate layer section and any other unit, and the new unit of the output layer section outputs the correct output value. This is achieved by a neural network pattern identification device characterized in that it is configured to change the weight of only the newly established connection.

[Effect]

In the neural network pattern recognition device according to the present invention, at a stage when learning has progressed, new units are added to the intermediate layer section and the output layer section, and a new unit in the intermediate layer section is connected to an arbitrary unit in the input layer section. connection, and
We created a connection between a new unit in the output layer and an arbitrary unit in the middle layer, and learned by changing the weight of only this new connection, so we could create a new one without destroying the already learned neural network. It has the advantage of being able to efficiently perform learning focused on the learning target (item).

〔Example〕

Embodiments of the present invention will be described in detail below with reference to the drawings.

FIG. 1 is a block diagram showing a schematic configuration of a neural network pattern identification device showing one embodiment of the present invention. In the figure, 10 indicates an input layer section 20 degrees, an intermediate layer section 30 . A neural network section consisting of an output layer section 40, 21
, 22, 23 and 24 are units that constitute the input layer section 20, 31 and 32 are units that were provided from the beginning among the units that constitute the intermediate layer section 30, and 33 is a newly added unit to the intermediate layer section 30. It shows the unit that has been installed.

Further, 41 is a unit that constitutes the output layer section 40,
42 indicates a unit newly added to the output layer section 40, which has been provided from the beginning. Similarly, regarding connections, connection 5 shown by a solid line
Connections 1 to 60 have been provided from the beginning, and connections 61 to 67 indicated by broken lines are newly added connections in response to the addition of units.

Further, 70 represents a comparison unit that performs a comparison with an expected value given in advance, 80 represents a control unit that controls the entire neural network pattern identification device, and 90 represents a comparator that performs a comparison between outputs.

In the neural network pattern recognition device shown in FIG.
, 41, and connections 51 to 60. It is assumed that learning has already been performed by the original neural network, and the weights of the connections 51 to 60 have been set so that correct output values are output from the oldest units in the output layer. That is, FIG. 1 further shows a state in which new units 33 and 42 and corresponding connections are provided in the intermediate layer section 30 and output layer section 40 in order to identify and output a new pattern.

In the neural network pattern recognition device shown in FIG. A new connection is established between all the units in the output layer section 40 and the intermediate layer section 30 (6).
Items other than 7) may be arbitrarily selected depending on the purpose. Although there is no limit to the number of units added, the number of units added at one time is usually less than or equal to the number of units in each layer of the neural network that has been trained up to that point.

The intermediate layer may be multilayered. However, although there is no restriction on the connection with the lower layer that is input to the additional unit, the connection from the additional unit to the upper layer unit is limited only to the connection to the upper layer additional unit.

In addition, in this embodiment, when adding a new unit after the learning of the existing unit and the additional unit has progressed, the entire unit that has been learned is treated as an existing unit, and a connection is established using the method described above. Learn.

The above-mentioned comparator 90 that performs a comparison between outputs compares and determines the output values of each unit of the output layer section 40.
Output corresponding to each unit of 0. Specific methods for comparison include, for example: (1) Setting the output value of the unit with the largest output value as '1''' (true) and setting the others as 110 It (false) (2) Comparing existing units and additional units If the output value exceeds a preset value (for example, 0.5), a logical formula such as a method in which the output value of the additional unit is set as It I 11 and the others as It O 71 can be used.

FIGS. 2(a) and 2(b) show flowcharts outlining the operation of the control section 80 described above. In addition, Figure 2 (
, ) shows the operation in the learning phase, and (b) shows the operation in the operation phase, respectively. In the embodiment described below, after executing the above-mentioned learning phase, an operation phase including normal operation and additional learning is executed to use a neural network pattern recognition device, and to improve the performance of the neural network. It is meant to improve.

Hereinafter, an example of the operation of voice recognition by the present neural network type pattern recognition apparatus configured as described above will be described as an example.

[Example 1] CV syllables uttered by a specific speaker / b a / r / d
a/t/ga/r/za/, /pa/
, /la 88 hehe, /sa/, /ha/, /ma/, /
13 pieces of na/, /wa/ and /ya/ at 12kHz
After sampling with a Hamming window with a window width of 16 m5 ec, the LPC cepstral coefficients were determined and used as input to the neural network.

The neural network has a 3-layer structure, the number of units in the middle layer is 10, and the number of units in the output layer is the above CV syllable/b.
a/ , /da/ , /ga/ , /za/ , /pa
/, /la/, /ka/.

/sa/, /ha/, /+Ila/, /nam/wa/
and /ya/, and the target values were set so that the unit expressing the syllable corresponding to each input was 0.9, and the other units were 0.1.

Out of the 20 utterances of audio data, 10 utterances were used for learning, and learning was performed 1000 times by pack propagation. After learning, the output value of Dekanit when the test data /pa/ is input is shown in Figure 3 (,).
When /pa/ is input, the unit corresponding to /pa/ shows a value close to 1'', and the other units show a value of 0.
”.

Next, add one unit each to the intermediate layer section and output layer section of this neural network, and create connections between the additional units of the intermediate layer section and all the units of the input layer section, and between the additional units of the output layer section and the intermediate layer section. Connections were established between all units in the layer (including additional units).

In this state, /bya/ was assigned to the additional unit of the output layer section, and learning was performed. In learning, /ba/, /da/.

/ga/ , /za/ , /pa/ , /ta/
, /ka/ , /sa/ , /ha/ + /m
When learning a/, /na/, /iia/ +/ya/, we did not change the weights of connections between existing units, but only changed the weights of newly created connections.

As a learning sample, syllables other than /bya/ are used appropriately, and when /bya/ is input, the output of the unit corresponding to /bya/ added to the output layer section is It 171
By pack propagation, 100
Learning was performed 0 times.

After learning, the output value of the unit in the output layer section when /bya/ is input is shown in FIG. 3(b). As shown in the figure,
Already learned items /ba/, /da/, /ga/, /z
For a/+/pa/, /lam8a/, /sa/, /ha, /ma/, /na/, /wa/, /ya/, the results were similar to those after the previous learning. Ta.

Also, when /bya/ is input, the unit corresponding to /bya/ shows a value exceeding 0.5, and the other units show a sufficiently low value, so it is easy to determine the desired /bya/. can do.

According to this embodiment, the neural network pattern recognition device shown in the above embodiment shows the same results for already learned items as before installing the additional unit, and also shows the same results for new learning items after installing the additional unit. was also confirmed to have sufficient discrimination ability.

[Example 2] Using the same method as in Example 1, learning was performed 1000 times for the vowels /e/ and /i/ using two units in the intermediate layer and two units in the output layer. After that, units were added to the intermediate layer and output layer, and /a/ was learned.

After learning, /e/, /i/ and /a/ are input, and the output values from each unit of the output layer section are input to the comparator 90.
The results were used as the output of the neural network. In other words, when the output values of the units in the output layer section are all 0.5 or less, the output value of the final unit corresponding to each unit is It OIt, and the output value of any existing unit and the output value of the additional unit are If it exceeds 0.5, set the final output value corresponding to the additional unit to "1", and set the others to LL Q 11. When the output value of any of the existing units exceeds 0.5 and the output value of the additional unit is 0.5 or less, the final output value corresponding to the unit with the highest output value among the existing units is tt. 1 u, others It
It was set as OIt.

FIG. 4(a) shows the output results of the output layer section for /e/i/ and /a/, and FIG. 4(b) shows the results of comparison by the comparator 90. Accordingly, the neural network pattern recognition device shown in the above embodiment shows the same results for already learned items as before installing the additional unit, and also shows sufficient results for new learning items after installing the additional unit. It was confirmed that it has a good discrimination ability.

[Example 3] Finally, as another example, 7 segment numbers (0 to 9
) Describe an example of its application to recognition. First, a neural network was trained to recognize five numbers from 0 to 4. The number of units in the input layer section is seven, the number of units in the intermediate layer section is three, and the number of units in the output layer section is five, corresponding to the numbers 0 to 4. Note that learning was performed 500 times with the output kaninit corresponding to each input set as 0.99 and the others as 0.01.

Let this network be No.

Next, the number of recognized digits was increased by 3 to 8 (0 to 7) for the network NO. This network is N
Set to 1. In network N1.

The number of units in the middle layer section is increased by three, all connections are made with each unit in the input layer section, and between the middle layer section and the output layer section, for the newly increased three units in the output layer section, All connections are provided with the intermediate layer section, but no connections are provided between the new unit of the intermediate layer section and the five output units of the network numbers. Learning was performed 500 times for each number, with the ratio of the three new numbers to the other numbers being 1=1.

Furthermore, for network N1, the number of recognized digits was increased by two to 10 (0 to 9). This network is N
Set it to 2. In network N2, the number of units in the intermediate layer is increased by two more than in N1, connections are established with each unit in the input layer, and connections between the intermediate layer and the output layer are connected to the new units in the output layer. Regarding the two items that increased to
All connections are provided between the intermediate layer section, but no connections are provided between the new unit of the intermediate layer section and the eight output units of the network N1. Learning was performed 500 times for each number, with the ratio of the two new numbers and the other numbers being 1 = 1. As a result of the above learning, the output crab unit added in the network N1 fires. (0, 9 or more), suppress the firing of the output layer of the network No., and when the output layer added in the network N2 fires, suppress the firing of the output layer of the network N02N1. It was confirmed that digit recognition was performed perfectly.

FIG. 5 shows the output results of the output layer section to the output unit with respect to the number of inputs.

According to this embodiment as well, the neural network pattern identification device shown in the embodiment described above can perform the following for already learned items:
In addition to showing the same results as before installing the additional unit,
It was confirmed that the system had sufficient discrimination ability even for new learning items after the addition of additional units.

〔Effect of the invention〕

As described above, according to the present invention, the input layer section, the intermediate layer section, and the output layer section each consisting of a plurality of units are arranged in this order, and the input layer section, which is composed of a plurality of units, is arranged in this order, and the input layer section, which connects the units of each section, is arranged in the direction of the output layer section. The connection has a tropic connection, and the weight of the connection is changed using the difference between the output at the output layer section and the desired output value for the input value input to each unit of the input layer section, and the correct value is determined. In a neural network pattern recognition device that is trained to output, at a stage when learning has progressed, a new unit is added to the intermediate layer section and the output layer section, and the new unit of the intermediate layer section and the input layer section are added. and a connection between the new unit of the output layer section and the additional unit of the intermediate layer section and any other arbitrary unit, and provide the correct output value to the new unit of the output layer section. In output learning, by configuring the weight of only the newly created connection to be changed, the neural network can efficiently learn new items while maintaining the discrimination ability of the neural network that has already been trained. This has the remarkable effect of realizing a mold pattern identification device.

In addition, when the initial two unit numbers of the input layer section, intermediate layer section, and output layer section are fl, m, and n, respectively, and one unit is added to each of the intermediate layer section and output layer section, each The number of learning times required for connection learning is the total number of connections (m+1) (Q+
n+1) is learned for (n+1) patterns, so A= α(n + 1)(m+ 1)(Q + n +
1) times, whereas in the case of the present invention, there only need to be two new items and other patterns for the number of connections (Q + m + 1), so the required number of learning times is: B = 2α ((1+m+1) times. Here, α is the number of repetitions of learning for one learning pattern.

For example, R=50. α=100. n=100. n
= m, the A/B valves are 5000, and the number of learning times, therefore the learning time, is approximately 1/5000, and if n = 1000, it is 5000.
It is shortened to 1/0 direction. Normally, it is often n

[Brief explanation of the drawing]

FIG. 1 is a block diagram showing a schematic configuration of a neural network pattern recognition device showing one embodiment of the present invention, FIG. 2 is a flowchart showing an overview of the operation, and FIGS. 3 to 5 are output results in the embodiment. , Figure 6 is a diagram showing the units that make up the neural network, Figure 7 is a diagram showing a configuration example when the unit shown in Figure 6 is realized electronically, and Figure 8 is a diagram showing the conventional neural network. It is a figure showing an example of composition of a type pattern identification device. 10: Neural net section, 20: Input layer section, 21-2
4: Unit of input layer section, 30: Intermediate layer section, 31 to 33
: Intermediate layer unit, 40: Output layer unit, 41.42:
Output layer unit, 51 to 67: Connection, 70
: comparison section, 80: control section, 90: comparator. Patent applicant: Nippon Telegraph and Telephone Corporation Figure 2 (a) Figure 2 (b) Operation phase Figure 4 (a) + 1nput=<<e>>; 0.9C160,09
40,057lL

JFigure 6 #X < (

Claims (3)

[Claims] (1) An input layer section, an intermediate layer section, and an output layer section composed of a plurality of units are arranged in this order, and have a directional connection in the direction of the output layer section connecting the units of each section, and the The weight of the connection is changed using the difference between the output at the output layer section and the desired output value for the input value input to each unit of the input layer section, and learning is made to output the correct value. In a neural network pattern recognition device, when learning has progressed,
A new unit is added to the intermediate layer section and the output layer section, and a connection between the new unit of the intermediate layer section and any unit of the input layer section, and a connection between the new unit of the output layer section and the intermediate layer section is established. In addition to providing a connection with the additional unit and any other arbitrary unit, in learning to output a correct output value to the new unit of the output layer section, the weight of only the newly provided connection is changed. Neural network pattern recognition device. (2) A comparison unit is provided to compare the output value of the existing unit of the output layer unit and the output value of the added unit, and the output value of the neural network is determined based on a predetermined logical formula. A neural network pattern identification device according to claim 1, characterized in that: (3) The comparison logical expression in the comparing section determines that the additional unit is true when the output value of the existing unit and the output value of the additional unit in the output layer section are both equal to or greater than a set value. A neural network pattern identification device according to claim 2, characterized in that: