JPH01230164A - Automatic learning type neural net system - Google Patents

Abstract

PURPOSE:To obtain a highly fast neural net at a high identification rate by generating the large quantities of deformed patterns from the small quantities of the patterns in a deformed pattern generating part, and executing learning with the use of the pattern. CONSTITUTION:A deformed pattern generating part 2 generates plural deformed patterns based on the basic pattern sent from a basic pattern input part 1, and stores them into a learning pattern set/expected value memory part 8 as a learning pattern set. A neural net part 3 receives the deformed patterns, with the use of the weight of a connection and the threshold of the output of a unit, the part 3 obtains the output value for the input patterns, and sends it to a comparing part 4. The weight of each connection is calculated by the comparing part 4 and a weight arithmetic part 5, and a neural net part 3 is updated. A pattern presentation control part 6 selects and presents the pattern to be used out of the pattern set stored in a basic pattern memory part 7 and the learning pattern set/expected value memory part 8. Thus, the neural net at the high identification rate can be obtained.

Description

[Detailed Description of the Invention] [Field of Industrial Application] The present invention relates to an automatic learning neural network system that performs learning by changing connection weights for presented patterns, and particularly relates to basic patterns and basic patterns. The present invention relates to an automatic learning neural network system that automatically learns to output a desired value in response to a transformed pattern input simply by inputting an expected value corresponding to the pattern.

[Prior art]

In recent years, I have been using biological neural networks as a model.

Research on neural networks, in which processing units M (units) imitating neurons are connected in a network, is once again gaining momentum.

In this neural network, each unit outputs a signal to the output side when the sum of the signals from the input side exceeds a certain value. It is also possible to train a neural network by repeatedly providing input patterns and output patterns. In this case, the network can be trained automatically without the need for humans to define the structure of the network.

As shown in FIG. 2, the units that make up such a neural network include a coupling section 21, a section that receives input from other units (input section 22), and a section that converts the input according to certain rules (conversion section 23). , and a part (output unit 24) that outputs the conversion result.

Further, a variable weight WIJ is attached to each of the coupling parts 21 with other units. A certain unit has multiple units 1~
When receiving input from , the sum of its output and weight is taken as the input value. The input value net is expressed by the following equation (1).

(1) net4=ΣW10° Further, the output of the unit is expressed by applying the input value net to the function f, for example, by the following equation (2).

(2) f 1=1 / (1+ e−”t+)
In formula (2), a sigmoid function is used, but other functions may be used as long as they are nonlinear saturation functions, such as a 5tep function or a broken line function. Also, in equation (2),
The following equation (2)' may be used by adding a threshold value O to net.

(2) / fl: 1 / (1+ 8-(nekai+6
1)) Furthermore, when this unit is realized electronically, it is configured as shown in FIG. 3, for example. In this case, the weight of the connection is realized by a variable resistor, and the nonlinearity of the unit utilizes the analog input/output characteristics of the inverter. Note that an operational amplifier can be used instead of the inverter.

In addition, the conventional neural network pattern recognition device has 1
For example, it is expressed as shown in FIG.

In this way, the conventional neural network type pattern identification device is composed of three layers, an input section 41, an intermediate section 42, and an output section 43 each having a plurality of units, and are connected in this direction. Note that each unit is indicated by a circle.

A typical method for learning the weights of each connection in such a configuration is the backpropagation method.

In this backpropagation method, when input data is given to each unit of the input section 41, each input data is outputted by the unit of the input section 41 to all or a plurality of units of the intermediate section 42, and each of the units after the intermediate section 42 The unit converts the sum of the load forces of all inputs (the sum of the products of the output value of the previous unit and the weight of the connection) using a nonlinear function, and finally the output part 4
Output to 3. Furthermore, this output value and the desired output value (
(teacher data or expected value) and change the weight W of the connection so as to reduce the difference.

That is, when a certain pattern p is given, the difference between the actual output value (OPJ) of a certain unit of the output section 43 and the desired output value (t□) is evaluated using the following equation (3).

(3) Ep=1/2(tpJ-0+-, t)" Therefore,
Learning is performed by changing the weight wB so that this difference becomes smaller.

Further, the amount of change in accumulation when p is given is shown by the following equation (4).

(4) ΔpW, = ηδ1. ○, Note that O7 is the input value from unit 1 to unit j. Further, δ7 differs depending on whether the unit j is an output section 13 unit or an intermediate section 42 unit.

In other words, in the case of the unit of the output section 43, the following (5)
It is expressed by the formula.

(5) δr=: (tr, -o, ,,)f,' (ne
trJ) Furthermore, in the case of the unit in the intermediate section 42, it is expressed by the following equation (6).

(6) δPJ='f j' (netPj)ΣδP I
c Wk j Note that the calculation of ΔW starts from the unit of the output section 43, moves to the unit of the intermediate section 42, and finally traces back to the input section 41.

In this way, we repeat the process of manually inputting the training data, outputting the results, and changing the connection weights to reduce the error in the results, and then converging the weights to produce the correct output for the input. .

Therefore, in order to develop a neural network that has a high classification rate for various input patterns, a large amount of learning data including various patterns is required.

In order to obtain these learning data, patterns have been collected manually, for example.

[Problem to be solved by the invention]

In the above-mentioned conventional technology, there is a limit to the number of patterns that can be collected, and a method for collecting patterns including various deformations has not been established, so that the quality is insufficient. Furthermore, there was a problem in that a huge amount of time was required for collection.

The purpose of the present invention is to improve such problems, automatically generate a large amount of deformed patterns from a small amount of learning patterns, and supply learning patterns with sufficient quality and pattern quantity in a short period of time. The purpose of this invention is to provide an automatic learning neural network with a high classification rate.

[Means to solve the problem]

In order to achieve the above objects, the automatic learning neural network system of the present invention includes a neural network composed of a plurality of units, and a neural network that performs learning by changing connection weights for presented patterns. In the network system, means (basic pattern input section) for taking in a small number of patterns belonging to each of a plurality of different categories as basic patterns, and means for accumulating the -W patterns (basic pattern register iα section); Means for generating a modified pattern obtained by modifying any one pattern among the basic patterns (modified pattern generation section), means for accumulating a set of the modified patterns, and expected values of output for input patterns (learning pattern set, an expected value storage section), a means (weight control information storage section) for storing the control information of the weight of the connection between the units and the threshold value at the output of the unit; Then, using the weight of the connection and the threshold value at the output of the unit, the neural net section is used as a means to obtain the output value for the input pattern, and the expected value given in advance according to the output value and the input pattern means for calculating the difference (comparison unit); if the difference is larger than a preset value, calculating a new weight for the connection using the weight of the connection and the output value of the unit;
The present invention is characterized in that it includes means for updating (weight calculating unit) and means for selecting and presenting a pattern to be used for learning from a set of basic patterns and modified patterns (pattern presentation control unit).

[Effect]

In the present invention, the operator generates various deformed patterns from the basic pattern by simply inputting the basic pattern and the expected output value, and learns to automatically identify the desired pattern using the deformed patterns. do.

〔Example〕

An embodiment of the present invention will be described below with reference to the drawings.

FIG. 5 is a configuration diagram of an automatic learning neural network system according to an embodiment of the present invention.

The automatic learning neural network system of this example is
Input/output terminal 51, central processing unit (hereinafter abbreviated as CPU) 5
2. and a storage device 53.

This input/output terminal 51 inputs the pattern to be learned and the expected value at which the system will read, recognize, and output the pattern through an operator's operation, and input it to the CPU 52.
give it to Further, if the difference between the output value for the input pattern and its expected value is smaller than a preset value, a message indicating that learning has been completed is output.

Further, the CPU 52 generates a plurality of modified patterns from the basic pattern obtained via the input/output terminal 51, and weights the modified patterns. Furthermore, the difference between the weighted value (output value) and the expected value is calculated, and if it is larger than the preset value, the current weighting is changed/updated.

In addition, the storage device 53 stores patterns (basic patterns) manually created by the operator, expected values, and the CPU 52.
A plurality of modified patterns (learning pattern cents) generated for each basic pattern, weighting control information for the modified patterns, and output value thresholds are stored.

FIG. 1 is a functional configuration diagram of an automatic learning neural network system according to an embodiment of the present invention, FIG. 6 is a diagram showing a deformed pattern in an embodiment of the present invention, and FIG. 7 is an embodiment of the present invention. It is an explanatory view of a pattern presentation method in an example.

The automatic learning neural network system of this example is
As shown in FIG. 1, it includes a storage section 10 and a processing section 11,
Perform optical character recognition.

The storage unit 10 also includes a basic pattern storage unit 7, a learning pattern set/expected value storage unit 8, and a weight control information storage unit 9.

The basic pattern storage unit 7 stores the patterns input to the basic pattern input unit 1, that is, binary patterns obtained by scanning photoelectric conversion and binarization, as basic patterns in small numbers for each category.

Further, the learning pattern set/expected value storage unit 8 stores the modified patterns generated from the basic pattern by the modified pattern generating unit 2 as a learning pattern set. At the same time, the expected value of the output input when inputting the basic pattern is also stored.

Further, the weight control information storage section 9 stores control information for updating the weight of the output value calculated by the neural network section 3 in the 1-weight calculation section 5.

Furthermore, the processing unit 11 includes a basic pattern input unit 1, a modified pattern generation unit 2, a neural network unit 3, a comparison unit 4,
It includes a weight calculation section 5 and a pattern presentation control section 6.

The basic pattern input unit 1- inputs a pattern to be learned, that is, a binary pattern obtained by scanning photoelectric conversion and binarization, as a basic pattern by an operator's operation, and stores it in the basic pattern storage unit 7. Furthermore, one of these basic patterns is sent to the modified pattern generation section 2.

Further, the modified pattern generation section 2 includes the basic pattern input section 1.
A plurality of modified patterns are generated based on the basic pattern sent from the controller and stored in the learning pattern set/expected value storage section 8 as a learning pattern set. Further, the deformed pattern is sent to the neural network section 3.

Regarding the generation method of this deformation pattern, for example,
Assign a number from 1 to N to each point of the basic pattern divided into BXB meshes on the plane according to the stroke order, and assign the coordinates of point i [1] to (X-work).

If the point moved from that point by the perturbation of r is defined as (x,', y'), then the following equation (7) is obtained.

However, 1≦j≦N (N: the total sum of points existing on the basic pattern), and θ□ is the angle that the base makes with the X axis.

In addition, to generate r, a random number R which takes a value of (-1, 1) is generated and convolved with a dispersive Gaussian function gk.

That is, rl is expressed by the following equation (8).

However, L is the distance between adjacent meshes in the X or Y direction.

Furthermore, gk is expressed by the following equation (9).

(9) gk=A-c-exp [-18 to 2L2/Z”
] However, C is represented by the following equation (10).

However, Z indicates the degree of smoothing, and the larger 2, the smoother the change in r. Further, A indicates the amplitude of the perturbation, and the larger A is, the larger the deviation from the basic pattern is.

A deformation pattern based on such an operating principle is shown, for example, as shown in FIG. In this case, the deformation parameter A(
amplitude of perturbation) is O~0, 9 am, deformation parameter W
(J, t; Distribution of deviation from the main pattern) Ha1, 05
mm ~10, 65 nni 1'a<'l. The frame for the letters is 9.6 mm x 9.6 mm, and the mesh is 64 x 64.

In this way, a modified pattern is generated for each category pattern, and is temporarily stored in the learning pattern set/expected value storage section 8 as a learning pattern set.

At the same time, when the basic pattern is manually input to the modified pattern generating section 2, the desired expected value as an output is also stored in the learning pattern set/expected value storage section 8.

In addition to this method of using perturbation for automatic character transformation, the same method can also be used when performing linear or nonlinear transformation from a basic pattern (standard character form), or when performing global or local transformation. It can be used for.

Further, the neural network section 3 is composed of units as shown in FIG. 3, and includes a learning pattern set/expected value storage section 8.
The deformed pattern accumulated in the deformed pattern is received from the deformed pattern generator 2, and the values of BXB meshes of this deformed pattern (
By inputting 1 for black and 0 for white, and using the weights in the neural network, the output values determined by equations (1) and (2) are obtained.

This output value is sent to the comparator 4 from each unit of the output section (see FIG. 4).

Further, the comparison unit 4 compares the output value of the neural network unit 3 with
The expected value of the corresponding pattern read from the learning pattern set/expected value storage section 8 is compared with the expected value of the corresponding pattern by 1 (cow), and the difference is calculated.

Furthermore, if the difference is greater than or equal to a preset value, the weight calculation unit 5 calculates the weight of each connection by the method shown in equations (4) to (6), and updates the neural network unit 3 as a new weight. do.

The pattern presentation control section 6 also includes a basic pattern storage section 7.
A pattern to be used for learning is selected and presented from among the basic patterns stored in the learning pattern set/expected value storage section 8 and the learning pattern cellar 1- stored in the learning pattern set/expected value storage section 8.

This pattern presentation method uses cyclic presentation (a), random number presentation (b), repeated presentation (C), long-term presentation of a specific pattern (d), etc. shown in FIG.

In this circular presentation (a), for example, when learning 50 categories of katakana, the modified pattern group of ``A'', the modified pattern group of ``A'', and so on, each modified pattern of each group is shown once. Repeat selection and presentation.

In the random number presentation (b), one pattern is selected from among all patterns of all groups by random numbers and presented.

In the repeated presentation (c), a pattern selected using random numbers is repeatedly presented a plurality of times (71 times).

In addition, in the long-term presentation of a specific pattern (d), a predetermined pattern is presented T2 times (T2>T,), and after the neural network completes these learnings, it repeats the same process for other patterns selected by random numbers. Repeat the presentation and learning.

The pattern presentation control unit 6 may perform only one of the methods (a) to (d), select one of a plurality of presentation methods, or control the progress of learning. The presentation method may be changed during the display.

FIG. 8 is a flowchart showing the processing of the automatic learning neural network system in one embodiment of the present invention.

The automatic learning neural network shown in Figures 5 to 7
Regarding the processing procedure of the system, when the operator inputs the basic pattern into the modified pattern generator 2 (801),
A large number of deformed patterns are generated (802), and these deformed patterns are stored as learning pattern cents (803).
). At this time, the expected value of the output input at the time of inputting the basic pattern is also stored at the same time.

Next, one pattern is read out from the pattern set and input to the neural network unit 3 (804).
.

The neural network unit 3 calculates the output value using the weight of each connection and the threshold value of the unit (805
).

Next, the comparison unit 4 compares the expected value corresponding to the pattern input to the neural network unit 3 with the output value of the neural network unit 3 (806), and when the difference is greater than a preset value, the weight is The calculation unit 5 performs neural network 1-
The weight is updated by calculating the weight (807).

Thereafter, other patterns are read from the learning pattern set, and these operations are repeated in the same manner until the difference between the output value and the expected value becomes equal to or less than a preset value.

Although this example deals with an automatic learning neural network system used for optical character recognition, similar methods can also be applied to recognition of patterns that can be expressed in vectors, such as online character recognition, voice recognition, and image recognition. Automatic learning can be performed depending on the configuration.

For example, when performing online character recognition, a pattern imported from an input device such as a tablet as a series of pen point coordinates is preprocessed using known techniques such as noise removal and size normalization. , are stored in the basic pattern input section 1.

Furthermore, the deformed pattern generation unit 2 creates a thinned binary pattern by connecting each writing point of the pattern of the writing point coordinate series with a straight line, and uses the same method as in optical character recognition. Perform processing.

In addition, when performing image recognition, a pattern imported from an input device such as a TV camera as a multivalued image is processed through well-known techniques such as differential processing and area division processing, and then converted into a binary line image with basic patterns. It is stored in the input section 1.

Furthermore, the deformed pattern generating section 2 obtains a deformed line missing image by varying the position coordinates of the bending point and the branching point, for example, by Aff1ne transformation. Note that the following processing is the same as that for optical character recognition.

Furthermore, when performing voice recognition, the voice waveform captured from a voice input device such as a microphone is subjected to digital processing using a known technique, and cepstrum coefficient values for each order are stored in the basic pattern input unit 1. Ru.

Furthermore, the deformation pattern generation unit 2 calculates each person's coefficient value (
Multiply by 1+α). Note that α is usually 1α; 0.5 and randomly selected. Further, the following processing is similar to optical character recognition.

〔Effect of the invention〕

According to the present invention, a large amount of deformed patterns are generated from a small amount of patterns in the deformed pattern generating section, and learning is performed using the large amount of patterns.

A neural network with high identification rate and high resistance can be realized.

Further, there is no need to manually collect a large amount of patterns including various deformations, and collection work that requires a huge amount of time and effort can be avoided.

Furthermore, by using multiple pattern presentation methods,
It is possible to efficiently realize the neural network desired by the designer.

[Brief explanation of the drawing]

Fig. 1 is a functional block diagram of an automatic learning neural network according to an embodiment of the present invention, Fig. 2 is a block diagram of a unit in a conventional neural network, Fig. 3 is a block diagram of unit realization, and Fig. 4 is a block diagram of a conventional neural network. Explanatory diagram of neural network, 5th
The figure is a block diagram of an automatic learning neural network system according to an embodiment of the present invention, FIG. 6 is a diagram showing a deformed pattern in an embodiment of the present invention, and FIG. 7 is a diagram of a pattern presentation method according to an embodiment of the present invention. The explanatory diagram, FIG. 8, is a flowchart showing the processing of the automatic learning neural network system in one embodiment of the present invention. Basic pattern input section, 2: Modified pattern generation section, 3:
Neural net section, 4: Comparison section, 5: Weight calculation section, 6
: pattern presentation control section, 7: basic pattern storage section, 8:
learning pattern set/expected value storage unit, 9: weight control information storage unit, 1o: storage unit, 11: processing unit, 21: coupling unit,
22: Input part of unit, 23: Conversion part, 24: Output part of unit, 41: Output part of neural network, 42:
Intermediate section, 43: input section of neural network, 51: input/output device, 52: central processing unit (CPU), 53: storage device. Patent Applicant: Nippon Telegraph and Telephone Corporation 'L''+ ',''R Figure 2 << Figure 4 Teacher↓ Input pattern p Figure 5 Figure 6 Figure 8

Claims (1)

[Claims] (1) A neural network that is equipped with a neural network composed of multiple units and that learns by changing connection weights for presented patterns.
In the system, means for incorporating a small number of patterns belonging to each of a plurality of different categories as basic patterns, means for accumulating the basic patterns, and generating a modified pattern that is a modification of any one pattern among the basic patterns. means for accumulating a set of the deformed patterns and expected values of outputs for the input patterns; means for storing control information of the weights of the inter-unit connections and threshold values for the outputs of the units; and the deformed patterns. Taking one pattern in the set as an input pattern, means for obtaining an output value for the input pattern using the weight of the connection and a threshold value at the output of the unit; , means for calculating the difference from a pre-given expected value, and if the difference is larger than the pre-set value, the weight of the connection;
and means for calculating and updating a new weight of the connection using the output value of the unit, and means for selecting and presenting a pattern to be used for learning from the basic pattern and the modified pattern set. An automatic learning neural network system characterized by: