JPH04367062A - Structure extracting method by neural network - Google Patents

Abstract

PURPOSE:To make useful for the characteristic understanding by presuming the grammar of a language from an example sentence with a neural network, using it for sentence structure analysis, etc., in the language processing and extracting the feature of a signal from a sample with the neural network even in the signal processing. CONSTITUTION:When a set 101 of an example sentence is inputted, conversion is performed with a pattern for learning and an identity mapping learning neural network part 103 learns the identity mapping of a pattern for learning. Thus, the structure of the pattern for learning is reflected to the linking weight between the units of the neural network, and by analyzing the linking weight between the units by a neural network analyzing part 104, the auxiliary and independent relation between the constituents of the pattern is found.

Description

[Detailed description of the invention]

0001

[Industrial Application Field] When the grammar of the target language is not obtained in advance, such as in language processing, neural networks can be used to obtain and use the grammar of the language from example sentences. Furthermore, in signal processing, when the characteristics of a signal cannot be easily discovered, neural networks can be used to acquire the characteristics from a sample, which can be useful for understanding the characteristics of the signal.

0002

2. Description of the Related Art A syntactic analyzer is one of the applications of neural networks to language processing. Among them, there are many related to context-free grammar, such as AIP Conference Proceedings 151, p. 140-p
．． 145, “Context-Free Purging with Connectionist Networks” (AIP Con
ferenceProceedings 151, p.
140-p. 145, “Context-Free P
asing with Connectionist
These methods use a method of constructing a network based on a grammar that is explicitly given in advance, and their purpose is to perform syntax analysis at high speed through parallel processing of neural networks. Therefore, it does not take advantage of the learning ability of neural networks.Morgan Kaufman Publishers, Advances in Neural Information Processing Systems 1, p.537-p.544, “A Massively Parallel・Self-tuning context-free parser” (Morgan Kaufmann)
Publisher, Advances in Ne
uralInformation Processin
gSystems1, p. 537-p. 544, “A
Massively ParallelSelf-Tu
ning Context-Free Parser”
) describes a syntactic analyzer with learning ability, but this too is given a grammar in advance and learns how to apply it. Others, JP-A-1-25
No. 5966 is also an example of using a neural network for language processing, but it also learns the order in which grammar rules are applied, and assumes that the grammar is given in advance.

0003

[Problems to be Solved by the Invention] In conventional applications of neural networks to parsing, the grammar of the language must be explicitly given. therefore,
For languages for which a grammar is not available in advance, a grammar must be created and provided by some means. An object of the present invention is to provide a structure extraction method for extracting the structure of example sentences using a neural network and estimating the grammar of an unknown language for which no grammar has been explicitly given.

Furthermore, we aim to be able to apply this method not only to symbol processing but also to feature extraction in signal processing.

[0005]

[Means for Solving the Problems] As shown in FIG. 1, the structure extraction means includes an input pattern generation section 102, an identity mapping learning neural network section 103, and a neural network analysis section 104. The input pattern generation unit 102 generates an input pattern set to be given to the neural network from the given example sentence set 101. The identity mapping learning neural network unit 103 uses an hourglass multilayer neural network (hereinafter referred to as an hourglass network) to learn an identity mapping of an input pattern (a mapping in which the input value is used as an output value). Neural networks are constructed by combining many units that perform simple calculations.
Each connection between units has a weight called a connection weight. As shown in FIG. 2, the multilayer neural network has a unit 206 that is a component of
2. It has a layered structure of an intermediate layer 203 and an output layer 204,
This is a neural network in which the units forming each layer have connections only with units in adjacent layers. Signals are transmitted through the network from the input layer to the output layer. As shown in FIG. 5, an hourglass network has the same number of units in the input layer 503 and output layer 505,
This is a multilayer neural network in which the number of units in the middle layer 504 is smaller than that. By making the hourglass network learn the identity mapping of the input pattern, the structure of the example sentence is reflected in the connection weights between the units of the hourglass network. The neural network analysis unit 104 is a means for analyzing the connection weights between units of the hourglass network that has been trained, and extracting relationships between elements constituting an example sentence from there.

[0006]

[Operation] When a set of example sentences 101 is given as input,
The input pattern generation unit 102 converts this into a set of numerical patterns that can be directly handled by the neural network. Next, using this as a learning pattern, the identity mapping learning neural network unit 103 causes an hourglass neural network to learn the identity mapping of the learning pattern. In an hourglass network, the number of units in the middle layer is smaller than the number of units in the input and output layers, so by learning the identity mapping, a mechanism for compressing information in the middle layer is formed, which is inherent in the learning pattern. The structure is reflected in the connection weights. After the learning is completed, the neural network analysis unit 104 analyzes the connection weights between the units of the output layer and the intermediate layer of the hourglass network, and discovers the dependence and independence relationships between the elements constituting the pattern. Regarding units in the output layer, a dependent relationship is concluded between units that are heavily coupled from the same unit in the intermediate layer, and an independent relationship is concluded between units that are not. This dependency/independence relationship is extracted as a language structure, and the extraction result 105
shall be.

Furthermore, this method can be applied not only to symbol processing but also to signal processing. When a set of signal samples is given instead of the set of example sentences 101, the input pattern generation unit 1
In step 02, this is converted into a numerical pattern suitable for handling by a neural network. Using this as a learning pattern, the identity mapping learning neural network unit 103 learns the identity mapping of the learning pattern. Thereafter, the neural network analysis unit 104 analyzes the connection weights of the hourglass network, extracts dependence and independence relationships between signal components from the connection weights, and extracts them as extraction results 105.

[0008]

DESCRIPTION OF THE PREFERRED EMBODIMENTS As shown in FIG. 2, a multilayer neural network is constructed by sequentially connecting an input layer 202, some intermediate layers 203, and an output layer 204. Each layer inputs a signal sent from an adjacent layer on the input layer side (previous layer), transforms it, and outputs it to an adjacent layer on the output layer side (next layer). However, the input layer receives signals from the outside and outputs them to the next layer without any conversion. The output layer receives the signal from the previous layer, transforms it, and outputs it to the outside. Signals input from the outside to the input layer are always transmitted through the network from the input layer side to the output layer side, and are never transmitted from the output layer side to the input layer side. Each layer consists of units that perform simple calculations. The units that make up the input layer will be called input units, the units that make up the output layer will be called output units, and the units that make up the intermediate layer will be called intermediate units.

The intermediate unit and the output unit are multi-input and one-output units as shown in FIG. Each unit has input/output characteristics given by equation 1.

0010

[Formula 1] oi=f(neti)

...(Equation 1) oi is the output of unit i, and f is the output function. neti is the weighted sum of the inputs of unit i,
It is given by the number 2.

[0011]

[Math 2]

Here, oj is the unit j in the previous layer.
, wij is the connection weight from unit j to unit i, and θi is the bias that unit i has. Each unit has bonds between units in adjacent layers, and there are no bonds between units in the same layer. As the output function f, a sigmoid function given by Equation 3 is usually used. Figure 4 shows the input/output characteristics of the sigmoid function.

[0013]

[Equation 3] f(x)=1/(1+exp(-x))

...(Math. 3) The input unit is a unit with one input and one output, and the input value is used as the output value.

As shown in FIG. 5, the hourglass neural network is a multilayer network in which the input layer 503 and output layer 505 have the same number of units, and the intermediate layer 504 has a smaller number of units. In this embodiment, a three-layer hourglass network with one intermediate layer is used, but an hourglass network with more intermediate layers is also possible. Let this hourglass network learn identity mapping. An identity mapping is a mapping whose output value is equal to the input value. When an hourglass network learns the identity mapping, a mechanism is created for information compression in the middle layer, and as a result, the structure inherent in the pattern is reflected in the connection weights.

Neural networks learn mappings between inputs and outputs by adjusting connection weights. Here, a learning algorithm called backpropagation method is used to adjust the connection weights. The backpropagation method presents a neural network with a set of an input pattern and a pattern to be output (this is called a teacher pattern), and combines them according to the difference between the pattern actually output by the neural network and the teacher pattern. This is supervised learning that modifies the weights. The modification amount Δwij of the connection weight wij from unit j to unit i is given by Equation 4.

[0016]

[Formula 4] Δwij=−ηδioj

...(Equation 4) η is a constant called learning constant, oj is unit j
This is the output of δi is the error signal of unit i, and how to obtain it differs depending on whether unit i is an output unit or an intermediate unit. δ if unit i is an output unit
i is given by equation 5.

[0017]

[Formula 5] δi=(oi-ti)f'(neti
)
...(Equation 5) Here, ti is the teacher pattern for unit i. Further, f' is a differential coefficient of the function f. If unit i is an intermediate unit, δi is given by Equation 6.

[0018]

[Math 6]

Unit j is a unit in the next layer to unit i. In this way, the error signal is transmitted from the output layer side to the input layer side. According to Equation 4, the mapping is learned by repeatedly presenting an input pattern and a teacher pattern corresponding to the input pattern and modifying the connection weights. FIG. 6 shows a PAD diagram of the backpropagation method.

Since the identity mapping outputs the same value as the input, no information other than the input pattern is required to prepare the teacher pattern. Therefore, it has the advantage that it can be easily applied to completely unknown patterns for which no prior knowledge has been obtained.

Next, a more specific example will be explained.

Example 1 As an example, the structure of a symbol string will be extracted. The symbol strings used were four types of subjects (I, YOU, HE, SHE), one verb (LIKE), and four types of objects (ME, YOU, H).
This is a three-word English sentence made from IM, HER). In addition, I LIKEME. Sentences like ``may not be natural English, but they are allowed here, so 16 sentences can be made from the combination of subject and object.

The structure of the neural network is shown in FIG. When the number of symbols used in a symbol string is S and the length of the symbol string is L, the input layer has S×L units, and each unit represents each symbol and its position in the symbol string. ing. In this example, one word is considered one symbol, and the words used are I, YOU, HE, SHE, ME, H.
There are 10 types, including 9 types: IM, HER, LIKE, and LIKES, plus blanks, and an English sentence consists of 3 words, so the number of input units is 30. For example, in this network I LIKE YOU. When inputting the sentence, enter the value 1 in the units corresponding to I in the first column, LIKE in the second column, and YOU in the third column, and enter the value 0 in the other units. The configuration of the output layer is the same as that of the input layer. The middle layer is one layer. Each intermediate unit has connections with all input units and all output units.

The maximum learning error is used to determine whether the identity mapping is realized as a result of learning. maximum learning error
maxerr is defined by equation 7.

[0025]

[Math 7]

Here, oip is the output value of output unit i when pattern p is input, and tip is input pattern p.
This is the teacher pattern of the output unit i for the output unit i. max
If err<0.1, it is assumed that the identity mapping has been realized.

In order to determine the number of intermediate units, we investigated whether the structure of the language could be properly acquired for various numbers of intermediate units. We determined whether the language structure had been acquired as follows. 1 out of 16 English sentences
Learn identity mapping using 5 pieces. It is assumed that the structure of the language has been acquired if one sentence that was not used for learning is input to the network thus learned, and the identity mapping is realized even for the unlearned pattern. Since all 16 sentences are considered to have the same structure, if the structure of the 15 sentences has been acquired, the structure should be applicable to one unlearned sentence.

Furthermore, Akaike's information criterion AIC is introduced. AIC is defined by equation 8.

[0029]

[Formula 8] AIC=NlogeE+2m

...(Equation 8) Here, N is the number of learning patterns, and m is the number of intermediate units. E is the sum of squares of errors and is given by Equation 9.

[0030]

[Math. 9]

[0031] However, M is the number of output units. AI
C is a quantity used to evaluate whether the number of parameters of the approximation function is appropriate in the problem of finding an approximation function of the original function from given sample points; in that case,
N is the number of sample points, and m is the number of parameters. In the ideal case, as the number of parameters increases, the AI
The value of C decreases, reaches a minimum at a certain stage, and then shows a tendency to increase slightly. Therefore, the number of parameters for which the AIC value is the minimum is determined to be the optimum number. N is the number of learning patterns,
Although it may be a bit unreasonable to replace m with the number of intermediate units and introduce AIC into neural network problems, we have incorporated it as a reference. Regarding AIC, please refer to Iwanami Lecture Software Science 9 "Numerical Processing Programming" p. 169-p. 171, p. 176-p. 1
78 was used as a reference.

[0032] For each case where the number of intermediate units is from 3 to 9, an experiment in which the network learns the identity mapping is performed 20 times by changing the initial value of the connection weight, and the maximum error for the unlearned pattern and the AIC A comparison was made. The learning constant is 0.5, the number of repetitions of connection weight correction in one learning is 100,000 times for each learning pattern, and the initial value of connection weights is (-1, 1).
given as a uniform random number. Table 1 shows the number of intermediate units from 3 to 9.
For each case, the number of times the identity mapping was realized for the learning data out of 20 learnings and the number of times the identity mapping was realized for the unlearned data are also shown. Further, FIG. 8 shows the relationship between the number of intermediate units and AIC. FIG. 8 plots the average value of AIC when the identity mapping can be realized for the learning data.

[0033]

[Table 1]

When there were three intermediate units, the identity mapping could not be learned correctly in any of the 20 learnings. When there are four or more intermediate units, the more intermediate units there are, the easier it is to realize identity mapping for learning patterns, but it tends to be rather difficult to realize identity mapping for unlearned patterns. From this result, it can be said that four intermediate units is appropriate for estimating the structure of a language. In addition, A
Looking at the value of IC, it is not the minimum when the number of intermediate units is 4, but it is not so far off, it decreases as the number of intermediate units increases, and increases when the number of intermediate units is increased. Since the general trend is correct, it can be used as a reference when determining the number of intermediate units.

Table 2 shows an example of the firing pattern of the intermediate layer with respect to the input pattern when the number of intermediate units is 4. Further, Table 3 shows the connection weights between the input unit and the intermediate unit at that time, and Table 4 shows the connection weights between the intermediate unit and the output unit. In Table 3, connection weights whose absolute value is less than 1 are omitted because they do not affect the output of the intermediate unit. Similarly, in Table 4, connection weights with absolute values less than 2 are omitted. In Tables 3 and 4, the input unit and
Output unit numbers 1 to 4 correspond to the first column of the symbol string, unit numbers 18 and 19 correspond to the second column, and unit numbers 22 to 27 correspond to the third column. In addition, input and output units that never fire were omitted throughout all the patterns.

[0036]

[Table 2]

[0037]

[Table 3]

[0038]

[Table 4]

From the connection weight between the intermediate unit and the output unit, it can be seen that there is the following dependent/independent relationship between the output units. - The first and second columns are influenced by intermediate units 1 and 4. The third column is influenced by intermediate units 2,3. Since there is no intermediate unit that affects both the first, second and third columns, the first, second and third columns are independent. - Since the first and second columns are both influenced by the intermediate unit 1, the first and second columns are dependent.

In general, if there are too many intermediate units, information compression will not be sufficient in the intermediate layer, and therefore pattern redundancy will remain and the essential structure will not be obtained. On the other hand, if there are too few, the necessary information cannot be expressed in the intermediate layer, and the identity mapping will not be realized. Therefore, the number of intermediate units is selected to be as small as possible without increasing the output error with respect to the learning pattern.

Example 2 Next, how many example sentences are needed for learning to realize the identity mapping for 16 sentences, or in other words, how many example sentences are needed to estimate the structure of the language? I looked into it. In Example 1, we obtained a result that 4 is the appropriate number of intermediate units, so here we set the number of intermediate units to 4, and set 20 combination weights as in Example 1 for various numbers of learning patterns. Learning was performed from initial values.

The results are shown in Table 5. Each column in the table represents the number of learning patterns, the number of times the identity mapping was realized for the learning data out of 20 patterns, and the number of times the identity mapping was realized for the unlearned data as well. As a result, the structure of the language could be acquired even from 12 example sentences.

[0043]

[Table 5]

When an unknown sentence is input to an hourglass network that has learned the structure of a certain language, if the sentence matches the structure of the learned language, the hourglass network can output the same sentence as the input. If the sentence does not match the structure, it will not be possible to output the same sentence as the input. Therefore, it can be used as a judge of whether the input sentence matches the learned structure. Furthermore, by looking at the firing patterns of the intermediate units of the hourglass network, we can see how the input sentence was processed by the hourglass network, which can be useful for syntactic analysis.

[0045]

[Effects of the Invention] For unknown languages whose grammars have not been obtained in advance, the grammar of the language can be estimated from example sentences using a neural network, and can be used for syntactic analysis. In addition, not only in symbol processing but also in signal processing, neural networks can be used to extract features of signals whose features cannot be easily discovered, and this can be used to help understand the characteristics of the signal.

[Brief explanation of the drawing]

FIG. 1 is a flowchart of structure extraction.

[Figure 2] Multilayer neural network.

FIG. 3: Units configuring a neural network.

[Figure 4] Input-output characteristics of a sigmoid function.

FIG. 5: Hourglass network.

FIG. 6 is a PAD diagram of the backpropagation method.

FIG. 7 shows the configuration of the neural network used in the example.

FIG. 8 shows the relationship between the number of intermediate units and AIC.

[Explanation of symbols]

101...Set of example sentences used for structure extraction, 102...Input pattern generation section, 103...Identity mapping learning neural network section, 104...Neural network analysis section, 1
05...Structure extraction result, 701...Input layer in which input units are arranged in a matrix, 702...Intermediate layer with fewer units than the input and output layers, 703...Output layer with the same structure as the input layer.

Claims (8)

[Claims] Claim 1: Using a sample of a pattern belonging to a certain pattern set as input, a neural network consisting of a large number of units that perform simple calculations of multiple inputs and one output are connected via weighted combinations, and the sample pattern is constant. A neural network characterized by comprising a process of learning an identity mapping, and a process of estimating independence and dependency relationships between constituent elements of patterns belonging to a pattern set by analyzing the connection weights of the neural network that has learned the identity mapping. structure extraction method. 2. In the structure extraction method according to claim 1, the target pattern set is a set of symbol strings belonging to a certain language, and from a given example sentence, the independence and dependence relationships between the symbols constituting the symbol string are determined. A structure extraction method using a neural network that is characterized by estimating. 3. The structure extraction method according to claim 1, wherein the target pattern set is a signal set, and independence and dependence relationships between signal components are estimated from a given signal sample. A structure extraction method using a neural network. 4. In the structure extraction method according to claim 1, the neural network for learning the identity mapping comprises an input layer,
It is a multilayer neural network that consists of a set of units called a middle layer and an output layer connected in order, and a signal given to the input layer is transmitted only in one direction from the input layer side to the output layer side. and the number of units in the output layer is equal,
A structure extraction method using a neural network characterized by an hourglass-shaped multilayer neural network in which the number of units in the middle layer is smaller than that. 5. In the structure extraction method according to claim 1, the process of analyzing the connection weights of the neural network that has learned the identity mapping of the sample pattern includes analyzing the connection weights between the intermediate layer and the output layer, and analyzing the connection weights between the intermediate layer and the output layer. A structure extraction method using a neural network, which is characterized in that, regarding units in a layer, a dependent relationship is concluded between units that are heavily connected to each other from the same unit in an intermediate layer, and an independent relationship is concluded between units that are not. 6. The structure extraction method according to claim 1, wherein the number of units in the intermediate layer of the neural network that learns the identity mapping of the sample pattern can be determined with reference to Akaike's information criterion AIC. Structure extraction method using neural network. 7. In the structure extraction method according to claim 1, the number of units in the intermediate layer of the neural network that learns the identity mapping of the input pattern is set as small as possible within a range that can realize the identity mapping for all sample patterns. A structure extraction method using a neural network. 8. In the structure extraction method for symbol strings according to claim 2, the input layer of the neural network for learning the identity mapping has units arranged in a matrix and located in the i-th row and the j-th column. The unit is the symbol i at position j in the symbol string
Due to this shape of the input layer, inputting a symbol string into the neural network requires no information other than all the symbols that can appear in the symbol string and the maximum length of the symbol string. A structure extraction method using a neural network.