JP2008217592A - Language analysis model learning device, language analysis model learning method, language analysis model learning program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a means capable of strictly determining an error estimation function in segment unit in estimation of a parameter vector with the same calculation amount as that of an approximate calculation method used in a conventional method. <P>SOLUTION: The language model learning device comprises a storage part which stores an output candidate graph in which an output label candidate showing a type of classification tag is associated with learning data 144, and an initial value of a parameter vector 145; and a parameter learning part 1413 which calculates a difference between the appearance probability of a correct answer element of the learning data 114 and the appearance probability of an element which is most likely to be output other than the correct answer element as an error estimation function, using a determination function which shows the appearance probability of a predetermined evaluation unit of the learning data 144 by a peripheral probability with the parameter vector 145 as a variable, and calculates the parameter vector 145 which optimizes an intended function set using the estimation function. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a language analysis technique, and more particularly to a technique for creating a feature parameter vector used for assigning a grammatical or semantic label to a target text from learning data (corpus).

Mainly the subject text (hereinafter referred to as raw text) is given part-of-speech (referred to as terminology or labeling in technical terms), word / sentence breaks or extraction of meaningful words (chunking or segmentation in technical terms) Technology).
On a computer, raw text is processed as a single stream (one-dimensional symbol string) from the beginning to the end of a sentence, so text labeling and segmentation are in the natural language processing / machine learning field. Are classified into the problem types of sequence labeling or sequence segmentation.

An example of such sequence segmentation and sequence labeling is shown in FIG.
As shown in FIG. 1, sequence labeling is defined as a problem of labeling an input sequence (however, a sequence means a sequence of symbols, and in the case of raw text) Equivalent to character strings and word strings). Further, in the natural language processing field, this includes a problem of giving a part of speech tag to a word string.
Similarly, as shown in FIG. 1, sequence segmentation is defined as a problem of obtaining a partial sequence unit segment for an input sequence. Here, a segment is a (partial) symbol string composed of a sequence of one or more symbols. In the field of natural language processing, this includes a problem of adding a phrase break to a word string.

  The sequence segmentation can be easily reduced to sequence labeling, and is generally easier to solve as a sequence labeling problem than to deal with a model that directly solves the sequence segmentation problem. A method is used in which a problem is solved using a model of sequence labeling and then converted into segments.

As a method for solving the sequence labeling from the sequence segmentation, an “IOB labeling method” has been proposed. In this IOB labeling method, a segment is divided and converted into individual labels using labels B, I, and O representing the start (B) and continuation (I) of the segment and the outside (O) of the segment.
Here, FIG. 2 is a diagram for explaining the IOB labeling method. As shown in FIG. 2, by paying attention to the label B indicating the start of the segment, it is possible to easily and uniquely convert from the label to the segment or from the segment to the label.

  Thus, the sequence segmentation problem can be regarded as the same problem as the sequence labeling by the IOB labeling method, and therefore can be solved using the same model as the sequence labeling problem.

Here, in the sequence labeling problem, an input sequence (observation data) is represented as x = {x ₁ ,..., X _n }, and an output sequence is represented as y = {y ₁ ,..., Y _n }. For example, in the case of part-of-speech tagging, x corresponds to raw text and _xi is the i-th word. Also, y is a part of speech tag string corresponding to the raw text of x, and y _i is a part of speech given to the i th word. Here, FIG. 3 is a diagram showing an example of part-of-speech tagging. As shown in FIG. 3, it is considered that the part of speech for each word x _i is given to the raw text x “Taro Yamada is the prime minister of Japan”.

At this time, in the sequence labeling problem, any one output label y _i is determined depending on the surrounding labels (for example, y _i−1 or y _{i + 1} ). That is, it can be said that each label y _i is a variable determined depending on not only the input sequence x but also the output sequence y itself. Therefore, even if the input sequences are partially the same, a completely different label may be given depending on the estimated output labels in the surrounding area. A problem in which the output sequence y itself has an interdependency (or interdependence structure) is called a “structural learning problem” in the field of machine learning research, and is roughly combined with a local optimal solution. The obtained classic model and the global optimum solution of the entire output can be classified into models that directly learn / estimate. As typical models, there are a cascade model and a conditional random field.

The cascade model is a classic method for solving structural learning problems. Conventionally, in order to obtain a global optimum solution of a problem having an interdependent structure, the amount of calculation becomes very large, and thus obtaining a global optimum solution has not been a realistic solution. Therefore, a cascade model has been proposed as a method for obtaining an overall solution by combining local optimal solutions.
Here, FIG. 4 is an explanatory diagram showing an example of sequence labeling to which the cascade model is applied. As the local optimal solution, as shown in FIG. 4, each label y _i is independently learned / estimated. Since learning / estimation of the individual labels can be regarded as a simple classification problem, a generally used machine learning technique for classification problems can be used. In other words, in the cascade model, the problem can be divided for each label (multi-class) and solved as a classification problem using a general learning method.

On the other hand, the conditional random field is one of models devised to obtain a global optimum solution that has been considered difficult in the past. In the field of machine learning, as shown by the cascade model, learning has been performed on the assumption that individual instances in the data are independent. However, it is known that in actual data, there are many cases where there are complex dependencies between individual instances.
For example, in the case of Web homepage category classification, individual instances (homepages) can be handled and classified independently, but in part-of-speech classification for words in a sentence, labels given by context information before and after are different. Has characteristics.

  One method for learning such a structure having a dependency relationship is a Markov random field. However, the Markov random field is a model that is originally used as a method for obtaining the joint probability (p (y, x)) of the input and output, and is not the joint probability when the input x is known as in text, Since it is possible to directly model an analysis model of text by obtaining a conditional probability (p (x | y)), it can be inferred that the analysis performance is improved. Therefore, in the framework of the Markov random field, a conditional random field (Conditiona1 Random Fields) is proposed in Non-Patent Document 1 and Non-Patent Document 2 as an improved method so that the problem can be solved with a conditional probability instead of a joint probability Has been. These methods are models for learning / estimating the output depending on the surrounding context.

Numerous experiments have shown that optimization using global information, such as a conditional random field, performs better than a cascade model that builds up local information and solves the problem.
Here, FIG. 5 is an explanatory diagram illustrating an example of sequence labeling to which a conditional random field is applied. As shown in FIG. 5, when a series labeling problem is modeled with a conditional random field (the same is true for a Markov random field), the path is included in a lattice-like output candidate graph representing a possible output label series. This results in a problem of finding the optimum path (the path with the highest probability) from the start node to the end node.
Note that “learning (also referred to as parameter estimation)” a model using a conditional random field is an operation of determining the weight of a parameter vector corresponding to a feature extracted by feature extraction.

Also, feature extraction is processing that extracts the information that characterizes the relationship between input x and output y, although it is the same processing in a general machine learning problem. Basically, the feature extraction function for extracting features from the input x is generally determined manually depending on the target problem.
Here, FIG. 6 is an explanatory diagram showing an example of feature extraction. In the example shown in FIG. 6, the feature is extracted as a feature vector by combining the two words before and after the label to be estimated (i = 4) and the label to be estimated. As described above, in the case of the sequence labeling problem, it is common to extract features by a combination of the input x and the output y. In the example shown in FIG. 6, combinations of x _i−2 , x _i−1 , x _i , x _{i + 1} , x _{i + 2} and y _i are used as features (functions) used when estimating an arbitrary label y _i. Extract.

  Learning to determine the weight of the parameter vector is a problem of estimating the value of the parameter (vector) λ corresponding to each feature extracted by the feature extraction function. Here, “corresponding” means that each feature and parameter vector elements correspond one-to-one, the feature vector and the parameter vector have the same number of dimensions, and the i-th feature is the i-th parameter. The weight is determined by the elements of the vector.

The conditional random field is defined as follows: The random variable for the input (observed data) is x, and the random variable for the output is y = {y ₁ ,..., Y _n }. Here, when the random variable y satisfies the Markov property, that is, p (y _i | x, y ₁ ,..., Y _i−1 , y _{i + 1} ,..., Y _n ) = p (y _i | x, N ( When the condition of y _i )) is satisfied, (x, y) is a conditional random field. Here, N (y _i ) represents a set of random variables that are connected when the dependency relationship between the individual random variables y _i is represented by a graph. That is, the random variable y _i is a variable determined depending on the connected random variable N (y _i ) and x. In general, if c is a clique set in a graph representing the connection between random variables y _i , the conditional random field is a log-linear model of the potential function of each clique.

Here, since the description is specialized for the sequence labeling problem, only cliques in the case of sequence labeling will be discussed. In the case of series labeling, since a lattice-like output candidate graph as shown in FIG. 5 is used, the clique is composed of y _i−1 , y _i label pairs.
Here, FIG. 7 is a diagram for explaining a clique in a conditional random field. A clique is a set of nodes that become a complete graph among subgraphs in a graph. As shown in FIG. 7, in the lattice-like output candidate graph, only two adjacent nodes become cliques. For this reason, all adjacent nodes connected by links become cliques.
Note that a complete graph refers to a case where a node in the graph has links to all nodes other than itself.

As can be seen from FIG. 7, in sequence labeling, the number of cliques in one path is the sequence length n + 1 (because it includes BOS (start) nodes and EOS (end) nodes).
At this time, in the conditional random field, a local feature at an arbitrary clique c _i is represented as f (y, x, i), and λ is a parameter vector corresponding to the feature. Here, FIG. 8 is a drawing showing an example of features for each clique in the conditional random field. At this time, the conditional probability for an arbitrary output y on the conditional random field can be expressed as follows.

In equation (2), Z _λ (x) means a normalization term and is the sum of all output possibilities. That is, the conditional probability distribution p (x | y) in the conditional random field is the sum of the values with the weight of the local feature f (y, x, i) at each point i in the output sequence y as an index. Divided by the sum of all possible output sequences.
At this time, Formula (1) can be rewritten as follows.

  From the above equation (3), the global feature for one entire path can be written using F as follows.

  That is, it means that the global feature of the series is represented by the sum of local features at each point. According to this global feature F, the conditional probability by the conditional random field of Equation (1) can be rewritten as follows:

は、次式により推定できる。 The probability of an arbitrary output sequence y is the output of all possible output values obtained by converting the inner product of the feature vector extracted from the clique on the path represented by the output sequence in the output candidate graph using an exponential function. Divided by the value of the entire series.
The output sequence with the highest probability for the input sequence x on the conditional random field

Can be estimated by the following equation.

Here, y represents a set of output sequence candidates for the input sequence x. Further, since the normalized term Z _λ (x) in the equation (5) does not depend on y, it does not have any influence when obtaining the maximum likelihood output (argmax), so it is not necessary to use it. Thus, when the input sequence x is given to the conditional random field using Equation (7), the output sequence y with the highest probability can be obtained.

からの学習をおこなう。 In conditional random field learning, the training data is based on maximum likelihood estimation.

Learn from.

Here, y ^* represents a correct answer.

  In general, in a probability model, learning is performed so as to maximize the logarithm (log likelihood) of a probability from a practical viewpoint. This is because even if the logarithm is taken, the parameter value that takes the maximum value is the same (probability and log likelihood are in a proportional relationship), and thus such conversion is possible. This is generally called maximum likelihood learning, and the following objective function is maximized in learning on a conditional random field.

  The optimal solution of Equation (9), which is the objective function, may be found by finding the point at which the differentiation by the parameter λ of the objective function becomes zero.

は、大域的特徴Ｆの期待値を意味する。ここで用いる目的関数、数式（９）は、凸関数であるので唯一の最適解を持つ。実際の最適化には、最急降下法やニュートン法などの一般的な数値最適化アルゴリズムを用いて効率的に解くことができる。
J.Lafferty, A. MaCallum, and F.Pereira. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proc. of ICML-2001, pages 282-289, 2001. F. Sha and F. Pereira. Shallow Parsing with Conditiona1 Random Fields. In Proc. of HLT/NACCL-2003, pages 213-220, 2003. Jun Suzuki， Erik McDermott, and Hideki Isozaki. Training Conditiona1 Random Fields with Multivariate Evaluation Measures. In Association for Computational Linguistics(ACL), pages 217-224, 2006. 磯崎秀樹 鈴木潤，誤り最小化に基づく条件付き確率場の学習:言語解析への適用，言語処理学会第１２回年次大会，2006年 However,

Means the expected value of the global feature F. Since the objective function used here, Equation (9), is a convex function, it has only one optimal solution. In actual optimization, it can be solved efficiently by using a general numerical optimization algorithm such as steepest descent method or Newton method.
J. Lafferty, A. MaCallum, and F. Pereira. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data.In Proc. Of ICML-2001, pages 282-289, 2001. F. Sha and F. Pereira. Shallow Parsing with Conditiona1 Random Fields. In Proc. Of HLT / NACCL-2003, pages 213-220, 2003. Jun Suzuki, Erik McDermott, and Hideki Isozaki. Training Conditiona1 Random Fields with Multivariate Evaluation Measures. In Association for Computational Linguistics (ACL), pages 217-224, 2006. Hideki Amagasaki Jun Suzuki, Learning Conditional Random Fields Based on Error Minimization: Application to Linguistic Analysis, 12th Annual Conference of the Association for Natural Language Processing, 2006

  As described above, generally, in the conditional random field, learning (estimation of the parameter vector λ) is performed using likelihood maximization learning. However, in Non-Patent Document 3 and Non-Patent Document 4, the conditional random field is maximized (or minimized) based on the evaluation index of the task to be solved, rather than learning (parameter estimation) by likelihood maximization. A learning method has been proposed.

  Performance evaluation when using a model learned by a conditional random field for a real problem is almost always performed by using an evaluation index specialized for the target problem. However, likelihood maximization learning is learning that maximizes the likelihood (logarithm of the output probability), and does not guarantee that the performance of the target task after learning is maximized. In other words, “evaluation index maximization learning” proposed in Non-Patent Document 3 and Non-Patent Document 4 eliminates inconsistencies between the objective function optimized during learning and the evaluation index actually used for task evaluation. Making it possible.

  A specific example is given. In the sequence segmentation, the performance of the sequence segmentation is evaluated using an index called F value in each segment unit.

  In Equation (11), TP, FP, and FN are true positive (number of correct answers for positive examples), false positive (number of errors for positive examples of negative examples), and false negative (number of errors for negative examples of positive examples), respectively. ) Represents the number of segments. In addition, γ is a hyperparameter that determines the degree of trade-off between the recall rate and the matching rate in calculating the F value, and is a value that is manually set according to the target task. In general, the F value is calculated using γ = 1.

  The F value is an evaluation index that is considered on the premise of two types of outputs, such as positive examples and negative examples. The “positive example” is the output class that is to be evaluated, and the “negative example” “(Negative)” is the other (not evaluated) output class. In the case of series segmentation, the case where O of IOB labeling is given is considered as a negative example, and all other labels are handled as positive examples. That is, the maximum number of TPs and FNs is the number of positive segments, and the total number of segments of FP is the maximum number.

As can be seen from the comparison between Equation (11) and Equation (9), although there is a certain degree of correlation between the F value and the likelihood of the sequence, an indicator that is closely related such as a proportional relationship Absent. As a specific comparison, the right side of Equation (9) uses the sum of likelihood (linear) for each sample, but the right side of Equation (11) considers positive and negative examples instead of simple summation. This is a fractional (non-linear) formula. In addition, the numerical formula (9) is evaluated with the whole series as one unit, but the numerical formula (11) is evaluated with the segment as one unit.
As described above, in the sequence segmentation problem, even if learning is performed by likelihood maximization learning, learning that maximizes the performance of the problem to be solved is not necessarily performed. Non-Patent Document 3 and Non-Patent Document 4 show that the performance is further improved by performing learning that maximizes the evaluation index of the problem to be solved rather than performing normal likelihood maximization learning. .

  In addition, by using this framework, any evaluation function can be used as long as the evaluation function of the target actual problem is calculated based on the “number of correct answers (or errors)”. It can be introduced as an objective function used for learning of a conditional random field. In other words, generally used evaluation indicators for actual problems are basically calculated based on the number of correct answers (or errors), so by using this framework, most of the currently used evaluation indicators can be calculated. It is possible to learn a conditional random field using this.

  In “evaluation index maximization learning”, an optimization method at the time of learning must be devised according to the evaluation index to be handled. In the evaluation index maximization learning, when considering an optimization method (including an algorithm) for a certain evaluation index, it is necessary to first define the (minimum) evaluation unit used by the evaluation index. Here, since series labeling or segmentation is targeted, there are mainly three types of evaluation units: (1) whole series, (2) segments, and (3) single labels.

As a basic means, the evaluation index maximization learning uses a difference between the correct output y ^{* for} each evaluation unit and the output candidate y (y ≠ y ^* ) other than the correct answer. Then, learning is performed so as to maximize the difference for each evaluation unit based on the evaluation index. In other words, increasing the difference based on the evaluation index that handles the difference between the output candidate that is most likely to be mistaken and the correct output maximizes the final evaluation index.

  The decision function g () of the output y as a whole sequence in the conditional random field is given as the inner product of the feature vector and the parameter vector as follows.

  The decision function is a function representing the likelihood (or score) that a certain output y is output when an input x and a parameter vector λ are given. However, the output y here represents the case where the unit of evaluation is the entire series. The sum i on the right side represents the sum of feature vectors in a single clique unit. At this time, the error estimation function can be defined as follows using a decision function.

  The error estimation function is a function that outputs an estimation amount indicating how much the error is likely to be mistaken compared to the correct answer attached to the learning data and the output other than the correct answer. Here, it represents with the difference of an output candidate other than a correct answer and a correct answer. In other words, this is an estimated amount that means that the larger the difference between the decision function between the output candidate that is most likely to be mistaken and the correct answer, the smaller the mistake.

Finally, an objective function based on the evaluation index to be used is constructed using this error estimation function d (). That is, the evaluation index is calculated at the time of evaluation, and at the time of learning, the evaluation index is calculated using the value returned by the error estimation function. However, when Equation (13) is used, the error estimation function takes a range of [−∞, ∞], but in the actual evaluation index, only the range of [0, 1] may be used. For example, when evaluating with the correct answer rate, since the evaluation index is composed of the number of errors, it is desired to calculate the correct answer as 1 and the error as 0. Therefore, generally, the range is converted using a smoothing function. An example of the smoothing function is a sigmoid function. Here, FIG. 9 is a diagram illustrating an example of the smoothing function. FIG. 9 shows a step function, a sigmoid function, and a logistics function as examples of the smoothing function.
However, the smoothing function is one means for preventing incompatibility and the like, and is not an essential method or condition for evaluation index maximization learning.

When the evaluation unit deals with an evaluation index for the entire sequence, an error estimation function d () as expressed by Equation (13) may be used. Similar to the case of handling the entire sequence, an error estimation function d _{[i, j]} () (in the case of the section from position _{i to j} ) may be defined in units of segments.
Here, FIG. 10 is a diagram for explaining the error estimation calculation methods described in Non-Patent Document 3 and Non-Patent Document 4. In the method shown in FIG. 10, a difference between a correct answer path and a path that is most likely to be output other than the correct answer path is calculated, and an error estimation function for each segment is approximately calculated using the error estimation function for the entire sequence. Is doing.
However, the path that is most likely to be output other than the correct path is selected from paths that do not match the correct path between the target segments y ₂ y ₄ .

  As described above, the error estimation function of the segment is configured using the error estimation function of the entire sequence. The conditional random field originally provides a framework of likelihood maximization learning for the entire sequence. Therefore, there is a background that an efficient calculation method for the decision function g () and the error estimation function d () for the entire sequence has been established. Conversely, in sequence labeling / segmentation in the evaluation index maximization learning framework, a method / algorithm for strictly calculating an error estimation function for each segment has not been invented so far, and an approximate method is used. There is also a background that. That is, in Non-Patent Document 3 and Non-Patent Document 4, the calculation of the error estimation function in units of segments is approximately realized by using almost the same algorithm as that used in learning of the conditional random field.

However, on the other hand, if the error estimation function for the entire sequence is approximately used as in the prior art, there is no guarantee that the segment-by-segment evaluation index to be optimized is truly maximized.
This will be described with reference to FIG. 11, which is a drawing for explaining the problems of the prior art. In the target section [i = 2, j = 2] of the output candidate graph shown in FIG. 11, when a path unit value is used, a C node of c−l−o: 3.8 is output. However, when a segment unit value is used, a B node having a segment likelihood of 8.8 is output.

  When viewed in segment units, there is a mismatch between the segment output probability and the segment output probability based on the sequence output probability because there is a correlation but no match. That is, there is a possibility that the evaluation index that is the original purpose is not maximized in a strict sense.

  In order to solve the above problems, the present invention accurately calculates an error estimation function of an evaluation unit (including a segment unit, a label unit, or an entire sequence) when estimating a parameter vector, and uses an approximation used in the conventional method. It is an object to provide a means that can be realized with the same amount of calculation as a typical calculation method.

The language analysis model learning apparatus according to the present invention, which has been made to solve the above-mentioned problem, is text data in which a classification tag is assigned in advance to a parameter vector used for assigning a predetermined classification tag to text data. A storage unit that is calculated from learning data and stores an output candidate graph in which output label candidates indicating classification tag types are associated with the learning data, an initial value of the parameter vector, and a predetermined evaluation of the learning data Using a decision function that expresses the occurrence probability of a unit as a variable with a parameter vector as a marginal probability, the difference between the occurrence probability of the correct element in the training data and the appearance probability of the element that is most likely to be output other than the correct element is incorrect. A parameter learning unit that calculates a parameter vector that is calculated as an estimation function and that optimizes an objective function that is set using the estimation function. It is characterized in that.
According to the language analysis model learning device of the present invention, it is possible to perform evaluation index maximization learning based on the strict appearance probability of elements of evaluation units of learning data.

  Other aspects of the present invention will be described in detail in the best mode described later.

According to the present invention, various points are improved as compared with the prior art.
Since it is possible to perform evaluation index optimization learning based on a strict appearance probability, it is possible to obtain the best learning result in the meaning of the evaluation index desired to be optimized. The prior art has shown a methodology for replacing the objective function with an evaluation index. However, when actually obtaining the output probability in the evaluation unit, an approximate value is used by diverting a calculation algorithm that has been used conventionally. It was. For this reason, the framework is a learning method that maximizes the evaluation index, but there is a possibility that the evaluation index is not actually maximized. On the other hand, the present invention improves the problem and makes it possible to optimize the evaluation index based on a strict output probability.

  Further, since the output probability of the evaluation unit can be obtained, the output value can be treated as a probability when further processing is performed using this output result. For example, when performing a specific expression extraction using the present invention and using it for information retrieval, it can be used as an index of how reliable the output specific expression is, or information retrieval based on probability theory If it is a system, the named entity system using the present invention can be directly coupled to the information retrieval system. In the conventional method, there is a possibility that the output value does not satisfy the definition of the probability. Therefore, in such a situation, the output value cannot be handled as it is.

  Hereinafter, the system of the present invention will be described, and then the best mode (hereinafter referred to as an embodiment) for carrying out the invention to which the system is applied will be described.

(Outline of learning method of language model)
Here, FIG. 12 is a diagram for explaining the output probability of a certain segment.
In the output candidate graph shown in FIG. 12, the output probability of the (partial) output sequence y _{[i, j]} of the segment between the position i and the position j in the output sequence y is as follows using the marginal probabilities: Can be defined.

は、セグメント［ｉ，ｊ］間のパスを通る全てのパスの重みの総和を表している。また、α_{［０，ｉ］}（ｙ_ｉ）及びβ_{［ｊ，ｎ］}（ｙ_ｊ）は、それぞれ、前側確率及び後側確率であり、［ｉ，ｊ］間の周辺確率を表している。図１２に示した例では、α_{［０，ｉ］}（ｙ_ｉ）は、セグメント［０，２］の確率を、β_{［ｊ，ｎ］}（ｙ_ｊ）は、セグメント［４，６］の確率をそれぞれ表す。そして、前側確率は、次に示す数式（１６）のように定義される。 In Formula (15),

Represents the sum of the weights of all the paths passing through the path between the segments [i, j]. Further, α _{[0, i]} (y _i ) and β _{[j, n]} (y _j ) are a front probability and a rear probability, respectively, and represent a peripheral probability between [i, j]. In the example shown in FIG. 12, α _{[0, i]} (y _i ) is the probability of segment [0, 2], and β _{[j, n]} (y _j ) is the probability of segment [4, 6]. Respectively. The front probability is defined as in the following formula (16).

However, when i = 0, α _{[0, 0]} (y ₀ ) = 1. Also, let Y be a set of single labels (all possible labels). Similarly, the rear probability is also defined as the following formula (17).

However, when j = n, β _{[n, n]} (y _n ) = 1.
Since the denominator of Equation (15) does not depend on y, the segment unit decision function based on the appearance probability of the segment unit can be expressed as follows.

  Therefore, using a decision function based on this segment appearance probability, the error estimation function d () can be expressed as follows.

Here, FIG. 13 is a diagram for explaining the meaning of Equation (19) in the output candidate graph shown in FIG. As shown in FIG. 13, the term of −g _{[i, j]} (y ^* , x, λ) in the equation (19) indicates the appearance probability of the correct segment, and the term below it is the most output other than the correct answer. It shows the appearance probability of the segment that is easy to be performed. However, the segment that is most likely to be output other than the correct answer is selected from the segments that do not match the correct segment in the target section [2, 4].

  By using the marginal probabilities, it is possible to strictly determine the probability that the target segment is selected from the series. Thereby, the error estimation function performs learning using the difference between the probability that the correct segment is output and the probability that other segments are output.

  Comparing Equation (13) and Equation (19) of the prior art as an error estimation function, the surface layer has approximated the segment unit decision function by the value of the entire series in the surface layer. It becomes the form replaced with the accurately calculated value. However, comparing the formula (12) and formula (18) of the prior art as a decision function, the weight of the sequence is used in the prior art, whereas in the present invention, the segment appearance probability using the marginal probability is used. You can see the difference of using the weight based on.

As an actual calculation algorithm, α _{0, k} (y _k ) and β _{l, n} (y _l ) can be efficiently calculated by using a forward-backward algorithm. That is, in the prior art, the Viterbi algorithm is calculated in two passes from the front and the back, and the target segment decision function and the error estimation function are approximated from the output probability of the entire sequence. Instead, the forward-backward algorithm is used to calculate the segment unit decision function and error estimation function. In addition, since the calculation amount of the two-pass Viterbi algorithm according to the prior art and the forward-backward algorithm according to the present invention are exactly the same, according to the present invention, the exact output probability of the segment unit with the same calculation amount as the prior art. Optimization based on the above becomes possible.

(Text analyzer)
Next, a text analysis apparatus according to an embodiment of the present invention to which the above method is applied will be described. FIG. 14 is a block diagram of the text analysis apparatus of this embodiment. As shown in FIG. 14, the text analysis apparatus 1 includes a CPU (Central Processing Unit) 11 that is an arithmetic unit, a RAM (Random Access Memory) 12 in which a program to be described later is expanded and temporarily stored data is held, an external The input / output unit 13 and the storage 14 including a hard disk drive are connected to each other, and are implemented using a personal computer or the like.

Also, the storage 14 of the text analysis apparatus 1 has a program that operates as a learning device 141, an estimator 142, and a tag / segment converter 143, and learning data 144 that is used or generated by each program. , Parameter vector 145, raw text 146, tagged text 147 and segmented tagged text 148 are stored.
The operation of each program and the contents of each data will be described later. Each program and each data stored in the storage 14 can be stored in various computer-readable recording media (CD-ROM or the like).

(Outline of operation of text analysis device)
Next, FIG. 15 is an explanatory diagram for explaining the outline of the processing operation of the text analysis apparatus 1 having the above-described configuration. The processing operation of the text analysis device 1 will be described with reference to FIG.

  The operation of the text analysis apparatus 1 is largely divided into a learning phase in which a parameter vector is calculated based on learning data that is a tagged corpus and an evaluation phase in which tagging or chunking is performed on the raw text using the calculated parameter vector. Divided.

As shown in FIG. 15, in the learning phase, the learning device 141 receives input of learning data 144 that is a tagged corpus and outputs a parameter vector 145.
Then, in the evaluation phase, the estimator 142 performs tagging on the raw text 146 using the output parameter vector 145 and outputs the tagged text 147. Then, the tag / segment converter 143 performs chunking on the tagged text 147 output from the estimator 142 and outputs the segmented tagged text 148. Here, for reference, FIG. 16 shows an example of learning data 144 input to the learning device 141 and an output parameter vector 145, and FIG. 17 shows raw text 146 input to the estimator 142 and an output tag. Examples of the attached text 147 are shown.

In the processing in the estimator 142, the raw text 146 is input instead of the learning data 144 in the processing in the learning device 141, and the parameter vector 145 calculated by the learning device 141 is input instead of the initial value of the parameter vector 145. Therefore, detailed description thereof is omitted. Further, since the processing in the tag / segment converter 143 can apply a known algorithm as described above, the detailed description thereof is omitted.
Further, in the operation example shown in FIG. 15, the estimator 142 executes tagging and outputs the tagged text 147. However, the estimator 142 executes chunking to output the segmented text and outputs the tag / segment. The converter 143 may execute tagging and output the segmented tagged text 148.

(Configuration and operation of learning device)
Next, the learning device 141 among the components of the text analysis device 1 will be described in detail. The learning device 141 corresponds to the language analysis model learning device in the claims, and the processing procedure in the learning device 141 corresponds to the language analysis model learning method.

Here, FIG. 18 is a diagram illustrating a detailed block of the learning device 141. As illustrated in FIG. 18, the learning device 141 includes an output candidate graph generation unit 1411, a feature extraction unit 1412, and a parameter learning unit 1413.
The output candidate graph generation unit 1411 generates an output candidate graph based on the learning data 144 and the output label candidates input from the input / output unit 13. The feature extraction unit 1412 acquires the feature extraction template input from the input / output unit 13, extracts features in units of cliques on the output candidate graph generated by the output candidate graph generation unit 1411, and sets the number of dimensions of the parameter vector 145. Calculate and set its initial value. The parameter learning unit 1413 learns the parameter vector 145 by a learning criterion algorithm based on the output candidate graph and the initial value of the parameter vector 145 set by the feature extraction unit 1412.

  Next, FIG. 19 is a flowchart showing a processing procedure of the learning device 141. The processing procedure in the learning device 141 will be described in detail with reference to FIG. In the following procedure, information generated or calculated by each component is temporarily stored in the RAM, and the description thereof is omitted.

  First, as preparation for operating the learning device 141, an evaluation index used for learning is determined (mainly using an evaluation index of a target problem), and a (minimum) unit of evaluation is determined based on the determined evaluation index. . Then, an objective function used for learning is determined based on the determined evaluation index and evaluation unit.

を所得して（ステップＳ１０１）、ＲＡＭ１２に記憶する。 Then, the operation of the learning device 141 is started, and the output candidate graph generation unit 1411 of the learning device 141 receives the learning data 144 from the storage 14.

Is obtained (step S101) and stored in the RAM 12.

  Next, the output candidate graph generation unit 1411 acquires an output label candidate set depending on the target problem via the input / output unit 13 (step S102) and stores it in the RAM 12. Then, an output candidate graph is generated based on the learning data and the output label candidate set stored in the RAM 12 (step S103).

  Next, the output candidate graph generation unit 1411 passes the processing to the feature extraction unit 1412, and the feature extraction unit 1412 acquires a feature extraction template via the input / output unit 13 (step S104) and stores it in the RAM 12. Then, a feature vector is extracted for each clique on the output candidate graph generated by the output candidate graph generation unit 1411, and the number of dimensions of the parameter vector 145 is calculated (step S105). Further, the feature extraction unit 1412 sets an initial value of the parameter vector 145. The initial value of the parameter vector 145 is obtained by setting each element of the parameter vector 145 to 0, for example.

  Next, the feature extraction unit 1412 passes the processing to the parameter learning unit 1413, and the parameter learning unit 1413 outputs the parameter vector having the output candidate graph generated by the output candidate graph generation unit 1411 and the number of dimensions calculated by the feature extraction unit 1412. Based on the initial value of 145, the parameter vector 145 is learned by the learning reference algorithm (step S106), and the learning result is stored in the storage 14.

(Details of processing in the parameter learning unit)
Next, FIG. 20 is a flowchart showing a detailed processing procedure of the parameter learning unit 1413. The process of the parameter learning unit 1413 in step S106 will be described in further detail with reference to FIG.

  When the process moves to step S107 in the flowchart shown in FIG. 19, the parameter learning unit 1413 calculates a correct output score (weight) for each evaluation unit using the parameter vector 145 at that time (step S201). ). Then, using the parameter vector 145 at that time, a score of an output candidate other than the correct answer (maximum likelihood output candidate) is calculated for each evaluation unit (step S202).

  Next, the parameter learning unit 1413 calculates, for each evaluation unit, the difference between the correct output score calculated in step S201 and the maximum likelihood output candidate score calculated in step S202 (step S203). Then, the objective function L is calculated based on the evaluation index determined in advance (step S204).

Next, the parameter learning unit 1413 may calculate the slope ∇L _lambda of the objective function L calculated in step S204 (step S205), determines whether or not the objective function L is converged (step S206). This determination may be, for example, gradient ∇L _lambda is determined by whether more than a predetermined value. Here, if there converge the objective function L (at step S206 'No'), updates the score using the value of the gradient ∇L _lambda returns to step S201. On the other hand, when the objective function L has converged (“Yes” in step S206), the parameter vector 145 at that time is stored in the storage 14, and the process is terminated (return to step S106 in FIG. 19).

  Next, the operation (learning phase) of the learning device 141 will be described in more detail while showing two specific examples of performing sequence segmentation (see FIGS. 18 to 20 as appropriate).

(Specific example 1: Series segmentation with accuracy rate as evaluation index)
Specific Example 1 shows a specific example in the case of “Segmentation Unit” as “Segment” and “Evaluation Index” as “Segment Correct Rate” in the sequence segmentation problem. The determination of the evaluation unit and the evaluation index corresponds to determining the evaluation index (learning standard) to be applied in step S204 shown in FIG. 20, and is determined in advance as a preparation for the processing of the learning device 141.
In the sequence segmentation problem, as described above, the objective function when the “evaluation unit” is “segment” and the “evaluation index” is “the accuracy rate per segment” is as follows.

  However, the correct answer rate is inversely proportional to the error rate, and the error estimation function of Equation (19) is defined to take a large negative value when the correct answer score is large (generally, in the machine learning field). In this case, error rate minimization is actually performed.

  Therefore, the objective function is defined as follows.

Since the total number of segments is a constant, it is not included in the objective function.
After that, according to the flowchart shown in FIG. 19, the output candidate graph generation unit 1411 of the learning device 141 acquires learning data (tagged corpus) (step S101). At the same time, an output label candidate set is acquired (step S102).

  Here, FIG. 21 is a diagram illustrating an example of learning data or the like acquired by the output candidate graph generation unit 1411. FIG. 21 (a) shows an example of learning data, and a label is given to text such as “Tanaka, Ichiro, is land, federation, president, president” (setting that morpheme breaks are in advance). Things are entered. Further, as shown in FIG. 21B, the output label candidates at this time are five types of labels “B-person name, I-person name, B-organization name, I-organization name, O”. It becomes.

  Next, based on the acquired learning data, the output candidate graph generation unit 1411 generates an output candidate graph (step S103). Here, FIG. 21C is an example of an output candidate graph generated by the output candidate graph generation unit 1411 based on the learning data in FIG. 21A and the output label set in FIG. The output candidate graph shown in FIG. 21C takes a lattice form in which all possible output candidates are connected by a path. That is, one path between the BOS node and the EOS node in the output candidate graph corresponds to one output, and the output candidate graph is a graph including all possible output candidates.

Further, a feature vector is assigned to each node (or link) of the output candidate graph using the output candidate graph and the feature extraction template.
The feature extraction template has a format as shown in FIG. 6, and here, a feature vector of a node at a target position is created using features of two words before and after. At this time, since the feature is generated by the combination of the two words before and after and the output label to which the node belongs, the feature vectors between the nodes having different output labels at the same position are orthogonal to each other (the inner product is 0).

As a specific example, a feature vector as shown in FIG. 22 is given to each node for the output candidate graph shown in FIG.
Through the above pre-processing, the output candidate graph with feature vector and the initialized parameter vector (vector in which all elements are 0) are input to the parameter learning unit 1413, and learning is started.

Next, the operation of the parameter learning unit 1413 will be described (see FIG. 20). For example, the text “Ichiro Tanaka is the President of the Land Federation” shown in FIG. 23 and the correct output are “B-person name, I-person name, O, B-organization name, I-organization name, O, O, O ”Indicates a case where the interval [4, 5] is evaluated.
Since the correct answer in this section is “B-organization name, I-organization name”, using the scores of all paths passing through these two nodes, “B-organization name, I Determine a decision function based on the marginal probability that “organization name” is output. This process corresponds to the process of calculating the correct output score (step S201).

Formula (19) is used as the calculation formula at this time. The front probability α _{[0, i]} (y _i ) becomes i = 4 and y ₄ = B-organization name. That is, the scores of all the paths on the front side connected to the B-organization name node at position 4 are represented. This is determined by the forward algorithm.
Similarly, β _{[j, n]} (y _j ) of the rear probability is j = 5 and y ₅ = I-organization name, and the rear path connected to the node of the I-organization name at position 5 Is the score. This is also determined by the backward algorithm.

Next, the segment with the highest score is obtained except for the correct segment. This process corresponds to the process of calculating the score of the maximum likelihood output candidate (step S202).
The calculation method itself is the same as the calculation of the correct segment, and it is only necessary to calculate all possible segments other than the correct answer in the same section and select the segment with the maximum value.

Equation (19) is calculated using the correct segment and the maximum likelihood output candidate segment score obtained here. This process corresponds to a process for calculating a difference (step S203).
Thus, error estimation for the interval [4, 5] was performed. Here, since the accuracy rate in segment units is used, it is considered to use a 1 ogistic function as a smoothing function (see FIG. 9). As a result, the value of the error estimation function takes the range [0, ∞], and the larger the value of the correct segment decision function, the closer the value is to 0.

  Since the segment error rate in the equation (22) can be calculated by simply summing the error estimation functions in segment units, the above processing is performed for each segment. This process corresponds to the process of calculating the objective function L based on the evaluation index (step S204).

What is finally desired is a parameter vector λ that minimizes the objective function. Therefore, a parameter vector that is differentiated to 0 is obtained. Next, in order to update parameters, the gradient of the objective function (formula (22)) is obtained. This process corresponds to the process of calculating the gradient of the objective function (step S205).
The differentiation of Equation (22) can be considered by decomposing as follows using a chain rule.

  The objective function differentiated by the smoothing function l is as follows according to the definition of the 1 ogistic function.

  Here, the partial differentiation with respect to the parameter of the error estimation function d () is shown.

  It can be seen that the partial differentiation of the parameter of the error estimation function d () is an expected value at which each node (link) on the path passing through the target segment is output. Therefore, the gradient of each parameter can be efficiently calculated by using the Forward-Backward algorithm.

Based on the gradient calculated here, the parameters are updated using a generally used numerical optimization method. At this time, if the parameter update amount has converged, the learning is terminated (“Yes” in step S206), the parameter vector 145 is output to the storage 14 (step S207), and the process is terminated.
On the other hand, if it has not converged (“No” in step S206), the process returns to the beginning of parameter learning. This convergence determination is considered to have converged, for example, when the amount of change in the parameter vector is sufficiently small (when it is below a predetermined threshold).

  According to this specific example, since the difference between the appearance probability of the correct segment and the appearance probability of the segment that is most likely to be output other than the correct segment is used as the objective function, all the probabilities are used for learning, and the objective function becomes a continuous convex function. . In the prior art, since the objective function may be a discontinuous (convex) function, the usual numerical optimization method cannot be simply applied. In other words, it is necessary to use a numerical optimization method corresponding to discontinuous points. However, according to this specific example, it is possible to use a general numerical optimization method without restriction, and it is possible to improve calculation efficiency.

(Specific example 2: Series segmentation using F value as evaluation index)
Next, a specific example in the case where “evaluation unit” is “segment” and “evaluation index” is “F value” in sequence segmentation will be described. Here, the case where the F value shown in Formula (11) is maximized is shown. This is equivalent to minimizing the following equation:

However, M is the number of positive example segments. Further, FP ₁ and FN ₁ represent estimated values of FP (false positive) and FN (false negative) during learning. This is a denominator when Equation (11) is transformed as follows, and is derived from the fact that minimizing the denominator maximizes the whole.

If the estimated values FP _l and FN _l of FP and FN can be calculated from the equation (29), the F value can be maximized. Therefore, when it is desired to maximize the F value shown in the formula (11) by the sequence segmentation instead of the correct answer rate, the learning is performed using the formula (28) as an objective function. Hereinafter, a procedure for deriving FP _l and FN _l will be described.

First, FN _l will be described. As described above, FN (false negative), that is, FN _l , which is an estimated amount of a positive example, can be defined as follows using an error estimation function.

However, since FN _l is an estimated amount that makes a mistake in the correct example, it is only necessary to calculate for the correct example, so the second sum is calculated only for the correct answer segment of the correct example.

Next, the FP _l. Since FP _l is an estimated amount that is mistaken for a positive example, an error for a negative example is not considered. Therefore, when calculating the error estimation function, it is necessary to modify the equation so that errors in negative examples are not included in the calculation.

ただし、正解セグメントがＯであった場合は、負例への間違いというのは存在しないので、数式（１９）の計算と等価となる。よって、正例へ間違える推定量であるＦＰ_ｌは、数式（３１）の誤り推定関数を用いて以下のように定義できる。

However, when the correct answer segment is O, there is no error in the negative example, and this is equivalent to the calculation of Expression (19). Therefore, FP ₁ , which is an estimation amount mistaken for a positive example, can be defined as follows using the error estimation function of Equation (31).

For FN _l FP _l computes for all possible segments.
Hereinafter, the equation for obtaining the gradient in the parameter learning unit is as follows.

The following is shown for each component.
First, the partial differentiation of the objective function by FN _l and FP _l is as follows.

However, each _{Z N} and _{Z D} shows the numerator and denominator values of the equation (28).
Next, the partial differentiation of FN _l and FP _l by the smoothing function l becomes a constant because FN _l and FP _l are represented by a simple linear sum of l.
The partial differentiation of the smoothing function l (here the sigmoid function is applied) is as follows according to the definition.

  Finally, the partial differentiation with respect to the parameters of the error estimation functions d () and d ′ () is shown.

  Hereinafter, the processing other than the definition of the evaluation unit and the objective function, and the calculation of the gradient of the objective function and the objective function is the same as the case of learning using the segment correct answer rate, and is omitted here.

  Although the embodiment of the present invention has been described above, the mathematical formulas and the like used in the above description are merely examples, and can be appropriately changed according to the language model to be applied. Therefore, the scope of the present invention is defined by the technical idea described in the claims.

It is drawing which shows the example of series segmentation and series labeling. It is drawing explaining the IOB labeling method. It is drawing which shows the example of part-of-speech tagging. It is drawing which shows the example of the sequence labeling to which a cascade model is applied. It is drawing which shows the example of the sequence labeling which applied the conditional random field. It is drawing which shows the example of feature extraction. It is drawing explaining the clique in a conditional random field. It is drawing which shows the example of the characteristic for every clique in a conditional random field. It is drawing which shows the example of a smoothing function. 6 is a diagram illustrating a conventional error estimation calculation method. It is drawing explaining the problem of a prior art. It is drawing explaining the example of the output probability of a segment. It is drawing explaining the meaning of Numerical formula (19) in an output candidate graph. It is a block block diagram of a text analysis apparatus. It is explanatory drawing explaining the outline of the processing operation of a text analysis apparatus. It is drawing which shows the example of input and output of a learning device. It is drawing which shows the example of input and output of an estimator. It is drawing which shows the detailed block of a learning device. It is a flowchart which shows the process sequence of a learning device. It is a flowchart which shows the detailed process sequence of a parameter learning part. (A) shows an example of input learning data, (b) shows a main label candidate set to be extracted, and (c) shows an example of a generated output candidate graph. It is drawing which shows the example which provided the feature vector to each node of an output candidate graph. It is drawing explaining the case where interval [4, 5] is evaluated in the example of learning data.

Explanation of symbols

11 CPU
12 RAM
DESCRIPTION OF SYMBOLS 13 Acquisition power part 14 Storage 141 Learner 144 Learning data 145 Parameter vector 1411 Output candidate graph generation part 1412 Feature extraction part 1413 Parameter learning part

Claims (12)

A language analysis model learning device that calculates a parameter vector used for assigning a predetermined classification tag to text data from learning data that is text data to which the classification tag is assigned in advance,
An output candidate graph that associates an output label candidate indicating the type of the classification tag with the learning data, and a storage unit that stores an initial value of the parameter vector;
Using the decision function that expresses the appearance probability of the predetermined evaluation unit of the learning data by the peripheral probability using the parameter vector as a variable, the most likely output is the appearance probability of the correct element of the learning data and other than the correct element. A language analysis comprising: a parameter learning unit that calculates a difference from an appearance probability of an easy element as an error estimation function and calculates a parameter vector that optimizes an objective function set using the error estimation function Model learning device. The objective function is a sum of outputs of the error estimation function for all minimum evaluation units, and a parameter vector that optimizes the objective function is the parameter vector that minimizes the output of the objective function. The language analysis model learning device according to claim 1, wherein The objective function is a denominator of the F value shown in the following equation (1) set using the error estimation function, and the parameter vector that optimizes the objective function minimizes the output of the objective function. The language analysis model learning device according to claim 1, wherein the parameter vector is a parameter vector.

Where γ is a constant indicating the degree of trade-off between recall and precision, M is the number of positive segment, FP _l is an estimated number of errors to a negative positive example, and FN _l is a negative negative example The estimated amount of errors is shown respectively. The parameter learning unit
The language analysis model learning device according to any one of claims 1 to 3, wherein an output of the decision function is calculated using a Forward-Backward algorithm. The parameter learning unit
The gradient of the objective function is calculated, and the parameter vector for optimizing the objective function is calculated again using the parameter vector at that time until the gradient becomes a predetermined value or less. The language analysis model learning device according to claim 4. A language analysis model learning method in a language analysis model learning device that calculates a parameter vector used for giving a predetermined classification tag to text data from learning data that is text data to which the classification tag has been assigned in advance,
The storage unit of the language analysis model learning device stores an output candidate graph in which an output label candidate indicating the type of the classification tag is associated with the learning data, and an initial value of the parameter vector,
The parameter learning unit of the language analysis model learning device includes:
Using a decision function that represents the probability of occurrence of a predetermined evaluation unit of the learning data by a peripheral probability with the parameter vector as a variable, the probability of occurrence of the correct element of the learning data is calculated,
Using the decision function, calculate the appearance probability of the element that is most likely to be output other than the correct answer element,
Calculating the difference between the occurrence probability of the correct element and the appearance probability of the element that is most likely to be output other than the correct element as an error estimation function;
A language analysis model learning method, comprising: calculating a parameter vector that optimizes an objective function set by using the error estimation function. The objective function is a sum of outputs of the error estimation function for all minimum evaluation units, and a parameter vector that optimizes the objective function is the parameter vector that minimizes the output of the objective function. The language analysis model learning method according to claim 6, wherein: The objective function is a denominator of the F value shown in the following equation (1) set using the error estimation function, and the parameter vector that optimizes the objective function minimizes the output of the objective function. The language analysis model learning method according to claim 6, wherein the parameter vector is a parameter vector.

Where γ is a constant indicating the degree of trade-off between recall and precision, M is the number of positive segment, FP _l is an estimated number of errors to a negative positive example, and FN _l is a negative negative example The estimated amount of errors is shown respectively. The parameter learning unit
The language analysis model learning method according to any one of claims 6 to 8, wherein an output of the decision function is calculated using a Forward-Backward algorithm. The parameter learning unit
After calculating the parameter vector, calculating the gradient of the objective function, and calculating again the parameter vector that optimizes the objective function using the parameter vector at that time until the gradient becomes a predetermined value or less. The language analysis model learning method according to claim 6, wherein the language analysis model is learned. A language analysis model learning program that causes a computer to function as the language analysis model learning device according to any one of claims 1 to 5. A recording medium in which the language analysis model learning program according to claim 9 is recorded.

Images