JP2006330935A - Program, method, and system for learning data preparation - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform stacking learning by preparing stacking learning data without reducing learning data at a first step of an NE sampling device without increasing a load for preparing learning data. <P>SOLUTION: A learning control section 140 comprises a step of dividing the learning data into n sections, a step of preparing the approximate NE sampling device by using (n-1) pieces of the learning data, a step of preparing the stacking learning data by using the NE sampled results of the approximate NE sampling device as test data for the remaining one learning data, and a step of making the stacking NE sampling device 130 perform stacking learning by using the stacking learning data prepared by the learning control section 140. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention creates learning data for creating stacking learning data used for stacking learning for creating a classification program for performing classification by using classification results of a plurality of classification programs (classifiers) based on different machine learning algorithms as clues. TECHNICAL FIELD The present invention relates to a learning data creation program, a learning data creation method, and a learning data creation device that can create learning data for stacking without increasing the load of learning data creation. is there.

  When creating a classifier using a machine learning machine, create a classifier with different rules and models from the same learning data by changing the machine learning algorithm used such as SVMs, Boosting, ADTrees, etc. (For example, refer nonpatent literatures 1-3.).

  In addition, since these classifiers have different advantages, a technique called stacking has been developed to perform better classification by using the classification results of the plurality of classifiers as a clue (for example, Non-Patent Document 4). reference.).

Vladimir Vapnik, "Statistical Learning Theory", Wiley-Interscience Publication, 1998 R.E.Scapire and Y. Singer, `` BoosTexter: A boosting-based system for text categorization '', Machine Learning, 39 (2/3): 135-168, May / June 2000 Yoav Freund and Llew Mason, `` The alternating decision tree learning algorithm '', Proceeding of the sixteenth International Conference on Machine Learning, PP.124-133, 1999 By David H. Wolpert, "Stacked generalization.", Proceedings of Neural Networks, pages 241-259, 1992. Takehito Utsuro, Manabu Sassano and Kiyotaka Uchimoto, "Combining Outputs of Multiple Japanese Named Entity Chunkers by Stacking", Proceedings of EMNLP 2002, PP.281-288, 2002

  Stacking is a method of creating a classifier using learning data for stacking to create a classifier that performs classification by using the classification results of a plurality of classifiers as clues. Hereinafter, a classifier whose output is used as a clue is referred to as a first-stage classifier, and a classifier that performs classification based on the classification result of the first-stage classifier is referred to as a second-stage classifier. However, in order to realize stacking, it is necessary to prepare learning data for the second stage (learning data for stacking) separately from the learning data for creating the classifier for the first stage. There is a problem that it is expensive.

  If the learning data for creating the first classifier is reused as learning data for stacking in order to reduce the learning data creation load, some classifiers classify the learning data almost perfectly. Since it is created, there is a problem that the meaning of stacking learning is lost.

  In addition, when certain learning data is divided and used for creating the first-stage classifier and stacking learning, there is a problem that learning data for creating the first-stage classifier is reduced and learning accuracy is deteriorated. is there. There is also a problem that it is unclear what ratio should be used for preparing the learning data for creating the first-stage classifier and stacking learning.

  The present invention has been made to solve the above-described problems caused by the prior art, and a learning data creation program and learning data creation that can create learning data for stacking without increasing the load of creating the learning data An object is to provide a method and an apparatus for creating learning data.

  In order to solve the above-described problems and achieve the object, a learning data creation program according to the invention of claim 1 uses as a clue the classification results of a plurality of classification programs (first classifier) based on different machine learning algorithms. A learning data creation program for creating learning data for stacking used for stacking learning for creating a classification program (second classifier) for performing final classification by using the plurality of classification programs, wherein the plurality of classification programs are machine learning A learning data dividing procedure for dividing the learning data used for a plurality of partial learning data, and one partial learning data among a plurality of partial learning data obtained by dividing the learning data by the learning data dividing procedure As described above, the plurality of pieces of partial learning data excluding the one partial learning data are learned as learning data. It is characterized by causing a computer to execute a stacking learning data generation procedure for generating stacking learning data by repeating a procedure for acquiring test results obtained by testing a classification program as learning data for stacking.

  According to the first aspect of the present invention, learning data used for machine learning by a plurality of classification programs is divided into a plurality of partial learning data, and one partial learning data among a plurality of partial learning data obtained by the division. Is used as test data, and test results obtained by testing multiple classification programs that have been learned using partial learning data excluding one partial learning data as learning data, are generated as learning data for stacking. Since partial learning data excluding data is learned as learning data, learning for stacking is performed using an approximate classifier of a classification program created from all learning data by acquiring partial learning data in finer units. Data can be created.

  According to a second aspect of the present invention, there is provided a learning data creation program according to the first aspect of the invention, wherein the stacking learning data generation procedure is executed by a different computer for each of the plurality of classification programs.

  According to the second aspect of the present invention, since the plurality of classification programs are executed by different computers, the classification program can be learned at high speed, and the learning data for stacking can be acquired at high speed.

  According to a third aspect of the present invention, there is provided a learning data creation program according to the first or second aspect, wherein the classification program is a specific expression extraction program for extracting a specific expression from a sentence.

  According to the third aspect of the present invention, the learning data for stacking can be created using the approximated specific expression extraction program of the specific expression extraction program created from all the learning data.

  According to a fourth aspect of the present invention, there is provided the learning data creation program according to the third aspect of the invention, wherein the specific expression extraction program includes a word extraction procedure for extracting a word from a sentence and a word extracted by the word extraction procedure. It is characterized in that it is a classification program for performing a specific expression tagging procedure for determining whether or not each is a specific expression and tagging the specific expression.

  According to a fifth aspect of the present invention, there is provided a learning data generation method for stacking learning used for stacking learning for generating a classification program for performing classification by using classification results of a plurality of classification programs based on different machine learning algorithms. A learning data creation method for creating data, wherein the learning data is divided by a learning data dividing step for dividing learning data used by the plurality of classification programs for machine learning into a plurality of partial learning data, and the learning data dividing step. A test in which the plurality of classification programs learned by using one partial learning data out of a plurality of partial learning data obtained as a test data and learning the partial learning data excluding the one partial learning data as learning data Repeat the procedure to acquire the results as learning data for stacking. Characterized in that it includes a stacking learning data generating step of generating stacking learning data in Succoth, the.

  According to the fifth aspect of the present invention, learning data used for machine learning by a plurality of classification programs is divided into a plurality of partial learning data, and one partial learning data among a plurality of partial learning data obtained by the division. Is used as test data, and test results obtained by testing a plurality of classification programs learned using partial learning data excluding one partial learning data as learning data are generated as stacking learning data. The learning data for stacking can be created by using an approximate classification program of the created unique expression extraction program.

  According to a sixth aspect of the present invention, there is provided a learning data generating apparatus for stacking learning used for stacking learning for generating a classification program for performing classification by using classification results of a plurality of classification apparatuses based on different machine learning algorithms as clues. A learning data creation device for creating data, wherein the learning data is divided by learning data dividing means for dividing learning data used by the plurality of classification devices for machine learning into a plurality of partial learning data, and the learning data dividing means. A test that tests the plurality of classification devices that have learned one partial learning data of the plurality of partial learning data obtained as a test data and the partial learning data excluding the one partial learning data as learning data. Stacking is performed by repeating the procedure of acquiring the results as learning data for stacking. And stacking the learning data generating means for generating a grayed learning data, characterized by comprising a.

  According to the invention of claim 6, the learning data used by the plurality of classification devices for machine learning is divided into a plurality of partial learning data, and one partial learning data among the plurality of partial learning data obtained by the division. Is used as test data, and a test result obtained by testing a plurality of classifiers that have learned partial learning data excluding one partial learning data as learning data is generated as learning data for stacking. The learning data for stacking can be created using an approximate classification device of the created unique expression extraction program.

  According to the first, fifth and sixth aspects of the present invention, the learning data for stacking is created using the approximate classification program (classification apparatus), so that the learning data for the classification program (classification apparatus) is not affected. There is an effect that learning data can be created.

  Further, according to the invention of claim 2, since the classification program can be learned at high speed, there is an effect that the creation of the learning data for stacking can be speeded up.

  According to the invention of claim 3, since the learning data for stacking is created using the approximated unique expression extraction program, the learning data for stacking can be created without affecting the learning data of the specific expression extraction program. There is an effect that can be.

  Exemplary embodiments of a learning data creation program, a learning data creation method, and a learning data creation device according to the present invention will be described below in detail with reference to the accompanying drawings. 1 and 2 are explanations of a stacking learning method algorithm and a flowchart according to the present invention. By preparing learning data in accordance with the task to be applied, it can be applied to various tasks, and the machine learning algorithm to be used is changed. Thus, it is possible to create a classifier that performs classification based on the results of the classifier created by various machine learning algorithms.

First, the configuration of the apparatus according to the present invention will be described. FIG. 3 is a functional block diagram in the present invention. As shown in the figure, the classification device 1 includes classification devices 11 _{1 to} 11 m composed of m different machine learning algorithms, a learning data storage unit 12 for stacking, a classification device 13 for stacking, and a learning control unit 14. And a classification control unit 15.

Each of the classification devices 11 _{1 to} 11m is a processor that performs classification on inputs. These classifiers 11 _{1 to} 11 m have a learning function for creating a classifier from learning data. Therefore, when these classification devices 11 _{1 to} 11 m particularly indicate a case where learning is performed as a learning device, they are denoted as classification devices (learning devices) 11 _{1 to} 11 m.

Further, when creating the learning data for stacking, these classifiers 11 _{1 to} 11m perform learning except for a part of the learning data and operate as an approximate classifier.

The stacking classifier 13 is a function of the classifier _{1, and} uses a classification result from the classifiers 11 _{1 to} 11m to finally create a classifier that determines a classification result from stacking learning data, and stacking. This is a processor that performs classification using a classifier created by the classification apparatus 13 for use.

The learning control unit 14 is a control unit that performs stacking learning using the m classification devices (learning devices) 11 _{1 to} 11 m, the stacking learning data storage unit 12, and the stacking classification device 13.

  The learning control unit 14 creates learning data for stacking by the algorithm shown in FIG. 1 and performs stacking learning. Here, creation of learning data for stacking by the algorithm shown in FIG. 1 by the learning control unit 14 will be described.

  In the method for creating an approximate classifier in the algorithm shown in FIG. 1, it is possible to create a classifier having a high degree of approximation by dividing the learning data into fine units and deleting a part thereof. The learning data that was not used when creating the approximate classifier is data other than the learning data used for creating the first-stage classifier, and the result of the classification performed by the approximate classifier using the learning data is It is expected that the result will be almost the same as the result of classification by the classifier created from the learning data. Therefore, by changing the learning data to be deleted to create the approximate classifier in the learning data in order, the learning data for stacking that has a clue that is almost the same as the classifier created from all the learning data Can be created for the prepared learning data.

  In this embodiment, the case where the present invention is applied to a specific expression extraction apparatus that extracts a specific expression from Japanese text will be mainly described. FIG. 7 and FIG. 10 describe the algorithm and flow chart when FIG. 1 and FIG. 2 which are explanations of the stacking learning method algorithm and flowchart according to the present invention are applied to specific expression extraction. Here, the extraction of the specific expression is to extract “person name”, “place name”, “date”, “time”, etc. from the text. Thereafter, the specific expression is NE (Named Entity) and the specific expression is extracted. It shall be described as NE extraction.

First, the configuration of the NE extraction device (specific expression extraction device) according to the present embodiment will be described. FIG. 4 is a functional block diagram illustrating the configuration of the NE extraction apparatus according to the present embodiment. As shown in the figure, the NE extraction device 100 includes NE extraction devices 110 _{1 to} 110 _m each including m different machine learning algorithms, a stacking learning data storage unit 120, a stacking NE extractor 130, A learning control unit 140 and an extraction control unit 150 are included.

Each of the NE extraction devices 110 _{1 to} 110 _m has a function of performing NE extraction from the text using the NET unit 111. Details of the NET unit 111 will be described later.

These NE extraction devices 110 _{1 to} 110 _m also have a learning function as a learning device that creates an NE extraction device (NET unit 111) that performs NE extraction by learning learning data. Therefore, when these NE extraction devices 110 _{1 to} 110 _m particularly indicate a case where learning is performed as a learning device, the NE extraction devices (learning devices) 110 _{1 to} 110 _m are described.

In addition, when creating the learning data for stacking, these NE extraction devices 110 _{1 to} 110 _m perform learning by removing a part of the learning data, create an approximate NE extractor, and extract using that. To implement. The creation of learning data for stacking and details of the approximate NE extractor will be described later.

These NE extraction devices 110 _{1 to} 110 _m extract eight types of NEs defined by IREX (see, for example, IREX Committee, editor. 1999. Proceedings of the IREX workshop) from text. FIG. 5 is a diagram illustrating an example of NE defined by IREX. As shown in the figure, these NE extractors 110 _{1 to} 110 _m are “ARTIFACT”, “DATE”, “LOCATION”, “MONEY”, “ORGANIZATION”, “PERCENT”, “PERSON” and “PERSON” from the text. "TIME" is extracted as a unique expression.

  The stacking learning data storage unit 120 is a storage unit that stores stacking learning data. FIG. 6 is a diagram showing an example of learning data for stacking and is shown together with learning data for creating an NE extractor for comparison.

As shown in the figure, the learning data for creating the NE extractor associates a feature with a tag. Here, the feature is a clue used for NE extraction, and the tag is one of eight types of NE or “Other” indicating that the NE does not correspond to NE. The learning data for stacking is obtained by adding the output result of each NE extractor to the features and tags of the learning data used to create the NE extractors 110 _{1 to} 110 _m . In FIG. 6, “NE extractor 1” corresponds to NE extractor 110 ₁ , and “NE extractor 2” corresponds to NE extractor 110 ₂ .

The NE extractor 130 for stacking is a processor that finally performs NE extraction as the NE extractor 100 using the NE extraction results from the NE extractors 110 _{1 to} 110 _m , and learns learning data for stacking. Has a learner function that enables NE extraction. A post-processing unit 112 is applied to the output of the NE extraction device 100. The post-processing unit 112 will be described later.

The learning control unit 140 is a control unit that performs stacking learning using the m NE extractors (learners) 110 _{1 to} 110 _m , the stacking learning data storage unit 120, and the stacking NE extractor 130.

  The learning control unit 140 creates learning data for stacking by the algorithm shown in FIG. 7 and performs stacking learning. Here, creation of learning data for stacking by the algorithm shown in FIG. 7 by the learning control unit 140 will be described.

Learning data for stacking, since learning the NE extraction result by the first stage of NE extractor 110 ₁ to 110 _m, other than the learning data used to create the first stage of the NE extractor 110 ₁ to 110 _m It is desirable to be data. Therefore, in order to perform stacking learning, it is also conceivable to newly create learning data for stacking in addition to the learning data used to create the first stage NE extractors 110 _{1 to} 110 _m . However, with this method, the cost of creating new learning data becomes a problem.

  Therefore, in this embodiment, instead of creating new learning data, consider creating an approximate NE extractor that uses all of the learning data. That is, in this embodiment, by using the learning data from which a part has been deleted, an approximate NE extractor for the NE extractor when all the learning data is learned is created and used for creating learning data for stacking learning. To do.

  In the method for creating an approximate NE extractor in the algorithm shown in FIG. 7, a NE extractor with a high degree of approximation can be created by dividing the learning data into fine units and deleting a part thereof. The learning data that was not used at the time of creating the approximate NE extractor is data other than the learning data used for creating the first-stage NE extractor, and the approximate NE extractor NE-extracted using the learned data. The result is expected to be almost the same as the NE extraction by the NE extractor created from the entire learning data. Therefore, by changing the learning data to be deleted for the creation of the approximate NE extractor in the learning data in order, the stacking has a NE extraction result that is almost equivalent to the NE extractor created from all the learning data. This learning data can be created as much as the prepared learning data.

FIG. 7 is a diagram illustrating a stacking learning algorithm according to the present embodiment. As shown in the figure, in this embodiment, (1) the learning data LD is divided into n to be {LD1, LD2,..., LDn}, and (2) m learning devices {ML1, ML2 to learn , ..., MLm} are prepared. The learning devices {ML1, ML2,..., MLm} correspond to the _m NE extractors (learning devices) 110 _{1 to} 110 _{m shown} in FIG.

  (3) Using each of the prepared learners, an approximate NE extractor is created using the learning data of (N-1) for learning, and the remaining 1 is used as test data for each approximate NE extractor. NE is extracted, the result is added to the test data as a feature, and stacking learning data ST is created. Specifically, first, the i-th stacking learning data STi is initialized (STi = LDi). Then, using the learning device MLj (1 ≦ j ≦ m), an approximate NE extractor CLji is created from learning data (LD-LDi) obtained by removing LDi (1 ≦ i ≦ n) from LD, and from LDi using CLji The result of NE extraction is added as a feature to the i-th stacking learning data STi. Here, the approximate NE extractor CLji means an NE extractor created from the learner MLj using the learning data (LD-LDi) with respect to the NE extractor CLj created from the learner MLj using the learning data LD. To do. Then, the approximate NE extractor CLji is created for all the learners MLj (1 ≦ j ≦ m), and the result of NE extraction from LDi using the created approximate NE extractor CLji is used as the i-th stacking learning data STi. As a feature, STi is added to the learning data ST for stacking. Then, STi (1 ≦ i ≦ n) is created for all learning data LDi (1 ≦ i ≦ n) and added to the ST.

Then, (4) NE extractors {CL1, CL2,..., CLm} are created from the learning data LD using the learners {ML1, ML2,. Here, the NE extractors {CL1, CL2,..., CLm} correspond to the _m NE extractors 110 _{1 to} 110 _{m shown} in FIG. Then, the stacking NE extractor 130 is learned from the stacking learning data ST.

  As described above, in this embodiment, the procedure of creating an approximate NE extractor and testing is repeated at a ratio of learning: test = (n−1): 1, and the test result is used as learning data for stacking. That is, in this embodiment, a part of learning data LDi is used as test data, and the result of testing the approximate NE extractor CLji (1≤j≤m) created from (LD-LDi) is used as learning data for stacking.

  Therefore, only the learning data LD for creating the NE extractor {CL1, CL2,..., CLm} is used, and the learning data is created when the NE extractor {CL1, CL2,. The learning data for stacking can be generated without reducing the LD. Further, by making n sufficiently large, the approximate NE extractor CLji (1 ≦ j ≦ m) is made sufficiently close to the NE extractors {CL1, CL2,..., CLm} created from all the learning data LD. Can do.

The extraction control unit 150 is a control unit that controls the NE extraction apparatus 100 to perform NE extraction using the _m NE extractors 110 _{1 to} 110 _m and the stacking NE extractor 130. FIG. 8 is an explanatory diagram for explaining stacking using a plurality of NE extractors. This figure shows a case where _four NE extractors 110 _{1 to} 110 ₄ are used.

As shown in the figure, each NE extractor performs NE extraction by inputting the text “Miyazaki / ha / Miyazaki / no / from”. For example, the NE extractor 110 ₁ outputs “Miyazaki (Other) / Has (Other) / Miyazaki (LOCATION) / (Other) / Owned from Others]” as the NE extraction result. Further, NE extractor 110 ₂ outputs the "Miyazaki (PERSON) / the (Other) / Miyazaki (Other) / the (Other) / former (Other)" as the NE extraction results.

  The stacking NE extractor 130 receives the NE extraction result of each NE extractor and outputs the NE extraction result as the NE extraction device 100. In FIG. 8, the stacking NE extractor 130 outputs “Miyazaki (PERSON) / ha (Other) / Miyazaki (LOCATION) / (Other) / origin (Other)” as the NE extraction result as the NE extraction device 100. is doing.

Next, details of the NET unit 111 and the post-processing unit 112 of the NE extractors 110 _{1 to} 110 _m will be described. The NE extractors 110 _{1 to} 110 _m perform NE extraction based on morphological analysis results. However, the word boundary and NE boundary by morphological analysis do not always match. Therefore, the NE extractors 110 _{1 to} 110 _m perform NE tagging in which the NET unit 111 determines the NE of the word, and the post-processing unit 112 performs post-processing when the word boundary does not match the NE boundary.

  The NET unit 111 performs morphological analysis using a morphological analyzer, and determines which NE class the word output from the morphological analyzer is. Here, a ChaSen morphological analyzer (see, for example, http://chasen.naist.jp/hiki/ChaSen/) is used. In addition, notation, parts of speech, and character types are used as features.

  Here, the notation is a notation of a word returned as an analysis result by ChaSen. As part of speech, the result of ChaSen is used. However, if it is determined as an unknown word, the ChaSen setting is changed so that it is output as a proper noun-general.

  Further, hiragana, katakana, kanji, uppercase alphabetic characters, lowercase alphabetic characters, symbols / others, and combinations of 6 types and numbers are used as character types. If the same character continues more than once, add “+” to distinguish them. For example, in the case of “visiting the morning”, two kanji characters are used, so that they are expressed in the form of “kanji +”, and in the case of “eating”, they are expressed in the form of “kanji-hirakana +”. Numbers are normalized for both Chinese numerals and Arabic numerals, and the result of classification “12 or less”, “24 or less”, “100 or less”, “2000 or less”, “other than that” is used as the character type.

  When NE extraction is performed in units of words, there are cases where one word becomes NE alone or a plurality of words constitute NE. For this reason, in NE tagging, NE tags represented by “morphological“ NE class ”and“ position within NE ”are attached to words.

For example, the sentence "<ORGANIZATION> Iwakura Mission </ ORGANIZATION> visited <LOCATION> US </ LOCATION>"
Iwakura “BEGIN-ORGANIZATION”
Envoy “INSIDE-ORGANIZATION”
Team “END-ORGANIZATION”
Is “O”
Visit “O”
US “SINGLE-LOCATION”
“O”
“O”
It is expressed.

  There are 8 types of NE tags to be discriminated, and 4 types: single (S-), top (B-), middle (I-), end (E-), which are "positions within NE" shown in FIG. There are a total of 33 types including 32 types in combination with types and one type other than NE (O).

  The morpheme range used for the clues of NE tagging is a total of 5 morphemes including the target morpheme and the front and back 2 morphemes. In addition, the NE tags are grouped into NE classes by the Viterbi algorithm. The input to the Viterbi algorithm is a probability value, but here the NE extractor created from the learner to be used returns a certainty factor. Therefore, each confidence factor is expressed as a sigmoid function S (X) = 1 / (1 + exp Normalization is performed in (-βX)) to replace the probability value. Note that the value of β is “5”.

  The post-processing performed by the post-processing unit 112 is processing for cutting out from a word when a part of the word is NE. For example, if NE extraction is performed from a sentence that has been morphologically analyzed as “Iwakura / mission / group / ga / visit / shi / ta”, “Iwakura mission” can be extracted as “ORGANIZATION”, but “LOCATION” is “rice”. "Cannot be extracted from the word" visiting the United States ". Therefore, here, NE tagging in character units is performed as post-processing after NE tagging in word units.

  FIG. 9 is a diagram illustrating an example of conversion from NE tagging in units of words to NE tagging in units of characters. In the figure, the upper NE tagging result is converted into lower NE tagging.

  The NE tags to be discriminated in the post-processing include 16 types of combinations of the eight types shown in FIG. 5 and the two types of the first (B-) and middle (I-) which are “positions in the NE”, and the NE tag. There are a total of 17 types of one type other than (O). The features used in post-processing are the part of speech and position of the word to which the character belongs, the character, the character type, and the NE tagging result. For each feature, four character positions (single (S-), head (B-), middle (I-), end (E-)) are used. The analysis is performed in the direction from the end of the sentence to the beginning of the sentence, and the NE tag determined two times before is dynamically used as a feature.

  Next, the processing procedure of the stacking learning process by the NE extraction apparatus 100 according to the present embodiment will be described. FIG. 10 is a flowchart illustrating the processing procedure of the stacking learning process performed by the NE extraction device 100 according to the present embodiment.

  As shown in the figure, in the NE extraction apparatus 100, the learning control unit 140 divides the learning data LD into n (step S101), and initializes the subscript i of the learning data LDi obtained by the division to “1”. (Step S102).

  Then, the learning control unit 140 determines whether i ≦ n is satisfied (step S103), and if i ≦ n, initializes the i-th stacking learning data. That is, STi = LDi (step S104). Further, the subscript j of the learning device MLj is initialized to “1” (step S105).

  Then, the learning control unit 140 determines whether or not j ≦ m (step S106). If j ≦ m, the learning data (LD-LDi) obtained by removing LDi from LD using MLj. ) To create an approximate NE extractor CLji, and the result of NE extraction from LDi by CLji is used as the feature of the NE extraction result from LDi by the NE extractor CLj created from the learning device MLj. Add to STi (step S107). Then, “1” is added to j (step S108), and the process returns to step S106.

  On the other hand, if j ≦ m is not satisfied, STi has been created, so STi is added as stacking learning data ST. That is, ST = ST + STi (step S109). Note that ST is stored in the learning data storage unit 120 for stacking. Then, “1” is added to i (step S110), and the process returns to step S103.

  Further, when i ≦ n is not satisfied, since the creation of the ST is completed, the learning control unit 140 uses the learners {ML1, ML2,..., MLm} to perform the NE extractor {CL1 from the LD. , CL2,..., CLm} are created (step S111). Then, the stacking NE extractor 130 performs stacking learning using the ST stored in the stacking learning data storage unit 120 (step S112).

  In this way, the learning control unit 140 creates the approximate NE extractor CLji, and the result of NE extraction from LDi by CLji is used as the feature of the NE extraction result from LDi by the NE extractor CLj, and the i-th learning data for stacking. By repeating the addition to STi within the range of 1 ≦ j ≦ m and 1 ≦ i ≦ n, stacking learning data ST can be created.

  FIG. 11 is a diagram illustrating an example of creating learning data for stacking by the NE extraction device 100 according to the present embodiment. The figure shows an example in which the learning data LD is divided into three. First, an approximate NE extractor CLj1 is created from the learning data LD2 and LD3 using the learner MLj, and the result of testing the approximate NE extractor CLj1 using LD1 as test data is combined with LD1 to create stacking learning data ST1. .

  Similarly, an approximate NE extractor CLj2 is created from the learning data LD1 and LD3 using the learning device MLj, and the result of testing the approximate NE extractor CLj2 using LD2 as test data and LD2 are combined to create learning data ST2 for stacking. A learning NE MLj is used to create an approximate NE extractor CLj3 from the learning data LD1 and LD2, and the result of testing the approximate NE extractor CLj3 using LD3 as test data is combined with the learning data ST3 for stacking. Can be created.

That is,
“Go to <PERSON> Miyazaki </ PERSON> and <LOCATION> Miyazaki </ LOCATION>.”
“<PERSON> Miyazaki </ PERSON> and go for a meal.”
“Go to <LOCATION> Miyazaki </ LOCATION> with him.”
2 of the learning data composed of the three sentences is learned to create an approximate NE extractor, and the NE extraction result of the remaining one sentence is used as stacking learning data using the approximate NE extractor. By repeating this, stacking learning data can be created for all learning data using an approximate NE extractor.

  Next, evaluation results of the NE extraction apparatus 100 according to the present embodiment will be described. In the evaluation, SVMs, Boosting, and ADTrees were used as learning devices. Regarding the parameters given to SVMs, the value of C is “1”, the polinomial kernel is secondary, and the multi-class is learning with One Vs the Rest. For Boosting, BoosTexter, which extends Boosting, which is a binary classifier, to multi-class, was modified so that multi-class can be learned with One Vs the Rest.

  For ADTrees, we used an improved ADTrees algorithm that creates a classification tree with a decision stump as a node in the boosting framework. In ADTrees, the newly selected feature for constructing the classification tree is used as a parent, and two cases of “examples where the feature appeared” and “examples where the feature did not appear” are used as new search spaces. In NE extraction, since the number of features and the number of cases is large, the search space rapidly expands as learning progresses. Therefore, here, we focus on the fact that the number of cases that do not appear is extremely large compared to cases where a certain feature appears, and the addition of the search space is limited to “cases where the feature appears”. Along with this change, the feature selection condition to be a node has been changed to BoosTexter's real-valued predictions and abstaining. Multi-class learning was done with One Vs the Rest, boosting repetition was fixed at 1000 times for all classes, and the maximum tree depth was set to “2”.

  Further, SVMs, Boosting, and ADTrees were used for NE tagging, and SVMs were used for post-processing and stacking learning. The post-processing is applied to the output of the NE extractor for stacking corresponding to 130 in FIG. 4. In all experiments, a common one created from SVMs was used.

  The evaluation was performed by comparing the learning data creation method by the NE extraction apparatus 100 according to the present embodiment with the following three methods (a) to (c). The methods (b) and (c) are the stacking learning data acquisition methods used in Non-Patent Document 5.

  Method (a): Learning data for stacking is created by using the correct answer (tag) to be learned as an output result of all NE extractors. In this method, since the outputs of all NE extractors are the same as the correct answer, it is impossible to learn what NE extraction each NE extractor performs. FIG. 12 is a diagram illustrating an example of creating learning data for stacking by the method (a). As shown in the figure, in this method, the NE extractors perform stacking learning on the assumption that all correct tags are output. For example, the learning data for stacking for the learning data “Miyazaki (PERSON) / and (Other) / Miyazaki (LOCATION) / Ni (Other) / Go (Other) /. (Other)” is the output result of all NE extractors. Is created under the assumption that “PERSON / Miyazaki (PERSON) / and (Other) / Miyazaki (LOCATION) / (Other) / Go (Other) /. (Other)”.

  Method (b): The result of testing the NE extractor using the learning data used for the NE extractor learning is used for stacking. That is, a NE extractor is created from the learning data, and NE extraction of the learning data is performed by the created NE extractor to obtain learning data for stacking. In this method, since the NE extractor performs NE extraction on learned data, in most cases, correct NE extraction can be performed, and only the portion where learning fails is different from the tag. Therefore, there is a possibility that the result is almost the same as the method (a). FIG. 13 is a diagram illustrating an example of creating learning data for stacking by the method (b). As shown in the figure, in this method, the NE extractor outputs a correct tag except for a portion where learning has failed. For example, in the learning data for stacking of learning data “Miyazaki (PERSON) / and (Other) / Miyazaki (LOCATION) / Ni (Other) / Go (Other) /. (Other)”, the output results of two NE extractors In the stacking learning data for "Hi (Other) / and (Other) / Miyazaki (LOCATION) / To (Other) / Go (Other) /. (Other)]", the two NE extractors The output result of is incorrect.

  Method (c): Learning data for the first-stage learning device and learning data for stacking learning are prepared separately. In this method, each NE extractor performs NE extraction on new data that is not learned data, and the result becomes learning data for stacking. It is checked whether correct NE extraction is performed. However, with this method, the load of creating learning data becomes a problem. Also, it is difficult to determine at what ratio it is appropriate to prepare learning data for creating an NE extractor and learning data for stacking. FIG. 14 is a diagram illustrating an example of creating stacking learning data by the method (c). As shown in the figure, in this method, the NE extractor performs a test (NE extraction) on data different from the learned data, and the test result becomes learning data for stacking. In the figure, “Miyazaki (PERSON) / and (Other) / meal (Other) / To (Other) / Go (Other) /.” (Other) ”and“ He (Other) / and (Other) / Miyazaki (LOCATION) ) / Ni (Other) / Go (Other) / Go (Other) is the training data of the NE extractor, and “Miyazaki (PERSON) / and (Other) / Miyazaki (LOCATION) / Ni (Other) / Go (Other) ) /. (Other) ”is used to create learning data for stacking.

  In addition, the evaluation is based on the IREX general task (GENERAL) (NEERAL) (http://www.csl.sony.co.jp/person/sekine/IREX/Package/IREXfinalB.tar. This was done by comparing the extraction results (available from gz). Here, the CRL specific expression data is data in which a specific expression defined by IREX is given to the 1,174 articles of the Mainichi Shimbun 95 edition, about 11,000 sentences.

For comparison of evaluation results, Recall, Precision, and F-measure (F value) calculated by the following equations are used.
Recall = NUM / (number of NEs to be extracted)
Precision = NUM / (number of NEs extracted by NE extractor)
F-measure = 2 × Recall × Precision / (Recall + Precision)
Here, NUM is the number of NEs correctly extracted by the NE extraction device.

  The F values of single SVMs, Boosting, and ADTrees evaluated by the IREX GENERAL were 85.34 for SVMs, 82.27 for Boosting, and 83.34 for ADTrees, and SVMs showed the best results. Therefore, the result of SVMs was used as the baseline for evaluation.

  In stacking, four combinations of SVMs-Boosting, SVMs-ADTrees, Boosting-ADTrees, and SVMs-Boosting-ADTrees were used as combinations of the first stage NE extractors. As the division number n, “5”, “10”, and “20” were used.

  FIG. 15 is a diagram showing the evaluation results. The figure shows the result of SVMs that was the best alone (broken line), the method (a) and (c) that were the objects of comparison, the best result (broken line), and the four combinations according to this example. The result (solid line) is shown. The vertical axis represents the F value, and the horizontal axis represents the number of learning data divisions.

  In method (a), the best result among the four types of combinations was the combination of SVMs-Boosting-ADTrees, and the overall F value was 84.9. The best with method (b) was the same as the result with the best SVMs alone.

  In the method (c), the CRL specific expression data is divided into N parts for each article, and “N-1” is used for creating each NE extractor, and the remaining “1” is used for stacking. As N, 5, 10, and 20 were used. In this method, when the data is divided into N, N results are obtained, so a total result of (5 + 10 + 20) × 4 = 140 is obtained, and a better result than the SVMs that are the best results can be left alone. Two of the 140 results. FIG. 15 shows the results of dividing SVMs-Boosting-ADTrees with the best results among 5, 10, 20 divisions.

  As shown in FIG. 15, only the method (c) is able to leave a result exceeding the baseline among the methods (a) to (c) to be compared, and the method (c) is more than the baseline. Only 2 out of 140 results were found. On the other hand, in the NE extraction apparatus 100 according to the present embodiment, 6 out of 4 (combination type) × 3 (division type) = 12 results are equal to or higher than the baseline, and the methods (a) to A better evaluation result is obtained than in (c).

  Also, except for the SVMs-Boosting combination, the accuracy gradually increases by increasing the number of divisions, and the combination of SVMs-ADTrees and SVMs-Boosting-ADTrees increases the accuracy by increasing the number of divisions, After 10 divisions, the result of the SVMs with the highest accuracy alone is exceeded.

  Thus, the learning data creation method for stacking according to the present embodiment shows an average better result than methods (a) to (c), and learning is performed by increasing the number of divisions of the learning data. Accuracy can be increased.

As described above, in this embodiment, the learning control unit 140 divides the learning data into n parts, creates an approximate NE extractor using the learning data of (n−1), and uses the remaining one learning data. By creating the learning data for stacking using the result of the NE extraction by the approximate NE extractor as the test data, the learning data for stacking is created without reducing the learning data of the NE extractors 110 _{1 to} 110 _m. A NE extraction device 100 can be realized.

  In the present embodiment, the NE extraction device 100 has been described. However, by removing the stacking NE extractor 130 and the extraction control unit 150 from the NE extraction device 100, a learning data creation device that creates learning data for stacking is obtained. Can do.

  In the present embodiment, the NE extraction device has been described. However, an NE extraction program having the same function can be obtained by realizing the configuration of the NE extraction device with software. Therefore, a computer that executes this NE extraction program will be described. Note that a learning data creation program for creating stacking learning data from the NE extraction program can be obtained in the same manner as a learning data creation device for creating stacking learning data from the NE extraction device 100.

  FIG. 16 is a functional block diagram illustrating the configuration of a computer that executes the NE extraction program according to the present embodiment. As shown in the figure, the computer 200 includes a RAM 210, a CPU 220, an HDD 230, a LAN interface 240, an input / output interface 250, and a DVD drive 260.

  The RAM 210 is a memory that stores a program and a program execution result, and the CPU 220 is a central processing unit that reads the program from the RAM 210 and executes the program.

  The HDD 230 is a disk device that stores programs and data, and the LAN interface 240 is an interface for connecting the computer 200 to other computers via the LAN.

  The input / output interface 250 is an interface for connecting an input device such as a mouse or a keyboard and a display device, and the DVD drive 260 is a device for reading / writing a DVD.

  The NE extraction program 211 executed in the computer 200 is stored in the DVD, read from the DVD by the DVD drive 260, and installed in the computer 200.

  Alternatively, this NE extraction program 211 is stored in a database or the like of another computer system connected via the LAN interface 240, read from these databases, and installed in the computer 200.

  The installed NE extraction program 211 is stored in the HDD 230, read into the RAM 210, and executed by the CPU 220 as the NE extraction process 221.

  In this embodiment, the NE extraction program is executed by one computer. However, the present invention is not limited to this, and a program corresponding to each NE extractor in the first stage is The same can be applied to the case where each is executed by one computer.

  In the present embodiment, the NE extraction apparatus has been described. However, the present invention is not limited to this, and can be similarly applied to a classification apparatus using stacking.

(Supplementary note 1) A learning data creation program for creating stacking learning data used for stacking learning for creating a classification program for performing classification by using classification results of a plurality of classification programs based on different machine learning algorithms as clues. ,
A learning data dividing procedure for dividing the learning data used by the plurality of classification programs for machine learning into a plurality of partial learning data;
One partial learning data of a plurality of partial learning data obtained by dividing the learning data by the learning data dividing procedure is used as test data, and the partial learning data excluding the one partial learning data is used as learning data. A learning data generation procedure for stacking that generates learning data for stacking by repeating a procedure of acquiring test results tested by the plurality of classification programs learned by a machine learning algorithm as learning data for stacking,
A learning data creation program characterized by causing a computer to execute.

(Supplementary note 2) The learning data creation program according to supplementary note 1, wherein the stacking learning data generation procedure is executed by a different computer for each of the plurality of classification programs.

(Additional remark 3) The said classification program is a specific expression extraction program which extracts a specific expression from a sentence, The learning data creation program of Additional remark 1 or 2 characterized by the above-mentioned.

(Additional remark 4) The said specific expression extraction program determines whether it is a specific expression for every word extracted by the said word extraction procedure and the word extraction procedure which extracts a word from a sentence, and extracts a specific expression The learning according to attachment 3 for creating a classification program for extracting a specific expression by determining whether each word is a specific expression by using a plurality of classification programs and the output results of the plurality of classification programs as clues Data creation program.

(Appendix 5) A stacking classification program for performing classification by using classification results of a plurality of classification programs based on different machine learning algorithms as a clue,
A learning data dividing procedure for dividing the learning data used by the plurality of classification programs for machine learning into a plurality of partial learning data;
One partial learning data of a plurality of partial learning data obtained by dividing the learning data by the learning data dividing procedure is used as test data, and the partial learning data excluding the one partial learning data is used as learning data. A learning data generation procedure for stacking that generates learning data for stacking by repeating a procedure of acquiring test results obtained by testing the plurality of classification programs learned by the machine learning algorithm as learning data for stacking;
A stacking learning procedure for performing stacking learning using the learning data for stacking generated by the learning data generation procedure for stacking;
A stacking classification program for causing a computer to execute.

(Supplementary note 6) A learning data creation method for creating learning data for stacking used for stacking learning in which classification is performed by using classification results of a plurality of classification programs based on different machine learning algorithms as clues,
A learning data dividing step for dividing the learning data used for machine learning by the plurality of classification programs into a plurality of partial learning data;
One partial learning data among a plurality of partial learning data obtained by dividing learning data by the learning data dividing step is used as test data, and partial learning data excluding the one partial learning data is used as learning data. A learning data generation process for stacking that generates learning data for stacking by repeating a procedure of acquiring test results obtained by testing the plurality of classification programs learned by the machine learning algorithm as learning data for stacking;
The learning data creation method characterized by including.

(Supplementary note 7) The learning data creation method according to supplementary note 6, wherein in the stacking learning data generation step, the plurality of classification programs are executed by different computers.

(Supplementary note 8) The learning data creation method according to supplementary note 6 or 7, wherein the classification program is a specific expression extraction program for extracting a specific expression from a sentence.

(Additional remark 9) The said specific expression extraction program determines whether it is a specific expression for every word cut out by the word extraction process which extracts the word from a sentence, and the said word extraction process, and extracts a specific expression Supplementary learning 8 for creating a classification program for extracting a specific expression by determining whether each word is a specific expression by using a plurality of classification programs and output results of the plurality of classification programs as clues Data creation method.

(Appendix 10) A stacking classification method for performing classification by using classification results of a plurality of classification programs based on different machine learning algorithms as a clue,
A learning data dividing step for dividing the learning data used for machine learning by the plurality of classification programs into a plurality of partial learning data;
One partial learning data among a plurality of partial learning data obtained by dividing learning data by the learning data dividing step is used as test data, and partial learning data excluding the one partial learning data is used as learning data. A learning data generation process for stacking that generates learning data for stacking by repeating a procedure of acquiring test results obtained by testing the plurality of classification programs learned by the machine learning algorithm as learning data for stacking;
A stacking learning step of performing stacking learning using the stacking learning data generated by the stacking learning data generation step;
A stacking classification method characterized by including:

(Supplementary note 11) A learning data creation device for creating stacking learning data used for stacking learning in which classification is performed by using classification results of a plurality of classification devices based on different machine learning algorithms as clues,
Learning data dividing means for dividing the learning data used by the plurality of classification devices for machine learning into a plurality of partial learning data;
One partial learning data among a plurality of partial learning data obtained by dividing learning data by the learning data dividing means is used as test data, and partial learning data excluding the one partial learning data is used as learning data. Stacking learning data generating means for generating stacking learning data by repeating a procedure of acquiring test results obtained by testing the plurality of classifiers learned by the machine learning algorithm as stacking learning data;
A learning data creation device characterized by comprising:

(Additional remark 12) The said learning data production | generation means for stacking is a learning data production apparatus of Additional remark 11 characterized by running a classification program with a respectively different computer as said several classification apparatus.

(Supplementary note 13) The learning data creation device according to supplementary note 11 or 12, wherein the classification device is a specific expression extraction device that extracts a specific expression from a sentence.

(Additional remark 14) The said specific expression extraction apparatus determines whether it is a specific expression for every word cut out by the word extraction means which extracts a word from a sentence, and the said word extraction means, and extracts a specific expression The learning according to appendix 13 for creating a classification device that extracts a specific expression by determining whether each word is a specific expression by using a plurality of classification means as a clue based on output results of the plurality of classification means Data creation device.

(Supplementary note 15) A stacking classification device that performs classification by using classification results of a plurality of classification devices based on different machine learning algorithms as a clue,
Learning data dividing means for dividing the learning data used by the plurality of classification devices for machine learning into a plurality of partial learning data;
One partial learning data among a plurality of partial learning data obtained by dividing learning data by the learning data dividing means is used as test data, and partial learning data excluding the one partial learning data is used as learning data. Stacking learning data generating means for generating stacking learning data by repeating a procedure of acquiring test results obtained by testing the plurality of classifiers learned by the machine learning algorithm as stacking learning data;
Stacking learning means for performing stacking learning using the stacking learning data generated by the stacking learning data generating means;
A stacking and classifying apparatus comprising:

  As described above, the learning data creation program, the learning data creation method, and the learning data creation device according to the present invention are useful for the classification device that uses stacking, and are particularly suitable when the learning data creation load is high.

It is a figure which shows the stacking learning algorithm by this invention. 3 is a flowchart of the present invention. It is a functional block diagram which shows the structure of the classification device in this invention. It is a functional block diagram which shows the structure of the NE extraction apparatus which concerns on a present Example. It is a figure which shows the example of NE defined by IREX. It is a figure which shows the example of the learning data for stacking. It is a figure which shows the stacking learning algorithm which concerns on a present Example. It is explanatory drawing for demonstrating the stacking using a some NE extractor. It is a figure which shows the example of conversion from NE tagging of a word unit to NE tagging of a character unit. It is a flowchart which shows the process sequence of the stacking learning process by the NE extraction apparatus which concerns on a present Example. It is a figure which shows the example of preparation of the learning data for stacking by the NE extraction apparatus which concerns on a present Example. It is a figure which shows the example of creation of the learning data for stacking by the method (a). It is a figure which shows the example of preparation of the learning data for stacking by the method (b). It is a figure which shows the example of creation of the learning data for stacking by the method (c). It is a figure which shows an evaluation result. It is a functional block diagram which shows the structure of the computer which performs NE extraction program based on a present Example.

Explanation of symbols

1 Classifier 11 _{1 to} 11 _m Classifier 12 Stacking Learning Data Storage Unit 13 Stacking Classifier 14 Learning Control Unit 15 Extraction Control Unit 100 NE Extraction Device 110 _{1 to} 110 _m NE Extraction Device 111 NET Unit 120 Stacking Learning Data Storage unit 130 NE extractor for stacking 140 Learning control unit 150 Extraction control unit 160 Post-processing unit 200 Computer 210 RAM
211 NE extraction program 220 CPU
221 NE extraction process 230 HDD
240 LAN interface 250 I / O interface 260 DVD drive

Claims (6)

A learning data creation program for creating learning data for stacking used for stacking learning for creating a classification program for performing classification by using classification results of a plurality of classification programs based on different machine learning algorithms as a clue,
A learning data dividing procedure for dividing the learning data used by the plurality of classification programs for machine learning into a plurality of partial learning data;
One partial learning data of a plurality of partial learning data obtained by dividing the learning data by the learning data dividing procedure is used as test data, and the partial learning data excluding the one partial learning data is used as learning data. A learning data generation procedure for stacking that generates learning data for stacking by repeating a procedure of acquiring test results tested by the plurality of classification programs learned by a machine learning algorithm as learning data for stacking,
A learning data creation program characterized by causing a computer to execute.   The learning data creation program according to claim 1, wherein the learning data generation procedure for stacking is executed by a different computer for each of the plurality of classification programs.   The learning data creation program according to claim 1, wherein the classification program is a specific expression extraction program that extracts a specific expression from a sentence.   A plurality of classification programs for extracting a specific expression by determining whether the specific expression extraction program is a specific expression for each word extracted by the word extraction procedure; The learning data creation program according to claim 3 for creating a classification program for determining whether each word is a unique expression by using the output results of the plurality of classification programs as a clue and extracting the specific expression . A learning data creation method for creating learning data for stacking used for stacking learning for creating a classification program for performing classification by using classification results of a plurality of classification programs based on different machine learning algorithms as clues,
A learning data dividing step for dividing the learning data used for machine learning by the plurality of classification programs into a plurality of partial learning data;
One partial learning data among a plurality of partial learning data obtained by dividing learning data by the learning data dividing step is used as test data, and partial learning data excluding the one partial learning data is used as learning data. A learning data generation process for stacking that generates learning data for stacking by repeating a procedure of acquiring test results obtained by testing the plurality of classification programs learned by the machine learning algorithm as learning data for stacking;
The learning data creation method characterized by including. A learning data creation device for creating stacking learning data for use in stacking learning for creating a classification program for performing classification by using classification results of a plurality of classification devices based on different machine learning algorithms as clues,
Learning data dividing means for dividing the learning data used by the plurality of classification devices for machine learning into a plurality of partial learning data;
One partial learning data among a plurality of partial learning data obtained by dividing learning data by the learning data dividing means is used as test data, and partial learning data excluding the one partial learning data is used as learning data. Stacking learning data generating means for generating stacking learning data by repeating a procedure of acquiring test results obtained by testing the plurality of classifiers learned by the machine learning algorithm as stacking learning data;
A learning data creation device characterized by comprising:

Images