JP6269953B2 - Word segmentation apparatus, method, and program - Google Patents

Description

  The present invention relates to a word dividing device, method, and program, and more particularly, to a word dividing device, method, and program for dividing a word for an input character string.

  In a language processing system that targets languages that do not specify word breaks in Japanese or Chinese orthography, normal word division processing is performed at an early stage, and character strings that make up an input document or input sentence are converted to word strings. . A strict definition of what is a word is usually not easy, and it is common to determine and use word units based on a kind of part-of-speech system. In recent language processing systems, word units are determined using a part-of-speech system used in a dictionary called IPAdic or UniDic. As word division processing performed based on the definition of such words, the mainstream in recent years is for word division using language data to which word division information is added (hereinafter referred to as a word division corpus). This is a method of learning a statistical model of the above and performing word segmentation processing of the input sentence based on the statistical model (Non-patent Document 1, Non-patent Document 2).

  In addition, a technique called field adaptation is known. There are roughly two types of field adaptation methods for word division. One is a method of preparing a word division corpus in a target field and learning a statistical model by combining with the original word division corpus or additionally learning a learned model (Non-patent Document 3). The technology of Non-Patent Document 3 treats word division as a problem of binary classification of whether or not each character is a word division position, and uses a statistical model that has already been learned for sentences in the subject field. This method describes a method in which additional corrective learning of a statistical model is performed gradually by creating correct answers for partial word division by manually giving correct answers to places where the certainty of division is small. The other is a method of using a statistic of a character string obtained from a corpus that is not divided into words in the target field (hereinafter referred to as a raw corpus) as a feature amount (hereinafter referred to as a feature) during word division. (Non-Patent Document 4, Non-Patent Document 5). In Non-Patent Document 5, using Accessor Variety (Non-Patent Document 6), the difference in the number of characters connected to both ends of a partial character string indicates that the partial character string is an independent word, Using the value as a feature, the degree to which the Accessor Variety value contributes to word division is learned from the word division corpus of the original field.

Taku Kudo et al., Applying Conditional Random Fields to Japanese Morphological Analysis, Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, 2004 Maruyama Hiroshi et al., Stochastic Morphological Analysis, Proceedings of the 8th Annual Meeting of the Japan Society for Software Science, pp.177-180, 1991 Shinsuke Mori et al., Automatic word segmentation by point prediction, IPSJ Transactions Vol.52, No.10, pp. 2944-2952, 2011 Yiou Wang et al., Improving Chinese Word Segmentation and POS Tagging with Semi-supervised Methods Using Large Auto-Analyzed Data, Proceedings of 5th International Joint Conference on Natural Language Processing, pp 309-317,2011 Zhen Guo et al., Exploration of N-gram Features for the Domain Adaptation of Chinese Word Segmentation.Proceedings of the 1st CCF Confer-ence on Natural Language Processing & Chinese Computing, pp 121-131, 2012. Haodi Feng et al., Accessor Variety Criteria for Chinese Word Extraction.Computational Linguistics, volume 30, pp 75-93, 2004

  However, in the methods of Non-Patent Document 1 and Non-Patent Document 2, while it is possible to perform word segmentation with high accuracy for sentences similar to the word segmentation corpus, However, there is a problem that the accuracy of word division tends to be relatively low.

  In Japanese word segmentation, vocabulary information obtained from a word segmentation corpus or an external dictionary, and character information such as kanji, hiragana, katakana, and numbers in character strings are known to be effective as features. However, for long compound words composed of katakana and kanji, there is a problem that if there is insufficient vocabulary information, effective clues for word division cannot be obtained and it is difficult to correctly divide.

  In addition, in the above-described prior art, it is necessary to create even a small amount of correct answer data in the method based on additional learning using the correct answer data in the target field, and the method using character string statistics as a feature can be obtained from a raw corpus. Since the contribution degree of the character string statistic is learned from the word division corpus of the original field, there is a problem that it is not easy to learn the contribution degree when the common part between the raw corpus and the word division corpus of the original field decreases. .

  In addition, in the method used as a feature, it is necessary to assign a feature of a statistic based on a large-scale raw corpus every time to a character string in a sentence to be newly divided into words, which increases the calculation time of the word division processing. There is a problem of doing.

  The present invention has been made to solve the above problem, and an object of the present invention is to provide a word segmentation apparatus, method, and program capable of segmenting a word with high accuracy for a character string in a target field.

  In order to achieve the above object, a word segmentation device according to a first aspect of the present invention estimates a word segmentation position for each of character strings included in a raw corpus, which is a set of character strings in a target field, A division position estimation unit for assigning a label indicating a position to divide a word between each of them, and a set of character strings in an original field different from the target field, and indicating a position at which a word is divided between each character A feature for each of characters between which a label indicating a position to divide a word is assigned to each character string included in a word division corpus to which a label and a label indicating a position not to divide a word are assigned in advance, and word division Between each character with a label indicating a position to divide a word into each character string included in the raw corpus A learning feature extraction unit that extracts features of each of the characters, a feature that is extracted by the learning feature extraction unit, for each character between the characters that are given a label that indicates a position to divide the word, and a label that indicates a position where the word is not divided. A model learning unit that learns a word division model for determining a word division position for a character string in the target field based on a feature for each given character, and an input character string in the target field Based on a feature extraction unit that extracts a feature for each character included in the character, a feature for each of the characters extracted by the feature extraction unit, and the word division model learned by the model learning unit A binary classifying unit that determines a position to divide a word from each of the characters included in the input character string of the target field. There.

  A word division method according to a second aspect is a word division method in a word division device including a division position estimation unit, a learning feature extraction unit, a model learning unit, a feature extraction unit, and a binary classification unit. The division position estimation unit estimates a position where words are divided for each character string included in the raw corpus, which is a set of character strings in the target field, and indicates a position where the word is divided between each character. The learning feature extraction unit assigns a label, and is a set of character strings in an original field different from the target field, and indicates a label indicating a position where a word is divided into each character and a position where the word is not divided For each of the character strings included in the word division corpus to which a label is assigned in advance, a feature for each of the characters to which a label indicating a position for dividing the word is given and a label indicating a position where the word is not divided are assigned. Sentence Extracting a feature for each of the character strings, extracting a feature for each of the character strings included in the raw corpus, each having a label indicating a position to divide the word, and the model learning unit Are extracted by the learning feature extraction unit for the features between the characters to which the label indicating the position to divide the word is assigned and the features for each character to which the label indicating the position to not divide the word is assigned. Based on this, a word division model for determining a word division position for a character string in the target field is learned, and the feature extraction unit is configured to input each character between characters included in the input character string in the target field. The binary classification unit extracts features from each of the characters extracted by the feature extraction unit, and the word division module learned by the model learning unit. Based on the Le, from each of the inter-character in the string of the inputted said target field to determine the location of word segmentation.

  According to the first and second inventions, the division position estimation unit estimates the position at which the word is divided for each of the character strings included in the raw corpus that is a set of character strings in the target field, and A label indicating the position where the word is divided is assigned to each, and the learning feature extraction unit is a set of character strings in the original field different from the target field, and the label indicating the position where the word is divided into each between characters For each of the character strings included in the word division corpus to which a label indicating a position where no word is divided is assigned in advance, a feature between each of the characters provided with a label indicating a position where the word is divided, and a position where the word is not divided For each of the characters between the characters with the label indicating the position to divide the word into each character string included in the raw corpus The features are extracted, and the model learning unit extracts the features for each of the characters given the label indicating the position to divide the word, and the characters between the characters given the label indicating the position where the word is not divided. Based on the feature, learn a word division model for determining the position of word division for the character string in the target field, and by the feature extraction unit, the feature between each character included in the input character string of the target field And the binary classification unit, based on the feature between each of the extracted characters and the learned word division model, from each of the characters included in the input character string of the target field, The position where the word is divided is determined.

  In this way, each character string included in the raw corpus, which is a set of character strings in the target field, is assigned a label indicating the position of word division between each character, and the original field different from the target field. For each of the character strings included in the word segmentation corpus, which is a set of character strings and has a label indicating a position at which to divide a word between each character and a label indicating a position at which the word is not divided are given in advance. Each of the character strings included in the raw corpus is extracted by extracting the features for each of the characters between which the labels indicating the positions to be divided and the characters between the characters having the labels indicating the positions where the words are not divided are assigned. For each of the characters between the characters to which the label indicating the position to divide the word is extracted, the feature between each character to which the label to indicate the position to divide the word is assigned, and Learning a word division model for determining a word division position for a character string of a target field based on a feature for each character with a label indicating a position where word division is not performed. Characters included in the character string of the input target field are extracted based on the extracted features of each character between the characters included in the character string and the learned word division model. By determining the position where the word is divided from each of the intervals, it is possible to perform word division with high accuracy for the character string in the target field.

  The first invention further includes a statistic calculator for calculating a statistic of characters connected before and after the partial character string for each partial character string based on the character string included in the raw corpus. The division position estimation unit is configured to determine a character string included in the raw corpus based on a statistic of characters connected before and after the partial character string calculated for each partial character string by the statistic calculation unit. For each, a position where the word is divided may be estimated, and a label indicating the position where the word is divided may be given to each between the characters.

  In the first invention, the division position estimation unit estimates a position where words are divided and a position where words are not divided, for each character string included in the raw corpus, and word division is performed between characters. Any one of a label indicating a position to be performed, a label indicating a position where word division is not performed, and a label indicating a position where the division is unknown is provided, and the learning feature extraction unit applies each of the character strings included in the word division corpus On the other hand, the features for each of the characters between which the label indicating the position to divide the word is assigned and the features for each of the characters to which the label indicating the position not to divide the word are extracted are included in the raw corpus For each character string, a feature for each character between which a label indicating a position to divide a word is given and each character between a character to which a label indicating a position not to divide a word is given Identity may be the extraction of about.

  Further, in the first invention, the division position estimation unit estimates a position for dividing a word for each of the character strings included in the raw corpus, and a label indicating a position for dividing the word between each character. , And a label indicating the position where the division is unknown may be provided.

  Moreover, the program of this invention is a program for functioning a computer as each part which comprises said word division | segmentation apparatus.

  As described above, according to the word segmentation apparatus, method, and program of the present invention, for each character string included in the raw corpus that is a set of character strings in the target field, word segmentation is performed between each character. A word indicating a position to be divided, and a word division corpus of an original field different from the target field, and a raw corpus, a feature for each of characters between which a label indicating a word dividing position is assigned, and a word The feature between each character with a label indicating the position to not divide is extracted, the word division model for determining the word division position for the character string in the target field is learned, and the learned character in the target field Based on the word division model for determining the word division position for the column, the character string in the target field is determined in detail by determining the word division position of the character string in the target field. Well it can be a word division.

It is a block diagram which shows the functional structure of the word division | segmentation apparatus which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the functional structure of the model learning apparatus which concerns on the 1st Embodiment of this invention. The example of the character referred to the classification _| category of a certain character spacing ti is shown. It is a block diagram which shows the functional structure of the word division | segmentation determination apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart figure which shows the partial word division corpus processing routine in the model learning apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart figure which shows the model learning process routine in the model learning apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart figure which shows the word division | segmentation determination processing routine in the word division | segmentation determination apparatus which concerns on the 1st Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Principle of this embodiment>
First, the principle in the present embodiment will be described. In particular, this embodiment is made by paying attention to the fact that there are many errors in compound words composed of katakana and kanji in Japanese word division. This embodiment calculates the statistic of a character string that is a clue to word division in a raw corpus of a large target field, automatically determines a portion that can be expected to be a word boundary in the raw corpus, The location is used as additional learning data for word division.

  It is assumed that the word division program used in the present embodiment is similar to that described in Non-Patent Document 3 and determines whether or not a certain character space is a word boundary by binary classification. In the present embodiment, a statistical model is learned using a word division corpus and a raw corpus in a target field that is a field different from the word division corpus, and word division processing is performed on an input character string in the target field. It has a function.

  Next, the word division program used in this embodiment will be described. The word segmentation program determines whether or not a character segmentation position is a word segmentation position by using binary classification based on information on surrounding characters and the like, and outputs a character string segmented based on the result. Non-patent document 3 includes information on peripheral characters and character types, whether or not peripheral partial character strings are words registered in an external dictionary, and the like. As a binary classification method, a known statistical method such as logistic regression or support vector machine (SVM) can be applied. In learning the statistical model of binary classification, a word division corpus and a partial word division corpus can be used. Here, the word division corpus is distinguished from the one in which the information on the word division position is given over the entire document or sentence, and the partial word division corpus is distinguished from the one in which the information on the word division position is given in part. The word segmentation method based on binary classification has a great advantage in that the statistical model can be learned by a partial word segmentation corpus, and that the statistical model can be strengthened by only local information.

Next, use of the raw corpus in the present embodiment will be described. Non-Patent Document 6 Accessor Variety and Non-Patent Document 7 (Zhihui Jin et al. Unsupervised Segmentation of Chinese Text by Use of Branching Entropy, Proceedings of the COLING / ACL 2006, pp 428- 435, 2006.) and Branching Entropy. The Accessor Variety is the number of different characters connected to both ends of a certain partial character string. For the partial character string x ⁿ _{1 of} length n, the value of AV (x ⁿ ₁ ) represented by the following equation (1) Become.

Here, AV _L (x _n ) is the number of different characters connected to the left side of the partial character string x _{n in} the raw corpus, and AV _R (x _n ) is the number of different characters connected to the right side. Branching Entropy is the entropy of the character to be connected next to a partial character string, and H (X | X ⁿ ₁ = x ⁿ ) expressed by the following equation (2) with respect to the partial character string x ⁿ _{1 of} length ⁿ ₁ ).

It is also possible to calculate the entropy of the character connected before the partial character string by considering X and x as characters connected before the partial character string. The difference value of Branching Entropy can also be seen as shown in the following equation (3). This is the difference between the entropy of the next character of the partial character string x ⁿ ₁ and the entropy of the next character of x ⁿ⁻¹ ₁ (corresponding to the last character of x ⁿ ₁ ). Intuitively, it is expected that the longer the partial character string, the smaller the ambiguity of the next character and the smaller the entropy. However, if a word boundary immediately follows x ⁿ ₁ , the word that can appear subsequently Since there is a high possibility that entropy will increase due to the ambiguity of the number of words, this is a powerful clue for finding word boundaries (Non-patent Document 7).

  Where these values are large, the connected character is highly ambiguous, that is, it is expected to be a boundary between words connected by various words before and after, so estimate the word division position using only the raw corpus Can do. Non-Patent Document 6 aims to find Chinese words using Accessor Variety, and Non-Patent Document 7 uses Branching Entropy to perform word segmentation without using a word segmentation corpus. It is said. However, since it is difficult to perform high-precision word division for all words only by estimating the word division position based on these statistics, it is combined with learning of a statistical model by a word division corpus as in this embodiment. By using this method, the method using only the word segmentation corpus that is highly accurate in the field that matches the word segmentation corpus but does not allow sufficient learning of the target field, and the word segmentation in the target field is possible to some extent, but the overall word segmentation accuracy is sufficient It is important to reconcile methods that use only raw corpora. In the present embodiment, a case will be described as an example where an Accessor Variety feature amount is acquired from a raw corpus.

  In the present embodiment, the advantage of the word classification method based on binary classification that can be learned by the partial word division corpus is utilized by using a large number of raw corpora whose word division positions can be estimated. In other words, improve word segmentation for vocabulary specific to the subject area that could not be adequately applied only with word segmentation corpora in different fields, using word segmentation information automatically estimated in the raw corpus of the subject field. Is the approach of the present embodiment. As described above, the statistics obtained from the raw corpus alone cannot achieve a high precision of word segmentation, but in particular for long compound words, the words that make up a compound word are usually used in different contexts. Therefore, it can be expected that a significant difference appears in the statistics of the character string, which is suitable for the purpose of this embodiment.

  In estimating the word division position, a threshold is set for the statistic of the character string, a label meaning “word division position” is attached to a portion exceeding the threshold value, and “partition existence is unknown” for the other portion A label meaning “position” is attached. Here, a label that means “a position where words are not divided” may be attached to a location below the threshold or a location below another smaller threshold. In addition, in order to maintain consistency with the word division corpus of the original field, only the portions that match the word division presence / absence obtained by the word division program learned by the word division corpus of the original field are retained, and the portions that do not match are divided. It may be a “presence / absence unknown position”. Furthermore, in order to focus on improving compound word segmentation, which is expected to have a significant effect on field adaptation, the “word segmentation position” is limited to specific targets such as “between katakana and katakana” and “between kanji and kanji”. You may make it attach the label of "the position which does not divide a word". If this operation is performed over the entire raw corpus, a corpus having the labels “position for dividing words”, “position where division is unknown”, and “position where words are not divided” is obtained. The corpus becomes a partial word division corpus due to the presence of the label “divided / unknown position”. In the present embodiment, a label that means “a position to divide words” is attached to a portion where the statistic of the character string exceeds the threshold value, which is equal to or less than the first threshold value and higher than the second threshold value. A label meaning “a position where the division is unknown” is attached to a large portion, and a label meaning “a position where no word is divided” is attached to a portion below the second threshold.

  The obtained partial word division corpus can be used for learning a statistical model of word division. In learning, the statistical model may be re-learned from the beginning in combination with the original word segmentation corpus, or a mixed model with the model obtained from the added partial word segmentation corpus (mixing ratio is determined separately) is configured. May be. As a learning method, a method used in the well-known word division program KyTea can be used.

<Configuration of word segmentation apparatus according to first embodiment of the present invention>
Next, the configuration of the word segmentation device according to the first embodiment of the present invention will be described. As shown in FIG. 1, the word segmentation device 1 according to the first exemplary embodiment of the present invention includes a model learning device 100 and a word segmentation determination device 200.

<Configuration of Model Learning Device According to First Embodiment of the Present Invention>
Next, the configuration of the model learning device according to the first embodiment of the present invention will be described. As shown in FIG. 2, the model learning device 100 according to the first embodiment of the present invention includes a CPU, a RAM, and a program for executing a partial word division corpus processing routine and a model learning processing routine described later. And a ROM including various data stored in a ROM. Functionally, the model learning apparatus 100 includes an input unit 10, a calculation unit 20, and an output unit 90 as shown in FIG.

  The input unit 10 accepts a raw corpus that is a set of character strings in the target field and stores it in the raw corpus storage unit 22.

  The arithmetic unit 20 includes a raw corpus storage unit 22, a statistic storage unit 24, an additional learning corpus unit 30, a partial word division corpus storage unit 40, a word division corpus storage unit 42, and a word dictionary storage unit 44. The statistical model learning unit 50 and the model storage unit 60 are provided.

  The raw corpus storage unit 22 stores the raw corpus received by the input unit 10.

  The additional learning corpus unit 30 acquires a partial word division corpus based on the statistic of characters connected before and after each of the partial character strings included in the raw corpus stored in the raw corpus storage unit 22, This is stored in the partial word division corpus storage unit 40. Further, the additional learning corpus unit 30 includes a statistic calculation unit 32, a division position estimation unit 34, and a corpus output unit 36.

  The statistic calculation unit 32 first acquires each of the partial character strings of length N included in the raw corpus stored in the raw corpus storage unit 22. Here, the character string is scanned through a window of length N, and a partial character string of length N is acquired while shifting character by character. At this time, partial character strings composed of the same character string are treated as the same partial character string. Next, for each of the acquired partial character strings, the statistic calculation unit 32 acquires a character connected to the left side of the partial character string and a character connected to the right side. Then, for each of the partial character strings, based on each of the characters connected to the left side and the characters connected to the right side acquired for each of the partial character strings, it is connected to both ends of the partial character string according to the above formula (1) The statistic of Accessor Variety, which is the number of different characters to be calculated, is calculated and stored in the statistic storage unit 24 as character string statistic data. The extraction of characters connected to a partial character string can be performed by dividing the raw corpus into an appropriate size for parallel processing in order to speed up the processing, and entropy calculation for the partial character string can be performed at high speed. It is possible to parallelize each partial character string for conversion. Moreover, as the length N of the partial character string is used in Non-Patent Document 5 and Non-Patent Document 7, a plurality of characters may be used in parallel.

  The division position estimation unit 34 estimates the word division position based on the statistics of characters connected before and after the partial character string calculated for each partial character string stored in the statistic storage unit 24. For each partial character string, by giving a label indicating a position to be divided, a label indicating a position not to be divided, or a label indicating a position where the division is unknown, between characters before and after the partial character string, Label each character between all characters. Specifically, a predetermined first threshold value and second threshold value (first threshold value> second threshold value) are determined, and each partial character string is connected before and after the partial character string. If the character statistic is greater than a predetermined first threshold, a label indicating the position to be divided is assigned, and the character statistic connected before and after the partial character string is determined in advance. A label indicating the position where the division is unknown, and a statistic of characters connected before and after the partial character string is less than the first threshold and greater than a predetermined second threshold. When the value is equal to or less than a predetermined second threshold value, a label indicating a position that is not divided is assigned.

  The corpus output unit 36 stores the raw corpus given the label by the division position estimation unit 34 in the partial word division corpus storage unit 40 as a partial word division corpus. As the form of the partial word division corpus, for example, the word division corpus used in the known word divider KyTea uses “|” as the position where the word is divided, and the position where the word is not divided as the character. A character string in which “-” is inserted in between and a blank character or “?” Is inserted between characters at positions where word division is unknown.

  The partial word division corpus storage unit 40 stores the partial word division corpus acquired by the corpus output unit 36.

  The word division corpus storage unit 42 is a set of character strings in the original field that is a field different from the target field, and indicates a label indicating a position where each word is divided between characters and a position where no word is divided. A word division corpus to which labels are assigned in advance is stored.

  The word dictionary storage unit 44 stores a word dictionary composed of a plurality of predefined words.

  The statistical model learning unit 50 stores the partial word division corpus stored in the partial word division corpus storage unit 40, the word division corpus stored in the word division corpus storage unit 42, and the word dictionary storage unit 44. A word division model for determining a position to divide a word with respect to a character string in the target field is learned based on the word dictionary. The statistical model learning unit 50 includes a learning feature extraction unit 52 and a model learning unit 54.

Based on the word dictionary stored in the word dictionary storage unit 44, the learning feature extraction unit 52 applies each character string included in the partial word division corpus stored in the partial word division corpus storage unit 40. Then, a feature for each of the characters between which the label indicating the position to divide the word is assigned and a feature for each of the characters to which the label indicating the position where the word is not divided are assigned are extracted, and the word division corpus storage unit 42 is extracted. Each of the character strings included in the word division corpus stored in is provided with a feature for each of the characters to which a label indicating a word division position is assigned, and a label indicating a position where the word division is not performed. Extract features for each character. For example, as a feature, the following (a) to (c) are extracted for a certain character interval t _i (Non-patent Document 3). The character referred to in the classification _| category of the character spacing ti in FIG. 3 is shown.

(A) Character n-gram: a partial character string before and after the position i between characters, and when there are parameters of a window width m and a length n, a character string x _{i-m + 1} ... x _{i-1 with} a length of 2 _m x _i x _{i + 1} ... is a feature made up of all partial characters (character n-gram) of length n of x _{i + m} .
(B) Character type n-gram: a partial character string before and after a position i between characters, and when there are parameters of a window width m and a length n, a character string x _{i-m + 1} ... x _{i−1 having} a length of 2 _m x _i x _{i + 1} ... is a feature made up of character types included in all partial characters (character n-gram) of length n of x _{i + m} . Here, there are six types of characters: kanji, katakana, hiragana, romaji, numbers, and others.
(C) Word dictionary feature: the left partial character string x _{i−k + 1} x _{i−k + 2} ... X _i for each length k is included in the word dictionary stored in the word dictionary storage unit 44 as a word. Whether the right partial character string x _{i + 1} x _{i + 2} ... X _{i + k} between the characters is included in the word dictionary stored in the word dictionary storage unit 44 as a word, and the partial character straddling the character Row x _{i−j + 1} x _{i−j + 2} ... X _{i−j + k}

Is included in the word dictionary stored in the word dictionary storage unit 44 as a word.

  The model learning unit 54 includes a label for each character of the character string included in the partial word division corpus stored in the partial word division corpus storage unit 40 and the features extracted by the learning feature extraction unit, Based on the character-by-character labels of the character strings included in the word division corpus stored in the division corpus storage unit 42 and the features extracted by the learning feature extraction unit, word division is performed on the character strings in the target field. The word division model for determining the position to be learned is learned and output to the output unit 90 and stored in the model storage unit 60. For model learning, LibLinear for learning a classifier using logistic regression or a support vector machine, SVMlight for learning a classifier using a support vector machine, or the like is used.

  The model storage unit 60 stores a word division model for determining a position where a word division is performed for a character string in the target field learned by the model learning unit 54.

<Configuration of Word Division Determination Apparatus According to First Embodiment of the Present Invention>
Next, the configuration of the word division determination device according to the first embodiment of the present invention will be described. As shown in FIG. 4, the word division determination apparatus 200 according to the first embodiment of the present invention stores a CPU, a RAM, a program for executing a word division determination processing routine described later, and various data. It can be composed of a computer including a ROM. Functionally, the word division determination apparatus 200 includes an input unit 210, a calculation unit 220, and an output unit 290 as shown in FIG.

  The input unit 210 receives a character string in the target field.

  The calculation unit 220 includes a word dictionary storage unit 230, a model storage unit 232, a word division processing unit 240, and a word string storage unit 250.

  The word dictionary storage unit 230 stores the same word dictionary as the word dictionary storage unit 44 of the model learning device 100.

  The model storage unit 232 stores the same word division model for determining the position of word division for the character string in the target field, which is the same as the model storage unit 60 of the model learning device 100.

  The word division processing unit 240 divides the character string in the target field received by the input unit 210 into a word string. The word division processing unit 240 includes a feature extraction unit 242, a binary classification unit 244, and a data conversion unit 246.

  Similar to the learning feature extraction unit 52 of the model learning device 100, the feature extraction unit 242 uses a word dictionary stored in the word dictionary storage unit 230 for each character string received by the input unit 210. Extract features.

  The binary classification unit 244 positions each of the features between characters of the character string received by the input unit 210 extracted by the feature extraction unit 242 and the word division of the character string of the target field stored in the model storage unit 232. It is determined based on the word division model for determining whether or not each position between the characters is a position where the word is divided. It should be noted that various known classifiers represented by logistic regression and support vector machines can be used to determine whether or not the position is a word division position.

  The data conversion unit 246 divides the character string received by the input unit 210 into word strings based on the result determined by the binary classification unit 244, stores the character strings in the word string storage unit 250 as word string data, and outputs them. Output to the unit 290. Note that when outputting word string data to a terminal or a file as output, the division position is typically represented by inserting a blank character between the characters to be divided, but the data format is not particularly limited.

<Operation of Model Learning Device According to First Embodiment of the Present Invention>
Next, the operation of the model learning device 100 according to the first embodiment of the present invention will be described. First, a raw corpus that is a set of character strings in the target field is received and stored in the raw corpus storage unit 22. Then, when the raw corpus is read from the raw corpus storage unit 22, the model learning device 100 executes a partial word division corpus processing routine shown in FIG. When the partial word division corpus processing routine ends, the model learning device 100 executes the model learning processing routine shown in FIG.

  First, the partial word division corpus processing routine shown in FIG. 5 will be described.

  In step S102, each of the character strings included in the read raw corpus and having a character string length N is acquired.

  Next, in step S104, for each of the partial character strings acquired in step S102, characters connected to the left side and the right side are acquired.

  Next, in step S106, for each of the partial character strings acquired in step S102, based on the characters connected to the left side and right side acquired in step S104, before and after the partial character string, according to the above equation (1). Calculate the statistics for connected characters.

  Next, in step S108, for each of the partial character strings acquired in step S102, the statistical amount of characters connected before and after the partial character string acquired in step S104, a predetermined first threshold value, and The raw corpus is assigned with a label indicating a position to be divided, a label indicating a position where no division is performed, or a label indicating a position where the division is unknown based on a threshold value of 2. A label is assigned to each of all the characters in.

  Next, in step S110, the raw corpus acquired in step S108 and labeled between each character is stored in the partial word division corpus storage unit 40 as a partial word division corpus, and the partial word division is performed. The corpus processing routine is terminated.

  Next, the model learning process routine shown in FIG. 6 will be described.

  First, in step S200, the word division corpus stored in the word division corpus storage unit 42 is read.

  Next, in step S202, the partial word division corpus stored in the partial word division corpus storage unit 40 is read.

  In step S204, the word dictionary stored in the word dictionary storage unit 44 is read.

  Next, in step S206, based on the word dictionary acquired in step S204, a label indicating the position at which the word is divided is assigned to each of the character strings included in the partial word division corpus acquired in step S202. A feature for each of the characters and a feature for each of the characters to which a label indicating a position where the word is not divided are extracted, and for each of the character strings included in the word division corpus acquired in step S200, A feature for each character between which a label indicating a position to divide a word is assigned and a feature for each character to which a label indicating a position where no word is divided is assigned are extracted.

  Next, in step S208, a label between each character of the character string included in the word division corpus acquired in step S200 and each character interval of each character string included in the partial word division corpus acquired in step S202. , And the character segmentation position of the character string in the target field based on each of the features acquired for each character string included in the word segmentation corpus and the partial word segmentation corpus acquired in step S206. Learn the word division model for judgment.

  Next, in step S210, a word division model for determining a word division position for the character string in the target field acquired in step S208 is stored in the model storage unit 60 and output to the output unit 90 for model learning. The processing routine ends.

<Operation of the word division determination device according to the first embodiment of the present invention>
Next, the operation of the word division determination apparatus 200 according to the first embodiment of the present invention will be described. First, from the input unit 210, a word division model for determining a word division position for a character string in the target field learned by the model learning device 100 is input and stored in the model storage unit 232. When receiving the character string to be processed, the word division determination device 200 executes a word division determination processing routine shown in FIG.

  First, in step S300, a word dictionary stored in the word dictionary storage unit 230 is read.

  Next, in step S <b> 302, a word division model for determining a position where word division is performed for the character string in the target field stored in the model storage unit 232 is read.

  Next, in step S304, each feature between the characters in the character string received by the input unit 210 is extracted as in step S206.

  Next, in step S306, in order to determine each of the features between the characters of the character string received in the input unit 210 acquired in step S304 and the position where the word is divided for the character string of the target field acquired in step S302. Based on the word division model, it is determined whether or not each position between characters is a position where a word is divided.

  Next, in step S308, the character string received in the input unit 210 is divided into word strings based on the result determined for each character in the character string acquired in step S306 to obtain word string data.

  Next, in step S310, the word string data acquired in step S308 is stored in the word string storage unit 250 and output to the output unit 290, and the word division determination processing routine is terminated.

<Experimental example>
In an experiment using the method described in this embodiment, when using a word division corpus of general Japanese written words and a large number of Japanese patent students corpus, the division accuracy when the Japanese patent sentence is divided into words (F value) was improved from 96.14% to 97.42% by field adaptation according to the present embodiment. That is, the error rate is reduced by about 30% from 3.86% to 2.58%, and this effect is great. The definition of the division accuracy (F value) is shown in the following equation (4).

  As described above, according to the word segmentation device according to the first embodiment of the present invention, each character between characters is included in each character string included in the raw corpus that is a set of character strings in the target field. A word segmentation corpus of a source field different from the target field, by giving a label indicating a position to divide the word, a label indicating a position where the word is not divided, or a label indicating a position where the division is unknown to be a partial word division And a feature for each character between a character with a label indicating a position where a word is divided and a character between each character with a label indicating a position where the word is not divided are extracted. Learning a word division model for determining a position to divide a word with respect to a character string in the target field, and dividing a word to determine a position to divide the word with respect to the learned character string in the target field Based on Dell, by determining the position of word segmentation of the subject areas string, it is possible to accurately word segmentation for the subject areas string.

  In addition, from the statistic of characters connected before and after each of the partial character strings in the raw corpus, automatically determine a portion that can be expected to be a position where the characters between the partial character strings are sufficiently divided, By using the result as additional learning data of a statistical model of word division, it is possible to adapt the field of word division. Since the features used by the word division program do not change and only the learning data increases, the learning time can be expected to increase, but the time of the word division processing itself can be expected not to change significantly.

  In addition, by using a large number of raw corpora in the target field, even if the available word division corpus is different from the target field, it is possible to accurately perform word division in the target field.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, in the first embodiment, a case has been described in which the Accessor Variety statistic calculated for each partial character string is used as character string statistic data. However, the present invention is not limited to this. For each, the entropy calculated for the Branching Entropy statistic according to the above equation (2) may be used as the character string statistic data. In addition, for each partial character string, the difference value of Branching Entropy may be used as character string statistic data. In this case, it is necessary to calculate the statistic similarly for the partial character string of length N-1.

  Further, in the first embodiment, based on the statistic of characters connected before and after the partial character string, a label indicating a position to be divided, a label indicating a position that is not divided, or a position where the division is unknown is indicated. Although the case where a label is applied has been described, the present invention is not limited to this. For example, for the purpose of maintaining consistency with the word division corpus of the original field, labels are given only to positions where the word division matches with the result of word division performed by the word divider learned only with the word division corpus of the original field. You may make it do.

  Further, in the first embodiment, in the learning feature extraction unit, the word division position for each character string included in the partial word division corpus stored in the partial word division corpus storage unit 40 The case of extracting the feature between each of the characters assigned the label indicating the character and the feature between each of the characters assigned the label indicating the position where the word is not divided has been described, but the present invention is not limited to this. Absent. For example, each character string included in the partial word division corpus stored in the partial word division corpus storage unit 40 is further characterized with respect to each of the characters between which the label indicating the position where the division is unknown is given. May be extracted. In addition, for each of the character strings included in the partial word division corpus stored in the partial word division corpus storage unit 40, only the features between the characters to which the label indicating the position for word division is given are provided. It may be extracted.

  In the first embodiment, a case has been described in which the word segmentation device is configured by two devices, that is, a model learning device and a word segmentation determination device. However, the present invention is not limited to this. For example, you may be comprised from three apparatuses, the additional learning corpus apparatus which has the function of the additional learning corpus part 30, the statistical model learning apparatus which has the function of the statistical model learning part 50, and a word division determination apparatus. Further, the word dividing device may have the functions of a model learning device and a word division determining device, and may be configured as one device.

  Next, a word segmentation device according to a second embodiment will be described.

  In the second embodiment, in the division learning position estimation unit 34 in the additional learning corpus 30 of the model learning device 100, a label indicating a division position between characters before and after the partial character string, or whether or not there is division is unknown. The point which provides the label which shows a position differs from 1st Embodiment. In addition, about the structure and effect | action similar to the word division | segmentation apparatus 1 which concerns on 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The division position estimation unit 34 estimates the word division position based on the statistic of characters connected before and after the partial character string calculated for each partial character string by the statistic calculation unit 32, and the partial character string Each time, a label indicating a position to be divided or a label indicating a position where the division is unknown is given between characters before and after the partial character string, thereby giving a label to each of all characters in the raw corpus. . Specifically, a threshold value is set in advance, and for each partial character string, a position to be divided when the statistic of characters connected before and after the partial character string is larger than a predetermined threshold value. When a statistic of characters connected before and after the partial character string is equal to or less than a predetermined threshold value, a label indicating a division presence / absence unknown position is assigned.

  The corpus output unit 36 stores the raw corpus given the label by the division position estimation unit 34 in the partial word division corpus storage unit 40 as a partial word division corpus.

  As described above, according to the word segmentation device according to the second embodiment of the present invention, each character between characters is included in each character string included in the raw corpus that is a set of character strings in the target field. To a partial word division corpus by assigning a label indicating a word division position or a label indicating an unknown division presence position to the original word division corpus different from the target field, and the partial word division corpus Then, a feature for each of characters between which a label indicating a position to divide a word is assigned and a feature for each of a character to which a label indicating a position not to divide the word is extracted are extracted. Learning the word division model for determining the position to divide the word, and based on the word division model for determining the position to divide the word for the learned character string in the target field, By determining the position of dividing the word string, it is possible to accurately word segmentation for the subject areas string.

  Next, a word segmentation device according to a third embodiment will be described.

  In the third embodiment, the division position estimation unit 34 in the additional learning corpus unit 30 of the model learning device 100 assigns only a label indicating the division position between characters before and after the partial character string. Is different from the first embodiment. In addition, about the structure and effect | action similar to the word division | segmentation apparatus 1 which concerns on 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The division position estimation unit 34 estimates the word division position based on the statistic of characters connected before and after the partial character string calculated for each partial character string by the statistic calculation unit 32, and the partial character string Each time, a label is given to each of the characters in the raw corpus by giving a label indicating the position to be divided between the characters before and after the partial character string. Specifically, a threshold value is set in advance, and for each partial character string, a position to be divided when the statistic of characters connected before and after the partial character string is larger than a predetermined threshold value. A label indicating is attached.

  The corpus output unit 36 stores the raw corpus given the label by the division position estimation unit 34 in the partial word division corpus storage unit 40 as a partial word division corpus.

  As described above, according to the word segmentation device according to the third embodiment of the present invention, each character between characters is included in each character string included in the raw corpus that is a set of character strings in the target field. Is a partial word division corpus by assigning a label indicating the word division position to the word division corpus of the original field different from the target field and a label indicating the word division position for the partial word division corpus To extract the features for each of the characters to which the character is assigned and the features for each of the characters to which the label indicating the position where the word is not divided is extracted, and to determine the position at which the word division is performed for the character string in the target field Learn the word division model and determine the position of the character string of the target field to be divided based on the word division model for determining the position of word division for the learned character string of the target field The Rukoto, it is possible to accurately word segmentation for the subject areas string.

  Further, in the third embodiment, the portion where the statistical value is large is often the position where the word is divided, but the statistical value is particularly the position where the word is divided around the short word. In view of the fact that the estimation accuracy of positions where words are not divided is not necessarily high, only labels indicating positions where words are divided are used.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  Further, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium or provided via a network. It is also possible to do.

DESCRIPTION OF SYMBOLS 1 Word segmentation apparatus 10 Input part 20 Operation part 22 Raw corpus storage part 24 Statistics storage part 30 Additional learning corpus part 32 Statistics calculation part 34 Division position estimation part 36 Corpus output part 40 Partial word division corpus storage part 42 Word division Corpus storage unit 44 Word dictionary storage unit 50 Statistical model learning unit 52 Learning feature extraction unit 54 Model learning unit 60 Model storage unit 90 Output unit 100 Model learning device 200 Word division determination device 210 Input unit 220 Operation unit 230 Word dictionary storage unit 232 Model storage unit 240 Word division processing unit 242 Feature extraction unit 244 Binary classification unit 246 Data conversion unit 250 Word string storage unit 290 Output unit

Claims (5)

Based on a statistic about the number of different characters or ambiguity of characters connected before and after the partial character string calculated for each partial character string based on the character string included in the raw corpus that is a set of character strings in the target field A division position estimation unit that estimates a position to divide a word for each of the character strings included in the raw corpus , and gives a label indicating a position to divide the word between each character;
A set of character strings in an original field different from the target field, and a label indicating a position where a word is divided and a label indicating a position where no word is divided are included in each of the characters. For each of the character strings, a feature for each of the characters between which the label indicating the position to divide the word is assigned, and a feature for each of the characters to which a label indicating the position to which the word is not divided are assigned, are extracted. A learning feature extraction unit that extracts a feature for each of characters between each character string included in the raw corpus and a label indicating a position for word division;
Based on the features extracted from the learning feature extraction unit for each of the characters assigned with the label indicating the position to divide the word, and the features for each of the characters assigned the label indicating the position where the word is not divided. A model learning unit for learning a word division model for determining a position to divide a word in the target field character string;
A feature extraction unit that extracts a feature for each of the characters included in the input character string of the target field;
Based on the features for each of the characters extracted by the feature extraction unit and the word division model learned by the model learning unit, between the characters included in the input character string of the target field A binary classification unit for determining a position to divide a word from each;
A word segmentation device. The division position estimation unit estimates a position where words are divided and a position where words are not divided, for each character string composed of a specific character type included in the raw corpus, and word division is performed between each character. label indicating the position of, and any one of label indicating the position not word segmentation granted,
The learning feature extraction unit includes a feature for each character between the character strings included in the word division corpus and a label indicating a position for dividing the word, and a label indicating a position for not dividing the word. The features for each of the given characters are extracted, and for each of the character strings included in the raw corpus, the features and the words are not divided for each of the characters to which a label indicating a position for dividing the word is given. The word segmentation device according to claim 1, wherein a feature for each of characters between which a label indicating a position is assigned is extracted. The division position estimation unit estimates a position to divide words for each of the character strings included in the raw corpus, and among the estimated positions to divide words,
The estimation which estimated the position which divides a word with respect to each of the character string contained in the said raw corpus using the word division | segmentation program for judging the position which divides | segments the word learned based on the word division | segmentation corpus of the said original field The word segmentation device according to claim 1 , wherein a label indicating a word segmentation position is assigned to a position that matches the result, and a label that indicates a segmentation unknown position is imparted to a position that does not match the estimation result . A word division method in a word division device including a division position estimation unit, a learning feature extraction unit, a model learning unit, a feature extraction unit, and a binary classification unit,
The division position estimation unit is calculated for each partial character string based on the character string included in the raw corpus that is a set of character strings in the target field, or the number of different characters connected before and after the partial character string or Based on the statistic regarding ambiguity, for each of the character strings included in the raw corpus , a position to divide the word is estimated, and a label indicating the position to divide the word is assigned to each between the characters,
The learning feature extraction unit is a set of character strings in an original field different from the target field, and a label indicating a position where words are divided and a label indicating a position where words are not divided are assigned in advance between characters. For each of the character strings included in the word division corpus, the feature between each of the characters given a label indicating the position to divide the word, and between each character given a label indicating the position where the word is not divided For each of the character strings included in the raw corpus, and for each of the characters between the characters to which a label indicating a position to divide the word is added,
The model learning unit is extracted by the learning feature extraction unit, and the feature between each character with a label indicating a position to divide a word, and each character between a character with a label indicating a position not to divide a word Based on the feature about, learning a word division model for determining the position of the word division for the character string of the target field,
The feature extraction unit extracts a feature for each of the characters included in the input character string of the target field,
The binary classifying unit is configured to input the characters in the target field based on the features between the characters extracted by the feature extraction unit and the word division model learned by the model learning unit. A word segmentation method for determining a position at which a word is segmented from each character included in a sequence. The program for functioning a computer as each part which comprises the word division | segmentation apparatus of any one of Claims 1-3 .