JP6558863B2 - Model creation device, estimation device, method, and program - Google Patents

Description

  The present invention relates to a model creation device, an estimation device, a method, and a program.

  Conventionally, a method for acquiring a collapsed notation word, which is a distorted notation with respect to a regular notation word, has been proposed. Examples of the technique using the teacher data include techniques described in Non-Patent Document 1 and Non-Patent Document 2. In Non-Patent Document 1, an abbreviation pair is acquired. Further, in Non-Patent Document 2, acquisition of a collapsed notation of a specific word is performed.

  Examples of methods that do not use teacher data include the methods described in Non-Patent Document 3 and Non-Patent Document 4. In Non-Patent Document 3, acquisition of katakana notation shaking is performed. Further, in Non-Patent Document 4, acquisition of a broken notation (normalization rule) focusing on an unknown word in an output result of an existing analyzer is performed. In the technique described in Non-Patent Document 5, a synonym pair is acquired based on a semantic similarity and a sound similarity.

Nakan Okazaki, Jun-ichi Sakurai, “Automatic Acquisition of Abbreviation Definitions Using Alignment Discrimination Model”. 14th Annual Conference of the Association for Natural Language Processing (NLP2008), pp. 139-142 Yoshinari Fujinuma, Hikaru Yokono, Akiko Aizawa, “Detection and Analysis of Broken Notation Using“ Good Morning ”on Twitter (R).” 27th Annual Conference of the Japanese Society for Artificial Intelligence, 2013.06 Koji Masuyama, Satoshi Sekine, “Automatic construction of a katakana spelling list from a large corpus”, 14th Annual Meeting of the Association for Natural Language Processing (NLP2004) Kazufumi Ikeda, Tadashi Yanagihara, Kazunori Matsumoto, Yasuhiro Takishima, “Automatic Normalization Rule Generation Method for Analyzing Complex Expressions with High Accuracy”, IPSJ Journal, vol3. No.3 pp.68-77 , 2010 Saito, Sadamitsu, Asano, Matsuo, “Acquisition of swing words from text on the web”, Proc. Of the 22nd Annual Conference of the Language Processing Society of Japan

  In the technique described in Non-Patent Document 5, a plurality of word division candidates are acquired from a document set in which a plurality of candidates for regular notation words and collapsed notation words are listed, and regular notation words appearing in the obtained word division candidates Filtering is performed by calculating the semantic similarity and the sound similarity between the (notation existing in the dictionary) and the broken notation word (notation not existing in the dictionary) appearing in the division candidates. In this case, filtering is performed so that the following (1) and (2) are satisfied.

(1) The semantic similarity between the regular notation word and the collapsed notation word is not less than a predetermined threshold. (2) The sound similarity between the regular notation word and the notation notation word is not less than a predetermined threshold.

  The semantic similarity is calculated using the cosine similarity of the semantic vector for each word. The sound similarity distance is calculated based on the reading of the word, and specifically, is calculated according to the following (1) to (3).

(1) Kanji notation is estimated by reading (2) Katakana notation is converted to hiragana (3) Sound similarity is calculated based on cost setting. For example, the sound similarity is calculated according to the cost setting examples shown in the following (3-1) to (3-4).

(3-1) The replacement cost for the same character is assumed to be zero.
(3-2) Deletion of vowel, prompting sound (tsu), repellent sound (n), and long sound is assumed to have no cost. However, deletion at the beginning of a word is cost 1.
(3-3) Replacement of accompanying / same character, replacement between vowel and prompt sound, replacement between vowel and long sound, and replacement between vowel and vowel shall be zero.
(3-4) Conversions other than the above (3-1) to (3-3) are cost 1.
However, “vowels” include lowercase letters (Ah, I, H, E, H)
“Accompanying / same column” refers to the same column and column of the Japanese hiragana 50-syllabary table. The muddy sound / semi-turbid sound is considered to be the same position as the character before the muddy sound / semi-turbid sound is converted (for example, “bu”, “fu”) and the character string of the same line (hahifuhekutsutsumuyuru) cost 0).

  However, in the above-described technique, when using a filter method using threshold values of semantic similarity and sound similarity, it is difficult to identify pairs of regular written words and broken written words using various features, There is a problem that it is difficult to identify abbreviations and phrases with high accuracy. Examples 1 and 2 show that it is difficult to identify abbreviations and phrases.

Example 1: Seven-Eleven (correct pair)
Sound similarity: 0.57, semantic similarity: 0.76
Example 2: I'm happy-I'm happy (incorrect pair)
Sound similarity: 0.8, semantic similarity: 0.80

  Regarding Examples 1 and 2 above, Example 2 is an example in which both sound similarity and semantic similarity are higher, but Example 1 is to be identified as a correct pair and Example 2 is identified as an incorrect pair.

  In this case, unless rules other than the threshold values of the sound similarity and the semantic similarity are introduced, the two cannot be correctly identified. However, it is expensive to set rules while looking at individual cases.

  The present invention has been made to solve the above-described problem, and is a model capable of acquiring a model capable of accurately estimating whether or not a combination of a regular notation word and a collapsed notation word. An object is to provide a creation device, a method, and a program.

  It is another object of the present invention to provide an estimation device, method, and program capable of accurately estimating whether or not a combination of a regular notation word and a collapsed notation word.

  In order to achieve the above object, the model creation device according to the first aspect of the present invention provides a word division candidate from the text for each of a plurality of texts including a collapsed notation word that is a fluctuation notation with respect to a regular notation word. A plurality of generated word dividing units; a semantic vector calculating unit that calculates a semantic vector of each word based on the plurality of division candidates generated by the word dividing unit; and a dictionary that stores a plurality of regular notation words A negative example generation unit that generates a negative example data representing a pair of the regular notation word and a word different from the collapsed notation word corresponding to the regular notation word, and corresponds to the normal notation word and the normal notation word For each of predetermined positive example data representing a pair with the collapsed notation word, the similarity of the meaning of the pair calculated based on each of the pair semantic vectors represented by the positive example data is On the basis of the features including the semantic similarity of the pair and the features including the semantic similarity of the pair calculated for each of the negative example data based on the semantic vectors of the pair represented by the negative example data. And a model creation unit that creates a model for estimating whether or not a combination of a regular notation word and a collapsed notation word corresponding to the regular notation word.

  A model creation method according to a second aspect of the present invention is a model creation method in a model creation device including a word division unit, a semantic vector calculation unit, a negative example generation unit, and a model creation unit, wherein the word division unit includes a normal notation A step of generating a plurality of word division candidates from the text for each of a plurality of texts including a collapsed notation word that is a distorted notation for the word, and the semantic vector calculation unit is generated by the word division unit A step of calculating a semantic vector of each word based on the plurality of division candidates; and the negative example generation unit, based on a dictionary in which a plurality of regular notation words are stored, the regular notation word and the normal notation word Generating negative example data representing a pair of words different from the collapsed notation word corresponding to the normal notation word and the collapsed word corresponding to the regular notation word. Semantic similarity indicating the similarity of the meaning of the pair, calculated based on each of the semantic vectors of the pair represented by the positive example data, for each of the predetermined positive example data representing the pair with the word For each of the negative example data, and a feature including the semantic similarity of the pair calculated based on each of the pair semantic vectors represented by the negative example data. And creating a model for estimating whether or not the combination is a combination of the collapsed notation word corresponding to the regular notation word.

  The estimation apparatus according to the third invention represents a pair of a normal notation word and a word different from the corrupted notation word corresponding to the normal notation word, generated based on a dictionary storing a plurality of normal notation words. For each negative example data, a feature including a semantic similarity representing the similarity of the meaning of the pair calculated from each of the semantic vectors of the pair represented by the negative example data, a normal notation word, and the normal notation The semantic similarity of the pair calculated from each of the pair semantic vectors represented by the positive example data for each of the predetermined positive example data representing a pair with the collapsed notation word corresponding to a word. A model storage unit for storing a model for estimating whether or not a combination of a regular notation word and a collapsed notation word is created in advance from the features to be included, and a plurality including a collapsed notation word corresponding to the regular notation word Each of the text A word dividing unit that generates a plurality of word division candidates from the text, a semantic vector calculation unit that calculates a word semantic vector based on the plurality of division candidates generated by the word dividing unit, and the division For each of the candidate pairs, a feature including a semantic similarity representing the similarity of the meaning of the pair, calculated based on each of the semantic vectors of the pair of split candidates, and the feature stored in the model storage unit And an estimation unit that estimates whether or not the combination of the division candidates is a combination of a regular notation word and a collapsed notation word based on a model.

  The estimation method according to the fourth invention corresponds to a normal notation word and a normal notation word generated based on a word division unit, a semantic vector calculation unit, an estimation unit, and a dictionary storing a plurality of normal notation words. For each negative example data representing a pair with a word different from the collapsed notation word, the semantic similarity representing the similarity of the meaning of the pair calculated from each of the pair semantic vectors represented by the negative example data The meaning vector of the pair and the semantic vector of the pair represented by the positive example data for each of the predetermined positive example data representing a pair of a feature including a degree, a regular written word and the collapsed written word corresponding to the regular written word A model storage unit for storing a model for estimating whether or not a combination of a regular notation word and a collapse notation word is created in advance from the features including the semantic similarity of the pair calculated from each Including estimation equipment In this estimation method, the word dividing unit generates a plurality of word division candidates from the text for each of a plurality of texts including a corrupted written word corresponding to a regular written word; and the semantic vector calculating unit Calculating a semantic vector of a word based on the plurality of division candidates generated by the word division unit, and the estimation unit means the meaning of the pair of division candidates for each of the pair of division candidates Based on the features including the semantic similarity indicating the similarity of the meaning of the pair, calculated based on each of the vectors, and the model stored in the model storage unit, the combination of the division candidates is represented as a normal notation. Estimating whether it is a combination of a word and a broken notation word.

  An estimation device according to a fifth aspect of the present invention represents a pair of a normal notation word and a word different from the corrupted notation word corresponding to the normal notation word, generated based on a dictionary storing a plurality of normal notation words. For each negative example data, a feature including a semantic similarity representing the similarity of the meaning of the pair calculated from each of the semantic vectors of the pair represented by the negative example data, a normal notation word, and the normal notation The semantic similarity of the pair calculated from each of the pair semantic vectors represented by the positive example data for each of the predetermined positive example data representing a pair with the collapsed notation word corresponding to a word. A model storage unit for storing a model for estimating whether or not a combination of a regular notation word and a collapsed notation word is created in advance from the features to be included, and a plurality including a collapsed notation word corresponding to the regular notation word Each of the text A word dividing unit that generates a plurality of word division candidates from the text, a semantic vector calculation unit that calculates a word semantic vector based on the plurality of division candidates generated by the word dividing unit, and the division For each candidate pair, based on the semantic similarity representing the similarity of the meaning of the pair, calculated based on each of the semantic vectors of the split candidate pair, the split candidate for each of the split candidates Generating a neighboring node list composed of the division candidates similar to the above, performing a random walk on each of the neighboring node lists for each of the division candidates, and obtaining a set of adjacent candidates for the division candidates; and A division candidate and the adjacent candidate included in the set of adjacent candidates acquired by the random walk unit; For a combination, the combination is converted into a normal notation word and a collapsed notation based on the model stored in the model storage unit and a feature including the semantic similarity calculated based on each of the semantic vectors of the combination. And an estimation unit that estimates whether or not a combination with a word is included.

  An estimation method according to a sixth aspect of the present invention includes a word division unit, a semantic vector calculation unit, a random walk unit, an estimation unit, and a normal notation word generated based on a dictionary storing a plurality of normal notation words and the normal notation The similarity degree of the meaning of the pair calculated from each of the semantic vectors of the pair represented by the negative example data for each of the negative example data representing a pair with a word different from the collapsed written word corresponding to the written word The pair represented by the positive example data for each of predetermined positive example data representing a pair of a feature including a semantic similarity representing a normal notation word and the collapsed notation word corresponding to the normal notation word A model for estimating whether or not a combination of a regular notation word and a collapsed notation word is created from the features including the semantic similarity of the pair calculated from each of the meaning vectors of model An estimation method in an estimation device including a memory unit, wherein the word segmentation unit generates a plurality of word segmentation candidates from the text for each of a plurality of texts including a corrupted notation word corresponding to a regular notation word; The semantic vector calculation unit calculates a word semantic vector based on the plurality of division candidates generated by the word division unit; and Based on the semantic similarity representing the similarity of the meaning of the pair, calculated based on each semantic vector of the pair of division candidates, for each of the division candidates, from the division candidates similar to the division candidate A neighboring node list is generated, a random walk is performed on each of the neighboring candidates for each neighboring node list, and A step of acquiring a set of adjacent candidates for the candidate; and the model storage unit for a combination of the division candidate and the adjacent candidate included in the set of adjacent candidates acquired by the random walk unit Whether the combination is a combination of a regular notation word and a collapse notation word based on the model stored in the above and a feature including the meaning similarity calculated based on each of the semantic vectors of the combination Estimating whether or not.

  The negative example data includes a pair of a normal notation word stored in the dictionary and a normal notation word different from the normal notation word, a normal notation word having a utilization form stored in the dictionary, and the normal notation word. A pair of regular notation words different from the notation form of the notation word, a regular notation word randomly selected from the dictionary, and a suffix predetermined for the part of speech of the regular notation word are the normal notation. It can be generated as at least one of a pair of words combined with a word.

  A program according to a seventh aspect is a program for causing each model to function as each part of the model creation device or the estimation device.

  According to the model creation device, method, and program of the present invention, a negative example data is generated based on a dictionary in which a plurality of regular notation words are stored, and a feature including a semantic similarity of a pair represented by the positive example data; Creating a model for estimating whether a combination of a regular notation word and a collapsed notation word corresponding to the regular notation word based on the feature including the semantic similarity of the pair represented by the negative example data Thus, an effect is obtained that a model for estimating whether or not a combination of a regular notation word and a collapse notation word can be acquired.

  According to the estimation apparatus, method, and program of the present invention, for each pair of word division candidates, a feature including a pair semantic similarity calculated based on each of the semantic vectors of the division candidate pairs; Based on the model stored in the model storage unit, by estimating whether the combination of division candidates is a combination of a regular notation word and a collapsed notation word, There is an effect that it can be accurately estimated whether or not there is.

It is a figure which shows an example of a collapse notation word. It is a figure for demonstrating the collapse notation word described with various forms. It is a block diagram which shows the structure of the model production apparatus which concerns on embodiment of this invention. It is a figure which shows an example of negative example data. It is explanatory drawing for demonstrating the flow of a process of a model production apparatus. It is a block diagram which shows the structure of the estimation apparatus which concerns on embodiment of this invention. It is a figure which shows the example of the neighborhood node list | wrist of node "Usesse". It is a figure which shows the example of the result obtained by random walk. It is a figure which shows an example of the set of adjacent candidates obtained from the similarity matrix obtained as a result of the random walk. It is a figure which shows an example of the result obtained from the candidate set of a regular notation word. It is explanatory drawing for demonstrating the flow of a process of an estimation apparatus. It is explanatory drawing for demonstrating the flow of a process of an estimation apparatus. It is a flowchart which shows the model creation process routine in the model creation apparatus which concerns on embodiment of this invention. It is a flowchart which shows the estimation process routine in the estimation apparatus which concerns on embodiment of this invention.

<Outline of Embodiment of the Present Invention>
This embodiment is a technique for automatically acquiring a collapsed notation word on a web that does not exist in an existing dictionary. FIG. 1 shows an example of automatic acquisition of a collapsed notation word corresponding to the word “great-adjective” existing in the dictionary. As shown in FIG. 1, “great”, “great”, “great”, “great”, “great” and the like are acquired as collapsed notation words for “great”.

  In the present embodiment, terms are defined as shown below.

Regular notation word: Word break notation existing in the dictionary: Word synonym pair that does not exist in the dictionary and appears as a variant of the normal notation word: Pair of words that have the same meaning but different notation Positive data: Regular notation Pair data of correct answer between word and collapsed notation data Negative example data: Pair data of incorrect answer between regular notation and collapsed notation word

  There are various collapse notation words. For example, as shown in FIG. 2, there are collapse notation words in colloquial style, small letters, katakana, and hiragana.

  In this embodiment, when identifying a combination of a regular notation word and a collapsed notation word, a model using correct data is used to accurately identify an example that cannot be well identified without considering a plurality of features. Is introduced.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. A model creation device for creating a model for estimating whether the present invention is a combination of a regular notation word and a collapsed notation word corresponding to the regular notation word, and a corresponding normality using the created model The case where it applies to the estimation apparatus which estimates the combination of a written word and a collapsed written word is demonstrated to an example.

<Configuration of Model Creation Device According to Embodiment of the Present Invention>

  The configuration of the model creation device according to the embodiment of the present invention will be described. As shown in FIG. 3, a model creation apparatus 100 according to an embodiment of the present invention includes a CPU, a RAM, and a ROM that stores a program for executing a model creation processing routine described later and various data. Can be configured with a computer. Functionally, the model creating apparatus 100 includes a first input unit 10, a second input unit 12, and a calculation unit 20, as shown in FIG.

  The first input unit 10 receives a plurality of predetermined positive example data representing a pair of a regular written word and a broken written word corresponding to the regular written word. The second input unit 12 accepts a text group that is a plurality of texts including a collapsed notation word that is a fluctuating notation with respect to a regular notation word.

  The calculation unit 20 includes a dictionary 21, a negative example generation unit 22, a word division unit 24, a semantic vector calculation unit 26, a similarity calculation unit 28, a model creation unit 30, and a model storage unit 32. It is configured.

  The dictionary 21 stores a plurality of regular notation words. As the dictionary 21, a general morphological analysis dictionary is assumed. The dictionary 21 stores information on regular notation words, parts of speech, readings, and the like. Here, all the words existing in the dictionary 21 are defined as “regular notation words” (the collapse notation words are words that do not exist in the dictionary 21). For example, an example of information stored in a Jtag dictionary (a dictionary maintained by the applicant) is shown below.

Kappa-Sushi, Noun: Proper: Organization, 100, Kappa'-Zushi ,,,,, 374: 428,88

  The negative example generation unit 22 generates negative example data representing a pair of a normal notation word and a word different from the corrupted notation word corresponding to the normal notation word based on a plurality of normal notation words stored in the dictionary 21. To do.

  When a model for estimating whether or not a combination of a regular notation word and a collapse notation word corresponding to the regular notation word is created, learning cannot be performed without negative example data. Although there is a method of using other positive example data as negative example data, there is a tendency that the number of positive example data created by a person is small, and variations of negative example data cannot be covered.

  Therefore, in the present embodiment, variations that do not appear in the positive example data are generated by using dictionary words that are words stored in the dictionary, usage forms, high-frequency notation, etc. Generation is possible, and the accuracy of estimation by the model can be improved. Only patterns that are obvious to be negative examples are generated as negative example data.

  For example, the negative example generation unit 22 generates negative example data based on the following plurality of conditions (1) to (3).

(1) For a specific regular expression word stored in the dictionary 21, a regular expression word different from the specific regular expression word is randomly selected, and the specific regular expression word and the regular expression word randomly selected Are generated as negative example data. For example, “Yabai-Good morning” is generated as negative example data.

(2) For a specific regular expression word having a utilization form stored in the dictionary 21, a regular expression word having a utilization form different from the utilization form of the specific regular expression word is generated and utilized differently from the specific regular expression word. A pair with a regular expression word in the form is generated as negative example data. For example, “happy-happy” or the like is generated as negative example data.

(3) For a specific regular expression word randomly selected from the dictionary 21, a word in which a suffix predetermined for the part of speech of the specific regular expression word is combined with the specific regular expression word is generated. A pair of a specific regular expression word and a word in which a suffix is combined with the specific regular expression word is generated as negative example data. For example, “I-I am” is generated as negative example data. As a predetermined suffix, a high-frequency hiragana suffix can be used. As a method for obtaining a high-frequency suffix, it is extracted in advance for each part of speech with reference to the following non-patent literature reference 6 and the like.

Non-Patent Document 6: Murawaki et al., “Automatic Acquisition of Unknown Words Using Morphological Constraints”, Proc. Of the 14th Annual Conference of the Language Processing Society of Japan

  An example of negative example data generated from positive example data is shown in FIG. As shown in FIG. 4, negative example data as shown on the right side is generated from previously collected positive example data shown on the left side. The negative example generation unit 22 compares the generated negative example data with the positive example data using a plurality of positive example data received by the first input unit 10, and the negative example data matches the positive example data. In this case, a process for excluding the negative example data may be performed.

  The word dividing unit 24 generates a plurality of word division candidates from the text for each text in the text group received by the second input unit 12 and generates a divided corpus.

  The word division unit 24 uses a technique that can output a broken notation word that does not exist in the dictionary 21 as a break candidate. For example, the word division unit 24 uses the word division method using point estimation, the word division method using unsupervised analysis, or the analysis result of Mecab, which is free software of the morphological analyzer, based on a predetermined rule. Existing word segmentation techniques, such as techniques for generating partially combined candidates, can be used. Here, in the method of generating a candidate that partially combines analysis results such as Mecab based on a predetermined rule, it is possible to use predetermined rules such as combining unknown word continuations and noun continuations. .

  The semantic vector calculation unit 26 calculates the semantic vector of each word based on the plurality of division candidates generated by the word division unit 24.

  The semantic vector calculation unit 26 can use an existing method such as word2vec described in the following reference as a method for obtaining the semantic vector of each word that is a division candidate.

References: Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean., "Efficient Estimation of Word Representations in Vector Space.", In Proceedings of Workshop at ICLR, 2013.

  Based on the meaning vector of each word calculated by the semantic vector calculation unit 26, the similarity calculation unit 28 calculates a semantic similarity representing the similarity of the meaning of the word pair for each word pair. In addition, the similarity calculation unit 28 calculates a sound similarity representing the similarity of the sound of the word pair for each of the word pairs based on each of the sound of the words. As a method for calculating the sound similarity, for example, the method of Non-Patent Document 5 can be used.

  The model creation unit 30 calculates the semantic similarity and the sound similarity for each pair of the regular written word and the broken written word represented by the positive example data received by the first input unit 10 calculated by the similarity calculating unit 28. , Meaning similarity and sound similarity for each of a pair of a normal notation word represented by the negative example data generated by the negative example generation unit 22 and a word different from the corrupted notation word corresponding to the normal notation word Based on the feature including the degree, a model for estimating whether or not the combination of the normal notation word and the collapsed notation word corresponding to the normal notation word is created.

  In this embodiment, a case where semantic similarity and sound similarity are used as features will be described as an example. However, other information may be included in the features. For example, the presence / absence of conversion of the first character, the subsequent difference character string, and the part of speech of the regular notation word candidate may be included in the feature. Note that the feature of the subsequent difference character string can be created using the method described in Non-Patent Document 6, for example.

  The model created by the model creation unit 30 is stored in the model storage unit 32.

  FIG. 5 is a diagram for explaining the flow of processing of the model creation apparatus 100. As shown in FIG. 5, when positive example data (for example, W1-W5, W3-W7) is input by the first input unit 10, the negative example generation unit 22 uses the information stored in the dictionary 21. , Negative example data (for example, W1-W7) is generated.

Next, when the text group A is input by the second input unit 12, the word dividing unit 24, as a process 1, extracts word division candidates B (W ₁ ,..., W _n ) from the text in the text group. Generate. Further, the word dividing unit 24 generates a divided corpus from the text group A as process 1. Then, as the processing 2, the semantic vector calculation unit 26 processes each word (W ₁ ,..., W _n ) that appears as a word division candidate with respect to the divided corpus that is a text group to which a word break is given. The semantic vector of the word is calculated.

Next, the similarity calculation unit 28 calculates the sound similarity for each word combination based on each word sound. Note that the sound similarity for each combination is represented by a matrix C, and the sound similarity is stored in each element of the matrix C for a combination of words (W ₁ ,..., W _n ).

Also, the similarity calculation unit 28 calculates the semantic similarity for each combination of division candidates based on each word semantic vector. The semantic similarity for each combination is represented by a matrix D, and the semantic similarity is stored in each element of the matrix D for the combination of words (W ₁ ,..., W _n ).

  Instead of the sound similarity matrix C and the semantic similarity matrix D, a similarity matrix R subjected to random walk may be used. The similarity matrix R will be described later.

  Then, the model creation unit 30 calculates the regular notation word and the normal notation word based on the features including the elements of the semantic similarity matrix C and the sound similarity matrix D, the negative example data, and the positive example data. A model for estimating whether or not a combination with the collapsed notation word corresponding to is created. Then, the created model is stored in the model storage unit 32. As shown in FIG. 5, the feature may include a subsequent difference character string, a part of speech of a normal notation word candidate, and the like. Further, the feature of the subsequent difference character string can be created by using the method described in Non-Patent Document 6, for example.

<Configuration of estimation apparatus according to embodiment of the present invention>

  Next, the configuration of the estimation apparatus according to the embodiment of the present invention will be described. As shown in FIG. 6, the estimation apparatus 200 according to the embodiment of the present invention is a computer including a CPU, a RAM, and a ROM that stores a program for executing an estimation processing routine described later and various data. Can be configured. Functionally, the estimation device 200 includes an input unit 40, a calculation unit 50, and an output unit 70 as shown in FIG.

  Similar to the second input unit 12, the input unit 40 accepts a text group that is a plurality of texts including a collapsed notation word that is a fluctuating notation with respect to a regular notation word.

  The calculation unit 50 includes a word division unit 52, a semantic vector calculation unit 54, a similarity calculation unit 56, a random walk unit 58, a model storage unit 60, and an estimation unit 62.

  Similar to the word division unit 24, the word division unit 52 generates a plurality of word division candidates from the text for each text in the text group received by the input unit 40, and generates a divided corpus.

  Similar to the semantic vector calculation unit 26, the semantic vector calculation unit 54 calculates the semantic vector of each word based on the plurality of division candidates generated by the word division unit 52.

  Similar to the similarity calculation unit 28, the similarity calculation unit 56 uses the semantic vector of each word calculated by the semantic vector calculation unit 54 to make the semantic similarity of the division candidate pair to each of the division candidate pairs. Calculate the degree. Similarly to the similarity calculation unit 28, the similarity calculation unit 56 calculates the sound similarity of the division candidate pair for each of the division candidate pairs based on each of the division candidate sounds.

  The random walk unit 58 includes, for each of the division candidates, a division candidate similar to the division candidate based on the semantic similarity and the sound similarity of each of the division candidate pairs calculated by the similarity calculation unit 56. Generate a neighborhood node list. Then, the random walk unit 58 performs a random walk on each neighboring node list for each of the division candidates, and acquires a set of adjacent candidates for the division candidate.

  Specifically, first, the random walk unit 58 uses the semantic similarity and sound similarity of each pair of division candidates calculated by the similarity calculation unit 56 to mean semantic similarity for each of the division candidate pairs. And a matrix having each element as a sound similarity with respect to each pair of candidates for division.

  Then, based on each element of the semantic similarity matrix and each element of the sound similarity matrix, the random walk unit 58 determines that the semantic similarity and the sound similarity satisfy the reference value for each of the division candidates. Are divided, and a neighborhood node list corresponding to each division candidate is generated. Each node in the neighborhood node list corresponds to a pair of division candidates whose semantic similarity and sound similarity satisfy the reference value. As will be described later, even when a combination of a regular notation word and a collapsed notation word is estimated using a model, it is costly to discriminate between positive and negative among all the division candidate pairs. Therefore, first, a restriction is applied with a reference value that is a loose threshold between the sound similarity and the semantic similarity.

  FIG. 7 shows an example of a neighborhood node list of the node “Usesse”. The numerical value on the right side of FIG. 7 represents the value of semantic similarity. In the example shown in FIG. 7, it is a neighborhood node list obtained with 0.4 as a reference value.

  Next, the random walk unit 58 performs a random walk on the neighborhood node list generated for each of the division candidates for a specified number of times, obtains a similarity matrix representing the degree of similarity of each pair of division candidates, and results of the random walk Based on the obtained similarity matrix, only candidates that satisfy the conditions of the regular notation word are set as a set of adjacent candidates for each of the division candidates.

  In addition, as a random walk, the method of the following nonpatent literature 7 can be used, for example. In addition, the matrix of initial values of the random walk can use a value obtained by normalizing each element of the semantic similarity.

Non-Patent Document 7: Martin Szummer, Tommi Jaakkola, “Partially labeled classification with Markov random walks”, [searched on June 24, 2016], Internet <http://people.ee.duke.edu/~lcarin/SzummerJaakkola .pdf>

  FIG. 8 shows an example of the result obtained by the random walk. In the example shown in FIG. 8, a case where the correct answer pair is “Yabee-Yabai” is shown. As shown in FIG. 8, in the left-side neighboring node list generated based on the reference value, when only the primary connection is considered by the reference value, there is no correct regular expression word “Yabai”. However, as shown on the right side of FIG. 8, by performing 10 random walks, the correct regular expression word “Yabai” appears.

  Also, based on the value of the similarity matrix obtained as a result of the random walk, for example, as shown in FIG. 9, only candidates that satisfy the regular notation word condition are set as a set of adjacent candidates. The condition of the regular notation word can be set with reference to the method described in Non-Patent Document 5, for example.

  The model created by the model creation device 100 is stored in the model storage unit 60.

  For each of a plurality of division candidates, the estimation unit 62 uses a model stored in the model storage unit 60 for a combination of the division candidate and an adjacent candidate included in the set of adjacent candidates acquired by the random walk unit 58. A feature including the semantic similarity calculated by the similarity calculation unit 56 based on each of the semantic vectors of the combination, and the sound similarity calculated by the similarity calculation unit 56 based on each of the sounds of the combination; Based on the above, it is estimated whether or not the combination is a combination of a regular notation word and a collapse notation word.

  Specifically, the estimation unit 62 includes a set of adjacent candidates acquired by random walk, a feature set including semantic similarity and sound similarity for each pair of division candidates, and a model stored in the model storage unit 60. Based on the above, it is estimated whether or not the combination of division candidates is a combination of a regular notation word and a collapsed notation word. The estimation unit 62 outputs a pair of division candidates estimated as a combination of a regular notation word and a collapsed notation word as a synonym pair.

  FIG. 10 shows an example of combinations of division candidates estimated as to whether the combination is a regular written word and a broken written word. In the example illustrated in FIG. 10, “noisy” is determined to be correct with respect to “Ususeshi”.

  The output unit 70 outputs, as a result, a combination of division candidates estimated by the estimation unit 62 as a combination of a regular notation word and a collapsed notation word.

11 and 12 are diagrams for explaining the processing flow of the estimation apparatus 200. FIG. As illustrated in FIG. 11, when the text group A is input by the input unit 40, the word dividing unit 52 performs processing 1 as word division candidates B (W ₁ ,..., W _n from the text group text. ) Is generated. In addition, as processing 1, the word dividing unit 52 generates a divided corpus from the text group A. Then, the semantic vector calculation unit 54 processes each word (W ₁ ,..., W _n ) that appears as word division candidates with respect to the divided corpus that is a group of texts to which word breaks are assigned. For, calculate the semantic vector of the word.

Next, the similarity calculation unit 56 calculates the sound similarity for each of the division candidate pairs based on each of the division candidate sounds. Note that the sound similarity for each of the pairs is expressed by a matrix C, and the sound similarity is stored in each element of the matrix C for the pair of division candidates (W ₁ ,..., W _n ).

Further, the similarity calculation unit 56 calculates the semantic similarity for each of the division candidate pairs based on each of the division candidate semantic vectors. The semantic similarity for each of the pairs is represented by a matrix D, and the semantic similarity is stored in each element of the matrix D for the pair of division candidates (W ₁ ,..., W _n ).

  Next, the random walk unit 58 extracts, for each element of the semantic similarity matrix D and each element of the sound similarity matrix C, each element whose semantic similarity and sound similarity satisfy the reference value, Output as a neighborhood node list E of each division candidate.

  Then, as shown in FIG. 12, the random walk unit 58 performs a random walk for each node of the generated neighborhood node list E a specified number of times. Based on the similarity matrix R obtained as a result of the random walk, only candidates that satisfy the conditions of the regular notation word are output as a set of adjacent candidates for each division candidate. As a matrix used as an initial value in a random walk, for example, the similarity graph described in “3.3.2 Construction of similarity graph” of Non-Patent Document 5 can be used.

  The estimation unit 62 then combines the combinations of division candidates based on the model stored in the model storage unit 60 and the features including the semantic similarity matrix C and the sound similarity matrix D for each combination of division candidates. Is a combination of a regular notation word and a collapse notation word. Note that, as shown in FIG. 12, the feature may include a subsequent difference character string, a part of speech of a regular notation word candidate, and the like.

  Instead of the semantic similarity feature matrix C, a similarity matrix R generated by random walk may be used. When using the similarity matrix R, first, in the estimation process by the estimation apparatus 200, the target division candidates are increased by random walk to generate the similarity matrix R. Next, a model is created using the similarity matrix R in the model creation process by the model creation device 100. Then, in the estimation process by the estimation apparatus 200, a synonym pair is estimated using the created model.

<Operation of Model Creation Device According to Embodiment of the Present Invention>

  Next, the operation of the model creation device 100 according to the embodiment of the present invention will be described. Hereinafter, a case where a similarity matrix generated by random walk is used will be described as an example. When the first input unit 10 accepts a plurality of positive example data and the second input unit 12 accepts a text group, the model creation device 100 executes a model creation processing routine shown in FIG.

  First, in step S <b> 100, the word dividing unit 24 acquires the text group received by the second input unit 12.

  Next, in step S102, the negative example generation unit 22 creates a pair of a normal notation word and a word different from the corrupted notation word corresponding to the normal notation word based on a plurality of normal notation words stored in the dictionary 21. To generate negative example data.

  In step S104, the word division unit 24 generates a plurality of word division candidates from the text for each text of the text group acquired in step S100.

  In step S106, the semantic vector calculation unit 26 calculates the semantic vector of each word based on the plurality of division candidates generated in step S104.

  In step S108, the similarity calculation unit 28 calculates a semantic similarity representing the similarity of the meaning of the word pair for each word pair based on the semantic vector of each word calculated in step S106. To do. In addition, the similarity calculation unit 28 calculates a sound similarity representing the similarity of the sound of the word pair for each of the word pairs based on each of the sound of the words.

  In step S109, a similarity matrix generated as a result of the random walk is acquired. Then, based on the similarity matrix, for each division candidate, only candidates that satisfy the condition of the regular notation word are set as a set of adjacent candidates.

  In step S110, the model creation unit 30 calculates the adjacent candidate set acquired in step S109 and the pair of the normal notation word and the collapsed notation word represented by the positive example data calculated in step S108. Meaning for each of a pair of a feature including a semantic similarity and a sound similarity, and a normal notation word represented by the negative example data generated in step S102 and a word different from the corrupted notation word corresponding to the normal notation word Based on the features including the similarity and the sound similarity, a model for estimating whether or not a combination of a regular written word and a broken written word corresponding to the regular written word is created. Then, the model creation unit 30 stores the created model in the model storage unit 32 and ends the process.

<Operation of Estimation Device according to Embodiment of the Present Invention>

  Next, the operation of the estimation apparatus 200 according to the embodiment of the present invention will be described. When the model created by the model creation device 100 is input to the estimation device 200, it is stored in the model storage unit 60. And if the text group is received in the input part 40 of the estimation apparatus 200, the estimation apparatus 200 will perform the estimation process routine shown in FIG.

  In step S <b> 200, the word dividing unit 52 acquires the text group accepted by the input unit 40.

  In step S202, the word dividing unit 52 generates a plurality of word division candidates from the text for each text in the text group acquired in step S200.

  In step S204, the semantic vector calculator 54 calculates the semantic vector of each word based on the plurality of division candidates generated in step S202.

  In step S <b> 205, the similarity calculation unit 56 represents the semantic similarity indicating the similarity of the meaning of the pair of division candidates for each of the pair of division candidates based on the semantic vector of each word calculated in step S <b> 204. Is calculated. Further, the similarity calculation unit 56 calculates a sound similarity representing the similarity of the sound of the pair of division candidates for each of the pair of division candidates based on each of the sounds of the division candidates.

  In step S206, the random walk unit 58 determines the semantic similarity and sound for each of the division candidates based on each element of the semantic similarity matrix and each element of the sound similarity matrix obtained from step S205. Other division candidates whose degree of similarity satisfies the reference value are extracted, and a neighborhood node list corresponding to each division candidate is generated.

  In step S208, the random walk unit 58 performs a random walk on the neighborhood node list generated for each of the division candidates in step S206, and generates a similarity matrix. Then, based on the similarity matrix obtained as a result of the random walk, the random walk unit 58 sets only candidates that satisfy the condition of the regular notation word as a set of adjacent candidates for each division candidate.

  In step S <b> 210, the estimation unit 62 includes a set of adjacent candidates acquired in step S <b> 208, a feature set including semantic similarity and sound similarity for each pair of division candidates, and a model stored in the model storage unit 60. Based on the above, it is estimated whether or not the combination of division candidates is a combination of a regular notation word and a collapsed notation word. And the estimation part 62 outputs the pair of the division | segmentation candidate estimated to be a combination of a regular notation word and a collapsed notation word as a synonym pair.

  In step S212, the output unit 70 outputs the synonym pair output in step S210 and ends the process.

  As described above, according to the model creation device according to the embodiment of the present invention, negative example data is generated based on a dictionary in which a plurality of regular notation words are stored, and the semantic similarity of pairs represented by positive example data In order to estimate whether or not a combination of a regular notation word and a collapsed notation word corresponding to the regular notation word based on a feature including the degree and a feature including the semantic similarity of the pair represented by the negative example data By creating the model, it is possible to acquire a model for estimating whether or not a combination of a regular notation word and a collapse notation word.

  Further, according to the estimation apparatus according to the embodiment of the present invention, based on the semantic similarity of a pair of word division candidates, a neighborhood node list including division candidates similar to the division candidate is generated, and each neighborhood node list A random walk is performed, a set of adjacent candidates for the division candidate is acquired, a model created in advance for a combination of the division candidate and the adjacent candidate included in the acquired set of adjacent candidates, and a semantic vector of the combination Based on the features including semantic similarity calculated based on each of the above, by estimating whether the combination is a combination of a regular notation word and a collapsed notation word, a regular notation word and a collapsed notation word It is possible to accurately estimate whether or not the combination is.

  Moreover, candidates can be efficiently expanded by using a random walk even for a pair in which a regular notation word does not appear in a primary connection between word division candidates. As a result, when a regular notation word is included in a series of multiple orders, the regular notation word can be included in the candidate set of regular notation words.

  In the present embodiment, regular notation words and collapsed notation words are estimated using a model created based on positive example data and negative example data. In this case, a general model such as an SVM or a maximum entropy model can be used as a model, and the accuracy can be improved by generating obvious negative example data using positive example data.

  In addition, by constructing a model based on positive example data and negative example data, it is possible to flexibly incorporate various features, and it is possible to identify a regular notation word and a collapsed notation word with high accuracy. Further, when building a model, by generating negative example data by a plurality of methods, it is possible to improve identification performance even for candidates that are difficult to identify by sound or semantic similarity.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

  For example, in the above embodiment, the case where the feature includes the semantic similarity and the sound similarity is described as an example, but the present invention is not limited to this. For example, only the semantic similarity may be included in the feature, and as described above, the subsequent difference character string, the part of speech of the regular notation word candidate, and the like may be included in the feature.

  In the above embodiment, the case where the model creation device and the estimation device are configured as separate devices has been described as an example. However, the model creation device and the estimation device may be configured as one device.

  The model creation device 100 and the estimation device 200 described above have a computer system inside. However, if the “computer system” uses a WWW system, a homepage providing environment (or display environment) is also available. Shall be included.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10 1st input part 12 2nd input part 20,50 Operation part 21 Dictionary 22 Negative example production | generation part 24,52 Word division part 26,54 Semantic vector calculation part 28,56 Similarity calculation part 30 Model creation part 40 Input part 32 , 60 Model storage unit 58 Random walk unit 62 Estimation unit 70 Output unit 100 Model creation device 200 Estimation device

Claims (8)

A word dividing unit that generates a plurality of word division candidates from the text for each of a plurality of texts including a collapsed notation word that is a fluctuating notation with respect to a regular notation word;
A semantic vector calculation unit that calculates a semantic vector of each word based on the plurality of division candidates generated by the word division unit;
A negative example generation unit that generates negative example data representing a pair of a regular notation word and a word different from the collapsed notation word corresponding to the regular notation word based on a dictionary storing a plurality of regular notation words; ,
For each predetermined positive example data representing a pair of a regular written word and the collapsed written word corresponding to the regular written word, it is calculated based on each semantic vector of the pair represented by the positive data. The meaning of the pair calculated based on the feature including the semantic similarity indicating the similarity of the meaning of the pair and each of the negative example data based on each of the pair semantic vectors represented by the negative example data. A model creation unit that creates a model for estimating whether or not a combination of a regular notation word and a collapsed notation word corresponding to the regular notation word based on a feature including similarity;
Model creation device including The negative data for each of the negative example data representing a pair of a normal notation word and a word different from the corrupted notation word corresponding to the normal notation word, generated based on a dictionary storing a plurality of normal notation words. Calculated from each of the semantic vectors of the pair represented by example data, a feature including a semantic similarity representing the similarity of the meaning of the pair, and a regular written word and the collapsed written word corresponding to the regular written word For each of predetermined positive example data representing a pair, a normal created from the features including the semantic similarity of the pair calculated from each of the pair semantic vectors represented by the positive example data, A model storage unit that stores a model for estimating whether or not a combination of a written word and a collapsed written word;
A word dividing unit that generates a plurality of word division candidates from the text for each of the plurality of texts including the collapsed notation word corresponding to the regular notation word;
A semantic vector calculation unit that calculates a semantic vector of a word based on the plurality of division candidates generated by the word division unit;
For each of the pair of division candidates, a feature including a semantic similarity representing the similarity of the meaning of the pair calculated based on each semantic vector of the pair of division candidates is stored in the model storage unit. Based on the model, an estimation unit that estimates whether or not the combination of the division candidates is a combination of a regular notation word and a collapsed notation word;
Including the estimation device. The negative data for each of the negative example data representing a pair of a normal notation word and a word different from the corrupted notation word corresponding to the normal notation word, generated based on a dictionary storing a plurality of normal notation words. Calculated from each of the semantic vectors of the pair represented by example data, a feature including a semantic similarity representing the similarity of the meaning of the pair, and a regular written word and the collapsed written word corresponding to the regular written word For each of predetermined positive example data representing a pair, a normal created from the features including the semantic similarity of the pair calculated from each of the pair semantic vectors represented by the positive example data, A model storage unit that stores a model for estimating whether or not a combination of a written word and a collapsed written word;
A word dividing unit that generates a plurality of word division candidates from the text for each of the plurality of texts including the collapsed notation word corresponding to the regular notation word;
A semantic vector calculation unit that calculates a semantic vector of a word based on the plurality of division candidates generated by the word division unit;
For each of the division candidates, for each of the division candidates, based on a semantic similarity representing the similarity of the meaning of the pair, calculated based on each of the semantic vectors of the division candidate pairs. A random walk unit that generates a neighborhood node list composed of the division candidates similar to the division candidate, performs a random walk on each neighborhood node list for each of the division candidates, and acquires a set of adjacent candidates for the division candidates; ,
Based on each of the model stored in the model storage unit and the semantic vector of the combination for the combination of the division candidate and the adjacent candidate included in the set of adjacent candidates acquired by the random walk unit An estimation unit that estimates whether or not the combination is a combination of a regular notation word and a collapsed notation word based on the feature including the semantic similarity,
Including the estimation device. The negative example data is
A pair of a regular expression word stored in the dictionary and a regular expression word different from the regular expression word,
A pair of a regular expression word having a utilization form stored in the dictionary, a regular expression word having a utilization form different from the utilization form of the regular expression word, and a regular expression word randomly selected from the dictionary; and The model creation apparatus according to claim 1, wherein a suffix predetermined for a part of speech of a regular expression word is generated as at least one of a pair with a word combined with the regular expression word. A model creation method in a model creation device including a word division unit, a semantic vector calculation unit, a negative example generation unit, and a model creation unit,
Generating a plurality of word division candidates from the text for each of a plurality of texts including a collapsed notation word that is a fluctuating notation with respect to a regular notation word;
The semantic vector calculation unit calculates a semantic vector of each word based on the plurality of division candidates generated by the word division unit;
The negative example generation unit, based on a dictionary storing a plurality of regular notation words, negative example data representing a pair of the regular notation word and a word different from the collapsed notation word corresponding to the regular notation word Generating step;
Each of the semantic vectors of the pair represented by the positive example data for each of the predetermined positive example data representing a pair of a regular written word and the collapsed written word corresponding to the regular written word. Calculated based on the feature including the semantic similarity representing the semantic similarity of the pair and each of the negative example data is calculated based on each of the pair semantic vectors represented by the negative example data. Creating a model for estimating whether or not a combination of a regular notation word and a collapsed notation word corresponding to the regular notation word based on the feature including the semantic similarity of the pair;
Model creation method including A word segmentation unit, a semantic vector calculation unit, an estimation unit, and a normal notation word generated based on a dictionary in which a plurality of regular notation words are stored, and a word different from the corrupted notation word corresponding to the normal notation word For each of the negative example data representing a pair, a feature including a semantic similarity representing the similarity of the meaning of the pair calculated from each of the semantic vectors of the pair represented by the negative example data, and a regular notation word; Meaning of the pair calculated from each of the semantic vectors of the pair represented by the positive example data for each of the predetermined positive example data representing a pair with the collapsed written word corresponding to the regular written word An estimation method in an estimation apparatus including a model storage unit that stores a model for estimating whether or not a combination of a regular notation word and a collapsed notation word, which is created in advance from features including similarity,
Generating a plurality of word division candidates from the text for each of a plurality of texts including a collapsed notation word corresponding to a regular notation word;
The semantic vector calculation unit calculates a word semantic vector based on the plurality of division candidates generated by the word division unit;
A feature including a semantic similarity indicating the similarity of the meaning of the pair calculated by the estimation unit based on each of the semantic vectors of the split candidate pair for each of the split candidate pairs; Based on the model stored in the storage unit, estimating whether the combination of the division candidates is a combination of a regular notation word and a collapsed notation word; and
An estimation method including: A normal notation word and a corrupted notation word corresponding to the normal notation word generated based on a word division part, a semantic vector calculation part, a random walk part, an estimation part, and a dictionary storing a plurality of normal notation words For each of the negative example data representing pairs with different words, a feature including a semantic similarity representing the similarity of the meaning of the pair calculated from each of the pair semantic vectors represented by the negative example data; For each predetermined positive example data representing a pair of a regular notation word and the collapsed notation word corresponding to the regular notation word, calculated from each of the pair of semantic vectors represented by the positive example data, In an estimation apparatus including a model storage unit that stores a model for estimating whether or not a combination of a regular notation word and a collapse notation word is created in advance from a feature including the semantic similarity of the pair. A constant way,
Generating a plurality of word division candidates from the text for each of a plurality of texts including a collapsed notation word corresponding to a regular notation word;
The semantic vector calculation unit calculates a word semantic vector based on the plurality of division candidates generated by the word division unit;
The random walk unit calculates, for each of the pair of division candidates, based on the semantic similarity representing the similarity of the meaning of the pair calculated based on each of the semantic vectors of the pair of division candidates. For each candidate, generate a neighborhood node list composed of the division candidates similar to the division candidate, perform a random walk on each neighborhood node list for each of the division candidates, and set a set of adjacent candidates for the division candidates. A step to obtain,
For the combination of the division candidate and the adjacent candidate included in the set of adjacent candidates acquired by the random walk unit, the estimation unit stores the model stored in the model storage unit and the meaning of the combination Estimating whether the combination is a combination of a regular notation word and a collapsed notation word based on a feature including the semantic similarity calculated based on each of the vectors;
An estimation method including:   The program for functioning a computer as each part of the model production apparatus of Claim 1, or the estimation apparatus of any one of Claims 2-4.