JP6427466B2 - Synonym pair acquisition apparatus, method and program - Google Patents

Description

  The present invention relates to a synonym pair acquisition apparatus, method, and program, and more particularly, to a synonym pair acquisition apparatus, method, and program for acquiring synonym pairs.

  Heretofore, methods have been proposed for acquiring broken notation words that are expressions that are jumbled with respect to regular expression words. Examples of methods using teacher data include the methods described in Non-Patent Document 1 and Non-Patent Document 2.

  Examples of methods that do not use teacher data include the methods described in Non-Patent Document 3 and Non-Patent Document 4.

Okazaki Intuition, Junichi Asai, "Automatic Acquisition of Abbreviation Definition Using Alignment Discrimination Model". 14th Annual Conference of the Association for Language Processing (NLP2008), pp. 139-142 Fujinuma Yoshinari, Yokono Hikaru, Aizawa Akiko, "Detection and analysis of broken notations with" Good morning "on Twitter (R) as an example." The 27th Annual Conference of Japan Society for Artificial Intelligence, 2013.06 Yuji Masuyama, Kei Sekine, "Automatic Construction of Shaking List of Katakana Expressions from Large-scale Corpus," 14th Annual Meeting of the Association for Language Processing (NLP 2004) Ikeda Kazufumi, Yanagihara Tadashi, Matsumoto Kazunori, Takishima Yasuhiro, "A Method of Automatic Generation of Normalized Rules for Analyzing Shattered Expressions with High Accuracy", Journal of the Information Processing Society of Japan, vol3. No.3 pp.68-77 , 2010 Kudo, T., Japanese Morphological Analyzer, Internet <URL: http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html>

  However, in the case of extracting broken words by the method using teacher data described in Non-Patent Document 1 and Non-Patent Document 2, it is necessary to manually create correct pairs as shown in FIG. 7 from Web data. There is a problem that the generation cost of the pair is high.

  Also, when based on a method that does not use teacher data, the candidates for the corrupted word to be acquired candidates are limited to the limited candidates (Katakana, words that became unknown words in the existing analyzer, etc.) There is a problem that it can not acquire the notation. This is because the existing analyzer often analyzes broken words incorrectly, and it is difficult to obtain various broken words. This is because Japanese does not have a space between words, so that it is difficult to analyze the correct separation position of morphemes in generally existing text. Also, on the Web, there are many broken written words written in hiragana, kanji and hiragana, katakana and hiragana, etc., which makes analysis difficult. For example, "Suge", "Yaba", "Samii", "Samui", "Cold" etc. Further, FIG. 8 shows an example of a result of analyzing a sentence including broken words using Mecab (IPAdic) described in Non-Patent Document 5.

  The present invention has been made to solve the above-mentioned problems, and an object thereof is to provide a synonym pair acquiring apparatus, method, and program capable of efficiently acquiring synonym pairs.

  In order to achieve the above object, a synonym pair acquiring device according to a first aspect of the present invention is, from a document, a plurality of regular expressions or a plurality of regular expressions which are candidates for irregular expressions with respect to the regular expressions. Word semantic vectors are calculated for each of the plurality of word division candidates based on the word division candidate generation unit that generates word division candidates, and the plurality of word division candidates generated by the word division candidate generation unit. Based on the semantic similarity calculated based on the semantic vector and the sound similarity calculated based on word reading for each of the semantic vector calculation unit and the word division candidate which is a regular expression word, The word division candidate is selected from the plurality of word division candidates, and a pair of the word division candidate which is a regular expression word and the selected word division candidate is acquired as a synonym pair. It is configured to include the word pair acquisition unit.

  Further, in the synonym pair acquiring apparatus according to the first invention, the synonym pair acquiring unit is configured to determine, based on the semantic similarity and the sound similarity, for each of the word division candidates which are regular expression words. The word division candidate is selected from the plurality of word division candidates, and a pair of the word division candidate which is a regular expression word and the selected word division candidate is acquired as a synonym pair, and the selected word division candidate is selected. For each of the word division candidates, the word division candidates are selected from the plurality of word division candidates based on the semantic similarity and the sound similarity, and the word division candidates that are regular written words are selected The pair with the word division candidate may be acquired as a synonym pair.

  In the synonym pair acquiring method according to the second aspect of the present invention, the word division candidate generation unit is a regular expression word from the document, or a plurality of words as broken expression words that are candidates for expressions that are distorted with respect to the regular expression word. A step of generating division candidates, and a semantic vector calculation unit calculating a word semantic vector for each of the plurality of word division candidates based on the plurality of word division candidates generated by the word division candidate generation unit And a synonym pair acquisition unit is calculated based on the semantic similarity calculated based on the semantic vector and the sound similarity calculated based on word reading for each of the word division candidates that are regular expressions. The word division candidate is selected from the plurality of word division candidates, and a pair of the word division candidate that is a regular expression word and the selected word division candidate is a synonym pe And executes includes a step of acquiring, as.

  In the synonym pair acquisition method according to the second aspect of the invention, the synonym pair acquisition unit acquires the semantic similarity degree and the sound similarity degree for each of the word division candidates which are regular expression words. And selecting the word division candidate from the plurality of word division candidates, and acquiring a pair of the word division candidate that is a regular expression word and the selected word division candidate as a synonym pair. For each of the selected word division candidates, the word division candidates are selected from the plurality of word division candidates based on the semantic similarity and the sound similarity, and the word division candidates are regular written words And the selected word division candidate may be acquired as a synonym pair.

  A program according to a third invention is a program for functioning as each unit of the synonym pair acquisition device according to the first invention.

  According to the synonym pair acquiring apparatus, method, and program of the present invention, a plurality of word division candidates which are regular written words or broken notation words are generated from a document, and a plurality of word division candidates are generated based on the plurality of word division candidates. For each of the word division candidates, the semantic vector of the word is calculated, and for each word division candidate which is a regular expression word, the sound similarity calculated based on the semantic vector and the sound calculated based on the word reading By selecting a word division candidate from a plurality of word division candidates based on the degree of similarity, and acquiring a pair of a word division candidate that is a regular expression word and the selected word division candidate as a synonym pair The effect is obtained that synonym pairs can be obtained efficiently.

It is a block diagram which shows the structure of the synonym pair acquisition apparatus which concerns on embodiment of this invention. It is a figure which shows an example of a sound similarity. It is a conceptual diagram which shows the example of acquisition of a synonym pair. It is a figure which shows the example which selects a word division candidate by making a regular expression word into a starting point. It is a figure which shows the example which selects a word division candidate further by using the selected word division candidate as a starting point. It is a flowchart which shows the synonym pair acquisition process routine in the synonym pair acquisition apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the combination of a regular expression word and a collapse expression word. It is a figure which shows an example of the result of having analyzed the sentence which contains a collapse written word using Mecab.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Configuration of Synonym Pair Acquisition Device According to Embodiment of the Present Invention>

  Next, the configuration of the synonym pair acquisition apparatus according to the embodiment of the present invention will be described. As shown in FIG. 1, the synonym pair acquisition apparatus 100 according to the embodiment of the present invention is a ROM storing a CPU, a RAM, a program for executing a synonym pair acquisition processing routine described later, and various data. And can be configured with a computer. The synonym pair acquisition apparatus 100 functionally includes an input unit 10, an operation unit 20, and an output unit 50 as shown in FIG.

  The input unit 10 receives a document set consisting of documents including broken notation words.

  The calculation unit 20 includes a dictionary database 28, a word division candidate generation unit 30, a semantic vector calculation unit 32, and a synonym pair acquisition unit 34.

  The dictionary database 28 stores dictionaries (reading, notation, parts of speech) necessary for performing dictionary lookup.

  The word division candidate generation unit 30 selects, from each of the documents in the document set received by the input unit 10, a plurality of word division candidates that are regular written words or broken written words that are candidates for writings that are distorted with respect to the regular written words. Generate

  Specifically, the word division candidate generation unit 30 generates word division candidates by applying each of the following first to third methods, which are existing word division methods, to a document. At this time, a method is used which can also output broken words not present in the dictionary database 28 as segmentation candidates.

  The word division candidate generation unit 30 generates word division candidates by applying a word division method using point estimation as a first method to each of the documents included in the document set. In the word division method using point estimation, a document is divided into a plurality of word division candidates using a character separation model having characters ngram, character type ngram, and the like as features.

  The word division candidate generation unit 30 generates a word division candidate by applying a word division method using unsupervised analysis as a second method to the document set. In the word division method using unsupervised analysis, the appearance frequency etc. are calculated for the sampled word division candidates, and each of the documents is divided into word division candidates so that the objective function is optimized.

  The word division candidate generation unit 30 acquires, as a third method, an analysis result by Mecab or the like for each of the documents included in the document set, and is a word division candidate partially combined based on a predetermined rule. Generate the As a rule, for example, unknown word continuation is connected, noun continuation is connected, etc. The following method may be used as a rule. For example, when a short sentence is cut out from Twitter (R) or the like and used as a word division candidate, cutting out a short sentence may be performed by a plurality of delimiters (for example, line feed, symbolic expression ("!", "W", "♪"), punctuation marks (",", ".", Etc.) may be set, and short sentences may be divided by delimiters. By setting in this manner, for example, in the case of the sentence “Yabe wwwwwwwwwww”, “Yabe” before “w” can be acquired as a word division candidate. In addition, in the case of the sentence "Oohayo ♪ I have lunch ... I just got up ...", "Oohayo" before "♪" can be acquired as a word division candidate. A character string having n or fewer characters acquired as described above may be added to the morpheme dictionary for analysis.

  The semantic vector calculation unit 32 calculates a semantic vector of a word for each of the plurality of word division candidates based on the plurality of word division candidates generated by the word division candidate generation unit 30.

  More specifically, the semantic vector calculation unit 32 appears as a word division candidate in a document set to which word breaks have been added so as to enumerate a plurality of word division candidates generated by the word division candidate generation unit 30. Calculate the semantic vector of each word. Under the present circumstances, the existing method can be used as a method of calculating | requiring the semantic vector of each word. For example, word2vec and the like described in Non-Patent Document 6 can be mentioned as a representative method.

[Non-patent document 6]: Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013.

  The synonym pair acquiring unit 34 determines, for each word division candidate that is a regular expression word, the semantic similarity calculated based on the semantic vector is equal to or higher than a threshold, and the sound calculated based on the word reading. A word division candidate whose similarity is equal to or higher than a threshold is selected from a plurality of word division candidates, and a pair of a word division candidate that is a regular expression word and the selected word division candidate is acquired as a synonym pair. The synonym pair acquiring unit 34 further determines, for each of the selected word division candidates, the semantic similarity calculated based on the semantic vector is equal to or higher than a threshold, and the sound is calculated based on the word reading. A word division candidate whose similarity is equal to or higher than a threshold is selected from a plurality of word division candidates, and a pair of a word division candidate that is a regular expression word and the selected word division candidate is acquired as a synonym pair. FIG. 3 shows a conceptual diagram of processing of the synonym pair acquisition unit 34. As shown in FIG.

  Specifically, the synonym pair acquiring unit 34 first calculates, for each of the word division candidates which are regular expression words, the semantic similarity with each of the other word division candidates. The semantic similarity is calculated using the cosine similarity of the semantic vector for each word obtained in the semantic vector calculator 32.

  Next, the synonym pair acquiring unit 34 calculates, for each word division candidate which is a regular expression word, the sound similarity with another word division candidate. In the present embodiment, the sound similarity distance is calculated as the sound similarity based on the reading of the word division candidate. Here, kanji notation is used for reading estimation and katakana notation is converted to hiragana. The conversion cost is set as follows. The conversion cost of the same character is 0. In addition, vowels (including small letters (eg, ぁ, ぃ, ぅ, ぇ, ぉ)), 促 (音), 撥 ((), and deletion of long notes are regarded as cost 0. However, deletion at the beginning of a word counts up the sound similarity distance as cost 1. In addition, same or same line (refers to the same line or line of Japanese Hiragana 50 phonogram. Duzziness or Hemitone is considered to be the same position as the character before Hakuon or Hakuon conversion) substitution of characters, vowel-speech Substitution, vowel-long tone, vowel-vowel conversion is cost 0. For example, it is assumed that the cost (0) is a string of the same line or string (such as “bu” or “pu” → “fu”). Conversion other than the above counts up the sound similarity distance as cost 1. FIG. 2 shows an example of calculation of the sound similarity distance. In the present embodiment, in order to filter sound similarities that are equal to or higher than the threshold, those having a sound similarity distance equal to or lower than the threshold are filtered.

  Next, the synonym pair acquiring unit 34 performs the first acquisition processing and the second acquisition processing described below on each of the word division candidates of the regular expression word acquired from the document set, and thereby the synonym pair is acquired. To earn In the first acquisition process of the synonym pair acquisition unit 34, the following process is performed on each of the word division candidates of the regular expression word acquired from the document set.

  First, for the word split candidates of the regular written word, filtering out word split candidates whose semantic similarity with other word split candidates is equal to or more than a predetermined threshold from other word split candidates appearing in the document set Do. Next, from the filtered word division candidates, the word division candidates for which the sound similarity with another word division candidate is equal to or more than a predetermined threshold (the sound similarity distance is equal to or less than the threshold) are filtered. . Furthermore, from the filtered word division candidates, in the dictionary database 28, the word division candidate is present as a regular expression word in the dictionary and is the same as the part of speech of the regular expression word in the dictionary delete. Then, the synonym pair acquiring unit 34 selects the word division candidate after deletion. In this way, a pair of the word segmentation candidate of the regular expression word and the selected word segmentation candidate is obtained as a synonym pair. FIG. 4 shows an example of the first acquisition process. In FIG. 4, word division candidates are selected starting from the word division candidate “Samui” of a regular expression word.

  Next, the synonym pair acquiring unit 34 is selected as a synonym pair for the word segmentation candidate of the regular expression word in the first acquisition process described above for the word segmentation candidate of the regular expression word as follows: A second acquisition process is performed starting from the word division candidate. First, for each of the word division candidates selected as synonym pairs for the word division candidates of the regular written word in the first acquisition process described above, calculation of semantic similarity with other word division candidates, and sound Calculate similarity distance. Next, the following processing is performed on each of the word division candidates selected as synonym pairs for the word division candidates of the regular written word.

  For the word division candidates, the word division candidates whose semantic similarity with each of the other word division candidates appearing in the document set is equal to or more than a predetermined threshold value are filtered. Next, from the filtered word division candidates, the word division candidates for which the sound similarity distance with the word division candidate is equal to or less than a predetermined threshold value are filtered. Furthermore, from the filtered word division candidates, in the dictionary database 28, the word division candidate notation is present as a regular expression word in the dictionary, and the part of speech of the regular expression word in the dictionary is the same as the part of speech Do. Then, the synonym pair acquiring unit 34 selects the word division candidate after deletion. In this way, a pair of the word segmentation candidate of the regular expression word and the selected word segmentation candidate is obtained as a synonym pair. FIG. 5 shows an example of the second acquisition process. In FIG. 5, the word division candidates are selected starting from the word division candidate "Samichi" selected for the word division candidate "Samui" of a regular expression word in the first acquisition processing. Furthermore, the synonym pair acquiring unit 34 repeats the same processing as the second acquisition processing a predetermined number of times, starting from the word division candidate selected in the second acquisition processing, as a starting point, and divides the word of the regular written word. A pair of the candidate and the selected word division candidate is acquired as a synonym pair.

<Operation of Synonym Pair Acquisition Device According to Embodiment of the Present Invention>

  Next, the operation of the synonym pair acquisition apparatus 100 according to the embodiment of the present invention will be described. When the input unit 10 receives a document set consisting of documents including broken notation words, the synonym pair acquisition apparatus 100 executes a synonym pair acquisition processing routine shown in FIG.

  First, in step S100, a plurality of word division candidates are generated from each of the documents of the document set accepted by the input unit 10.

  Next, in step S102, a word semantic vector is calculated for each of the word division candidates based on the plurality of word division candidates generated in step S100.

  In step S104, semantic similarity with each of the other word division candidates is calculated based on the semantic vector calculated in step S102 for each of the word division candidates that are regular expression words generated in step S100.

  In step S106, the sound similarity distance with each of the other word division candidates is calculated based on the reading of the word division candidates for each of the word division candidates that are regular expression words generated in step S100.

  In step S108, the semantic similarity calculated in step S104 is greater than or equal to the threshold and the sound similarity distance calculated in step S106 is less than or equal to the threshold for each of the word division candidates that are regular expressions. A word division candidate is selected from a plurality of word division candidates, and a pair of a word division candidate that is a regular expression word and the selected word division candidate is acquired as a synonym pair.

  In step S110, for each word division candidate that is a regular expression word, the semantic similarity calculated in the same manner as step S104 is equal to or greater than the threshold value for each of the word division candidates selected in step S108 or the previous step S110. And a word division candidate whose sound similarity distance calculated in the same manner as in step S106 is equal to or less than a threshold is selected from a plurality of word division candidates, and the word division candidate which is the regular written word is selected A pair with the word division candidate is acquired as a synonym pair.

  In step S112, it is determined whether the process of step S110 is repeated a predetermined number of times, and if it is repeated, the process proceeds to step S114, and if it is not repeated, the process returns to step S110 to repeat the process.

  In step S114, the synonym pair acquired in step S108 and step S110 is output to the output unit 50, and the process ends.

  As described above, according to the synonym pair acquiring apparatus according to the embodiment of the present invention, a plurality of word division candidates which are regular written words or broken written words are generated from a document, and a plurality of word division candidates are generated. The semantic vector of the word is calculated for each of the plurality of word division candidates, and the semantic similarity calculated based on the semantic vector is equal to or greater than the threshold value for each of the word division candidates that are regular written words, And, a word division candidate whose sound similarity distance calculated based on word reading is equal to or less than a threshold is selected from a plurality of word division candidates, and the word division candidate which is a regular written word and the selected word By acquiring a pair with a division candidate as a synonym pair, it is possible to efficiently acquire a synonym pair.

  Also, by considering both the semantic similarity and the sound similarity, it is possible to obtain pairs of synonymous candidates with high accuracy.

  In addition, even with respect to word division candidates that have been filtered only by acquisition based on regular written words, a variety of broken transcription words can be obtained by acquiring new synonym pairs starting from the selected word division candidate. It will be possible to earn

  Also, compared with the conventional method, it becomes possible to generate word division candidates as correct divisions of various broken words.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the scope of the present invention.

Reference Signs List 10 input unit 20 operation unit 28 dictionary database 30 word division candidate generation unit 32 meaning vector calculation unit 34 synonym pair acquisition unit 50 output unit 100 synonym pair acquisition device

Claims (5)

A word division candidate generation unit that generates, from a document, a plurality of word division candidates including a word division candidate that is a regular expression word and a word division candidate that is a broken expression word that is a substitution candidate that is a fluctuation candidate for the regular expression word When,
A semantic vector calculation unit that calculates a semantic vector of a word for each of the plurality of word division candidates based on the plurality of word division candidates generated by the word division candidate generation unit;
For each of said word candidate dividing a normalized notation word, the meaning similarity is calculated based on the mean vector, based on a sound similarity is calculated based on the reading of the word, before Symbol plurality of word segmentation The word division from the plurality of word division candidates filtered and filtered, which is the same expression as the predetermined regular expression word and is the same as the part of speech with the regular expression word of the same expression A synonym pair acquiring unit which selects a candidate excluding a candidate and acquires a pair of the word division candidate which is a regular expression word and the selected word division candidate as a synonym pair;
Synonyms pair acquisition device including. The synonym pair acquisition unit
For each selected by said word segmentation candidate, on the basis of said sound similarity to the mean similarity, filtering the previous SL plurality of word segmentation candidate, from the filtered plurality of word segmentation candidate, predetermined The word segmentation candidate is a regular notation word, and the word segmentation candidate is a regular notation word, and the word segmentation candidate is selected except for the word segmentation candidate having the same notation as the regular notation word and the same part of speech with the regular notation word. The synonym pair acquisition apparatus according to claim 1, wherein a pair with the selected word division candidate is further acquired as a synonym pair. From the document, the word division candidate generation unit generates, from the document, a plurality of word division candidates including a word division candidate which is a regular expression word and a word division candidate which is a broken expression word which is a candidate of a transcriptional expression with respect to the regular expression word. Generating steps,
Calculating a semantic vector of a word for each of the plurality of word division candidates based on the plurality of word division candidates generated by the word division candidate generation unit;
For each of the word division candidates that are regular expression words, the synonym pair acquisition unit is calculated based on the semantic similarity calculated based on the semantic vector and the sound similarity calculated based on word reading filters the previous SL plurality of word segmentation candidate, from the filtered plurality of word segmentation candidates have the same notation and the normal notation word predetermined, equal to the normal notation word of the same notation Selecting the word division candidate which is a part of speech, and acquiring a pair of the word division candidate which is a regular expression word and the selected word division candidate as a synonym pair;
How to get synonym pairs including. The steps acquired by the synonym pair acquisition unit are:
For each selected by said word segmentation candidate, on the basis of said sound similarity to the mean similarity, filtering the previous SL plurality of word segmentation candidate, from the filtered plurality of word segmentation candidate, predetermined The word segmentation candidate is a regular notation word, and the word segmentation candidate is a regular notation word, and the word segmentation candidate is selected except for the word segmentation candidate having the same notation as the regular notation word and the same part of speech with the regular notation word. The synonym pair acquisition method according to claim 3, wherein a pair with the selected word division candidate is further acquired as a synonym pair.   The program for functioning a computer as each part of the synonym pair acquisition apparatus of Claim 1 or Claim 2.