JP2016224482A - Synonym pair acquisition device, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire a synonym pair efficiently.SOLUTION: A word division candidate creation part 30 creates plural word division candidates which are normal notation words, or informal notation words, from a document. A meaning vector calculation part 32 calculates a word meaning vector for each of the plural word division candidates, based on the plural word division candidates. A synonym pair acquisition part 34 selects word division candidates in which, meaning similarity calculated based on the meaning vector is equal to or higher than a threshold, and a sound similarity distance calculated based on reading of the word is equal to or lower than a threshold, for each of the word division candidates which are the normal notation words, from the word division candidates, then, acquires a pair of the word division candidate being the normal notation word, and the selected word division candidate as a synonym pair.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a synonym pair acquisition apparatus, method, and program, and more particularly, to a synonym pair acquisition apparatus, method, and program for acquiring synonym pairs.

  Conventionally, a method for acquiring a collapsed notation word, which is a distorted notation with respect to a regular notation word, has been proposed. Examples of the technique using the teacher data include techniques described in Non-Patent Document 1 and Non-Patent Document 2.

  Examples of methods that do not use teacher data include the methods described in Non-Patent Document 3 and Non-Patent Document 4.

Nakan Okazaki, Jun-ichi Sakurai, “Automatic Acquisition of Abbreviation Definitions Using Alignment Discrimination Models”. The 14th Annual Conference of the Association for Natural Language Processing (NLP2008), pp. 139-142 Yoshinari Fujinuma, Hikaru Yokono, Akiko Aizawa, “Detection and Analysis of Broken Notation Using“ Good Morning ”on Twitter (R).” 27th Annual Conference of the Japanese Society for Artificial Intelligence, 2013.06 Koji Masuyama, Satoshi Sekine, “Automatic construction of a katakana spelling list from a large corpus”, 14th Annual Meeting of the Association for Natural Language Processing (NLP2004) Kazufumi Ikeda, Tadashi Yanagihara, Kazunori Matsumoto, Yasuhiro Takishima, “Automatic Normalization Rule Generation Method for Analyzing Complex Expressions with High Accuracy”, IPSJ Journal, vol3. No.3 pp.68-77 , 2010 Kudo, T., Japanese Morphological Analyzer, Internet <URL: http: //mecab.googlecode.com/svn/trunk/mecab/doc/index.html>

  However, when a broken notation word is extracted by the method using the teacher data described in Non-Patent Document 1 and Non-Patent Document 2, it is necessary to manually create a correct answer pair as shown in FIG. There is a problem that the cost of generating pairs is high.

  In addition, based on a method that does not use teacher data, the candidates for corrupted words that can be obtained are limited to limited candidates (such as katakana and words that have become unknown words with existing analyzers), and there are various types of corrupted words. There is a problem that the notation cannot be acquired. This is because, with existing analyzers, broken notation words are often mistakenly analyzed, and it is difficult to acquire various broken notation words. This is because, in Japanese, there is no separation such as a space between words, so it is difficult to analyze the correct separation position of morphemes in existing text. Also, on the Web, there are many broken notation words written in hiragana, kanji and hiragana, katakana and hiragana, etc., and analysis is difficult. For example, “Suge”, “Yaba”, “Samii”, “Samui”, “Cold”, etc. FIG. 8 shows an example of a result of analyzing a sentence including a collapsed notation word using Mecab (IPAdic) described in Non-Patent Document 5.

  The present invention has been made to solve the above-described problems, and an object thereof is to provide a synonym pair acquisition apparatus, method, and program capable of efficiently acquiring synonym pairs.

  In order to achieve the above object, the synonym pair acquisition apparatus according to the first invention is a regular notation word from a document, or a plurality of collapsed notation words that are candidates for a notation that fluctuates with respect to the regular notation word. Based on the word division candidate generation unit that generates word division candidates and the plurality of word division candidates generated by the word division candidate generation unit, a word semantic vector is calculated for each of the plurality of word division candidates. Based on a semantic vector calculation unit, a semantic similarity calculated based on the semantic vector, and a sound similarity calculated based on a word reading for each of the word division candidates that are regular notation words, The word division candidate is selected from the plurality of word division candidates, and a pair of the word division candidate that is a regular notation word and the selected word division candidate is acquired as a synonym pair. It is configured to include the word pair acquisition unit.

  Further, in the synonym pair acquisition device according to the first invention, the synonym pair acquisition unit, for each of the word division candidates that are regular notation words, based on the semantic similarity and the sound similarity, The word division candidate is selected from the plurality of word division candidates, and a pair of the word division candidate that is a normal notation word and the selected word division candidate is acquired as a synonym pair, and the selected For each word division candidate, based on the semantic similarity and the sound similarity, the word division candidate is selected from the plurality of word division candidates, and the word division candidate that is a normal written word is selected. Alternatively, a pair with the word division candidate may be acquired as a synonym pair.

  In the synonym pair acquisition method according to the second invention, the word division candidate generation unit includes a plurality of words that are normal notation words or collapsed notation words that are fluctuation candidates for the normal notation words from the document. A step of generating a division candidate, and a semantic vector calculation unit calculates a word semantic vector for each of the plurality of word division candidates based on the plurality of word division candidates generated by the word division candidate generation unit And a synonym pair acquisition unit, for each of the word division candidates that are regular notation words, a semantic similarity calculated based on the semantic vector and a sound similarity calculated based on the reading of the word Based on the above, the word division candidate is selected from the plurality of word division candidates, and a pair of the word division candidate that is a normal notation word and the selected word division candidate is defined as a synonym pair. And executes includes a step of acquiring, as.

  In the synonym pair acquisition method according to the second invention, the synonym pair acquisition unit acquires the semantic similarity and the sound similarity for each of the word division candidates that are regular notation words. The word division candidate is selected from the plurality of word division candidates, and a pair of the word division candidate that is a normal notation word and the selected word division candidate is obtained as a synonym pair, For each of the selected word division candidates, the word division candidates are selected from the plurality of word division candidates based on the semantic similarity and the sound similarity, and the word division candidates that are regular notation words And the selected pair of word division candidates may be acquired as a synonym pair.

  A program according to a third invention is a program for causing each unit to function as each unit of the synonym pair acquisition device according to the first invention.

  According to the synonym pair acquisition device, method, and program of the present invention, a plurality of word division candidates that are regular notation words or collapsed notation words are generated from a document, and a plurality of word division candidates are generated based on the plurality of word division candidates. A word semantic vector is calculated for each word division candidate, and for each word division candidate that is a regular notation word, a semantic similarity calculated based on the semantic vector and a sound calculated based on the word reading Based on the similarity, a word division candidate is selected from a plurality of word division candidates, and a pair of the word division candidate that is a normal notation word and the selected word division candidate is acquired as a synonym pair. The effect that a synonym pair can be acquired efficiently is acquired.

It is a block diagram which shows the structure of the synonym pair acquisition apparatus which concerns on embodiment of this invention. It is a figure which shows an example of a sound similarity. It is a conceptual diagram which shows the example of acquisition of a synonym pair. It is a figure which shows the example which selects a word division | segmentation candidate starting from a regular notation word. It is a figure which shows the example which selects a word division candidate further from the selected word division candidate as a starting point. It is a flowchart which shows the synonym pair acquisition process routine in the synonym pair acquisition apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the combination of a regular notation word and a collapse notation word. It is a figure which shows an example of the result of having analyzed the sentence containing collapse notation word using Mecab.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Configuration of Synonym Pair Acquisition Device According to Embodiment of the Present Invention>

  Next, the structure of the synonym pair acquisition apparatus which concerns on embodiment of this invention is demonstrated. As shown in FIG. 1, a synonym pair acquisition apparatus 100 according to an embodiment of the present invention includes a CPU, a RAM, and a ROM that stores a program and various data for executing a synonym pair acquisition processing routine described later. And a computer including Functionally, the synonym pair acquisition apparatus 100 includes an input unit 10, a calculation unit 20, and an output unit 50 as shown in FIG.

  The input unit 10 accepts a document set made up of documents including corrupted notation words.

  The calculation unit 20 includes a dictionary database 28, a word division candidate generation unit 30, a semantic vector calculation unit 32, and a synonym pair acquisition unit 34.

  The dictionary database 28 stores dictionaries (reading, notation, parts of speech) necessary for dictionary lookup.

  The word division candidate generation unit 30 includes a plurality of word division candidates that are regular notation words or collapsed notation words that are fluctuation candidates for the normal notation words from each of the documents in the document set received by the input unit 10. Is generated.

  Specifically, the word division candidate generation unit 30 generates word division candidates by applying each of the following first to third methods, which are existing word division methods, to a document. At this time, a technique is used in which corrupted notation words that do not exist in the dictionary database 28 can also be output as delimiter candidates.

  The word division candidate generation unit 30 generates a word division candidate by applying a word division method using point estimation as a first method to each of the documents included in the document set. In the word segmentation technique using point estimation, a document is segmented into a plurality of word segmentation candidates using a delimiter model between characters whose features are character ngram and character type ngram.

  The word division candidate generation unit 30 applies a word division method using unsupervised analysis to the document set as a second method to generate word division candidates. In the word division method using unsupervised analysis, the appearance frequency is calculated for the sampled word division candidates, and each of the documents is divided into word division candidates so that the objective function is optimized.

  The word division candidate generation unit 30 acquires, as a third method, an analysis result by Mecab or the like for each of the documents included in the document set and partially combines them based on a predetermined rule. Is generated. For example, unknown word sequences are combined, noun sequences are combined, and the like. The following method may be used as a rule. For example, when a short sentence is cut out from Twitter (R) or the like and used as a word division candidate, the short sentence is cut out by using a plurality of delimiters (for example, line breaks, symbolic expressions (“!”, “W”, "♪"), punctuation marks (",", "."), Etc.) are set, and short sentences can be divided by delimiters. By setting in this way, for example, if it is a sentence “Yaybee wwwwwwwww”, “Yaybee” before “w” can be acquired as a word division candidate. In addition, if the sentence “Ohyoyo ♪ is noon, but now ...”, “Ohahayo” before “♪” can be acquired as a word division candidate. Analysis may be performed by adding a character string having n or fewer characters acquired as described above to the morpheme dictionary.

  The semantic vector calculation unit 32 calculates a word semantic vector for each of the plurality of word division candidates based on the plurality of word division candidates generated by the word division candidate generation unit 30.

  Specifically, the semantic vector calculation unit 32 appears as a word division candidate for a document set to which word breaks are given so as to enumerate a plurality of word division candidates generated by the word division candidate generation unit 30. Calculate the semantic vector for each word. At this time, an existing method can be used as a method for obtaining the semantic vector of each word. For example, word2vec described in Non-Patent Document 6 can be cited as a representative technique.

[Non-Patent Document 6]: Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013.

  The synonym pair acquisition unit 34, for each word division candidate that is a regular notation word, the semantic similarity calculated based on the semantic vector is equal to or greater than a threshold, and the sound calculated based on the reading of the word A word division candidate having a similarity equal to or greater than a threshold is selected from a plurality of word division candidates, and a pair of the word division candidate that is a normal notation word and the selected word division candidate is acquired as a synonym pair. Further, the synonym pair acquisition unit 34 has, for each of the selected word division candidates, a sound having a semantic similarity calculated based on a semantic vector equal to or greater than a threshold and calculated based on a word reading. A word division candidate having a similarity equal to or greater than a threshold is selected from a plurality of word division candidates, and a pair of the word division candidate that is a normal notation word and the selected word division candidate is acquired as a synonym pair. FIG. 3 shows a conceptual diagram of processing of the synonym pair acquisition unit 34.

  Specifically, the synonym pair acquisition unit 34 first calculates the semantic similarity with each of the other word division candidates for each of the word division candidates that are regular notation words. The semantic similarity is calculated using the cosine similarity of the semantic vector for each word obtained by the semantic vector calculation unit 32.

  Next, the synonym pair acquisition unit 34 calculates, for each word division candidate that is a regular notation word, a sound similarity with other word division candidates. In the present embodiment, the sound similarity distance is calculated as the sound similarity based on the reading of the word division candidates. Here, kanji notation is estimated by reading, and katakana notation is converted to hiragana. The conversion cost is set as follows. The conversion cost for the same character is assumed to be zero. In addition, the cost of deleting vowels (including lowercase letters (eg, aaa, i, ぅ, eh, ぉ)), prompting sounds (tsu), repelling sounds (n), and long sounds is 0. However, deletion at the beginning of a word counts up the sound similarity distance as cost 1. In the same row or in the same row (refers to the same row or row in the Japanese hiragana 50 syllabary. The cloudy or semi-turbid sound is considered as the same position as the character before making it muddy or semi-voiced). Replacement, vowel-long sound, and vowel-vowel conversion are assumed to have no cost. For example, a character string in the same row or column (such as “bu” or “pu” → “fu”) is assumed to have a cost of zero. For conversions other than the above, the sound similarity distance is counted up as cost 1. FIG. 2 shows a calculation example of the sound similarity distance. In the present embodiment, in order to filter sound similarities that are equal to or greater than the threshold, those having a sound similarity distance equal to or less than the threshold are filtered.

  Next, the synonym pair acquisition unit 34 performs a first acquisition process and a second acquisition process described below for each of the word segmentation candidates of the regular notation words obtained from the document set, and synonym pair acquisition To win. In the first acquisition process of the synonym pair acquisition unit 34, the following process is performed for each word division candidate of the regular notation word obtained from the document set.

  First, for word division candidates for the regular notation word, word division candidates whose semantic similarity with other word division candidates is greater than or equal to a predetermined threshold are filtered from other word division candidates that appear in the document set. To do. Next, from the filtered word division candidates, word division candidates for which the sound similarity with other word division candidates is equal to or greater than a predetermined threshold (sound similarity distance is equal to or less than the threshold) are filtered for the regular notation word. . Further, from the filtered word segmentation candidates, in the dictionary database 28, the notation of the word segmentation candidate exists as a regular notation word in the dictionary and has the same part of speech as the part of speech of the regular notation word in the dictionary. delete. And the synonym pair acquisition part 34 selects the word division candidate after deletion. In this way, a pair of the word division candidate of the regular notation word and the selected word division candidate is acquired as a synonym pair. FIG. 4 shows an example of the first acquisition process. In FIG. 4, the word division candidate is selected starting from the word division candidate “Samui” of the regular notation word.

  Next, the synonym pair acquisition unit 34 selected the word division candidate of the regular notation word as the synonym pair for the word division candidate of the regular notation word in the first acquisition process as described below. A second acquisition process is performed starting from the word division candidate. First, for each word division candidate selected as a synonym pair for the word division candidate of the regular notation word in the first acquisition process, calculation of semantic similarity with other word division candidates, and sound Calculate the similarity distance. Next, the following processing is performed for each word division candidate selected as a synonym pair for the word division candidate of the regular notation word.

  For the word division candidates, word division candidates whose semantic similarity with each of the other word division candidates appearing in the document set is equal to or greater than a predetermined threshold are filtered. Next, the word division candidate whose sound similarity distance with the word division candidate is equal to or less than a predetermined threshold is filtered from the filtered word division candidates. Further, from the filtered word segmentation candidates, in the dictionary database 28, the word segmentation candidate notation is present as a regular notation word in the dictionary and the part of speech that is the same as the part of speech of the regular notation word in the dictionary is deleted. To do. And the synonym pair acquisition part 34 selects the word division candidate after deletion. In this way, a pair of the word division candidate of the regular notation word and the selected word division candidate is acquired as a synonym pair. FIG. 5 shows an example of the second acquisition process. In FIG. 5, the word division candidate is selected starting from the word division candidate “Samie” selected for the word division candidate “Samui” of the regular notation word in the first acquisition process. Further, the synonym pair acquisition unit 34 repeats the same process as the second acquisition process a predetermined number of times starting from the word division candidate selected in the second acquisition process, and performs word division of the regular notation word A pair of the candidate and the selected word division candidate is acquired as a synonym pair.

<Operation of Synonym Pair Acquisition Device According to Embodiment of the Present Invention>

  Next, the operation of the synonym pair acquisition apparatus 100 according to the embodiment of the present invention will be described. When the input unit 10 receives a document set made up of documents including corrupted notation words, the synonym pair acquisition apparatus 100 executes a synonym pair acquisition processing routine shown in FIG.

  First, in step S100, a plurality of word division candidates are generated from each of the documents in the document set received by the input unit 10.

  Next, in step S102, a word semantic vector is calculated for each of the word division candidates based on the plurality of word division candidates generated in step S100.

  In step S104, the semantic similarity with each of the other word division candidates is calculated based on the semantic vector calculated in step S102 for each of the word division candidates that are regular notation words generated in step S100.

  In step S106, for each word division candidate that is the normal notation word generated in step S100, the sound similarity distance with each of the other word division candidates is calculated based on the reading of the word division candidates.

  In step S108, the semantic similarity calculated in step S104 is greater than or equal to the threshold for each word division candidate that is a regular notation word, and the sound similarity distance calculated in step S106 is less than or equal to the threshold. A word division candidate is selected from a plurality of word division candidates, and a pair of a word division candidate that is a regular notation word and the selected word division candidate is acquired as a synonym pair.

  In step S110, for each word division candidate that is a regular notation word, the semantic similarity calculated in the same manner as in step S104 is greater than or equal to the threshold for each word division candidate selected in step S108 or previous step S110. In addition, a word division candidate whose sound similarity distance calculated in the same manner as in step S106 is equal to or less than a threshold is selected from a plurality of word division candidates, and the word division candidate that is the regular notation word is selected. The pair with the word division candidate obtained is acquired as a synonym pair.

  In step S112, it is determined whether the process of step S110 has been repeated a predetermined number of times. If it has been repeated, the process proceeds to step S114, and if not, the process returns to step S110 to repeat the process.

  In step S114, the synonym pair acquired in step S108 and step S110 is output to the output unit 50, and the process ends.

  As described above, according to the synonym pair acquisition device according to the embodiment of the present invention, a plurality of word division candidates that are regular notation words or collapsed notation words are generated from a document, and a plurality of word division candidates are obtained. Based on the above, a word semantic vector is calculated for each of a plurality of word division candidates, and the semantic similarity calculated based on the semantic vector for each word division candidate that is a regular notation word is greater than or equal to a threshold value In addition, a word division candidate having a sound similarity distance calculated based on the reading of the word is equal to or less than a threshold is selected from a plurality of word division candidates, and the word division candidate that is a normal notation word and the selected word By acquiring a pair with a division candidate as a synonym pair, a synonym pair can be efficiently acquired.

  Further, by considering both the semantic similarity and the sound similarity, a pair of synonym candidates can be obtained with high accuracy.

  In addition, even for word segmentation candidates that have been filtered only by acquisition based on regular notation words, more diverse collaborative notation words can be obtained by acquiring new synonym pairs starting from the selected word segmentation candidates. It becomes possible to acquire.

  In addition, compared to the conventional method, it is possible to generate word division candidates as correct delimiters of various corrupted notation words.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 Input part 20 Operation part 28 Dictionary database 30 Word division candidate production | generation part 32 Semantic vector calculation part 34 Synonym pair acquisition part 50 Output part 100 Synonym pair acquisition apparatus

Claims (5)

A word division candidate generation unit that generates a plurality of word division candidates that are regular notation words or collapsed notation words that are fluctuation notation candidates with respect to the regular notation word from a document;
A semantic vector calculation unit that calculates a word semantic vector for each of the plurality of word division candidates based on the plurality of word division candidates generated by the word division candidate generation unit;
Based on the semantic similarity calculated based on the semantic vector and the sound similarity calculated based on word reading for each of the word division candidates that are regular notation words, A synonym pair acquisition unit that acquires a pair of the word division candidate that is a normal notation word and the selected word division candidate as a synonym pair, selected from the plurality of word division candidates;
A synonym pair acquisition device. The synonym pair acquisition unit
For each word division candidate that is a regular notation word, the word division candidate is selected from the plurality of word division candidates based on the semantic similarity and the sound similarity, and the word that is a normal notation word A pair of the division candidate and the selected word division candidate is acquired as a synonym pair,
For each of the selected word division candidates, the word division candidates are selected from the plurality of word division candidates based on the semantic similarity and the sound similarity, and the word division candidates that are regular notation words The synonym pair acquisition apparatus according to claim 1, wherein a pair of the selected word division candidate is acquired as a synonym pair. A word division candidate generation unit, from a document, generating a plurality of word division candidates that are regular notation words or collapsed notation words that are candidates for notation that fluctuates with respect to the regular notation words;
A semantic vector calculation unit calculating a word semantic vector for each of the plurality of word division candidates based on the plurality of word division candidates generated by the word division candidate generation unit;
The synonym pair acquisition unit, based on the semantic similarity calculated based on the semantic vector and the sound similarity calculated based on the reading of the word, for each of the word division candidates that are regular notation words Selecting the word division candidate from the plurality of word division candidates, and obtaining a pair of the word division candidate that is a normal notation word and the selected word division candidate as a synonym pair;
Synonym pair acquisition method including The synonym pair acquisition unit acquires,
For each word division candidate that is a regular notation word, the word division candidate is selected from the plurality of word division candidates based on the semantic similarity and the sound similarity, and the word that is a normal notation word A pair of the division candidate and the selected word division candidate is acquired as a synonym pair,
For each of the selected word division candidates, the word division candidates are selected from the plurality of word division candidates based on the semantic similarity and the sound similarity, and the word division candidates that are regular notation words The synonym pair acquisition method according to claim 3, wherein a pair of the selected word division candidate is acquired as a synonym pair.   The program for functioning a computer as each part of the synonym pair acquisition apparatus of Claim 1 or Claim 2.