JP2019016162A - Morphological analysis program, morphological analysis device, and morphological analysis method - Google Patents

Abstract

To improve the analysis accuracy of morphological analysis.SOLUTION: A morphological analysis device comprises: a storage unit for storing a matching dictionary that includes a morphological analysis dictionary, character strings included in each of a plurality of sentences, and a first morphological analysis result of the character strings that are commonly obtained for each of the plurality of sentences; a first analysis unit for, from an analysis target text, outputting the first morphological analysis result for the character strings matching with the character strings included in the matching dictionary; and a second analysis unit for, from the analysis target text, generating a lattice including a plurality of morphological analysis result candidates using the morphological analysis dictionary for the remaining character strings unmatched with the character strings included in the matching dictionary, morphologically analyzing the remaining character strings using the lattice, and outputting a second morphological analysis result for the remaining character strings.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a morpheme analysis program, a morpheme analyzer, and a morpheme analysis method.

  In recent years, information on the Internet has increased dramatically, and business using big data has increased. Therefore, it is desired to process big data efficiently. In the case of a document in which a word and a word are not separated by a delimiter such as a space, such as a Japanese, Chinese, or Korean document, morphological analysis is performed to calculate the appearance frequency of the word.

  Morphological analysis is a process of dividing text into morphemes and adding part-of-speech information to each morpheme. A morpheme obtained by morpheme analysis may be treated as a word. By performing such morphological analysis, the relationship between words in the document and the part of speech of the word are determined, and the text in the document can be divided into words. However, since morphological analysis has a heavy processing load, it takes a long time to process a large amount of text.

  In morphological analysis, all words that partially match the notation of the character string to be analyzed are extracted from the dictionary, and analysis using a lattice that is a graph structure in which word candidates (analysis candidates) are listed is performed.

FIG. 1 is a diagram illustrating an example of a lattice.
FIG. 1 shows a case where a lattice for an input sentence = “sent” that is an analysis target is constructed. In the morphological analysis, a correct morpheme sequence is determined in consideration of the context (front and back morphemes) for the constructed lattice. As a result, the result of the morphological analysis of “sent” is “sent (verb / form)” | (suffix) | kita (suffix). Lattice construction is computationally expensive and time consuming.

  In morphological analysis, there is known a method of increasing the speed by using pattern matching without constructing a lattice having a high calculation cost (for example, see Non-Patent Document 1). 2. Description of the Related Art A word dividing device that divides a sentence into two or more words at high speed is known (for example, see Patent Document 1). A dictionary registration device that obtains a word segmentation dictionary with high accuracy is known (see, for example, Patent Document 2).

JP 2014-106707 A JP 2014-120007 A

Manabu Sassano, “Deterministic Word Segmentation Using Maximum Matching with Fully Lexicalized Rules”, Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 79-83, Gothenburg, Sweden, April 26-30 2014

  However, although the morpheme analysis method described in Non-Patent Document 1 can perform morpheme analysis at high speed by pattern matching, there is a problem that an incorrect analysis result may be output and analysis accuracy is low.

  In one aspect, an object of the present invention is to improve analysis accuracy of morphological analysis.

  The morpheme analysis program according to the embodiment includes a morpheme analysis dictionary, a character string included in each of a plurality of sentences, and a first morpheme analysis result of the character string obtained in common for each of the plurality of sentences. A computer having a storage unit that stores a matching dictionary including the following processes is executed.

  The computer outputs the first morpheme analysis result for a character string that matches the character string included in the matching dictionary in the analysis target text.

The computer generates a lattice including a plurality of morphological analysis result candidates using the morphological analysis dictionary for the remaining character strings that do not match the character strings included in the matching dictionary in the analysis target text. The computer performs a morphological analysis on the remaining character string using the lattice, and outputs a second morpheme analysis result on the remaining character string

  According to the embodiment, the analysis accuracy of morphological analysis can be improved.

It is a figure which shows the example of a lattice. It is a block diagram of the morpheme analyzer which concerns on embodiment. It is an example of a context independent dictionary. It is an example of a context-dependent character string and an analysis result. It is a figure which shows the morphological analysis process which concerns on embodiment. It is a figure which shows the morphological analysis process which concerns on embodiment. It is a flowchart of the context independent dictionary production | generation process which concerns on embodiment. It is a flowchart of the morphological analysis process which concerns on embodiment. It is a figure which shows the lattice about an unanalyzed character string and the morpheme before and behind it. It is a figure which shows the lattice about an unanalyzed character string and the morpheme before and behind it. It is a figure which shows the morpheme string which is the analysis result with respect to an unanalyzed character string. It is a figure which shows the morpheme string which is the analysis result with respect to an unanalyzed character string. It is a flowchart of the modification of the morphological analysis process which concerns on embodiment. It is a figure which shows the lattice of the input sentence containing an unparsed character string. It is a figure which shows the morpheme string of the input sentence containing the analysis result with respect to an unanalyzed character string. It is a block diagram of information processing apparatus.

Hereinafter, embodiments will be described with reference to the drawings.
First, a case where morphological analysis is performed using the conventional technique described in Non-Patent Document 1 will be described. In the prior art, first, a morpheme string corresponding to a sentence to be analyzed is output by a longest match method using a dictionary, and a morpheme string that matches a replacement pattern among morpheme strings output in error is correct according to the replacement pattern. Replaced with a column.
(First example of morphological analysis according to the prior art)
For the input sentence to be analyzed = “very good reputation”, the result of the morphological analysis by the longest match method is “very | reputation | is | good reason | is”. In the prior art, in order to correct an erroneous analysis result, a replacement pattern is referred to and a matching morpheme string is corrected.

  Here, it is assumed that there is a replacement pattern that corrects “ga | good reason | da” to “ga | good matter”. In that case, "very | reputation | is | good reason | is" is corrected to "very | reputation | is | good | As described above, when there is an appropriate replacement pattern, the analysis result “very good reputation” is obtained as the analysis result “very good reputation” as the analysis result “very good reputation”.

However, if there is no appropriate replacement pattern as described above, the morpheme sequence is not corrected. Therefore, as a result of the morphological analysis for “Very Reputable”, the erroneous analysis result “Very | Reputation | | Good reason |
(Second example of conventional morphological analysis)
For the input sentence = “It is good to say that there is a shortage of manpower”, the result of the morphological analysis by the longest match method is “manual | shortage | and | say | In the prior art, in order to correct an erroneous analysis result, a replacement pattern is referred to and a matching morpheme string is corrected.

  Here, if there is no replacement pattern for correcting the morpheme sequence included in “manual | shortage | and | say | is | good reason | da”, the analysis result is not corrected and the correct analysis result is “manual | shortage” | And | say | is | good reason |

  Here, it is assumed that there is a replacement pattern that corrects “ga | good reason | da” to “ga | good matter”. In this case, “manual | insufficient | and | say | is | good reason | da” is corrected to “manual | insufficient | and | say | is | good”. By applying the replacement pattern, the result of the morphological analysis for “manual shortage is good” is obtained as the result of erroneous analysis “manual | shortage | and | say | is | good”.

  As described above, the replacement pattern is not a rule considering the context, and an erroneous analysis result may be obtained by applying the replacement pattern.

FIG. 2 is a configuration diagram of the morpheme analyzer according to the embodiment.
The morpheme analyzer 101 includes a dictionary generator 201, a morpheme analyzer 301, and a storage unit 401.

  The dictionary generation unit 201 includes a context independent dictionary construction unit 211, a morpheme analysis unit 221, and a dependency determination unit 231.

  The context independent dictionary construction unit 211 generates the context independent dictionary 421 using the determination results of the morphological analysis unit 221 and the dependency determination unit 231.

  The morpheme analysis unit 221 performs morpheme analysis of the corpus 411. The morpheme analysis unit 221 performs morpheme analysis using, for example, an existing morpheme analysis method.

  The dependency determination unit 231 determines whether the character string is a character string whose morpheme analysis results differ (depends on the context) depending on the context.

  In the embodiment, a character string having a different morphological analysis result depending on a context is referred to as a context-dependent character string. In the embodiment, a character string whose morphological analysis result does not change depending on the context is referred to as a context-independent character string.

  The morpheme analysis unit 301 includes a context independent character string analysis unit 311 and a context dependent character string analysis unit 321.

  The context independent character string analysis unit 311 performs morphological analysis of the input sentence 431 by pattern matching using the context independent dictionary 421. The context-independent character string analysis unit 311 performs morphological analysis of the context-independent character string in the input sentence 431.

  The context-dependent character string analysis unit 321 includes a lattice construction unit 322 and a morpheme sequence selection unit 323. The context-dependent character string analysis unit 321 performs morphological analysis of a character string (that is, a context-dependent character string) that has not been analyzed by the context-independent character string analysis unit 311 in the input sentence 431.

  The lattice constructing unit 322 constructs a lattice of an unanalyzed character string. The lattice (also referred to as a word lattice) is a graph structure in which all words partially matching the notation of the character string to be analyzed are extracted from the morphological analysis dictionary and word candidates (analysis result candidates) are listed.

  The morpheme string selection unit 323 selects a word sequence (path) that is most likely to be a sentence in the constructed lattice. The morpheme sequence selection unit 323 selects a path that minimizes the evaluation value using, for example, the Viterbi algorithm. Note that the morpheme sequence selection unit 323 is not limited to the Viterbi algorithm, and may use a method such as a beam search.

  The storage unit 401 stores data, programs, and the like used by the morphological analyzer 101. The storage unit 401 stores a corpus 411, a context independent dictionary 421, an input sentence 431, and an analysis result 441. In addition, the storage unit 401 stores a morpheme analysis dictionary (not shown) including a plurality of words (morphemes) used when the lattice construction unit 322 and the morpheme analysis unit 221 construct a lattice.

  The corpus 411 is a set of a plurality of sentences. The corpus 411 is used by the dictionary generation unit 201 to generate the context independent dictionary 421.

  The context independent dictionary 421 is information indicating the result of morphological analysis for the context independent character string and the context independent character string. The context independent dictionary 421 is an example of a matching dictionary.

  The input sentence 431 is a sentence that is a target of morphological analysis by the morphological analysis unit 301. The input sentence 431 is an example of analysis target text.

The analysis result 441 is a result of morphological analysis of the input sentence 431.
FIG. 3 is an example of a context independent dictionary.

  The context independent dictionary 421 is information indicating a context independent character string that is a character string whose morphological analysis result does not change depending on the context. The context independent dictionary 421 includes a character string and a morpheme string. In the context independent dictionary 421, character strings and morpheme strings are recorded in association with each other.

The character string is a context-independent character string.
The morpheme string is a result of morpheme analysis on the character string. A morpheme string is a set of a plurality of morphemes obtained by dividing a character string by morpheme analysis. In the specification and drawings, “|” in the morpheme string indicates a break between morphemes. Note that the morpheme string may be added with information indicating the part of speech or the utilization form of each morpheme.

  For example, the context-independent dictionary 421 in FIG. 3 includes “nighttime or holiday” as a character string, and “nighttime | or | holiday” as a corresponding morpheme string. Also, the context-independent dictionary 421 in FIG. 3 includes “debris mountain” as a character string and “debris | no | mountain” as a corresponding morpheme string.

  “Night or holiday” is a character string whose morphological analysis result does not change depending on the context before and after “night or holiday”. That is, the result of the morphological analysis for “nighttime or holiday” is always the same. When the morphological analysis is performed on “nighttime or holiday”, it is divided into “nighttime | or | holiday”.

  Also for “debris mountain”, the result of morphological analysis is always divided as “debris | no | mountain”.

  The context-independent character string as described above always has the same morphological analysis result regardless of the context before and after the character string. Therefore, it is possible to obtain a correct analysis result using only the context-independent character string.

  In the context-independent dictionary 421, a model number, a person name, an emoticon, a fixed phrase, an English word, or a control token indicating a tab or a line feed may be registered as a character string. The model number, personal name, emoticon, fixed phrase, English word, and control token are character strings that always have the same morphological analysis results regardless of the context before and after the character string. The context independent dictionary 421 includes information indicating the result of morphological analysis for parentheses when the character string is parentheses and the result of morphological analysis for numerical expressions when the character string is a numerical expression such as a continuous numerical value. But you can. Parentheses and numerical expressions are character strings whose morphological analysis results are always the same regardless of the context before and after the character string.

  Next, a context-dependent character string, which is a character string whose morphological analysis results differ depending on the context, will be described.

FIG. 4 is an example of a context-dependent character string and an analysis result.
Here, three examples of context-sensitive character strings “because you know well”, “holiday and night”, and “snow mountain” will be described.
(1) “Because I know well”
If you perform morphological analysis on "Because you know well", depending on the context before and after "Because you know well", "Well | Know | From | Da" or "Well | Know | Body) ”.
(2) "Holidays and nights"
If "this week" is preceded by "holiday or night", morphological analysis is performed on "this week's holiday or night". The

If “hospital fixed” precedes “holiday or night”, the morphological analysis of “hospital closed holiday or night” would result in “hospital | no | fixed holiday | or | night” It is divided into.
(3) “Snowy Mountain”
When “see” is attached after “snow mountain”, if morphological analysis is performed on “see snow mountain”, it is divided into “snow | no | mountain | see” .

  When “large” is added before “snow mountain” and “gata prefecture” is added after “snow mountain”, when “morphological analysis” is performed on “snow mountain”, It is divided like |

  The context-dependent character string as described above has different morphological analysis results depending on the context before and after the character string, and it is difficult to obtain a correct analysis result only with the context-dependent character string.

Next, an example of morphological analysis processing according to the embodiment will be described.
FIG. 5 is a diagram illustrating morphological analysis processing according to the embodiment.

  FIG. 5 illustrates a case where a morphological analysis of “very good reputation” is performed as the input sentence 431. Further, it is assumed that the context independent dictionary 421 includes a character string = “very good reputation” and a morpheme string = “very | reputation | is | good”.

  The context-independent character string analysis unit 311 performs analysis by the longest match method using the context-independent dictionary 421 for the input sentence = “very good reputation”. In FIG. 5, “very good reputation” in the input sentence = “very good reputation” matches the character string of the context independent dictionary 421.

  Therefore, the result of the morphological analysis of “very good reputation” in the input sentence = “very good reputation” is “very | reputation | is good”.

  Next, the context-dependent character string analysis unit 321 performs morphological analysis of the remaining character strings that have not been analyzed by the context-independent character string analysis unit 311 in the input sentence. That is, the context-dependent character string analysis unit 321 performs a morphological analysis of the remaining character string = “translation” which is not analyzed by the context-independent character string analysis unit 311 among the input sentence = “very popular”. Do.

  The lattice constructing unit 322 constructs a lattice of the remaining (unanalyzed) character string = “translation” and the analyzed character string before and after that = “very good reputation”.

  The morpheme string selection unit 323 selects a word sequence (path) that is most likely to be a sentence in the constructed lattice. As a result, the analysis result of the unparsed character string = “Daida” becomes “Daida”.

  As described above, the result of the morphological analysis of the input sentence = “very good reputation” becomes “very | reputation | is |

  Next, a case where the character string included in the input sentence is not included in the context independent dictionary 421 will be described. The case where the size of the corpus used for the context independent dictionary 421 is small and the character string = “very popular” is not included in the context independent dictionary 421 will be described.

FIG. 6 is a diagram illustrating morphological analysis processing according to the embodiment.
FIG. 6 illustrates a case where the morphological analysis of “very good reputation” is performed as the input sentence 431 as in FIG. 5. In addition, the context independent dictionary 421 does not include the character string = “very popular”.

  The context-independent character string analysis unit 311 performs analysis by the longest match method using the context-independent dictionary 421 for the input sentence = “very good reputation”. In FIG. 6, in the input sentence = “very popular”, the matching character string is not included in the context independent dictionary 421.

  Therefore, any character string of the input sentence = “it is very popular” is not analyzed by the context independent character string analysis unit 311.

  Next, the context-dependent character string analysis unit 321 performs morphological analysis of the remaining character strings that have not been analyzed by the context-independent character string analysis unit 311 in the input sentence. That is, the context-dependent character string analysis unit 321 performs a morphological analysis of input sentence = “very good reputation”.

  The lattice constructing unit 322 constructs a lattice of the remaining (unanalyzed) character string = “very popular”.

  The morpheme string selection unit 323 selects a word sequence (path) that is most likely to be a sentence in the constructed lattice. As a result, the analysis result of the unanalyzed character string = “very good reputation” becomes “very | reputation | is good”.

  As described above, the result of the morphological analysis of the input sentence = “very good reputation” becomes “very | reputation | is |

  Thus, even when the character string included in the input sentence is not included in the context independent dictionary 421, the morphological analysis can be performed correctly.

FIG. 7 is a flowchart of context-independent dictionary generation processing according to the embodiment.
Here, it is assumed that the corpus 401 includes sentences s _i (i = 0 to N). In the embodiment, sentences s ₁ , s ₂ , s ₁₂ , s ₁₅ , s ₂₀ , s ₃₀ , and s ₃₅ are shown below.
Sentence s ₁ = “The Asahi Shimbun Tokyo headquarters reported to the Tsukiji station that“ two suspicious cardboard boxes were sent by courier ”. "
Sentence s ₂ = “Because it is necessary to do so.”
Sentence s ₁₂ = “The person in charge answered the Asahi Shimbun.”
Sentence s ₁₅ = “Let's take care of our body.”
Sentence ₂₀ = “The Asahi Shimbun Tokyo headquarters is in front of Tsukiji market station on the Oedo Line.”
Sentence s ₃₀ = “Head office has decision-making power.”
Sentence ₃₅ = “The publisher, Japan, is responsible.”
In addition, i of the sentence _{s i} is the sentence ID of the sentence _{s i.}

  Step S501 is the beginning of a loop corresponding to the end of step S506. The initial value of the variable i is 0, the condition for executing the loop is that i is N or less, and i is incremented by 1 at the end of the loop.

In step S502, the morphological analysis unit 221 reads the corpus 401, performs morphological analysis of a sentence _{s i} included in the corpus 401. For example, the morphological analysis unit 221 constructs a lattice for the sentence s _i and performs morphological analysis. A morpheme sequence s ′ _i is a morpheme sequence that is a result of the morpheme analysis on the sentence s _i . As a result of morphological analysis of sentence s ₁ , s ′ ₁ is s ′ ₁ = “Asahi | newspaper | Tokyo | head office ||||| In addition, morpheme string s 'i of _i is, morpheme string s' the statement ID of _i.

In step S503, it is the beginning of a loop corresponding to the end of step S505. The dependency determination unit 231 selects one unselected continuous partial morpheme sequence from among the continuous partial morpheme sequences included in the morpheme sequence s ′ _i . The selected partial morpheme string n is expressed as n = (character string p, morpheme string m, sentence ID). The character string p is a character string obtained by connecting the morpheme strings m, the morpheme string m is a morpheme string constituting the selected partial morpheme string, and the sentence ID is a morpheme string s ′ including the selected partial morpheme string n. _This is the sentence ID of _i or sentence s _i . For example, n = (Asahi Shimbun, Asahi | Newspaper, 1). Further, in step S504 where n = (newspaper Tokyo headquarters, newspaper | Tokyo | headquarters, 1), the dependency determining unit 231 sets the array of morpheme strings m and sentence IDs to T [p]. M, T [p]. Save each to H. For example, if the character string p = “Asahi Shimbun”, T [Asahi Shimbun]. M = [Asahi | newspaper], T [Asahi newspaper]. H = [1, 12, ...]. When the character string p = “body”, T [body]. M = [from |, body (body)], T [body]. H = [2,15,-]. That is, the result of the morphological analysis of the character string = “body” is “from |” or “body (body)”. Also, T [Body]. H = [2, 15, ...] indicates that the character string = “body” is included in the sentences s ₂ and s ₁₅ .

In step S505, it is the end of the loop corresponding to the start of step S503. If all the continuous partial morpheme sequences included in the morpheme sequence s ′ _i have been selected, the control proceeds to step S506. If there is an unselected continuous partial morpheme sequence in the morpheme sequence s ′ _i , the control proceeds to step S503. Return.

  Step S506 is the end of the loop corresponding to the start of step S501. If i is larger than N, the process proceeds to step S507. If i is N or less, i is incremented by 1, and the control returns to step S501.

  In step S507, it is the beginning of the loop corresponding to the end of step S516. The dependency determination unit 231 selects one unselected character string p from the character strings p in the array T [p]. Hereinafter, it is assumed that the character string p in steps S508 to S515 is the selected character string p.

  In step S508, the dependency determining unit 231 determines that the array T [p]. Number of elements of M | T [p]. It is determined whether M | is 1. Array T [p]. If the number of elements of M is 1, control proceeds to step S510, where T [p]. If the number of elements of M is other than 1, control proceeds to step S509. For example, when the character string p = “body”, T [body]. Since M = [from |, body (body)], | T [p]. M | = 2, and control proceeds to step S509. For example, if the character string p = “Tokyo head office is“ courier ””, T [Tokyo head office is “courier”. M = [Tokyo head office is “courier”], so | T [p] .M | Then, control proceeds to step S510, where it is checked whether there are a plurality of morphological analysis results of the character string p, in other words, whether the morphological analysis of the character string p is always the same.

In step S509, the character string p is discarded.
In step S510, the dependency determining unit 231 determines whether the array T [p]. Number of elements of H | T [p]. It is determined whether H | Array T [p]. If the number of elements of H is greater than 1, control proceeds to step S512 where T [p]. If the number of elements of H is 1 or less, control proceeds to step S511. For example, when the character string p = “Tokyo head office is“ courier service ”, T [Tokyo head office is“ courier service ”.H = [1], and therefore | T [p] .H | = 1. The process proceeds to step S511. For example, if the character string p = “Asahi Shimbun Tokyo head office”, T [Asahi Shimbun Tokyo head office]. Since H = [1, 12, 30], | T [p]. H | = 3, and control proceeds to step S512.

In step S511, the character string p is discarded.
In step S512, the dependency determination unit 231 obtains H _p ′ that is the sentence ID of the sentence set including the character string p. For example, when the character string p = “head office is”, the sentences including “head office is” are the sentences s ₁ , s ₂₀ , s ₃₀ , s ₃₅ , and thus H _p ′ = ₁ , ₂₀ , ₃₀ , ₃₅ It becomes.

In step S513, the dependency determining unit 231 determines that the array T [p]. It is determined whether H and the sentence set H _p ′ are equal. Array T [p]. If H and sentence set H _p ′ are equal, control proceeds to step S515 and array T [p]. If H and sentence set H ′ are not equal, control proceeds to step S514. For example, when the character string p = “head office is”, the array T [p]. H = [1, 12, 30], H _p ′ = 1, 20, 30, 35, and the array T [p]. Since H and H _p ′ are not equal, control proceeds to step S514. In step S513, it is detected whether the boundary of the morpheme string may be different.

In step S514, the character string p is discarded.
In step S515, the character string p and the morpheme string that is the analysis result of the character string p are registered in the context independent dictionary 421. When the character string p = “Asahi Shimbun Tokyo head office”, T [Asahi Shimbun Tokyo head office]. M = [Asahi | Newspaper | Tokyo | Headquarters], T [Asahi Shimbun Tokyo Headquarters]. H = [1, 12, 30], and the character string p = “Asahi Shimbun Tokyo head office” and the morpheme string = “Asahi | newspaper | Tokyo | head office” are registered in the context independent dictionary 421.

In step S516, the end of the loop corresponding to the start of step S507.
FIG. 8 is a flowchart of morpheme analysis processing according to the embodiment.

  In step S601, the context independent character string analysis unit 311 reads the input sentence 431. Characters included in the input sentence 431 are expressed as c0, c1,..., CN in order from the top. Further, it is assumed that the variable i = 0. In the embodiment, it is assumed that the input sentence = “Asahi Shimbun Tokyo headquarters is“ suspicious cardboard box by courier ”.

  In step S602, the context-independent character string analysis unit 311 performs pattern matching between the input sentence 431 and the context-independent dictionary 421, and detects a character string that matches the character string included in the context-independent dictionary 421 from the input sentence 431. Specifically, the context-independent character string analysis unit 311 searches the context-independent dictionary 421 and searches for the longest character strings ci to cj that start with ci matching the character string included in the context-independent dictionary 421.

  For example, when i = 0, c0 to c7 = “Asahi Shimbun Tokyo head office”. When i = 8, there is no matching character string. When i = 9, c9 to c11 = “home delivery”. When i = 12, there is no matching character string. When i = 13, c13 to c22 = “Suspicious cardboard box”.

  In step S603, the context-independent character string analysis unit 311 checks whether there is a character string that matches the character string starting with ci in the context-independent dictionary 421. If there is a matching character string in the context independent dictionary 421, the control proceeds to step S605, and if there is no matching character string in the context independent dictionary 421, the control proceeds to step S604.

  In step S604, the context-independent character string analysis unit 311 increments the variable i by 1.

  In step S605, the context-independent character string analysis unit 311 sets the variable i to j + 1. For example, when i = 0, as described in step S602, c0 to c7 = “Asahi Shimbun Tokyo head office” and j = 7, so i is set to 8 (= 7 + 1).

  In step S606, the context-independent character string analysis unit 311 stores the analysis results for the character strings ci to cj in the storage unit 401 as the analysis results 441. For example, when i = 0, c0 to c7 = analysis result for “Asahi Shimbun Tokyo head office” = “Asahi | newspaper | Tokyo | head office” is stored in the storage unit 401 as an analysis result 441. When i = 9, c9 to c11 = “analysis result for“ home delivery ”=“ “| home delivery” is stored in the storage unit 401 as the analysis result 441. When i = 13, the analysis result for c13 to c22 = “and suspicious corrugated cardboard box” = “de | suspicious | corrugated cardboard | box |” is stored in the storage unit 401 as the analysis result 441.

  In step S607, the context-independent character string analysis unit 311 determines whether the variable i is greater than N. If the variable i is greater than N, the control proceeds to step S608, and if the variable i is N or less, the control returns to step S602.

In step S608, _{it s 0,} s 1 string unparsed context independent string analysis unit 311 of the input sentence 431, ~, and _{s M.} Further, it is assumed that the variable k = 0. In the embodiment, the input sentence = “Asahi Shimbun Tokyo head office is“ suspicious cardboard box by courier ”,“ Asahi Shimbun Tokyo head office ”,“ Home delivery ”, and“ Suspicious cardboard box is analyzed ” Therefore, the unanalyzed character string is s ₀ = “ga” and s ₁ = “stool”.

In step S609, the lattice constructing unit 322 constructs a lattice for the character string s _k and the analyzed morphemes before and after the character string s _k using a dictionary including a plurality of words. FIG. 9 shows a lattice of the character string s ₀ = “ga” and the analyzed morphemes before and after that. FIG. 10 shows the lattice of the character string s ₁ = “stool” and the analyzed morphemes before and after the character string s ₁ = “stool”.

In step S610, the morpheme sequence selection unit 323 selects a word sequence (path) that is most likely to be a sentence in the constructed lattice. The morpheme sequence selection unit 323 selects a path that minimizes the evaluation value using, for example, the Viterbi algorithm. For example, “ga (particle)” is selected as the analysis result for the character string s ₀ = “ga”. FIG. 11 shows a morpheme string including an analysis result of the character string s ₀ = “ga” and an analyzed morpheme before and after the character string s ₀ . For example, for the character string s ₁ = “stool”, “stool (bottle)” is selected as the analysis result. FIG. 12 shows an analysis result of the character string s ₁ = “feces” and a morpheme string including the analyzed morpheme before and after the character string s ₁ .

  In step S611, the morpheme string selection unit 323 increments the variable k by 1.

  In step S612, the morpheme string selection unit 323 determines whether the variable k is greater than M. If the variable k is larger than M, the control proceeds to step S613, and if the variable k is equal to or smaller than M, the control returns to step S609.

In step S613, stored in the storage unit 401 the result of the morphological analysis for a character string Sl to S _M as an analysis result 441.

  As in the morpheme analysis process shown in FIG. 8, morpheme analysis is not performed for each unanalyzed character string for an unanalyzed character string, but using a lattice of the entire input sentence including all unanalyzed character strings. Analysis may be performed.

FIG. 13 is a flowchart of a modification of the morphological analysis process according to the embodiment.
As in the morphological analysis process of FIG. 8, it is assumed that the input sentence = “Asahi Shimbun Tokyo head office is“ suspicious cardboard box by courier ”.

  The processes in steps S1601 to S1608 are the same as the processes in steps S601 to S607 in FIG.

In step S1609, the lattice constructing unit 322 constructs a lattice for the character string s _k and the analyzed morphemes before and after the character string s _k using a dictionary including a plurality of words.

  In step S1610, the morpheme string selection unit 323 increments the variable k by 1.

In step S <b> 1611, the morpheme string selection unit 323 determines whether the variable k is larger than M. If the variable k is larger than M, the control proceeds to step S1612. If the variable k is equal to or smaller than M, the control returns to step S1609. In the embodiment, when the variable k is larger than M, a lattice of the entire input sentence including the unparsed character string s ₀ = “ga” and s ₁ = “feces” as shown in FIG. 14 is constructed.

In step S <b> 1612, the morpheme string selection unit 323 selects a word sequence (path) that seems to be most likely as a sentence in the constructed lattice. The morpheme sequence selection unit 323 selects a path that minimizes the evaluation value using, for example, the Viterbi algorithm. For example, “ga (particle)” is selected as the analysis result for the character string s ₀ = “ga”. For example, for the character string s ₁ = “stool”, “stool (bottle)” is selected as the analysis result. FIG. 15 shows a morpheme string of the entire input sentence including the analysis result of the character string s ₀ = “ga” and s ₁ = “feces”. By constructing a lattice of the entire input sentence and performing morpheme analysis, the accuracy can be improved compared to performing morpheme analysis from only the unparsed character string and the lattice of the morpheme strings before and after that as shown in FIGS.

In step S613, stored in the storage unit 401 the result of the morphological analysis for a character string Sl to S _M as an analysis result 441.

  According to the morphological analysis device of the embodiment, morphological analysis is performed by pattern matching, and a lattice is constructed and analyzed for text that does not match pattern matching, thereby speeding up while maintaining the accuracy of morphological analysis it can.

  According to the morphological analysis device of the embodiment, the lattice is constructed only for the text that does not match the pattern matching and the morphological analysis is performed. Therefore, the lattice of the entire text to be analyzed is constructed and the morphological analysis is performed. Compared with, the calculation cost can be reduced.

FIG. 16 is a configuration diagram of the information processing apparatus.
The morphological analysis apparatus 101 in FIG. 2 can be realized using, for example, an information processing apparatus (computer) 10 as shown in FIG.

  The information processing apparatus in FIG. 16 includes a central processing unit (CPU) 1, a memory 2, an input device 3, an output device 4, an auxiliary storage device 5, a medium driving device 6, and a network connection device 7. These components are connected to each other by a bus 8.

  The memory 2 is a semiconductor memory such as a read only memory (ROM), a random access memory (RAM), or a flash memory, for example. The memory 2 stores a program and data for morphological analysis processing. The memory 2 can be used as the storage unit 401.

  The CPU 1 (processor) executes, for example, a program using the memory 2 to thereby execute a context independent dictionary construction unit 211, a morpheme analysis unit 221, a dependency determination unit 231, a context independent character string analysis unit 311, and a lattice construction unit. 322 and the morpheme string selection unit 323 operate.

  The input device 3 is, for example, a keyboard, a pointing device, and the like, and is used for inputting instructions and information from a user or an operator. The output device 4 is, for example, a display device, a printer, a speaker, and the like, and is used for outputting an inquiry to a user or an operator and a processing result. The processing result may be a result of morphological analysis.

  The auxiliary storage device 5 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, or the like. The auxiliary storage device 5 may be a hard disk drive or a flash memory. The information processing apparatus can store programs and data in the auxiliary storage device 5 and load them into the memory 2 for use. The auxiliary storage device 5 can be used as the storage unit 401.

  The medium driving device 6 drives a portable recording medium 9 and accesses the recorded contents. The portable recording medium 9 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, or the like. The portable recording medium 9 may be a compact disk read only memory (CD-ROM), a digital versatile disk (DVD), a universal serial bus (USB) memory, or the like. A user or an operator can store programs and data in the portable recording medium 9 and load them into the memory 2 for use.

  As described above, the computer-readable recording medium for storing the program and data is a physical (non-transitory) recording medium such as the memory 2, the auxiliary storage device 5, and the portable recording medium 9.

  The network connection device 7 is a communication interface that is connected to a communication network such as a local area network (LAN) or the Internet and performs data conversion accompanying communication. The information processing apparatus can receive a program and data from an external apparatus via the network connection apparatus 7 and load them into the memory 2 for use.

  The information processing apparatus can also receive instructions and information from the user terminal via the network connection apparatus 7, perform morphological analysis processing, and transmit the processing result to the user terminal.

  Note that the information processing apparatus does not have to include all the components illustrated in FIG. 16, and some of the components may be omitted depending on the application and conditions. For example, the input device 3 may be omitted when an instruction or information is not input from the user or operator, and the output device 4 is omitted when an inquiry or processing result is not output to the user or operator. May be. When the information processing apparatus does not access the portable recording medium 9 or the communication network, the medium driving device 6 or the network connection device 7 may be omitted.

Regarding the above embodiment, the following additional notes are disclosed.
(Appendix 1)
A storage unit for storing a matching dictionary including a morpheme analysis dictionary, a character string included in each of a plurality of sentences, and a first morpheme analysis result of the character string obtained in common for each of the plurality of sentences; The computer includes the output of the first morpheme analysis result for the character string that matches the character string included in the matching dictionary in the analysis target text,
Among the analysis target text, for the remaining character strings that do not match the character strings included in the matching dictionary, using the morphological analysis dictionary, generate a lattice including a plurality of morphological analysis result candidates,
Performing morphological analysis on the remaining character string using the lattice, and outputting a second morphological analysis result on the remaining character string;
A morphological analysis program that executes processing.
(Appendix 2)
Performing a morphological analysis of the plurality of sentences, and causing the computer to further execute a process of registering the character strings in the matching dictionary when all the morphological analysis results of the character strings included in the plurality of sentences are the same. The morphological analysis program according to attachment 1.
(Appendix 3)
The matching dictionary includes a plurality of character strings and a plurality of morphological analysis results of the plurality of character strings, and the plurality of character strings include a character string included in each of the plurality of sentences and another character string. The plurality of morpheme analysis results include the first morpheme analysis result and the morpheme analysis result of the other character string,
Among the analysis target text, for the plurality of character strings that match each of the plurality of character strings included in the matching dictionary, the plurality of morpheme analysis results are output,
Of the analysis target text, for the remaining character strings that do not match the plurality of character strings included in the matching dictionary, the lattice is generated using the morphological analysis dictionary, and the plurality of characters using the lattice The morpheme analysis program according to appendix 1, further causing the computer to execute a process for performing a morphological analysis on the remaining character string that does not match the string.
(Appendix 4)
A storage unit for storing a morphological analysis dictionary, a matching dictionary including a character string included in each of a plurality of sentences, and a first morphological analysis result of the character string obtained in common for each of the plurality of sentences; ,
A first analysis unit that outputs the first morpheme analysis result for a character string that matches the character string included in the matching dictionary in the analysis target text;
Using the morpheme analysis dictionary, a lattice including a plurality of morpheme analysis result candidates is generated for the remaining character strings that do not match the character strings included in the matching dictionary in the analysis target text, and the lattice A second analysis unit that performs morpheme analysis on the remaining character string using and outputs a second morpheme analysis result on the remaining character string;
A morphological analyzer comprising:
(Appendix 5)
Performing morphological analysis of the plurality of sentences, and when the morphological analysis results of the character strings included in each of the plurality of sentences are all the same, a dictionary generation unit that registers the character strings in the matching dictionary;
The morphological analysis device according to appendix 4, further comprising:
(Appendix 6)
The matching dictionary includes a plurality of character strings and a plurality of morphological analysis results of the plurality of character strings, and the plurality of character strings include a character string included in each of the plurality of sentences and another character string. The plurality of morpheme analysis results include the first morpheme analysis result and the morpheme analysis result of the other character string,
The first analysis unit outputs the plurality of morpheme analysis results for a plurality of character strings that match each of the plurality of character strings included in the matching dictionary in the analysis target text,
The second analysis unit generates the lattice using the morphological analysis dictionary for the remaining character strings that do not match the plurality of character strings included in the matching dictionary in the analysis target text, and The morpheme analyzer according to appendix 4, wherein morpheme analysis is performed on the remaining character strings that do not match the plurality of character strings using a lattice.
(Appendix 7)
A storage unit for storing a matching dictionary including a morpheme analysis dictionary, a character string included in each of a plurality of sentences, and a first morpheme analysis result of the character string obtained in common for each of the plurality of sentences; The morpheme analyzer comprising the output of the first morpheme analysis result for the character string that matches the character string included in the matching dictionary in the analysis target text,
Among the analysis target text, for the remaining character strings that do not match the character strings included in the matching dictionary, using the morphological analysis dictionary, generate a lattice including a plurality of morphological analysis result candidates,
Performing morphological analysis on the remaining character string using the lattice, and outputting a second morphological analysis result on the remaining character string;
A morphological analysis method having processing.
(Appendix 8)
The appendix 7 further includes a process of performing morphological analysis of the plurality of sentences and registering the character strings in the matching dictionary when the morphological analysis results of the character strings included in the plurality of sentences are all the same. Morphological analysis method.
(Appendix 9)
The matching dictionary includes a plurality of character strings and a plurality of morphological analysis results of the plurality of character strings, and the plurality of character strings include a character string included in each of the plurality of sentences and another character string. The plurality of morpheme analysis results include the first morpheme analysis result and the morpheme analysis result of the other character string,
Among the analysis target text, for the plurality of character strings that match each of the plurality of character strings included in the matching dictionary, the plurality of morpheme analysis results are output,
Of the analysis target text, for the remaining character strings that do not match the plurality of character strings included in the matching dictionary, the lattice is generated using the morphological analysis dictionary, and the plurality of characters using the lattice The morpheme analysis method according to appendix 7, further comprising a process of performing a morpheme analysis on the remaining character string that does not match the string.

DESCRIPTION OF SYMBOLS 101 Morphological analyzer 201 Dictionary generation part 211 Context independent dictionary construction part 221 Morphological analysis part 231 Dependency determination part 301 Morphological analysis part 311 Context independent character string analysis part 321 Context dependent character string analysis part 322 Lattice construction part 323 Morphological string selection part 401 storage unit 411 corpus 421 context independent dictionary 431 input sentence 441 analysis result

Claims (5)

A storage unit for storing a matching dictionary including a morpheme analysis dictionary, a character string included in each of a plurality of sentences, and a first morpheme analysis result of the character string obtained in common for each of the plurality of sentences; The computer includes the output of the first morpheme analysis result for the character string that matches the character string included in the matching dictionary in the analysis target text,
Among the analysis target text, for the remaining character strings that do not match the character strings included in the matching dictionary, using the morphological analysis dictionary, generate a lattice including a plurality of morphological analysis result candidates,
Performing morphological analysis on the remaining character string using the lattice, and outputting a second morphological analysis result on the remaining character string;
A morphological analysis program that executes processing.   Performing a morphological analysis of the plurality of sentences, and causing the computer to further execute a process of registering the character strings in the matching dictionary when all the morphological analysis results of the character strings included in the plurality of sentences are the same. The morphological analysis program according to claim 1. The matching dictionary includes a plurality of character strings and a plurality of morphological analysis results of the plurality of character strings, and the plurality of character strings include a character string included in each of the plurality of sentences and another character string. The plurality of morpheme analysis results include the first morpheme analysis result and the morpheme analysis result of the other character string,
Among the analysis target text, for the plurality of character strings that match each of the plurality of character strings included in the matching dictionary, the plurality of morpheme analysis results are output,
Of the analysis target text, for the remaining character strings that do not match the plurality of character strings included in the matching dictionary, the lattice is generated using the morphological analysis dictionary, and the plurality of characters using the lattice The morpheme analysis program according to claim 1, further causing the computer to execute a process of performing a morphological analysis on a remaining character string that does not match a string. A storage unit for storing a morphological analysis dictionary, a matching dictionary including a character string included in each of a plurality of sentences, and a first morphological analysis result of the character string obtained in common for each of the plurality of sentences; ,
A first analysis unit that outputs the first morpheme analysis result for a character string that matches the character string included in the matching dictionary in the analysis target text;
Using the morpheme analysis dictionary, a lattice including a plurality of morpheme analysis result candidates is generated for the remaining character strings that do not match the character strings included in the matching dictionary in the analysis target text, and the lattice A second analysis unit that performs morpheme analysis on the remaining character string using and outputs a second morpheme analysis result on the remaining character string;
A morphological analyzer comprising: A storage unit for storing a matching dictionary including a morpheme analysis dictionary, a character string included in each of a plurality of sentences, and a first morpheme analysis result of the character string obtained in common for each of the plurality of sentences; The morpheme analyzer comprising the output of the first morpheme analysis result for the character string that matches the character string included in the matching dictionary in the analysis target text,
Among the analysis target text, for the remaining character strings that do not match the character strings included in the matching dictionary, using the morphological analysis dictionary, generate a lattice including a plurality of morphological analysis result candidates,
Performing morphological analysis on the remaining character string using the lattice, and outputting a second morphological analysis result on the remaining character string;
A morphological analysis method having processing.

Images