JP6145027B2 - Model learning device, morphological analysis device, and program - Google Patents

Description

  The present invention relates to a model learning device, a morphological analysis device, and a program, and in particular, a model learning device, a morphological analysis device, and a program for robustly analyzing a broken notation that does not appear in a normal writing method such as colloquial tone About.

  As shown in FIG. 24, a conventional Japanese morpheme analyzer simultaneously performs input sentence segmentation (word division) and part-of-speech assignment using a morpheme connection probability (cost) and occurrence probability (cost) ( Select the combination with the lowest total cost.)

  Moreover, there is a technique called transliteration as a technique used when estimating the correspondence between character conversions between different character types and the correspondence between kanji and reading. As shown in FIG. 25, transliteration is a technique for obtaining the correspondence between two character strings. For example, various extension techniques such as estimating the correspondence of each character by using dynamic programming. Has been proposed (Non-Patent Document 1).

  Conventionally, in order to cope with the broken notation, a character string conversion rule as shown in FIG. 26 is manually created and incorporated into morphological analysis. Here, “broken notation” refers to notation that does not appear in the orthography of newspapers, such as over-spoken words or web-specific written words, as shown in FIG. Specifically, manual maintenance is performed for characteristic patterns such as lowercase letters (a → a, i → i), long sound (a → −, u → −), and rewriting of input sentences and dictionaries in morphological analysis. Pull extension is performed (Non-Patent Document 2, Non-Patent Document 3).

  The expansion of dictionary lookup is to perform dictionary lookup in consideration of a case where the input character string is changed to another character string. For example, for an input sentence “light-light”, the adjective “lightly” is not usually enumerated because the character strings do not match, but when the rule “delete“-”” is considered, “light-light” And “lightly” both strings are dictionary-drawn into one lattice. In this case, the morpheme “light” is also listed, so that a lattice as shown in FIG. 28 is generated. Here, the lattice is “a graph structure of a set of matched morphemes” as a result of performing dictionary lookup on the input character string. An example is shown in FIG.

Kuniko Saito, Akio Shinohara, Masaaki Nagata, Ei Ohara, “VCWeb's English-Japanese Seamless Web Browser” (2002), Journal of the Japanese Society for Artificial Intelligence, pp.343-347 Kenta Katsuki, Shinpei Kanno, Daisuke Kawahara, Ikuo Kurohashi, “Robust Morphological Analysis for Various Language Variations on the Web”, (2011), Proc. Of the 17th Annual Conference Teruoka Oka, Mamoru Komachi, Tomonobu Ogi, Yuji Matsumoto, “Morphological Analysis of Modern Japanese Considering Variations of Notation”, (2013), Proc.

  In the conventional Japanese morphological analyzer, the analysis is performed based on the dictionary lookup. Therefore, there is a high probability that the analysis failure occurs when a word that does not exist in the dictionary appears.

  In addition, the conventional methods for dealing with corrupted notations have a variety of collapse patterns, which cannot be covered by manual maintenance and have a low recall rate. Further, although the collapse probability varies depending on the phenomenon, the conventional method has a problem that the same cost is given to all the collapse patterns.

  In the present invention, a morpheme analysis apparatus and a program capable of accurately performing morpheme analysis even on a character string including a broken word that is a distorted notation are made to solve the above problems. The purpose is to provide.

  In addition, it is possible to provide a model learning apparatus and a program that can learn a model that can accurately perform morphological analysis even on a character string that includes a broken word that is a fuzzy notation.

  In order to achieve the above object, a model learning device according to a first aspect of the present invention includes a plurality of regular words that are input normalized expressions and corrupted words that are notations that fluctuate with respect to the regular words. Based on a pair, for each of the plurality of pairs, each of a regular phrase that is a partial character string in which the regular word is separated at a character string delimiter position, and a partial character in which the collapsed word is delimited at a character string delimiter position Based on the correspondence relationship between each of the plurality of pairs obtained by the phrase alignment unit and the phrase alignment unit obtained by the phrase alignment unit, the optimum correspondence relationship with each of the collapsed phrases that are columns, A conversion probability calculation unit that calculates a conversion probability that the regular phrase is converted into the collapsed phrase for each pair of the regular phrase and the collapsed phrase. It is.

  In the model learning device according to the first aspect of the present invention, initializing a conversion probability table storing the conversion probabilities for each pair of the regular phrase and the collapsed phrase based on the plurality of input pairs. A setting unit and an iterative determination unit that repeats the processing by the phrase alignment unit and the calculation by the conversion probability calculation unit until a predetermined repetition end condition is satisfied, and outputs the conversion probability table as a normal collapsed phrase model. Further, the phrase alignment unit obtains an optimal correspondence for each of the plurality of pairs based on the conversion probability table according to dynamic programming, and the conversion probability calculation unit includes the regular phrase and the For each pair of collapsed phrases, calculate the conversion probability and update the conversion probability table It may be.

  The model learning device according to the second aspect of the present invention is based on at least one of a character n-gram composed of n characters and a word surface layer n-gram composed of n words based on a plurality of inputted corpora. A statistic calculation unit for calculating a statistic, a plurality of pairs of a normal word that is an input normalized expression and a broken word that is a distorted expression with respect to the regular word, and the broken word as a character string Based on a pre-determined regular collapsible phrase model for converting a collapsed phrase that is a partial character string delimited by a delimiter position into a regular phrase that is a partial character string delimited by the delimiter position of the character string For each of the plurality of pairs, the collapsed phrase included in the collapsed word of the pair is converted into the regular phrase, and the converted regular phrase and the regularized pair Statistic calculated for at least one of the character n-gram and the word surface layer n-gram corresponding to the collapsed phrase, and the character n-gram and the word surface layer n- corresponding to the regular phrase A data generation unit that generates learning data that is one of positive example data and negative example data, including a statistic calculated for at least one of the gram, and generated for each of the plurality of pairs by the data generation unit A learning unit that learns a phrase filter model for determining the likelihood of converting the collapsed phrase into the regular phrase based on the learned data.

  The morpheme analyzer according to the third aspect of the present invention provides a normal phrase that is a normalized expression that is a partial character string that is a partial character string delimited by a character string delimiter, and that is a notation that is a distorted expression with respect to the normal word. A partial character string that matches the collapsed phrase with respect to the input character string, based on a normal disrupted phrase model obtained in advance for converting a broken phrase that is a partial character string obtained by dividing a word at a character string separation position. For each of the character string of the analysis candidate generated by the analysis candidate generation unit that generates a character string replaced with the corresponding regular phrase as an analysis candidate, and the input character string and the analysis candidate generation unit This is a graph structure consisting of edges that connect nodes corresponding to each partial character string to which parts of speech are assigned and nodes corresponding to connected partial character strings. A lattice generation unit that generates a lattice, a selection unit that selects an optimal path connecting the nodes according to dynamic programming based on the lattice generated in the lattice generation unit, and outputs the result as an analysis result; It is comprised including.

  The morpheme analyzer according to the fourth aspect of the present invention provides a normal phrase that is a normalized expression that is a partial character string that is a partial character string that is delimited at a character string delimiter position, and is a notation that is a distorted expression with respect to the normal word. A partial character string that matches the collapsed phrase with respect to the input character string, based on a normal disrupted phrase model obtained in advance for converting a broken phrase that is a partial character string obtained by dividing a word at a character string separation position. For each of the analysis candidate generated by the analysis candidate generation unit and the analysis candidate generation unit that generates a character string in which the corresponding character string is replaced with the corresponding normal phrase, the collapsed phrase is converted into the normal phrase A calculation unit that calculates the likelihood of the analysis candidate based on a phrase filter model obtained in advance for determining the likelihood of the analysis An analysis candidate that extracts the likely analysis candidate from the analysis candidate generated by the analysis candidate generation unit based on the likelihood of the analysis candidate calculated by the calculation unit and a predetermined threshold value A dictionary for each of the character strings of the extraction candidates, the input character strings and the analysis candidate character strings extracted by the analysis candidate extraction unit, and nodes corresponding to the partial character strings to which parts of speech are assigned; A lattice generation unit that generates a lattice that is a graph structure composed of edges connecting nodes corresponding to connected partial character strings, and the nodes according to dynamic programming based on the lattice generated by the lattice generation unit And a selection unit that selects an optimum route connecting the two and outputs the result as an analysis result.

  Moreover, the program of this invention is a program for functioning a computer as each part which comprises said model learning apparatus and morphological analyzer.

  As described above, according to the morphological analysis device and the program of the present invention, it is possible to accurately perform morphological analysis even on a character string including a broken word that is a distorted notation.

  Further, according to the model learning apparatus and the program of the present invention, it is possible to learn a model that can accurately perform morphological analysis even on a character string that includes a broken word that is a fuzzy notation.

It is a figure which shows the example of a broken word and a regular word. It is a figure which shows the example of a regular phrase and a collapse phrase. It is a block diagram which shows the functional structure of the model learning apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the example of a regular word breaking word pair. It is a figure which shows the example of the table used when searching the path | route of the minimum cost based on a dynamic programming. It is a figure which shows the example of a cost function. It is a figure which shows the example of a regular collapse phrase model. It is a block diagram which shows the functional structure of the morphological analyzer which concerns on the 1st Embodiment of this invention. It is a figure which shows the example which produces an analysis candidate sentence. It is a figure which shows the example of a lattice. It is a flowchart which shows the regular break phrase model learning process routine in the model learning apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the phrase filter model learning process routine in the model learning apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the morpheme analysis process routine in the morpheme analyzer which concerns on the 1st Embodiment of this invention. It is a figure which shows the example of the result using the morpheme analyzer which concerns on the 1st Embodiment of this invention. It is a figure which shows the example of the result using the morpheme analyzer which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the functional structure of the model learning apparatus which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the functional structure of the morphological analyzer which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the phrase filter model learning process routine in the model learning apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the morpheme analysis process routine in the morpheme analyzer which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the functional structure of the model learning apparatus which concerns on the 3rd Embodiment of this invention. It is a block diagram which shows the functional structure of the morphological analyzer which concerns on the 3rd Embodiment of this invention. It is a block diagram which shows the functional structure of the model learning apparatus which concerns on the 4th Embodiment of this invention. It is a block diagram which shows the functional structure of the morphological analyzer which concerns on the 4th Embodiment of this invention. It is a figure which shows the example of the conventional Japanese morphological analyzer. It is a figure which shows the example of transliteration. This is an example of manually creating a character string conversion rule. It is a figure which shows the example of the notation which collapsed. It is a figure which shows the example of a lattice. It is a figure which shows the example of what made the matched morpheme set into the graph structure.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Principle of the invention>
In the present embodiment, the words “collapse phrase / regular phrase” and “collapse phrase / regular word” are used. As shown in FIG. 1, “collapsed word / regular word” is defined as a regular character string extracted by manual annotation and a unit of corrupted character string when the collapse model is estimated. In addition, as shown in FIG. 2, the unit of the partial character string in the broken word / normal word extracted by phrase alignment using the broken word / normal word pair (correct data) It is defined as

  As a basic solution policy, as shown in the following equation (1), by applying the probability that a regular phrase is broken and converted into a phrase to the objective function of conventional morphological analysis, the part of speech of the normal notation and the correct word break are obtained. Estimate at the same time.

  However, P (V | W) is a word conversion probability that the normal word V is converted into the observed word W, and is equal to the probability that the normal phrase is broken and converted into the phrase. Further, P (T) is a connection probability of the part T of regular notation part of speech, and P (W | T) is a normal probability of the observed word W when the part T of regular part of speech part is given. .

  Further, the word conversion probability is decomposed for each character string (phrase), and the word conversion probability is approximated by the product of the phrase conversion probabilities as shown in the following equation (2), and the phrase probability is obtained by a transliteration model.

<Configuration of Model Learning Device According to First Embodiment>
Next, the configuration of the model learning device according to the first embodiment of the present invention will be described. As shown in FIG. 3, the model learning device 100 according to the first embodiment of the present invention executes a CPU, a RAM, a normal collapse phrase model learning process routine and a phrase filter model learning process routine, which will be described later. And a ROM including a ROM storing various programs and various data. Functionally, the model learning apparatus 100 includes an input unit 10, a calculation unit 20, and an output unit 50 as shown in FIG.

  The input unit 10 is for phrase estimation in which a corresponding regular word is given to a broken word manually extracted from a sentence including a broken notation such as Twitter (registered trademark) or a blog as shown in FIG. Regular word break word pair data, which is correct answer data, is received. Further, the input unit 10 accepts a large number of document groups in a plurality of fields on the web and uses them as a corpus of a plurality of fields.

  The calculation unit 20 includes an initial setting unit 30, a phrase alignment unit 31, a conversion probability calculation unit 32, an iterative determination unit 34, a normal collapsed phrase model storage unit 36, a statistic calculation unit 42, and a model learning unit 44. And a phrase filter model storage unit 46.

  The initial setting unit 30 obtains all pairs of the regular phrase fv and the corrupted phrase fw based on the regular word corrupted word pair data received by the input unit 10, and randomly calculates the conversion probability P (fw | fv) for each pair. A conversion probability table that stores the conversion probabilities of pairs of regular phrases and collapsed phrases is generated and stored in a memory (not shown). Note that the conversion probability P (fw | fv) of each pair of the regular phrase fv and the collapsed phrase fw may be set heuristically.

  The phrase alignment unit 31 uses dynamic programming based on the conversion probability table generated in the initial setting unit 30 or updated in the conversion probability calculation unit 32 for each of the pairs included in the regular word broken word pair data. Find the optimum alignment, which is the best correspondence. Specifically, based on dynamic programming, a route as shown in FIG. 5 is used to search for a route with the lowest cost, and a correspondence between characters is obtained. The total cost of the route is represented by the sum of the corresponding costs of the character strings on the route. As the cost function, for example, a logarithm of probability values as shown in FIG. 6 is used. In this case, if the path with the lowest cost is path 1, the output alignment is (ka, ka), (na, na), (aa, null), (ri, ri). However, (null, x) represents insertion of x, and (y, null) represents deletion of y.

The conversion probability calculation unit 32 calculates the P in the t-th calculation according to the following equation (3) based on the optimum alignment of each pair included in the regular word collapsed word pair data obtained by the phrase alignment unit 31 at the t-th time. An expected value P _t (fw | fv) of (fw | fv) is obtained and stored in the memory. Then, the conversion probability table is stored in the update memory based on the obtained expected value P _t (fw | fv). However, N (fw, fv) represents the number of times that the normal phrase fv is converted into the broken phrase fw in the optimum alignment of the normal word broken word pair data.

The iterative determination unit 34 is based on the likelihood function based on P _t (fw | fv) calculated by the conversion probability calculation unit 32 and the previously calculated P _t−1 (fw | fv) stored in the memory. It is determined whether or not the difference from the likelihood function is equal to or less than a predetermined threshold value ε (for example, ε = 0.05). When the difference is less than or equal to the threshold ε, it is determined that the iteration end condition is satisfied, and the current conversion probability table as illustrated in FIG. 7 is stored in the storage unit 36 as a normal collapsed phrase model, and the output unit Output to 50. If the difference is larger than the threshold ε, it is determined that the iteration end condition is not satisfied, and the process of the phrase alignment unit 31 and the process of the conversion probability calculation unit 32 are repeated. The likelihood function is represented by the following equation (4). Further, the repetition end condition may be that the number of repetitions reaches the upper limit number.

  The regular broken phrase model storage unit 36 stores a regular broken phrase model.

  The statistic calculation unit 42 calculates the character n-gram statistic for all the character n-grams based on the corpus of a plurality of fields received by the input unit 10. For example, if the target document is “That's not done”, P (n | so), P (na | n), etc. are calculated as character 2 gram statistics, and as character 3 gram statistics, P (n | so, n), P (n | n, n), etc. are calculated. Note that the word surface n-gram statistics may be calculated for all word surface n-grams based on a plurality of corpora of morphological analysis processing.

  The data generation unit 43 generates a part of the corrupted word based on the normal corrupted phrase model stored in the normalized corrupted phrase model storage unit 36 for each of the pairs included in the normalized word corrupted word pair data received by the input unit 10. Of the character string, a partial character string corresponding to the collapse phrase is converted into a partial character string of a regular phrase corresponding to the collapse phrase. Then, it is determined whether or not the converted regular phrase is included in the regular word corresponding to the corrupted word, and if included, the character n-gram statistic corresponding to the regular phrase and the corrupted phrase are supported. Learning data including character n-gram statistics is generated as positive example data. If not included, character n-gram statistics corresponding to the regular phrase and character n-gram statistics corresponding to the collapsed phrase are included. Learning data is generated as negative example data.

  The model learning unit 44 performs identification learning using a support vector machine or the like based on learning data composed of positive example data and negative example data generated by the data generation unit 43, and converts a collapsed phrase into a regular phrase. A phrase filter model for determining the likelihood of the phrase is learned, stored in the phrase filter model storage unit 46, and output to the output unit 50.

  The phrase filter model storage unit 46 stores the phrase filter model learned by the model learning unit 44 and the character n-gram statistic calculated by the statistic calculation unit 42.

<Configuration of morphological analyzer according to the first embodiment>
Next, the configuration of the morphological analyzer according to the first embodiment of the present invention will be described. As shown in FIG. 8, the morpheme analyzer 200 according to the first embodiment of the present invention includes a CPU, a RAM, a ROM for storing a program and various data for executing a morpheme analysis processing routine to be described later, , Can be configured with a computer including. Functionally, the morphological analyzer 200 includes an input unit 210, a calculation unit 220, and an output unit 250 as shown in FIG.

  The input unit 210 receives an input sentence.

  The calculation unit 220 includes an analysis candidate generation unit 230, a denormalized phrase model storage unit 232, a dictionary database 233, a calculation unit 235, a phrase filter model storage unit 236, an analysis target extraction unit 238, and a lattice generation unit 239. And a selection unit 240.

  The analysis candidate generation unit 230 generates each analysis candidate sentence from the input sentence received by the input unit 10 based on the normal breakdown phrase model stored in the normal breakdown phrase model storage unit 232. Specifically, when the input sentence received by the input unit 210 includes a partial character string that matches the broken phrase included in the regular broken phrase model, the partial character string is converted into a regular phrase corresponding to the broken phrase. To generate an analysis candidate sentence. It is assumed that the analysis candidate sentence is changed only in one of the collapsed phrases included in the input sentence. If there are a plurality of collapsed phrases, an analysis candidate sentence is generated for each collapsed phrase. A specific example is shown in FIG. Moreover, when there are a plurality of regular phrases corresponding to the collapsed phrase, an analysis candidate sentence is generated for each regular phrase.

  The normal broken phrase model storage unit 232 stores the same normal broken phrase model as the normal broken phrase model learned by the model learning device 100.

  The dictionary database storage unit 233 stores a dictionary (reading, notation, part of speech, cost (occurrence probability)) and connection probability of each part of speech part necessary for performing morphological analysis.

  The phrase filter model storage unit 236 stores the same phrase filter model and character n-gram statistic as the phrase filter model and character n-gram statistic learned by the model learning device 100.

  The analysis target extraction unit 238 converts the input sentence, the phrase filter model and the character n-gram statistic stored in the phrase filter model storage unit 236 for each of the analysis candidate sentences generated by the analysis candidate generation unit 230. Based on this, it is determined whether or not the collapsed phrase is an analysis candidate sentence that has been correctly converted into a regular phrase. If it is determined that the collapsed phrase is an analysis candidate sentence that has been correctly converted, it is extracted as an analysis target sentence and converted correctly. If it is determined that the analysis candidate sentence has not been analyzed, the analysis candidate sentence is deleted.

  For example, the character n-gram statistic of the regular phrase converted from the broken phrase of the input sentence, the character n-gram statistic of the broken phrase of the input sentence, the phrase filter model stored in the phrase filter model storage unit 236, and Based on the above, a score indicating the likelihood of conversion of the collapsed phrase into the regular phrase is calculated, and if the calculated score is equal to or greater than a threshold, it is determined that the sentence has been correctly converted.

  The lattice generation unit 239 uses the dictionary stored in the dictionary database storage unit 233 for the input sentence received by the input unit 210 and each of the analysis target sentences extracted by the analysis target extraction unit 238. Then, a lattice is generated, which is a graph structure including nodes corresponding to each partial character string to which the part of speech is assigned and nodes corresponding to the connected partial character strings. An example of the created lattice is shown in FIG.

  The selection unit 240 includes the lattice generated by the lattice generation unit 239, the conversion probability of the normal broken phrase model stored in the normal broken phrase model storage unit 232, the cost of the dictionary stored in the dictionary database storage unit 233, and Based on the connection probability of part-of-speech pairs, the optimal path connecting the nodes of the lattice that maximizes the objective function of equation (1) is selected according to dynamic programming, and the normal represented by the selected path The notation part of speech and word break are output as morphological analysis results.

<Operation of the model learning device according to the first embodiment>
Next, the operation of the model learning device 100 according to the first embodiment of the present invention will be described. When the regular word broken word pair data is received by the input unit 10, the model learning device 100 executes a regular broken phrase model learning processing routine shown in FIG.

  First, in step S100, all pairs of the normal phrase fv and the broken phrase fw are obtained based on the normal word broken word pair data received by the input unit 10, and the conversion probability P (fw | fv) is randomly determined for each pair. A conversion probability table storing the conversion probabilities of each pair is generated and stored in the memory.

  Next, in step S104, for each of the pairs included in the regular word corrupted word pair data, according to the dynamic programming, based on the conversion probability table generated in step S100 or updated last time in step S106, Find the optimal alignment that is the correspondence.

Next, in step S106, an expected value P _t (fw | fv) of P (fw | fv) is obtained based on the optimum alignment of each pair included in the regular word corrupted word pair data in step S104, and stored in the memory. The conversion probability table is updated with the stored expected value P _t (fw | fv).

Next, in step S108, based on the expected value P _t (fw | fv) of each P (fw | fv) of the regular phrase and the collapsed phrase pair acquired in step S106, the likelihood is calculated according to the above equation (4). Calculate the function.

  Next, in step S110, it is determined whether or not the difference between the value of the likelihood function acquired in step S108 and the value of the likelihood function acquired in previous step S108 is equal to or less than a predetermined threshold ε. If the difference is equal to or smaller than the threshold ε, it is determined that the iteration end condition is satisfied, and the process proceeds to step S110. If the difference is greater than the threshold ε, it is determined that the iteration end condition is not satisfied, and step S104 is performed. Step S104 to Step S110 are repeated.

  Next, in step S112, the conversion probability table finally updated in step S106 is stored in the normal broken phrase model storage unit 36 as a normal broken phrase model.

  Next, in step S114, the normal broken phrase model acquired in step S112 is output by the output unit 50, and the process ends.

  Next, the operation of the model learning device 100 according to the first embodiment of the present invention will be described. When the input unit 10 receives regular word collapsed word pair data and a corpus of a plurality of fields, the model learning device 100 executes a phrase filter model learning processing routine shown in FIG.

  First, in step S <b> 201, a normal broken phrase model stored in the normal broken phrase model storage unit 36 is read.

  Next, in step S203, the character n-gram statistic is calculated for all the character n-grams based on the corpus of a plurality of fields received by the input unit 10.

  Next, in step S204, for each of the pairs included in the regular word corrupted word pair data received in the input unit 10, based on the regular corrupted phrase model acquired in step S201, the corrupted character partial character strings are corrupted. The partial character string corresponding to the phrase is converted into a partial character string of a regular phrase corresponding to the collapsed phrase.

  Next, in step S205, for each pair of regular phrases converted in step S204, whether or not each of the regular phrases converted in step S204 is included in the regular words of the pair to be processed for the pair of regular word collapsed word data received in the input unit 10. If it is determined and included, the learning data including the character n-gram statistics corresponding to each of the regular phrase and the collapse phrase acquired in step S203 is created as positive example data. Creates learning data including character n-gram statistics corresponding to each of the regular phrase and the collapse phrase acquired in step S203 as negative example data.

  In Step 206, it is determined whether or not the processing of Steps S204 and S205 has been executed for all pairs of regular word corrupted word pair data, and there is a pair that has not executed the processing of Steps S204 and S205. In step S204, the pair is set as a processing target. On the other hand, if the processes of steps S204 and S205 have been executed for all pairs, the process proceeds to step S207.

  Next, in step S207, it is possible to perform identification learning using a support vector machine or the like based on the learning data including the positive example data and the negative example data acquired in step S205, and to convert the collapsed phrase into a regular phrase. The phrase filter model for judging the likelihood is learned and stored in the phrase filter model storage unit 46. Further, the character n-gram statistic calculated in step S203 is stored in the phrase filter model storage unit 46.

  Next, in step S208, the phrase filter model learned in step S207 and the character n-gram statistic calculated in step S203 are output by the output unit 50, and the processing of the phrase filter model learning processing routine is terminated.

<Operation of the morphological analyzer according to the first embodiment>
Next, the operation of the morphological analyzer 200 according to the first embodiment of the present invention will be described. First, the normal broken phrase model output by the model learning device 100 is input to the morphological analyzer 200 and stored in the normal broken phrase model storage unit 232. Further, the phrase filter model and the character n-gram statistic output by the model learning device 100 are input to the morphological analyzer 200 and stored in the phrase filter model storage unit 236. Then, when an input sentence is received by the input unit 210, the morpheme analyzer 200 executes a morpheme analysis process routine shown in FIG.

  First, in step S300, a normal broken phrase model stored in the normal broken phrase model storage unit 232 is read.

  Next, in step S301, the dictionary stored in the dictionary database storage unit 233 is read.

  Next, in step S302, the phrase filter model and character n-gram statistics stored in the phrase filter model storage unit 236 are read.

  Next, in step S304, each of the analysis candidate sentences is generated based on the normal broken phrase model acquired in step S300 for the input sentence received by the input unit 210.

  Next, in step S308, for each of the analysis candidate sentences acquired in step S304, a collapse is performed based on the input sentence received in the input unit 210 and the phrase filter model and character n-gram statistics acquired in step S302. It is determined whether or not the phrase is an analysis candidate sentence that has been correctly converted into a regular phrase, and if it is determined that the phrase is an analysis candidate sentence that has been converted correctly, it is extracted as an analysis target sentence and the analysis candidate that has been converted correctly When it is determined that the sentence is not a sentence, the analysis candidate sentence is deleted.

  Next, in step S310, dictionary analysis is performed on each of the analysis target sentences extracted in step S308 and the input sentence received in the input unit 210 using the dictionary acquired in step S301, and parts of speech are given. A lattice, which is a graph structure composed of edges connecting the nodes corresponding to the partial character strings and the nodes corresponding to the connected partial character strings, is generated.

  Next, in step S312, based on the lattice acquired in step S310, the conversion probability of the regular collapsed phrase model acquired in step S300, the dictionary cost and the part-of-speech pair connection probability acquired in step S301, dynamic According to the programming method, the optimum path connecting the nodes of the lattice that maximizes the objective function of the above equation (1) is selected.

  Next, in step S314, the part of speech and the word break in the normal notation represented by the route selected in step S312 are output as a morpheme analysis result by the output unit 250, and the morpheme analysis processing routine is terminated.

  By executing the above morpheme analysis processing routine, for example, when an input sentence “That's what you do” is input, a morpheme analysis result as shown in FIG. 14B is output. As a comparative example, FIG. 14A shows a morphological analysis result when a conventional morphological analyzer is used.

  In addition, when the input sentence “I live well on the next day-w” is input, a morphological analysis result as shown in FIG. 15B is output. On the other hand, when a conventional morphological analyzer is used, the morphological analysis result shown in FIG.

As described above, according to the morphological analysis device according to the first embodiment of the present invention, by using the normal collapsed phrase model and the phrase filter model, the character string including the collapsed word that is a distorted notation is used. However, the morphological analysis can be performed with high accuracy.

  In addition, according to the model learning device according to the first exemplary embodiment of the present invention, a normal collapsed phrase model and a phrase that can accurately perform morphological analysis even on a character string that includes a corrupted word that is a fuzzy notation. A filter model can be learned.

  In addition, by automatically constructing a fluctuation model at the character string level from correct data and incorporating it into morphological analysis, it is possible to provide a framework for a morphological analyzer that is robust against broken notation.

  In addition, by introducing a phrase conversion model by transliteration using correct pair data in normal notation and broken notation, it is possible to automatically set rules and cost for each rule.

  In addition, by introducing a phrase filter model that determines whether or not a part that has been converted into a regular phrase as a broken phrase has been correctly converted, the number of candidates for analysis can be reduced, and the calculation at the time of analysis can be reduced. Cost can be reduced.

  Moreover, a pair pair of a regular phrase and a collapsing phrase and a conversion probability can be automatically acquired by a transliteration model, and processing speed and processing accuracy can be improved by a combination with dynamic programming.

  Next, a second embodiment will be described. In addition, about the part which becomes the structure and effect | action similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The second embodiment is different from the first embodiment in that the character n-gram statistic is a phrase filter model.

<Configuration of Model Learning Device According to Second Embodiment>
Next, the configuration of the model learning device 300 according to the second embodiment will be described.

  As shown in FIG. 16, the model learning device 300 according to the second exemplary embodiment of the present invention includes an input unit 10, a calculation unit 320, and an output unit 50.

  The calculation unit 320 includes an initial setting unit 30, a phrase alignment unit 31, a conversion probability calculation unit 32, an iterative determination unit 34, a normal disruption phrase model storage unit 36, a statistic calculation unit 342, and a phrase filter model storage. Part 346.

  The statistic calculation unit 342 calculates the character n-gram statistic for all the character n-grams based on the corpus of a plurality of fields received by the input unit 10, and sets the calculated character n-gram statistic. Is stored in the phrase filter model storage unit 346 as a phrase filter model.

  The phrase filter model storage unit 346 stores the phrase filter model generated by the statistic calculation unit 342.

<Configuration of Morphological Analyzer according to Second Embodiment>
Next, the configuration of the morphological analyzer 400 according to the second embodiment will be described.

  As shown in FIG. 17, the morphological analyzer 400 according to the second embodiment of the present invention includes an input unit 210, a calculation unit 420, and an output unit 50.

  The calculation unit 420 includes an analysis candidate generation unit 230, a denormalized phrase model storage unit 232, a dictionary database storage unit 233, a calculation unit 425, a phrase filter model storage unit 436, an analysis target extraction unit 438, and a lattice generation. A part 239 and a selection part 240 are included.

  For each of the analysis candidate sentences generated by the analysis candidate generation unit 230, the calculation unit 435 includes an input sentence and a phrase filter model that is a set of character n-gram statistics stored in the phrase filter model storage unit 436. Based on the above, using the character n-gram statistic P (before conversion) corresponding to the collapsed phrase in the input sentence and the character n-gram statistic P (after conversion) corresponding to the regular phrase in the analysis candidate sentence, P (after conversion) / P (before conversion) is calculated as a value indicating the likelihood of phrase conversion. For example, if there is an input sentence “I do not do it” and an analysis candidate sentence “I do not do it”, the character n corresponding to the partial character string consisting of the conversion character string of the conversion part and the surrounding character string in the input sentence A character n-gram statistic corresponding to a partial character string consisting of a return character string of the conversion sentence in the analysis candidate sentence and a peripheral character string, P ( The value of P (yo |, na, i) / P (yo |, ne, i) is calculated using yo |, n, i).

  The analysis target extraction unit 438 determines, for each analysis candidate sentence, whether the value indicating the likelihood of phrase conversion calculated by the calculation unit 435 for the analysis candidate sentence is greater than a threshold value T. When the value indicating the likelihood of phrase conversion is larger than the threshold T, the analysis candidate sentence is extracted as an analysis target sentence, and when the calculated value indicating the likelihood of phrase conversion is equal to or less than the threshold T, the analysis candidate Delete the sentence.

<Operation of Model Learning Device According to Second Embodiment>
Next, the operation of the model learning device 300 according to the second embodiment of the present invention will be described. When the input unit 10 receives a corpus of a plurality of fields, the model learning device 300 executes a phrase filter model learning processing routine shown in FIG.

  In step S400, the set of character n-gram statistics acquired in step S203 is stored in the phrase filter model storage unit 346 as a phrase filter model.

<Operation of the morphological analyzer according to the second embodiment>
Next, the operation of the morphological analyzer 400 according to the second embodiment of the present invention will be described. When an input sentence is received by the input unit 210, the morpheme analyzer 400 executes a morpheme analysis processing routine shown in FIG.

  In step S500, the likelihood of phrase conversion is calculated for each of the analysis candidate sentences based on the input sentence received in the input unit 210, each of the analysis candidate sentences acquired in step S304, and the phrase filter model acquired in step S302. A value indicating the likelihood is calculated.

  In step S502, each analysis candidate sentence whose value indicating the likelihood of phrase conversion calculated in step S500 is larger than the threshold value T is extracted as an analysis target sentence.

As described above, according to the morpheme device according to the second exemplary embodiment of the present invention, the morpheme apparatus uses the phrase filter model including a set of a normal collapsed phrase model and a character n-gram statistic, and the expression is fluctuating. Morphological analysis can be performed with high accuracy even for a character string including a broken word.

  In addition, according to the model learning device according to the second embodiment of the present invention, a normal collapsed phrase model and a character that can accurately perform morphological analysis even on a character string including a corrupted word that is a distorted notation. A phrase filter model consisting of a set of n-gram statistics can be learned.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  In the second embodiment, a value indicating the likelihood of phrase conversion is calculated as P (after conversion) / P (before conversion), and an analysis candidate sentence whose value is greater than the threshold T is used as an analysis target sentence. Although the case of extracting has been described, the present invention is not limited to this. For example, an analysis candidate sentence having a value of P (after conversion) −P (before conversion) equal to or greater than a predetermined threshold may be extracted as an analysis target sentence.

  Next, a third embodiment will be described. In addition, about the part which becomes the structure and effect | action similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The third embodiment is different from the first embodiment in that only the regular disruption phrase model is used.

<Configuration of Model Learning Device According to Third Embodiment>
Next, the configuration of the model learning device 500 according to the third embodiment will be described.

  As shown in FIG. 20, the model learning device 500 according to the third exemplary embodiment of the present invention includes an input unit 10, a calculation unit 520, and an output unit 50.

  The calculation unit 520 includes an initial setting unit 30, a phrase alignment unit 31, a conversion probability calculation unit 32, an iterative determination unit 34, and a normal disruption phrase model storage unit 36.

<Configuration of morphological analyzer according to the third embodiment>
Next, the configuration of the morphological analyzer 600 according to the third embodiment will be described.

  As shown in FIG. 21, the morphological analyzer 600 according to the third embodiment of the present invention includes an input unit 10, a calculation unit 620, and an output unit 50.

  The calculation unit 620 includes an analysis candidate generation unit 230, a denormalized phrase model storage unit 232, a dictionary database storage unit 233, a lattice generation unit 639, and a selection unit 240.

  The lattice generation unit 639 performs dictionary lookup using the dictionary stored in the dictionary database storage unit 233 for the input sentence received by the input unit 210 and each of the analysis candidate sentences generated by the analysis candidate generation unit 230. And generating a lattice that is a graph structure including nodes corresponding to each partial character string to which the part of speech is assigned and nodes connecting nodes corresponding to the connected partial character strings.

As described above, according to the morpheme device according to the third embodiment of the present invention, a morphological analysis is performed even for a character string including a broken word that is a distorted notation using a regular broken phrase model. It can be done accurately.

  In addition, according to the model learning device according to the third exemplary embodiment of the present invention, a regular collapsed phrase model that can accurately perform a morphological analysis on a character string including a broken word that is a distorted notation is learned. can do.

  Next, a fourth embodiment will be described. In addition, about the part which becomes the structure and effect | action similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The fourth embodiment is different from the first embodiment in that a normal broken phrase rule created in advance by hand is used instead of the regular broken phrase model.

<Configuration of Model Learning Device According to Fourth Embodiment>
Next, the configuration of the model learning device 700 according to the fourth embodiment will be described.

  As shown in FIG. 22, the model learning device 700 according to the fourth exemplary embodiment of the present invention includes an input unit 10, a calculation unit 720, and an output unit 50.

  The calculation unit 720 includes an irregular phrase phrase storage unit 736, a statistic calculation unit 42, a data generation unit 743, a model learning unit 44, and a phrase filter model storage unit 46.

  The regular broken phrase rule storage unit 736 stores a plurality of pairs of a broken phrase and a regular phrase corresponding to the broken phrase in advance as a regular broken phrase rule.

  For each of the pairs included in the normal-word-breaking word pair data received by the input unit 10, the data generation unit 743 breaks down the pair based on the normal-breaking phrase rule stored in the normal-breaking phrase rule storage unit 736. Among the partial character strings of the word, the partial character string corresponding to the collapsed phrase is converted into the partial character string of the corresponding regular phrase. Then, it is determined whether or not the converted regular phrase is included in the pair of regular words. If included, the character n-gram statistic corresponding to the regular phrase and the character n− corresponding to the collapsed phrase are included. Learning data including gram statistic is generated as positive example data, and if not included, learning data including character n-gram statistic corresponding to the regular phrase and character n-gram statistic corresponding to the collapsed phrase Are generated as negative example data.

<Configuration of Morphological Analyzer according to Fourth Embodiment>
Next, the configuration of the morphological analyzer 800 according to the fourth embodiment will be described.

  As shown in FIG. 23, the morphological analyzer 800 according to the fourth embodiment of the present invention includes an input unit 210, a calculation unit 820, and an output unit 50.

  The calculation unit 820 includes an analysis candidate generation unit 830, a denormalized phrase rule storage unit 832, a dictionary database storage unit 233, a phrase filter model storage unit 236, an analysis target extraction unit 238, a lattice generation unit 239, and a selection Part 840.

  The analysis candidate generation unit 830 generates an analysis candidate sentence from the input sentence received by the input unit 210 based on the normal breakdown phrase rule stored in the normal breakdown phrase rule storage unit 832.

  The phrase rule storage unit 832 stores the normal broken phrase rule that is the same as the normal broken phrase rule stored in the phrase rule storage unit 736 of the model learning device 700.

  Based on the lattice generated by the lattice generation unit 239, the cost of the dictionary stored in the dictionary database storage unit 233, and the connection probability of the part-of-speech pair, the selection unit 840 uses the above equation (1). The optimal path connecting the lattice nodes is selected, and the part-of-speech part and the word break of the normal notation represented by the selected path are output as the morphological analysis result. In the above equation (1), a constant value may be used as each conversion probability.

As described above, according to the morpheme device according to the fourth exemplary embodiment of the present invention, even with respect to a character string including a broken word that is a distorted notation, using a regular broken phrase rule and a phrase filter model. The morphological analysis can be performed with high accuracy.

  In addition, according to the model learning device according to the fourth exemplary embodiment of the present invention, a phrase filter model that can accurately perform a morphological analysis on a character string including a broken word that is a fuzzy notation is learned. be able to.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, in the present specification, the program has been described as an embodiment in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium or provided via a network. It is also possible to do.

DESCRIPTION OF SYMBOLS 10 Input part 20 Operation part 30 Initial setting part 31 Phrase alignment part 32 Conversion probability calculation part 34 Iteration determination part 36 Normal break phrase model storage part 42 Statistics amount calculation part 43 Data generation part 44 Model learning part 46 Phrase filter model storage part 50 Output unit 100 Model learning device 200 Morphological analysis device 210 Input unit 220 Operation unit 230 Analysis candidate generation unit 232 Normal broken phrase model storage unit 233 Dictionary database storage unit 235 Calculation unit 236 Phrase filter model storage unit 238 Analysis target extraction unit 239 Lattice generation Unit 240 selection unit 250 output unit 300 model learning device 320 calculation unit 342 statistic calculation unit 346 phrase filter model storage unit 400 morpheme analysis device 420 calculation unit 425 calculation unit 435 calculation unit 436 phrase filter model storage unit 438 Analysis target extraction unit 500 Model learning device 520 Calculation unit 600 Morphological analysis device 620 Calculation unit 639 Lattice generation unit 700 Model learning device 720 Calculation unit 736 Normal broken phrase rule storage unit 743 Data generation unit 800 Morphological analysis device 820 Calculation unit 830 Analysis Candidate generator 832 Regularly broken phrase rule storage 840 selector

Claims (14)

Based on a plurality of pairs of a normal word that is an input normalized expression and a collapsing word that is a notation that fluctuates with respect to the regular word, the regular word is a character string for each of the plurality of pairs. Dynamic programming is used to determine the optimal correspondence between each regular phrase that is a partial character string delimited by a delimiter position and each broken phrase that is a partial character string delimited by the delimiter position of the character string. The phrase alignment part to be determined according to
Based on the correspondence relationship for each of the plurality of pairs obtained by the phrase alignment unit, a conversion probability that the regular phrase is converted to the collapsed phrase is calculated for each pair of the regular phrase and the collapsed phrase. A conversion probability calculation unit to
The including, a model learning device,
An initial setting unit that initializes a conversion probability table that stores the conversion probability for each pair of the regular phrase and the collapsed phrase based on the plurality of the input pairs;
It further includes a repetition determination unit that repeats the processing by the phrase alignment unit and the calculation by the conversion probability calculation unit until a predetermined repetition end condition is satisfied, and outputs the conversion probability table as a normal collapsed phrase model,
The phrase alignment unit obtains the optimal correspondence for each of the plurality of pairs according to the dynamic programming based on the conversion probability table,
The said conversion probability calculation part is a model learning apparatus which calculates the said conversion probability about each pair of the said normal phrase and the said collapse phrase, and updates the said conversion probability table. A statistic calculator that calculates a statistic for each of at least one of a character n-gram consisting of n characters and a word surface layer n-gram consisting of n words based on a plurality of input corpora;
A plurality of pairs of a normal word that is an input normalized expression and a broken word that is a swaying notation with respect to the regular word, and a partial character string in which the broken word is delimited at a character string delimiter position For each of the plurality of pairs, the broken phrase is converted into a regular phrase that is a partial character string obtained by dividing the regular word at a character string separation position. The collapsed phrase included in the collapsed word is converted into the regular phrase, and based on the comparison result between the converted regular phrase and the paired regular word, the character n-gram corresponding to the collapsed phrase and Statistics calculated for at least one of the word surface n-grams, and fewer characters n-gram and word surface n-grams corresponding to the regular phrases Also includes a statistics calculated for one, and a data generation unit for generating learning data is either the positive example data and a negative example data,
A learning unit that learns a phrase filter model for determining the likelihood of converting the collapsed phrase into the regular phrase based on learning data generated for each of the plurality of pairs by the data generation unit;
A model learning device. Based on a plurality of pairs of a normal word that is an input normalized expression and a collapsing word that is a notation that fluctuates with respect to the regular word, the regular word is a character string for each of the plurality of pairs. Dynamic programming is used to determine the optimal correspondence between each regular phrase that is a partial character string delimited by a delimiter position and each broken phrase that is a partial character string delimited by the delimiter position of the character string. The phrase alignment part to be determined according to
Based on the correspondence relationship for each of the plurality of pairs obtained by the phrase alignment unit, a conversion probability that the regular phrase is converted to the collapsed phrase is calculated for each pair of the regular phrase and the collapsed phrase. A conversion probability calculation unit that outputs a conversion probability table storing the conversion probability for each pair as a normal collapsed phrase model ;
A model learning device. The program for functioning a computer as each part which comprises the model learning apparatus of any one of Claims 1-3 . A portion obtained by dividing a regular phrase that is a substring of a regular word that is a normalized expression at a delimiter position of a character string, and a broken word that is a distorted expression relative to the regular word at a delimiter position of the character string Based on a conversion probability table that stores conversion probabilities that the normal phrase and the collapse phrase are converted into the collapse phrase for each pair of the regular phrase and the collapse phrase, which are obtained in advance for conversion to the collapse phrase that is a character string, An analysis candidate generation unit that generates, as an analysis candidate, a character string obtained by replacing a partial character string that matches the collapsed phrase with the corresponding regular phrase for the input character string;
Dictionaries are applied to each of the input character strings and the analysis candidate character strings generated by the analysis candidate generation unit, nodes corresponding to the partial character strings to which parts of speech are assigned, and connected parts A lattice generation unit that generates a lattice that is a graph structure composed of edges connecting nodes corresponding to character strings;
Based on the lattice generated in the lattice generation unit, in accordance with dynamic programming, select an optimal path connecting the nodes, and outputs a selection result,
A morpheme analyzer.   The selection unit optimizes an objective function including a conversion probability that the regular phrase is converted into the collapsed phrase according to dynamic programming based on the lattice generated in the lattice generation unit and the conversion probability table. 6. The morphological analyzer according to claim 5, wherein a path connecting the nodes is selected and output as an analysis result. A portion obtained by dividing a regular phrase that is a substring of a regular word that is a normalized expression at a delimiter position of a character string, and a broken word that is a distorted expression relative to the regular word at a delimiter position of the character string Based on a conversion probability table that stores conversion probabilities that the normal phrase and the collapse phrase are converted into the collapse phrase for each pair of the regular phrase and the collapse phrase, which are obtained in advance for conversion to the collapse phrase that is a character string, An analysis candidate generation unit that generates, as an analysis candidate, a character string obtained by replacing a partial character string that matches the collapsed phrase with the corresponding regular phrase for the input character string;
For each of the analysis candidates generated by the analysis candidate generation unit, based on a phrase filter model obtained in advance for determining the likelihood of converting the collapsed phrase into the regular phrase, A calculation unit for calculating the likelihood,
Analysis candidate extraction for extracting the likely analysis candidate from the analysis candidate generated by the analysis candidate generation unit based on the likelihood of the analysis candidate calculated by the calculation unit and a predetermined threshold And
Dictionaries for each of the input character string and the analysis candidate character string extracted by the analysis candidate extraction unit, nodes corresponding to each partial character string given a part of speech and connected parts A lattice generation unit that generates a lattice that is a graph structure composed of edges connecting nodes corresponding to character strings;
Based on the lattice generated in the lattice generation unit, in accordance with dynamic programming, select an optimal path connecting the nodes, and outputs a selection result,
A morpheme analyzer.   The selection unit optimizes an objective function including a conversion probability that the regular phrase is converted into the collapsed phrase according to dynamic programming based on the lattice generated in the lattice generation unit and the conversion probability table. The morpheme analyzer according to claim 7, wherein a path connecting the nodes is selected and output as an analysis result. The program for functioning a computer as each part which comprises the morphological analyzer of any one of Claims 5-8.   The initial setting unit, based on a plurality of pairs of a regular word that is an input normalized expression and a broken word that is a swaying expression with respect to the regular word, the regular word The conversion probability that the regular phrase is converted to the collapsed phrase for each pair of the regular phrase that is the partial string delimited by and the collapsed phrase that is the partial string obtained by separating the collapsed word at the position where the string is separated. Initialize the stored conversion probability table,
  The phrase alignment unit is a normal character string that is a partial character string obtained by dividing the regular word at a character string delimiter position for each of the plurality of pairs based on the plurality of input pairs and the conversion probability table. Finding the optimal correspondence between each of the phrases and each of the collapsed phrases that are partial character strings obtained by dividing the collapsed word at the position where the character string is separated, according to dynamic programming,
  The conversion probability calculation unit calculates the conversion probability for each pair of the regular phrase and the collapsed phrase based on the correspondence relationship for each of the plurality of pairs obtained by the phrase alignment unit, and the conversion Update the probability table,
  The iterative determination unit repeats the processing by the phrase alignment unit and the calculation by the conversion probability calculation unit until a predetermined repetition end condition is satisfied, and outputs the conversion probability table as a normal collapsed phrase model
  Model learning method.   The statistic calculator calculates a statistic for each of at least one of a character n-gram consisting of n characters and a word surface layer n-gram consisting of n words based on the plurality of inputted corpora. ,
  The data generation unit divides a plurality of pairs of a normal word that is an input normalized expression and a broken word that is a swaying expression with respect to the regular word, and the broken word at a character string delimiter position. Based on a pre-determined regular collapsed phrase model for converting a regular phrase that is a partial character string into a regular phrase that is a partial character string obtained by dividing the regular word at a character string separation position, For each, convert the collapsible phrase included in the collapsible word of the pair into the regular phrase, and correspond to the collapsible phrase based on a comparison result between the converted regular phrase and the regular word of the pair Statistics calculated for at least one of a character n-gram and a word surface layer n-gram, and a character n-gram and a word surface layer n- corresponding to the regular phrase And a statistic least one calculated for the ram, and generates learning data is either the positive example data and a negative example data,
  The learning unit learns a phrase filter model for determining the likelihood of converting the collapsed phrase into the regular phrase based on the learning data generated for each of the plurality of pairs by the data generation unit.
  Model learning method.   For each of the plurality of pairs, the phrase alignment unit is based on a plurality of pairs of a regular word that is an input normalized expression and a broken word that is a swaying expression with respect to the regular word. Optimal correspondence between each regular phrase that is a partial character string obtained by dividing a regular word at a character string delimiter position and each broken phrase that is a partial character string obtained by delimiting the broken word at a character string delimiter position , According to dynamic programming,
  The conversion probability calculation unit converts the regular phrase into the collapsed phrase for each pair of the regular phrase and the collapsed phrase based on the correspondence relationship for each of the plurality of pairs obtained by the phrase alignment unit. A conversion probability table storing the conversion probabilities for each pair is output as a normal collapsed phrase model
  Model learning method.   The analysis candidate generator generates a regular phrase that is a partial character string obtained by dividing a regular word that is a normalized expression at a character string delimiter position, and a corrupted word that is a distorted expression with respect to the regular word. Conversion that stores the conversion probability that the normal phrase is converted into the collapsed phrase for each pair of the regular phrase and the collapsed phrase, which is obtained in advance for conversion into the collapsed phrase that is a partial character string that is delimited at a delimiter position. Based on the probability table, for the input character string, a character string in which a partial character string that matches the collapsed phrase is replaced with the corresponding regular phrase is generated as an analysis candidate,
  A node corresponding to each partial character string to which a part of speech is given by the lattice generation unit performing a dictionary lookup on each of the input character string and the analysis candidate character string generated by the analysis candidate generation unit And a lattice which is a graph structure composed of edges connecting nodes corresponding to the substrings to be connected,
  The selection unit selects an optimum path connecting the nodes according to the dynamic programming based on the lattice generated by the lattice generation unit, and outputs it as an analysis result.
  Morphological analysis method.   The analysis candidate generator generates a regular phrase that is a partial character string obtained by dividing a regular word that is a normalized expression at a character string delimiter position, and a corrupted word that is a distorted expression with respect to the regular word. Conversion that stores the conversion probability that the normal phrase is converted into the collapsed phrase for each pair of the regular phrase and the collapsed phrase, which is obtained in advance for conversion into the collapsed phrase that is a partial character string that is delimited at a delimiter position. Based on the probability table, for the input character string, a character string in which a partial character string that matches the collapsed phrase is replaced with the corresponding regular phrase is generated as an analysis candidate,
  For each of the analysis candidates generated by the analysis candidate generation unit, the calculation unit is based on a phrase filter model obtained in advance for determining the likelihood of converting the collapsed phrase into the regular phrase. Calculating the likelihood of the analysis candidate;
  The analysis candidate extraction unit is based on the likelihood of the analysis candidate calculated by the calculation unit based on the analysis candidate generated by the analysis candidate generation unit and a predetermined threshold, and the analysis candidate that is likely Extract
  A node corresponding to each partial character string to which a part of speech is assigned by the dictionary generator for each of the input character string and the analysis candidate character string extracted by the analysis candidate extraction unit. And a lattice which is a graph structure composed of edges connecting nodes corresponding to the substrings to be connected,
  The selection unit selects an optimum path connecting the nodes according to the dynamic programming based on the lattice generated by the lattice generation unit, and outputs it as an analysis result.
  Morphological analysis method.