JP2014232510A - Japanese kana character correction model learning device, japanese kana character correction device, methods therefore, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a Japanese kana character correction model learning device and a Japanese kana character correction device, which learn a Japanese kana character correction model as a statistical model for automatically correcting a Japanese kana character error due to notation variability.SOLUTION: A Japanese kana character correction model learning device includes: an N-1 series extraction unit for extracting an N-gram where N-1 characters of hiragana letters appear continuously in one Chinese character in a learning text by using the learning text in which Chinese characters are mixed with kanas, as input; and an N-gram model learning unit for learning a Chinese character-kana N-gram model provided with a probability according to an appearance frequency of the N-gram, and outputting the Chinese character-kana N-gram model as a Japanese kana character correction model to the outside. Also, the Japanese kana character correction model learning device inputs an N-gram of an appropriate Chinese character to the Japanese kana character correction model to find an occurrence probability, and corrects a Japanese kana character of the appropriate Chinese character to a Japanese kana character having an occurrence probability of a predetermined value or more and outputs it.

Description

  The present invention relates to a reading kana correction model learning device that generates a reading kana correction model used for automatic correction of reading kana errors, a reading kana correction device using the model, and a method and a program thereof.

  Conventionally, with kana reading for kanji, word candidates consisting of pairs of (word notation, part of speech, reading kana) are obtained from a word dictionary, and the most appropriate word sequence as a Japanese sentence based on the part of speech connection between words In general, a method of assigning a reading kana to a kanji based on the reading kana of the selected word series has been used (for example, Non-Patent Document 1).

  In texts that contain broken notation written by individuals, such as Twitter and blogs, for example, lowercase letters such as “happy” → “happy”, katakana such as “don't know” → “know lanai”, etc. Shake occurs. If the text that is subject to reading kana includes text that includes such a notation fluctuation, the dictionary cannot be correctly matched when selecting a word sequence, and a reading kana error occurs. In order to improve reading kana mistakes caused by typographical fluctuations, it has been common practice to rewrite the text according to rules before selecting a word series and correct the text containing typographical fluctuations to a dictionary-matchable notation. It was solved by making a choice.

Yuji Matsumoto, et al. "Japanese Morphological Analysis System" Cha "Version 2.0 Instruction Manual ~" NAIST-IS-TR99012 (1999).

  Since there are a wide variety of notation fluctuation patterns included in the broken notation text, there are many notation fluctuations that cannot be covered by rewriting text according to conventional rules. Also, since it is necessary to manually design the rules, it is expensive to design the rules every time a new notation fluctuation pattern appears.

  The present invention has been made in view of this problem, and a reading-kana correction model learning device that learns a reading-kana correction model, which is a statistical model for automatically correcting reading-kana errors caused by notation fluctuation, It is an object of the present invention to provide a reading kana correction device using the model, and a method and program thereof.

  The reading-kana correction model learning device of the present invention includes an N-1 sequence extraction unit and an N-gram model learning unit. The N-1 series extraction unit receives a learning text mixed with kanji and kana, and extracts an N-gram in which N-1 hiragana characters appear concatenated with one kanji character in the learning text. The N-gram model learning unit learns a kanji-kana N-gram model to which a probability is given according to the appearance frequency of the N-gram, reads the kanji-kana N-gram model, and outputs it as a kana correction model.

  The reading kana correction device of the present invention includes a reading kana correction model and a reading kana correction unit. The reading kana correction model is a reading kana correction model learned by the above-described reading kana correction model learning device. The reading kana correction unit extracts an N-gram in which N-1 characters of hiragana are concatenated and appear in one kanji character included in the input text, and the N-gram of the corresponding kanji is used as the reading kana correction model. Then, the occurrence probability of the N-gram is obtained, and the reading kana of the corresponding kanji is corrected to the reading kana with the occurrence probability being a predetermined value or more and output.

  The reading kana correction model learning apparatus of the present invention is an N-gram probability model consisting of one kanji character in a learning text, its reading kana and N-1 characters connected to the kanji, and is included in the text. A kana correction model that can be used to correct shaking is provided. The reading kana correction device of the present invention can automatically correct reading kana errors caused by the notation fluctuations included in the text using the reading correction model. Therefore, there is an effect of reducing the cost required for designing the rule every time a new notation fluctuation pattern appears.

The figure which shows the function structural example of the reading-kana correction model learning apparatus 100 of this invention. The figure which shows the operation | movement flow of the reading kana correction model learning apparatus 100. FIG. The figure which shows the function structural example of the reading kana correction apparatus 200 of this invention. The figure which shows the more specific functional structural example of the reading kana correction | amendment part 210. FIG. The figure which shows the function structural example of the reading kana correction apparatus 300 of this invention. The figure which shows the more specific function structural example of the reading kana candidate extraction part 310. FIG. The figure which shows the function structural example of the reading kana correction apparatus 400 of this invention. The figure which shows the function structural example of the reading kana correction apparatus 500 of this invention.

  Embodiments of the present invention will be described below with reference to the drawings. The same reference numerals are given to the same components in a plurality of drawings, and the description will not be repeated.

[Reading Kana correction model learning device]
FIG. 1 shows an example of the functional configuration of a reading-kana correction model learning apparatus 100 according to the present invention. The operation flow is shown in FIG. The reading-kana correction model learning device 100 includes an N-1 sequence extraction unit 110, an N-gram model learning unit 120, and a control unit 130. The reading-kana correction model learning apparatus 100 is realized by reading a predetermined program into a computer composed of, for example, a ROM, a RAM, a CPU, and the like, and executing the program by the CPU. The same applies to other embodiments described below.

  The N-1 series extraction unit 110 receives learning text mixed with kanji kana and extracts an N-gram in which N-1 hiragana characters appear concatenated with one kanji character in the learning text (step S110). ). In the learning text, only an N-gram in which N-1 hiragana characters are concatenated to one kanji character is set as a learning target. Those in which Kanji characters appear consecutively or N-1 characters that appear after Kanji characters include characters other than hiragana (katakana, kanji characters, symbols, etc.) are not subject to learning.

  For example, in the case of N = 3, in the learning text “Let's play outside today (Kyowa Soto de Asobi Mashoune)”, a 3-gram where two hiragana characters are connected to one kanji character is It is only the “Play” part. In this example, it is a set of triples consisting of a set of the first kanji and its reading kana (Yu, Aso) and “bima” which is a reading of the two hiragana characters “Bima” connected to the kanji. ([Yu, Aso], Bi, Ma) is counted as an N-gram. This N-gram extraction is performed for all words in the learning text, and is repeated until all of the N-grams in which two hiragana characters are connected to one kanji character in the learning text are extracted. (No in step S130). This repetitive operation is controlled by the control unit 130. The control unit 130 is a general unit that controls the time-series operation of each unit of the reading-a-kana correction model learning device 100, and does not perform a special process. The same applies to the other embodiments.

  The N-gram model learning unit 120 counts the frequency of all the N-grams extracted by the N-1 sequence extraction unit 110 and assigns a probability according to the frequency. The model is learned, and the kanji-kana N-gram model is read and output to the outside as the kana correction model 140 (step S120). The learning method of the N-gram model is well known as described in, for example, Reference 1 (Kenji Kita, “Language and Computation-4 Stochastic Language Model”, The University of Tokyo Press, pp.57-62). is there.

  The conventional general N-gram model is often used as a language model for speech recognition or morphological analysis by learning a combination of adjacent words. In the present invention, the N-gram model is new in that it learns a kanji, its reading kana, and a combination of readings connected to the kanji and uses it as a model for correcting reading kana errors.

  N may be any number as long as N is 2 or more. For example, assuming that N = 2, there may be a kanji-kana N-gram model that considers only kanji and kanji and readings connected to kanji. However, when N = 2, it is possible to determine the reading pseudonym almost uniquely by considering up to two readings concatenated with kanji, such as “Fun” and “Easy”. Even in the example, since only “fun” can be considered, there is a problem of becoming a model in which a large difference does not appear probabilistically between the reading kana “Tano” and “Raku”.

  In order to avoid such a model, a Kanji Kana N-gram model with a long N number of N-grams should be used for Kanji characters with a high appearance frequency that can obtain a statistically sufficient learning amount. Is desirable. However, in this case as well, there is a problem that a problem of data sparse occurs due to a lack of learning data if the number of N is set to be long (large) for a Chinese character with a low appearance frequency.

Therefore, the N number of N-grams may be fixed to the optimum N number corresponding to the learning text, or a plurality of N kanji-kana N-gram models may be used in combination.
[Reading Kana Correction Device]
FIG. 3 shows an example of the functional configuration of the reading-kana correction device 200 of the present invention. The reading-kana correction device 200 includes a reading-kana correction model 140, a reading-kana correction unit 210, and a control unit 230.

  The reading-kana correction model 140 is a kanji-kana N-gram model learned by the reading-kana correction model learning device 100 described above. The kanji-kana N-gram model is, for example, a 3-gram model.

  The reading kana correction unit 210 extracts an N-gram in which N-1 characters of hiragana are concatenated and appear in one kanji character included in the input text, and reads the N-gram of the corresponding kanji into the kana correction model 140. The N-gram occurrence probability is input and the kanji reading of the corresponding kanji is corrected to a reading kana with an occurrence probability of a predetermined value or more and output. The reading kana correction unit 210 corrects, for example, a 3-gram of ([Raku, Gaku], Shi, A) included in the input text to a high occurrence probability ([Raku, Tano], Shi, A). Output the modified text to the outside. Here, the corresponding kanji is one arbitrary kanji in the input text to be corrected by the reading kana correction device 200.

  The reading-kana correction unit 210 uses the occurrence probability calculated from the kanji-kana N-gram model learned by the reading-kana correction model learning device 100 as an index for correcting the reading kana error for the input text. The Kanji Kana N-gram model calculates the probability of occurrence of a combination of ([Kanji, reading Kana], N-1 readings connected to Kanji) according to the frequency of occurrence of the corresponding combination in the learning text. can do. A high probability is calculated for combinations that appear in the learning text with a high frequency, and a low probability is calculated for combinations that appear with a low frequency. In this embodiment, those with a high occurrence probability calculated from the Kanji Kana N-gram model are less likely to have a reading kana error, and conversely, those with a low occurrence probability have a reading kana error. Assuming that there is a high possibility, the reading kana error is corrected by correcting the reading kana with a low occurrence probability calculated from the Kanji N-gram model to a reading kana with a high occurrence probability.

  FIG. 4 shows a more specific functional configuration example of the reading-kana correction unit 210, and the operation will be described in more detail. The reading kana correction unit 210 includes a single kanji dictionary 211, input text reading kana occurrence probability calculation means 212, single kanji reading kana occurrence probability calculation means 213, and reading kana determination means 214.

  The single kanji dictionary 211 is a dictionary in which kanji appearing in Japanese text and candidates for reading kana for the kanji are listed. For example, it stores information such as easy, fun, joyful, easy, fun, and so on.

  The input text reading kana occurrence probability calculation means 212 extracts an N-gram in which N-1 characters of hiragana are concatenated and appear in one kanji character included in the input text, and reads the k-character N-gram of the corresponding kanji. Input to the reading-kana correction model 140 learned by the correction model learning device 100 to determine the occurrence probability P0 of the N-gram. For example, if the N-gram of the target input text is ([Raku, Gaku], Shi, A), the occurrence probability P0 is obtained. Then, the kanji information is output to the single kanji reading kana occurrence probability calculation means 213.

  The single kanji reading kana occurrence probability calculation means 213 acquires one or more other reading kana candidates for the corresponding kanji from the single kanji dictionary, inputs other reading kana candidates for the corresponding kanji into the reading kana correction model 140, and others. The occurrence probability Pk of the reading pseudonym candidate is obtained. When the corresponding kanji is ([Raku, Gaku]), other reading candidate candidates are k = 1 fun (Tanoshii), k = 2 fun (Rakushite), k = 3 fun (Tanoshiku), Respective occurrence probabilities P1, P2, and P3 are obtained.

  The reading-kana determination means 214 calculates a likelihood ratio Rk (= Pk / P0) between the occurrence probability Pk (k = 1,..., N) and the occurrence probability P0, and the likelihood ratio Rk is equal to or greater than a predetermined value T. And the largest reading kana is determined as the corrected reading kana of the corresponding kanji, and when the likelihood ratio Rk is not more than the predetermined value T, the reading kana with the occurrence probability P0 is determined as the reading kana of the corresponding kanji. To do. In the example in which the corresponding kanji is ([Raku, Gaku]), if the value of the likelihood ratio R1 between (Tanoshii) and (Gakushii) is greater than or equal to a predetermined value T, ([Raku, Gaku] ], Shi, and i) are corrected to ([Raku, Tano], shi, i) and output. Here, the predetermined value T is set to be about 2 to 3 times the occurrence probability of the reading pseudonym candidate having the maximum likelihood (T = 2 to 3). If the likelihood ratio Rk is 1.0 or more, it means that there is a reading with a higher occurrence probability. However, if the likelihood ratio Rk is too close to 1.0, the possibility of erroneous conversion increases. Therefore, the value of the predetermined value may be determined by a result of trial according to the input text.

  FIG. 5 shows an example of a functional configuration of the reading-kana correction device 300 of the present invention. The reading kana correction device 300 includes a reading kana correction model 140, a reading kana candidate extraction unit 310, a reading kana correction unit 320, a reading N-gram model 340, and a control unit 330. The reading kana candidate extraction unit 310 differs from the reading kana correction device 200 (FIG. 3) in that it outputs a plurality of reading kana candidates. The reading-kana correction model 140 is the same as the reading-kana correction device 200 (FIG. 3) of the first embodiment.

  The kana candidate extraction unit 310 extracts an N-gram in which N-1 hiragana characters are concatenated and appear in one kanji character included in the input text, and reads the k-character N-gram to learn a kana correction model. The N-gram occurrence probability is input to the reading kana correction model learned by the apparatus, and a plurality of reading kana having the occurrence probability equal to or greater than a predetermined value are output as reading kana candidates for the corresponding kanji.

  FIG. 6 shows a more specific functional configuration example of the reading kana candidate extraction unit 310. The reading kana candidate extraction unit 310 differs from the reading kana correction unit 210 (FIG. 4) only in that it includes a reading kana candidate selection means 311. The reading kana candidate selection means 311 includes the N-gram occurrence probability P0 of the corresponding kanji output from the input text reading kana occurrence probability calculation means 212 and other reading kana candidate candidates output from the single kanji reading kana occurrence probability calculation means 213. The likelihood ratio Rk (= Pk / P0) is obtained using the occurrence probability Pk (k = 1,..., N) as input, and a plurality of reading kana candidates whose likelihood ratio Rk is equal to or greater than a predetermined value T are output.

  The reading N-gram model 340 is a model in which the appearance frequency of readings that appear in the learning text connected in series is learned. In the case of N = 3, in the learning text “Let's play outside today (Kyowa Soto de Asobi Mashoune)”, three “Kyo”, “Kyowa”, “Uwaso” etc. are connected. All the readings that appear are counted and given a probability according to the frequency. When a reading sequence is input to the reading N-gram model 340, the occurrence probability of the reading can be calculated. The construction method of the reading N-gram model 340 is the same as the reading kana correction model 140 and is well known.

  The reading kana modification unit 320 obtains the occurrence probability of one sentence including the plurality of reading kana candidates by referring to the reading N-gram model 340, and outputs one sentence including the reading kana candidate having the highest occurrence probability. As an example, the kana character “Easy” in the text “Today is Fun (Kyowa Gakushiina)” is input to the kana candidate extraction unit 310 by the “Reading Kana” candidate extraction unit 310. It is assumed that two reading candidate names “)” have been output.

  In that case, the occurrence probability is calculated using the reading N-gram model for each of the series “Kyowarakushina” and “Kyowatanoshina” which are the reading series of the entire input text. In the case of this example, “Kyowatanoshina”, which is a reading sequence with a high occurrence probability, is output as a text whose reading is corrected.

  FIG. 7 shows an example of the functional configuration of the reading-kana correction device 400 of the present invention. The reading kana correction device 400 includes a kanji 2-gram model 142, a kanji 3-gram model 143, a kanji 4-gram model 144, a reading kana correction unit 410, and a control unit 430. The reading kana correction device 400 is different from the reading kana correction device 200 in that it includes a plurality of Kanji N-gram models 142 to 144.

  The Kanji Kana N-gram models 142 to 144 are probability models learned by the reading-kana correction model learning device 100. The reading-kana correction unit 410 extracts 2-gram, 3-gram, and 4-gram in which hiragana appears concatenated with one kanji character included in the input text, and the N-gram of the corresponding kanji is converted into the corresponding N-gram. gram kanji 2-gram model 142, kanji 3-gram model 143, and kanji 4-gram model 144 are input to each N-gram occurrence probability, the kana reading kana Is corrected to a kana reading greater than or equal to a predetermined value and output.

  As described above, it is desirable to use a kanji-kana N-gram model in which the N number of N-grams is set longer for kanji characters with a high appearance frequency that can obtain a statistically sufficient learning amount. However, if the number of N-grams is set longer for kanji characters with a low appearance frequency, there is insufficient learning data and a data sparse problem occurs. The reading-kana correction device 400 can solve this problem.

  The reading-kana correction device 400 uses a plurality of kanji-kana N-gram models together, and calculates the sum of likelihood ratios Rk_n-gram (= Pk_n-gram / P0_n-gram) separately calculated from each kana-kana N-gram model. However, the kana reading of the corresponding kanji is corrected and output to the maximum reading kana that is greater than or equal to a certain value. According to the reading kana correction device 400, the kanji with high appearance frequency can use the probability of the model in which the number of N-grams is set to be large, so that the reading kana can be corrected with higher accuracy. In addition, for kanji characters with a low appearance frequency, the probability of a model with a reduced number of N-grams can be used, which reduces the data sparse problem.

  FIG. 8 shows an example of the functional configuration of the reading-kana correction device 500 of the present invention. The reading kana correction device 500 includes a kanji-kana 2-gram model 142, a kanji-kana 3-gram model 143, a kanji-kana 4-gram model 144, a reading kana candidate extraction unit 510, a reading kana correction unit 320, An N-gram model 340 and a control unit 530 are provided. The reading-kana correction device 500 is a combination of the ideas of the second embodiment (the reading-kana correction device 300 (FIG. 5)) and 3 (the reading-kana correction device 400 (FIG. 7)).

  The reading kana candidate extraction unit 510 extracts 2-gram, 3-gram, and 4-gram in which hiragana appears concatenated with one kanji character included in the input text, and corresponds to the N-gram of the corresponding kanji. N-gram Kanji 2-gram model 142, Kanji 3-gram model 143 and Kanji 4-gram model 144 are input to each N-gram occurrence probability, and the occurrence probability is greater than or equal to a predetermined value A plurality of reading kana candidates for the corresponding kanji are output. The reading Kana correction unit 320 and the reading N-gram model 340 are the same as the reading Kana correction device 300 as is clear from the reference numerals.

  According to the reading kana correction device 500 that combines the ideas of the reading kana correction device 300 and the reading kana correction device 400, reading kana correction that is less dependent on the difference in the appearance frequency of kanji in the learning text and that is optimal for the entire sentence is performed. This makes it possible to correct the reading kana with higher accuracy.

  As described above, the reading kana correction model learning device 100 according to the present invention learns a kanji of a learning text, its reading kana, and a combination of readings connected to the kanji, and uses them as a model for correcting reading kana errors. Can provide a new statistical model. In addition, the kana correction devices 200, 300, 400, and 500 of the present invention use the new statistical model, so that various notations included in texts including broken notations written by individuals such as Twitter and blogs. The shake can be automatically corrected to the correct reading kana. The kana correction device 200, 300, 400, 500 of the present invention has an effect of reducing the cost of designing a rule each time a new notation fluctuation pattern that has been necessary in the past appears.

  When the processing means in the above apparatus is realized by a computer, the processing contents of the functions that each apparatus should have are described by a program. Then, by executing this program on the computer, the processing means in each apparatus is realized on the computer.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a recording device of a server computer and transferring the program from the server computer to another computer via a network.

  Each means may be configured by executing a predetermined program on a computer, or at least a part of these processing contents may be realized by hardware.

Claims (8)

An N-1 sequence extraction unit that extracts an N-gram in which hiragana N-1 characters appear contiguously to one kanji character in the learning text, using a kanji mixed kana learning text as input,
An N-gram model learning unit that learns a Kanji Kana N-gram model to which a probability is given according to the appearance frequency of the N-gram, reads the Kanji Kana N-gram model, and outputs it as a kana correction model;
A kana correction model learning device comprising: A reading-kana correction model learned by the reading-kana correction model learning device according to claim 1;
An N-gram in which N-1 characters of hiragana appear concatenated with one Kanji character included in the input text is extracted, and the N-gram of the corresponding Kanji character is input to the reading kana correction model and the N a reading kana correction unit that calculates the occurrence probability of -gram, corrects the kana reading of the corresponding kanji to a reading kana with the occurrence probability equal to or higher than a predetermined value,
A reading kana correction device comprising: In the reading kana correction device according to claim 2,
The above-mentioned kana correction part
A single kanji dictionary that lists kanji characters that appear in Japanese text and possible kana characters for kanji,
N-gram in which N-1 characters of hiragana appear concatenated to one kanji character included in the input text, and the kana correction corrected by reading the k-character N-gram and learning with the kana correction model learning device An input text reading pseudonym occurrence probability calculating means for inputting to the model and calculating the occurrence probability P0 of the N-gram;
One or more other kana characters for the corresponding kanji are obtained from the single kanji dictionary, and the other kana kana candidates are input into the kana correction model to determine the occurrence probability Pk of the other kana candidates. A reading pseudonym occurrence probability calculating means;
The likelihood ratio Rk between the occurrence probability Pk and the occurrence probability P0 is obtained, and the maximum reading kana candidate whose likelihood ratio Rk is equal to or greater than a predetermined value is determined as the corrected reading kana of the corresponding kanji, When the likelihood ratio Rk is less than or equal to the predetermined value, a reading kana determination means for determining the reading kana of the occurrence probability P0 as the reading kana of the corresponding kanji,
A reading kana correction device comprising: A reading-kana correction model learned by the reading-kana correction model learning device according to claim 1;
N-gram in which N-1 hiragana characters appear concatenated with one Kanji character included in the input text is extracted, and the above-mentioned N-gram of the corresponding Kanji character is read and learned by the kana correction model learning device. A reading kana candidate extraction unit that inputs to the model to determine the occurrence probability of the N-gram, and outputs a plurality of reading kana with the occurrence probability being a predetermined value or more as reading kana candidates for the corresponding kanji;
A reading N-gram model that has learned the frequency of occurrence of N concatenated readings in the learning text;
An occurrence probability of a sentence including the above-mentioned candidate for reading, with reference to the above-mentioned reading N-gram model,
A reading kana correction device comprising: A kana 2-gram model, a kana 3-gram model, and a kana 4-gram model of a kana 2-gram model learned by the reading kana correction model learning device according to claim 1;
2-grams, 3-grams and 4-grams in which hiragana characters appear concatenated with one kanji character included in the input text are extracted, and the above kanji for the corresponding kanji 2 -The input to each of the -gram model, the Kanji 3-gram model, and the Kanji 4-gram model is used to determine the occurrence probability of each N-gram, and the above-mentioned occurrence probability is corrected to a reading above the predetermined value and output. A kana correction part;
A reading kana correction device comprising: An N-1 sequence extraction process for extracting an N-gram in which hiragana N-1 characters appear concatenatingly appearing as one kanji character in the learning text, using a kanji-kana mixed learning text as input,
N-gram model learning process of learning a kanji Kana N-gram model to which a probability is given according to the appearance frequency of the N-gram, reading the Kanji Kana N-gram model and outputting it as a kana correction model,
A kana correction model learning method comprising: A reading kana correction model learned by the reading kana correction model learning method according to claim 6;
An N-gram in which N-1 characters of hiragana are concatenated and appear in one kanji character included in the input text is extracted, and the N-gram of the corresponding kanji is input to the reading kana correction model and the N-gram is input. A kana correction process for obtaining the occurrence probability of gram, correcting the kana reading of the corresponding kanji into a reading kana with the occurrence probability being a predetermined value or more,
A reading kana correction method comprising:   A program for causing a computer to function as the reading kana correction model learning device according to claim 1 and the reading kana correction device according to any one of claims 2 to 5.

Images