JP2007004637A - Kana-kanji conversion - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a KANA-KANJI conversion technology which does not damage a sensation, even if a user inputs by frequently pressing conversion keys, to realize a higher-precision conversion. <P>SOLUTION: The first processing is executed so that a probability that a word to be an object for an inputted sentence follows a specific symbol may become a corrected one, the probabilities being obtained from a statistic language model provided in a database. Further, the first processing is executed to a prescribed symbol among specific symbols only when the prescribed symbol is fixed. When a specific symbol is not included in the inputted sentence and a previously inputted character string is fixed, the second processing is executed so that a probability that a word following the fixed character string follows the fixed character string may be a corrected one. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to kana-kanji conversion using an n-gram model as a language model, and to a method and a computer-readable recording medium for storing the kana-kanji conversion program.

In general, when Japanese is typed into a computer system or the like, kana-kanji conversion processing is performed to convert the typed kana characters into kanji. As a typical kana-kanji conversion program, there is MS-IME (the related technology is disclosed in Patent Document 1) by the present applicant.
When typing in Japanese, it is not certain when the user presses the conversion key. However, it is known from the usability test and misconversion reports that many users input while pressing the conversion key in detail.

JP 2004-118461 A

  On the other hand, when such a shredded input is simulated and measured with a pure SLM (Statistical Language Model), the conversion accuracy (CER: Character Error Rate) is measured. It was. Actually, in the case of input with a small amount, if the input is re-input after a surprising conversion error, the conversion may be as expected, and the user is likely to be confused. In particular, conversion results differed between the case where a specific character was first input and the case where the character was not input.

  Even when there is no specific symbol, the conversion result expected for the subsequent character string may differ depending on whether or not there is a previously determined character string.

  As described above, there are a large number of users who input shredded pieces. Therefore, it is necessary to solve the above-mentioned problem that the conversion accuracy of the shredded input is low and to experience a conversion accuracy equivalent to or higher than that of the user who inputs a long time even if the user inputs the shredded input.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a kana-kanji conversion technique that can perform conversion with higher accuracy without impairing the sensation even when the user performs fine input while pressing the conversion key. There is.

  In order to achieve the above object, the invention described in claim 1 is a computer-readable recording medium storing a database having a statistical language model and a kana-kanji conversion program using the statistical language model, The kana-kanji conversion program executes a first process that makes a probability obtained from the statistical language model and a probability that an input sentence word follows a specific symbol becomes a corrected value. Features.

  The invention according to claim 2 is the computer-readable recording medium according to claim 1, wherein, for the predetermined symbol, the predetermined symbol is determined only when the predetermined symbol is determined. The first process is executed.

  The invention according to claim 3 is the computer-readable recording medium according to claim 1 or 2, wherein the kana-kanji conversion program does not include the specific symbol in the input sentence, and When the previously input character string is confirmed, a second process is executed such that the probability that the word following the confirmed character string follows the confirmed character string becomes a corrected value. It is characterized by that.

  According to a fourth aspect of the present invention, in the computer-readable recording medium according to the first or second aspect, the probability obtained by the first processing indicates the specific symbol and the beginning of a sentence. It is a probability obtained from the statistical language model as a sentence mark.

  According to a fifth aspect of the present invention, in the computer-readable recording medium according to the third aspect, the probability obtained by the second process is a probability obtained from the statistical language model. A probability that a word following the confirmed character string follows the confirmed character string, and a probability obtained from the statistical language model by regarding a word included in the confirmed character string as a head mark, and the confirmation The probability that a word following the determined character string follows the determined character string is a probability obtained by linear interpolation.

  According to a sixth aspect of the present invention, in the computer-readable recording medium according to any of the first to fifth aspects, the statistical language model is an n-gram model.

  The invention according to claim 7 is a kana-kanji conversion method using a statistical language model, which is executed on a computer system, and is input with a probability obtained from a statistical language model provided in a database. It is characterized by comprising the step of executing a first process so that a probability that a word of a sentence follows a specific symbol becomes a corrected value.

  The invention according to claim 8 is the computer-readable recording medium according to claim 7, wherein, for the predetermined symbol among the specific symbols, only when the predetermined symbol is determined. The first process is executed.

  In the method according to claim 9, in the method according to claim 7 or claim 8, the input sentence does not include the specific symbol, and the previously input character string is confirmed. In this case, the method further includes a step of executing a second process so that a probability that a word following the confirmed character string continues to the confirmed character string becomes a corrected value.

  The invention according to claim 10 is the method according to any one of claims 7 to 9, wherein the probability obtained by the first processing is the initial mark indicating the specific symbol, the initial mark. This is a probability obtained from the statistical language model.

  The invention according to claim 11 is the method according to claim 9, wherein the probability obtained by the second process is a probability obtained from the statistical language model and the determined character. A probability that a word following a string follows the confirmed character string, and a probability obtained from the statistical language model by regarding a word included in the confirmed character string as a head mark, and the confirmed character string It is a probability obtained by linearly interpolating the probability that the word following is followed by the confirmed character string.

  The invention according to claim 12 is the method according to any one of claims 7 to 11, wherein the statistical language model is an n-gram model.

According to the present invention, it is possible to improve the conversion accuracy of kana-kanji conversion even when the user presses the conversion key finely, and to prevent the user's experience from being impaired.
In addition, since the above effect can be enjoyed only by processing in which the probability for obtaining the conversion candidate is corrected, it is not necessary to make any changes to the statistical language model itself.

  FIG. 1 illustrates an example of a suitable computer system 100 on which the invention may be implemented. The computer system 100 is only one example of a suitable computer system and can be implemented in a distributed computer system where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computer system, program modules for causing a computer to execute predetermined processing can be arranged in recording media of local and remote computers.

  With reference to FIG. 1, a computer system 100, which is an exemplary system for implementing the invention, includes a general purpose computing device shown as computer 110. The components of the computer 110 include a processing unit 120, a system memory 130, and a system bus 121 that couples various system components including the system memory to the processing unit 120.

  The computer 110 generally includes various computer-readable recording media. The computer-readable recording medium may be any medium that can be accessed by the computer 110, and may be any of a volatile medium and a non-volatile medium, a removable medium, and a fixed medium. Such computer readable recording media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital multipurpose disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage (hard disk drive 141 ) Or other magnetic storage device, any medium that can be used to store the desired information and that can be accessed by the computer 110.

  The computer 110 also includes a drive for reading from and writing to removable media. Illustratively, FIG. 1 shows a magnetic disk drive 151 that reads or writes to a removable non-volatile magnetic disk 152, a CD-ROM or other optical that reads or writes to a removable non-volatile optical disk 156. An optical disk drive 155 such as a medium is shown. The hard disk drive 141 is generally connected to the system bus 121 via a fixed interface such as the interface 140, and the magnetic disk drive 151 and the optical disk drive 155 are generally connected to the system bus 121 by a removable interface such as the interface 150. It is connected to the.

  The system memory 130 includes a volatile or non-volatile memory such as a read only memory (ROM) 131 and a random access memory (RAM) 132. A basic input / output system 133 (BIOS) containing basic routines useful for transferring information between elements within the computer 110, such as at startup, is typically stored in the ROM 131. The RAM 132 typically includes data or program modules that are immediately accessible to or processed by the processing unit 120 at that time. As an illustration of these, FIG. 1 shows an operating system 134, application programs 135, other program modules 136 and program data 137.

  In FIG. 1, different numbers are assigned to the operating system 144, the application program 145, the other program modules 146, and the program data 147 at a minimum to indicate that they are different copies. A user may enter commands and information into the computer 110 through input devices such as a keyboard 162 and pointing device 161, commonly referred to as a mouse, trackball or touch pad. Other input devices (not shown) can include a microphone, joystick, game pad, scanner, and the like. These input devices are connected to the processing unit 120 via a user input interface 160 coupled to the system bus, but connected via other interfaces such as a parallel port, game port or universal serial bus (USB). You can also. A monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190. In addition to the monitor, the computer can also include other peripheral output devices such as a speaker 197 and a printer 196 that can be connected via an output peripheral interface 195.

  Computer 110 may also operate in a networked environment using logical connections to one or more remote computers, such as remote computer 180 as described above. The remote computer 180 may be another personal computer, server, router, network PC, peer device or other common network node, and only the storage device 181 is shown in FIG. Includes many or all of the elements described above. The logical connections shown in FIG. 1 include a local area network (LAN) 171 and a wide area network (WAN) 173, but can also include other networks. Such networking environments are common in computer networks such as intranets and the Internet.

  When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173, such as the Internet. A built-in or external modem 172 can be connected to the system bus 121 via a user input interface 160. In a network environment, the program modules illustrated for computer 110 or portions thereof may be stored in a remote storage device. Illustratively, FIG. 1 shows remote application program 185 as residing on storage device 181.

  In the following description, unless otherwise specified, the present invention can be executed by the processing unit 120 loaded with the computer executable instructions of the application program (kana-kanji conversion program) into the system memory 131 based on the computer executable instructions. The operation will be described. In this operation, the processing unit 120 refers to or updates the program data 137 based on computer-executable instructions. The kana-kanji conversion program according to the present invention can be recorded on the computer-readable recording medium and provided to the user, or can be distributed to the user via the communication medium.

  FIG. 2 is a diagram showing the details of the program data 137 according to the present embodiment in more detail, and schematically shows only the parts related to the present invention (details will be described later).

  The program data 137 includes a corpus 202, a dictionary 204, and a user dictionary 206. The corpus 202 is large-scale text data used for natural language processing or the like, in which a character string is divided for each morpheme, and a part of speech is determined for each morpheme (that is, part-of-speech tagging). In addition, information to which syntactic information such as dependency is added can also be used as the corpus 202. The dictionary 204 is data defining an identifier (ID) for each word and part of speech. Here, the word includes notation of characters and their reading. The user dictionary 206 is one of registered dictionaries that are created by registering words and fixed phrases for the convenience of individual users. Here, in addition to the user dictionary, the registration dictionary may be registered by a vendor such as a specialized dictionary or a field-specific dictionary.

Here, the n-gram model will be briefly described.
n-1 word strings W ₁ W ₂ . . . The probability that the _nth word Wn follows Wn ₋₁ is indicated by the following conditional probability.
_{P = P (W n | W} 1 W 2 ... W n-1)
_{= P (W 1 W 2 ...} W n) / P (W 1 W 2 ... W n-1)
This is called an n-gram model. Hereinafter, a trigram model (that is, n = 3 in the above formula) will be described as an example, but the present invention is another multigram model such as a bigram model (hereinafter collectively referred to as a statistical language model). Needless to say, this is also applicable.

FIG. 3 is a diagram showing an outline of a kana-kanji conversion method according to the present embodiment.
The computer system shown in FIG. 1 is used as the computer system used in the following description.

  In step S3002 shown in FIG. 3, the computer system uses the dictionary 202 and the user dictionary to create a set (lattice) of ID combinations (paths) in which word IDs and part-of-speech IDs are mixed from the input hiragana 704. In step S3004, the probability that each path occurs is extracted from the statistical language model 304, and a path-probability correspondence table 708 in which each path is associated with the probability is generated. In step S3006, the path with the highest probability is selected as the conversion candidate 710 from the path-probability correspondence table 708. In step S3008, the selected path is converted into a kana-kanji character string 712 using the dictionary 202 and the user dictionary 206 (for details of the kana-kanji conversion method, refer to, for example, Patent Document 1). ).

  As described above, Kana-Kanji conversion using a statistical language model is performed on a computer system. However, as described above, conversion results are different depending on whether there is a specific symbol in the first segment input or not. There is a problem of being different. This point will be described below.

For example, it is assumed that a space (blank character: represented here by □) is input at the beginning (start of sentence) of a line, and “Kyo” is further input. The word string in this case is
<S> □ Today
It becomes. Here, the beginning mark indicating the beginning of the sentence is represented by <s> and the end of sentence mark is represented by </ s>. The sentence head mark <s> is automatically added after a line feed.
On the other hand, it is assumed that “Kyo” is input from the beginning of the sentence. in this case,
<S> Today </ s>
It becomes.

  In the above two cases, the respective trigrams P (today | <s>, □) and P (today | <s>, <s>) (in this case, <s 2> are different). Therefore, the result of Kana-Kanji conversion may be different. Similarly, the conversion result differs depending on whether or not the bracket ““ ”comes at the beginning of the sentence. For example, ““ Kyoto ”is converted to“ “Strong” instead of “Today”.

If the user wishes to convert “today” to “today” and there is a specific symbol such as a space or an angle bracket at the beginning of the sentence (or in the sentence), “ , "Today" should be converted.
In order to solve this problem, in the present embodiment, when a specific symbol such as a space or an angle bracket appears in the beginning of a sentence or a sentence, the specific symbol is regarded as a beginning mark <s> under a predetermined condition, and a word trigram P for W ₃ (W ₃₎ and to that corrected, (details will be described later) for determining the probability of occurrence of a path in the kana-kanji conversion described above.

  Here, the specific symbol will be described. In this embodiment, specific symbols are: Open bracket group (Open brackets group), 2. The two bullet groups are handled (in this case, the space can be treated the same as the open bracket group). That is,

  In the case of a symbol of 1, regardless of whether the symbol is confirmed or not yet confirmed, the symbol is regarded as a head mark and a corrected probability value is obtained. In the case of the symbol of 2, it is determined whether the symbol has been confirmed or not yet confirmed, and if it has been confirmed, the symbol is regarded as a head mark and a corrected probability value is obtained.

  For example, when ““ AB ”or“ “” determined “AB” is typed (in the case of a symbol of 1), a corrected probability value is obtained by regarding ““ <s> <s> AB ””. On the other hand, when “● AB” is typed (in the case of symbol 2), a normal probability value is obtained, and when “●” is confirmed and “AB” is typed, this is replaced with “● <s>”. <S> AB ”is regarded as a corrected probability value. Here, “confirm” on the right side of the symbol indicates that the previous symbol has been confirmed. In addition, the symbol “” is a symbol belonging to the open bracket group, but in this specification, it is also used to emphasize and indicate a symbol / character string (for example, “●”). It should be understood that this symbol does not indicate that it is being typed.

In the above, the case where a specific symbol exists is mentioned, but even when there is no specific symbol, the expected conversion for the subsequent character string differs depending on whether there is a predetermined character string or not. There is.
For example, suppose that “to go to the doctor today” corresponds to the path having the highest probability for “going to today”. However, when “Today” is confirmed and the user inputs “go to the hospital”, the user expects not to “go to the doctor” but to “go to the dentist” in most cases. However, in the statistical language model, “Go to the doctor” has the highest probability. On the other hand, “going to the dentist” has the highest probability of “going to saishani” connected to the sentence mark. In this way, once the user has confirmed, there is a case where it is better to consider the probability when it is connected to the sentence head mark. In order to solve this problem, in such a case, linear interpolation is performed as in the following equation.
(1) When a word / symbol W ₃ is inserted after PrevW ₁ PrevW ₂ of a confirmed word / symbol:
P (W ₃ ) = P (W ₃ | <s>, <s>) * (1-α)
+ P (W ₃ | PrevW ₁ , PrevW ₂ ) * α
(2) When uncertain words / symbols W ₂ and W ₃ are inserted after PrevW ₁ of the confirmed word / symbol:
P (W ₃ ) = P (W ₃ | <s>, W ₂ ) * P (W ₂ | <s>, <s>) * (1-α)
+ P (W ₃ | PrevW ₁ , W ₂ ) * α
As a correction. In the above equation, 0 ≦ α ≦ 1, and α is appropriately set as a fixed value so that a better kanji conversion is performed. For example, in the above example, α = 0 is preferable.

Next, an example of a process for obtaining the corrected probability value will be described with reference to a conceptual program source shown in FIG.
In the trigram, the probability that the _third word W ₃ continues from the three word strings W ₁ W ₂ W ₃ is obtained. Here, these three words are represented by Left, Middle, and Right.

  In the process shown in FIG. 4, first, it is determined whether or not a previously confirmed character string (that is, a confirmed character string) exists in the Middle, and whether the Middle is a bullet, a space, or an open bracket (condition 1). ing. When this condition 1 is satisfied, a sentence head mark is set for each of Middle and Left.

  Next, when the above condition 1 is not satisfied, that is, when there is no pre-determined character string in Middle, or when Middle is not any of bullet, space, or open bracket, a pre-determined character string exists in Left, In addition, it is determined whether Left is a bullet, a space, or an open bracket (condition 2). At this stage, when the condition 2 is satisfied, the head mark <s> is set in Left.

  Next, when the above condition 2 is not satisfied, that is, when there is no preceding fixed character string in Middle, or when Middle is not any of bullet, space, or open bracket, and there is no preceding fixed character string in Left. If Left is not a bullet, space, or open bracket, it is further determined whether Middle is an open bracket (condition 3). At this stage, if the condition 3 is satisfied, the head mark <s> is set in each of Middle and Left.

  Next, when the above condition 3 is not satisfied, that is, there is no pre-defined character string in Middle, or Middle is not any of bullet, space, open bracket, and there is no pre-defined character string in Left, Alternatively, if Left is not a bullet, space, or open bracket, and Middle is not an open bracket, it is determined whether Left is an oven bracket (condition 4). At this stage, if Left is an open bracket, only the Left character is set with a head mark <s>.

  If none of the above conditions 1, 2, 3, and 4 is satisfied, Middle and Left are left at their original settings (contents of Middle and Left of the original word string).

  The above-described trigram is calculated using Middle and Left set as described above. In the figure, it is represented by the function GetTrigram (Left, Middle, Right), which is set as “normal probability” (if the above conditions are not satisfied, it is just a normal probability, and either condition is satisfied) The corrected probability is set to this “normal probability”).

  Next, if the word string consisting of Left and Middle does not include any of spaces, bullets, spaces, and open brackets (these are referred to as “special char” in FIG. 4) and includes a pre-determined character string, the following processing is further performed I do.

When Middle is a pre-determined character string (in this case, of course, Left is also a pre-determined character string), P (W ₃ | <s>, <s>) is used as the probability of a sentence start (sentence mark). Ask. On the other hand, when Middle is not a pre-determined character string, only Left is a pre-determined character string, and in this case, P (W ₃ | <s>, W ₂ ) * P (W ₂ ) as the probability of Sentence Start. | <S>, <s>).
And as a normal probability
Normal probability = normal probability * α + probability with Sentence Start * (1−α)
Ask for. Here, the coefficient α is appropriately set so that preferable kana-kanji conversion is performed in the range of 0 ≦ α ≦ 1, as described above.

  Using the “normal probability” calculated as described above, the final probability (= normal probability * probability to previous word) is calculated. This final probability is used as the probability of occurrence of the corrected corresponding path, and kana-kanji conversion is performed as described above.

By processing as described above, the corrected probability value of a word W ₃ when there is a specific symbol on the beginning of a sentence or statement (trigram value), a word W ₃ is equal to the probability in a case in beginning of a sentence, Therefore, it is possible to solve the problem that the conversion result differs depending on whether or not there is a specific symbol such as a space or an angle bracket in the beginning or sentence.
In addition, even if there is no specific symbol, the expected conversion for the subsequent character string may be different depending on whether there is a pre-determined character string or not. A more preferable conversion result can be obtained by performing linear interpolation.
Therefore, according to the present embodiment, the problem described above can be dealt with only by obtaining the probability value corrected as described above, without training (adjusting) the statistical language model itself. .

FIG. 2 illustrates a computer system that constitutes an exemplary system for implementing the invention. It is a block diagram which shows the content of the program data in embodiment by this invention. It is a figure which shows the outline | summary of operation | movement of the kana-kanji conversion method in the same embodiment. It is a conceptual program source which shows the process example for obtaining the corrected probability value (trigram) by this invention.

Explanation of symbols

100 Computer System 110 Computer 120 Processing Unit 121 System Bus 130 System Memory 131 Read Only Memory 132 Random Access Memory 133 Basic Input / Output System 134 Operating Module 135 Application Program 136 Other Program Modules 137 Program Data 140 Non-Removable Non-volatile Memory Interface 141 Hard disk drive 144 Operating system 145 Application program 146 Other program modules 147 Program data 150 Removable non-volatile memory interface 151 Magnetic disk drive 152 Removable non-volatile magnetic disk 155 Optical disk drive 156 Removal Capacity, nonvolatile optical disk 160 user input interface 161 a pointing device 162 keyboard 170 adapter 171 local area network (LAN)
172 Modem 173 Wide Area Network (WAN)
180 remote computer 181 storage device 184 multi-level cache 185 remote application program 190 video interface 191 monitor 195 output peripheral interface 196 printer 197 speaker 202 corpus 204 dictionary 206 user dictionary 304 statistical language model 704 input hiragana 706 lattice 708 path-probability correspondence table 710 Conversion candidate 712 Kana-Kanji character string

Claims (12)

A computer-readable recording medium for storing a database including a statistical language model and a kana-kanji conversion program using the statistical language model,
The kana-kanji conversion program executes a first process that makes a probability obtained from the statistical language model and a probability that an input sentence word follows a specific symbol becomes a corrected value. A computer-readable recording medium characterized by the above.   2. The computer-readable recording according to claim 1, wherein the first process is executed for a predetermined symbol among the specific symbols only when the predetermined symbol is confirmed. Medium.   When the input sentence does not include the specific symbol and the previously input character string is confirmed, the kana-kanji conversion program determines that the word following the confirmed character string is the confirmed character string. 3. The computer-readable recording medium according to claim 1, wherein the second process is performed so that the probability following the first value becomes a corrected value. 4.   The probability obtained by the first processing is a probability obtained from the statistical language model by regarding the specific symbol as a head mark indicating a head of the sentence. The computer-readable recording medium as described.   The probability obtained by the second process is a probability obtained from the statistical language model, and a probability that a word following the confirmed character string follows the confirmed character string, and the confirmed character By linearly interpolating the probabilities obtained from the statistical language model with the words included in the sequence as sentence head marks, and the probabilities that the words following the confirmed character string follow the confirmed character string The computer-readable recording medium according to claim 3, wherein the probability is obtained.   The computer-readable recording medium according to any one of claims 1 to 5, wherein the statistical language model is an n-gram model. A kana-kanji conversion method using a statistical language model executed on a computer system,
And a step of executing a first process in which a probability obtained from a statistical language model provided in a database and a word of an inputted sentence follows a specific symbol becomes a corrected value. Feature method.   The method according to claim 7, wherein the first process is executed only for a predetermined symbol among the specific symbols when the predetermined symbol is confirmed.   When the input sentence does not include the specific symbol and the previously input character string is confirmed, the probability that the word following the confirmed character string continues to the confirmed character string is corrected. The method according to claim 7, further comprising a step of executing a second process for obtaining the determined value.   The probability obtained by the first processing is a probability obtained from the statistical language model by regarding the specific symbol as a head mark indicating a head of the sentence. The method according to any one.   The probability obtained by the second process is a probability obtained from the statistical language model, and a probability that a word following the confirmed character string follows the confirmed character string, and the confirmed character By linearly interpolating the probabilities obtained from the statistical language model with the words included in the sequence as sentence head marks, and the probabilities that the words following the confirmed character string follow the confirmed character string The method of claim 9, wherein the probability is obtained. 12. A method according to any of claims 7 to 11, wherein the statistical language model is an n-gram model.

Images