Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F40/00—Handling natural language data
  • G06F40/20—Natural language analysis
  • G06F40/232—Orthographic correction, e.g. spell checking or vowelisation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F40/00—Handling natural language data
  • G06F40/10—Text processing
  • G06F40/12—Use of codes for handling textual entities
  • G06F40/126—Character encoding
  • G06F40/129—Handling non-Latin characters, e.g. kana-to-kanji conversion

Definitions

• the present invention relates generally to processing non-Roman based languages. More specifically, systems and methods to process and correct spelling errors for non- Roman based words such as in Chinese, Japanese, and Korean languages using a rule-based classifier and a hidden Markov model are disclosed.
• Description of Related Art Spell correction generally includes detecting erroneous words and determining appropriate replacements for the erroneous words.
• an effective spell checker for a non- Roman based language should make use of contextual information to determine which characters and/or words in context are not suitable.
• Spell correction for non-Roman languages such as CJK languages is also complex and challenging in that there are no standard dictionaries in such languages because the definition of CJK words are not clean. For example, some may regard "Beijing city" in Chinese as one word while others may regard them as two words.
• the English dictionary/wordlist lookup is a key feature in English spell correction and thus English spell correction methods cannot be easily adapted for use in CJK languages.
• the systems and methods use transformation rules, hidden Markov models and similarity matrix of confusing characters.
• the similarity between a pair of confusing characters may be a positive number if the characters have the same pronunciation and/or share some input keystrokes in simplified or traditional Chinese. Otherwise, the value is zero.
• the similarity may have a Boolean value, e.g., 1 for a pair of confusing characters and 0 for a pair of non-confusing characters.
• the systems and methods are particularly applicable to web-based search engines and downloadable applications at client sites, e.g., implemented in a toolbar or deskbar, but are applicable to various other applications.
• the present invention can be implemented in numerous ways, including as a process, an apparatus, a system, a device, a method, or a computer readable medium such as a computer readable storage medium or a computer network wherein program instructions are sent over optical or electronic communication lines.
• the term computer generally refers to any device with computing power such as personal digital assistants (PDAs), cellular telephones, and network switches.
• the method generally includes converting an input entry in a first language such as Chinese to at least one intermediate entry in an intermediate representation, such as pinyin, different from the first language, converting the intermediate entry to at least one possible alternative spelling of the input in the first language, and determining that the input entry is either a correct or questionable input entry when a match between the input entry and all possible alternative spellings to the input entry is or is not located, respectively.
• a first language such as Chinese
• an intermediate representation such as pinyin
• pinyin refers to all phonetic notations for Chinese, simplified or traditional, include zhuyin fuhao (Bopomofo), i.e., "The Notation of Annotated Sounds.” Similarity between pairs of confusing characters in the first language can be defined according to common tokens in the intermediate representation.
• the questionable input entry may be classified using, for example, a transformation rule based classifier based on transformation rules generated by a transformation rules generator.
• a transformation rule based classifier based on transformation rules generated by a transformation rules generator.
• Various other classifiers such as decision tree and neural network classifiers may be similarly employed.
• the converting may include converting multiple input entries, such as user queries in a query log.
• the method may further include classifying, e.g., by a transformation rule based classifier, the questionable entry as a correctly spelled or an incorrectly spelled entry based on a set of rules such as spell correction transformation rules. Users' votes, e.g., query logs and/or webpages, are preferably utilized to generate the transformation rules.
• the method may also include generating and training the spell correction transformation rules using a transformation rules generator using the questionable input entry and the possible alternative spellings.
• the method may further include receiving a user input in the first language, determining whether any of the rules apply to the user input, generating at least one alternate spelling in the first language corresponding to the user input upon determining that at least one rule applies to the user input, comparing a likelihood of the user input with a likelihood of at least one alternate spelling of the user input, and making a spell correction suggestion and/or a spell correction with at least one alternate spelling of the user input that has a higher likelihood than the user input.
• a system generally includes a first converter configured to convert an input in a first language to at least one intermediate representation of the input entry, the intermediate representation being different from the first language, a second converter configured to convert the intermediate representation to at least one possible alternative spelling of the input in the first language, locating a match by comparing the possible alternative spelling to the input entry, and determining that the input entry is a questionable input entry if a match is not located from all the possible alternative spellings and that the input entry is a correct input entry if a match is located.
• a computer program product for use in conjunction with a computer system having a computer readable storage medium on which are stored instructions executable on a computer processor, the instructions generally including receiving an input entry in a first language, converting the input entry to at least one intermediate representation of the input entry, the intermediate representation being different from the first language, converting the intermediate representation to at least one possible alternative spelling in the first language, locating a match by comparing at least one possible alternative spelling to the input entry, and determining that the input entry is a questionable input entry if a match is not located from all the possible alternative spellings and that the input entry is a correct input entry if a match is located.
• An application implementing the system and method may be implemented on a server site such as on a search engine or may be implemented on a client site such as a user's computer, e.g., downloaded, to provide spell corrections for text inputting into a document or to interface with a remote server such as a search engine.
• the client site application may optionally include a user-editable table of stop rule patterns that allows the user to customize the application by specifying that certain spell corrections are disallowed, e.g., never replace X and Y except when X precedes or follows Z.
• FIG. 1 is block diagram of an illustrative system and method for performing forward and reverse conversions to and from an intermediate form of the non-Roman based language to determine possible alternate spellings for questionable original inputs.
• FIG. 2 is block diagram of an illustrative system and method for generating spell correction transformation rules from a set of entries.
• FIG. 3 is a flowchart illustrating a process for automatically generating spell correction transformation rules.
• FIG. 4 is a flowchart illustrating a process utilizing the transformation rules for processing an entry to determine spell correction suggestions, if any.
• alternate spelling or alternate form of an input is used herein to refer to an alternate set of characters and/or words different from the input but in the same language as the input, whether the input is a single character or word, a series or collection of characters and/or words, a phrase, a sentence, etc.
• the questionable input entries are identified from input entries and possible alternate spellings are generated by the questionable input entry detector illustrated in FIG. 1.
• the spell correction transformation rules are then generated and trained and the questionable entries are classified as correct or incorrect by the transformation rules generator and classifier as shown in FIG. 2.
• the systems and methods use transformation rules, hidden Markov models and similarity matrix of confusing characters.
• the similarity between a pair of confusing characters may be a positive number if the characters have the same pronunciation and/or share some input keystrokes in simplified or traditional Chinese. Otherwise, the value is zero.
• the similarity may have a Boolean value, e.g., 1 for a pair of confusing characters and 0 for a pair of non-confusing characters.
• FIG. 1 is block diagram of an illustrative questionable input entry detector 100 for performing forward and reverse conversions to and from an intermediate form, e.g., pinyin, of simplified Chinese to identify questionable original inputs and to determine possible alternate spellings for questionable original inputs.
• the questionable input entry detector 100 illustrated in FIG. 1 makes use of the convenient fact that pinyin is a commonly-used input method for simplified Chinese. However, any other intermediate form, Roman-based or non-Roman based, may be implemented and utilized. Similarly, the questionable input entry detector 100 may be adapted for use with various other non-Roman based languages. As shown in FIG.
• a word-pinyin converter 104 converts each original entry 102 in Chinese characters into one or more pronunciations or pinyins 106 corresponding to the original entry 102.
• a pinyin- word converter 108 then converts the pinyins 106 to possible spellings 110 in Chinese characters.
• Other suitable converters 104, 106 for converting text in a first language to an intermediate representation and then back to the first language may be employed. Pinyin is merely a convenient intermediate representation for Chinese or simplified Chinese.
• a comparer 1 12 compares the original entry 102 with the possible spellings 110, both in the first language, to determine if there is a match.
• Pinyin is a phonetic input method used mainly for inputting simplified Chinese character. As referred to herein, pinyin generally refers to phonetic representation of Chinese characters, with or without representation of the tones associated with the Chinese characters.
• pinyin refers to all phonetic notations for Chinese, simplified or traditional, include zhuyin fuhao (Bopomofo), i.e., "The Notation of Annotated Sounds.” Pinyin uses Roman characters and has a vocabulary listed in the form of multiple syllable words. Because Chinese has numerous homographs and homophones, each original entry 102 may be converted into multiple pinyins 106 by the word-pinyin converter 104 and, similarly, each pinyin 106 may be converted into multiple possible spellings in Chinese characters 110 by the piny in- word converter 108.
• one phonetic syllable may correspond to many different Hanzi.
• the pronunciation of "yi" in Mandarin can correspond to over 100 Hanzi.
• the processes implemented by the word-pinyin converter 104 and the pinyin-word converter 108 of converting each original entry 102 to pinyin 106 and then back to Chinese characters 110 may be non-trivial given the large proportion of Chinese words that are homographs and/or homophones.
• the systems and methods as described herein use transformation rules, hidden Markov models and similarity matrix of confusing characters.
• the similarity between a pair of confusing characters may be a positive number if the characters have similar pronunciation, share similar input keystrokes, and/or are similarly spelled, i.e., visually similar. Otherwise, the value is zero.
• the similarity may have a Boolean value, e.g., 1 for a pair of confusing characters and 0 for a pair of non- confusing characters.
• the similarity between a pair of confusing characters in the first language can be defined according to common tokens in the intermediate representation.
• Various suitable mechanisms for converting Chinese words to pinyins and for converting pinyins to Chinese words may be implemented.
• various decoders are suitable for translating pinyin to Hanzi (Chinese characters).
• a Viterbi decoder using hidden Markov models may be implemented.
• the training for the hidden Markov models may be achieved, for example, by collecting empirical counts or by computing an expectation and performing an iterative maximization process.
• the Viterbi algorithm is a useful and efficient algorithm to decode the source input according to the output observations of a Markov communication channel.
• the Viterbi algorithm has been successfully implemented in various applications for natural language processing, such as speech recognition, optical character recognition, machine translation, speech tagging, parsing and spell checking.
• various other suitable assumptions may be made in implementing the decoding algorithm.
• the Viterbi algorithm is merely one suitable decoding algorithm that may be implemented by the decoder and various other suitable decoding algorithms such as a finite state machine, a Bayesian network, a decision plane algorithm (a high dimension Viterbi algorithm) or a Bahl-Cocke-Jelinek-Raviv (BCJR) algorithm (a two pass forward/backward Viterbi algorithm) may be implemented.
• the questionable entries detected by the questionable input entry detector 100 generally include nearly all spelling errors. However, the questionable entries also generally include relatively high false-alarm/false-positive rate, , i.e., ratio of the number of correct queries marked as incorrect to the number of incorrect queries.
• the questionable queries 116 as determined by the questionable entry detector 100 may then be classified as correct or incorrect.
• the classifier may be a Transformation Rule Based classifier, as is preferred, or may be a decision tree classifier, a neural network classifier, and the like.
• FIG. 2 is block diagram of an illustrative system and method 120 for generating spell correction transformation rules from a set of original entries 102 as processed by the questionable entry detector 100.
• the set of original entries 102 may include user input entries such as query logs for a web search engine and/or entries derived from documents such as those available on the Internet, for example.
• the set of original inputs 102 may include a collection of user queries from the past three weeks or two months, for example.
• Examples of documents may include web content and various publications such as newspaper, books, magazines, webpages, and the like.
• the set of original inputs 102 may be derived from a set, collection or repository of documents, for example, documents written in simplified and/or traditional Chinese available on the Internet. It is noted that the illustrative systems and methods as described herein are particularly applicable in the context of a web search engine and to a search engine for a database containing organized data. However, it is to be understood that the systems and method may be adapted and employed for various other applications for spelling error detection and correction, particularly for entries in a non-Romanized language.
• the system and method may be adapted for a CJK text input application, e.g., word processing application, that detects and corrects spelling errors.
• the transformation rules generator and classifier 120 implements a transformation based learning algorithm, introduced by Eric Brill, that, during the training process, automatically extracts (learns) and ranks transformation rules according to confidence measurements from training data, e.g., human annotated incorrect spellings. These transformation rules are used by the annotator/voter 124. Note that transformation rules are different from grammar rules used in linguistics in that the transformation rules are based on statistics rather than linguistic knowledge. Thus, for example, if most of the entries incorrectly spell certain words in the same incorrect way, the incorrect spelling would be classified as correct.
• Transformation Rule Based methods are presented in US Pat. No. 6,684201 issued on Jan. 27, 2004 to Eric Brill and entitled "Linguistic Disambiguation System and Method Using String-Based Pattern Training to Learn to Resolve Ambiguity Sites," the entirety of which is incorporated by reference herein.
• the transformation rules generator 120 generates rules automatically, i.e., unsupervised, by utilizing the users' votes. In other words, the correctness of a pattern of characters is determined according to the majority of votes in the database, e.g., the query logs, rather than human annotated data.
• Each transformation rule is associated with a confidence measurement such that rules with higher confidence measurements are applied later than rules with lower confidence measurements.
• a first transformation rule may specify replacing X with Y if B precedes X.
• a second transformation rule with a higher confidence measurement may specify replacing Y with X if E follows Y.
• the first transformation rule would first be applied to an entry BXE to generate BYE.
• the second transformation rule would then be applied to the resulting entry BYE to converted the entry back to BXE.
• the order that the transformation rules are applied can affect the outcome.
• the characters being replaced and the replacement characters may be any component of the entry and need not necessarily be words.
• the condition may be based on any context, part-of-speech tags or grammatical non-terminal labels (e.g., NP for noun phrase).
• each questionable entry 116 and its corresponding possible alternate spellings 110 output by the questionable entry detector 100 is received by the annotator 124 of the spell correction transformation rules generator 120.
• the annotator 124 classifies entries 128 based initially on the initial transformation rules 126 and eventually on the extracted and ranked transformation rules 130.
• the learning phase may be supervised, i.e., by human personnel, and/or unsupervised.
• an initial set of a few common manually created transformation rules is used to automatically annotate a small set of questionable entries, with some human monitoring or without any human monitoring by utilizing users' votes.
• additional transformation rules are generated, preferably also with some human monitoring, and additional questionable entries are annotated.
• the resulting rules which govern a significant amount of user traffic for example, with relatively few rules may be regarded as very reliable and thus correspond to a high confidence measurement. Note that since rules with higher confidence typically have less coverage than those with lower confidence, both rules with high confidence and rules with comparatively lower confidence are used.
• the relatively large number of remaining questionable entries that account for a relative small proportion of user traffic may be automatically generated without human monitoring, for purposes of cost efficiency.
• One illustrative process 150 for automatically generating such rules is shown in the flowchart of FIG. 3.
• a comparison of Q and the alternate spelling Q' is made at block 156 to determine characters in Q that are possibly improper and their substitutions C.
• a window of width 2N+1 is opened with N preceding characters and N succeeding characters of C.
• any suitable length of context e.g., 2N+1, may be implemented and the length of context before and after the character in question may but need not be equal.
• the frequencies F(pre-C, C, post-C) of all subsequences (pre-C, C, post -C) from C_ ⁇ -N ⁇ , ..., C,..., C_ ⁇ N ⁇ are counted to ensure that the rule is significant, i.e., if the rule can cover a reasonable large portion of spelling errors in the questionable entries.
• Decision block 162 determines whether the rule is reliable, e.g., by. using query logs and webpages, i.e., users' voting. If the rule is determined to be reliable, the transformation rule, i.e., substitute C for C given pre-C, post-C, is extracted. Specifically, the rule is deemed to be reliable if: F(pre-C, C, post-C) > Tl and F(pre-C, C, post-C) / F(pre-C, C, post-C) > T2, where Tl is a minimum significance threshold and T2 is a minimum confidence threshold.
• the process 150 implemented by the transformation rules generator generates rules automatically, i.e., unsupervised, by utilizing the users' votes such that the correctness of a pattern of characters is determined according to the majority of votes in the database, e.g., the query logs, rather than human annotated data. Because the most frequent transformation rules will govern a very large portion of the error patterns, the size of the rule set preferably does not increase rapidly with the number of questionable entries. A minimum occurrence of each rule may also be set to limit the size of the transformation rule set.
• An application implementing the systems and methods described herein may be implemented on a server site such as on a search engine or may be implemented on a client site such as an end user's computer, e.g., downloaded, to provide spell corrections for text inputting into a word processing document or to interface with a remote server such as a search engine.
• the client site application may be implemented, for example, in a toolbar, and may optionally include a user-editable table of stop rule patterns that allows the user to customize the application by specifying that certain spell corrections are disallowed, e.g., never replace X and Y except when X precedes or follows Z.
• FIG. 4 is a flowchart illustrating a process 200 utilizing the transformation rules for processing an entry to determine spell correction suggestions, if any.
• Decision block 202 determines if any spell correction rule applies to the user input.
• a hash table of the spell correction transformation rules may be examined to determine if any transformation rule applies to the user input. For example, for a given Chinese user input ABCDE, if a transformation rule dictates that character C be replaced with C if the preceding characters to C are AB, then this particular rule is applicable to the user input. If no rules are applicable to the user input, no spell correction suggestion is made for user input. Alternatively, for each spell correction transformation rule that is applicable to the user input, alternate spellings for the user input corresponding to the applicable spell correction transformation rule are generated at block 204. In the example above, an alternate spelling ABCDE is generated for the user input ABCDE corresponding to the applicable spell correction transformation rule.
• decision block 206 the likelihood of each alternate spelling is determined and compared to the likelihood of the user input.
• decision block 206 may utilize the hidden Markov model and the Viterbi decoder to compute the likelihood.
• the relative output probabilities of ABCDE and ABCDE are determined and compared.
• the alternate spelling has a higher likelihood than the user input and thus regarded as a valid correction if: P(ABCDE) * P(transformation rule) > P(ABCDE),
• P(transformation rule) may be defined as the ratio of the number of successful corrections and the total number of corrections.
• P(ABCDE) should take into account the ambiguity in segmentation. For example, if ABCDE has two possible segmentations AB-CDE and ABC-DE, then the probably is a sum of products of Bayesian probabilities:
• P(ABCDE) P(input-end
• the equation above is a Bayesian probability derived from the original Bayesian probability by applying the Markov assumption which determines the current word by the preceding word rather than by the entire history. The determination of P(ABCDE) may be similarly made.
• the particular spell correction suggestion is not made. However, if the given alternate spelling is more likely than the user input as determined at decision block 206, the corresponding alternate spelling for the user's input is suggested and/or automatically made at block 208.
• the systems and method for spell correction as described herein are particularly well suited for use with non-Roman based languages and can be highly effective in both detecting spelling errors and in generating alternate spelling suggestions or corrections.
• the systems and method for spell correction are also particularly applicable in the context of a web search engine and to a search engine for a database containing organized data in performing spell correction of various user inputs or queries.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Artificial Intelligence (AREA)
• Audiology, Speech & Language Pathology (AREA)
• Computational Linguistics (AREA)
• General Health & Medical Sciences (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Document Processing Apparatus (AREA)
• Machine Translation (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (2)

Family

ID=35427493

Family Applications (1)

Country Status (5)

Families Citing this family (156)

Family Cites Families (22)

• 2004
  • 2004-06-23 US US10/875,449 patent/US20050289463A1/en not_active Abandoned
• 2005
  • 2005-06-21 JP JP2007518226A patent/JP2008504605A/ja not_active Withdrawn
  • 2005-06-21 KR KR1020077001543A patent/KR101146539B1/ko not_active Expired - Fee Related
  • 2005-06-21 WO PCT/US2005/022027 patent/WO2006002219A2/en not_active Ceased
  • 2005-06-21 CN CN2005800263504A patent/CN101002198B/zh not_active Expired - Fee Related
• 2011
  • 2011-11-04 JP JP2011242872A patent/JP5444308B2/ja not_active Expired - Fee Related

Also Published As

Similar Documents

Legal Events