EP1055182A2 - Segmentation of chinese text into words - Google Patents
Segmentation of chinese text into wordsInfo
- Publication number
- EP1055182A2 EP1055182A2 EP99902779A EP99902779A EP1055182A2 EP 1055182 A2 EP1055182 A2 EP 1055182A2 EP 99902779 A EP99902779 A EP 99902779A EP 99902779 A EP99902779 A EP 99902779A EP 1055182 A2 EP1055182 A2 EP 1055182A2
- Authority
- EP
- European Patent Office
- Prior art keywords
- character
- words
- characters
- word
- facility
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/211—Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/268—Morphological analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/284—Lexical analysis, e.g. tokenisation or collocates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
- G06F40/53—Processing of non-Latin text
Definitions
- the invention relates generally to the field of natural language processing, and, more specifically, to the field of word segmentation.
- Word segmentation refers to the process of identifying the individual words that make up an expression of language, such as text. Word segmentation is useful for checking spelling and grammar, synthesizing speech from text, and performing natural language parsing and understanding, all of which benefit from an identification of individual words.
- Table 3 Despite the absence of punctuation and spaces from the sentence, a reader of Chinese would recognize the sentence in Table 3 as being comprised of the words separately underlined in Table 4 below.
- a word segmentation software facility (“the facility”) provides word segmentation services for text in unsegmented languages such as Chinese by (1) evaluating the possible combinations of characters in an input sentence and discarding those unlikely to represent words in the input sentence, (2) looking up the remaining combinations of characters in a dictionary to determine whether they may constitute words, and (3) submitting the combinations of characters determined to be words to a natural language parser as alternative lexical records representing the input sentence.
- the parser generates a syntactic parse tree representing the syntactic structure of the input sentence, which contains only those lexical records representing the combinations of characters certified to be words in the input sentence.
- the facility weights the lexical records so that longer combinations of characters, which more commonly represent the correct segmentation of a sentence than shorter combinations of characters, are considered by the parser before shorter combinations of characters.
- the facility adds to the dictionary, for each character occurring in the dictionary, (1) indications of all of the different combinations of word length and character position in which the word appears, and (2) indications of all of the characters that may follow this character when this character begins a word.
- the facility further adds (3) indications to multiple-character words of whether sub-words within the multiple- character words are viable and should be considered.
- the facility discards (1) combinations of characters in which any character is used in a word length/position combination not occurring in the dictionary, and (2) combinations of characters in which the second character is not listed as a possible second character of the first character.
- the facility further discards (3) combinations of characters occurring in a word for which sub-words are not to be considered.
- the facility both minimizes the number of character combinations looked up in the dictionary and utilizes the syntactic context of the sentence to differentiate between alternative segmentation results that are each comprised of valid words.
- Figure 1 is a high-level block diagram of the general -purpose computer system upon which the facility preferably executes.
- Figure 2 is an overview flow diagram showing the two phases in which the facility preferably operates.
- Figure 3 is a flow diagram showing the steps preferably performed by the facility in order to augment the lexical knowledge base in the initialization phase to include information used to perform word segmentation.
- Figure 4 is a flow diagram showing the steps preferably performed in order to determine whether a particular word can contain other, smaller words.
- Figure 5 is a flow diagram of the steps preferably performed by the facility in order to segment a sentence into its constituent words.
- Figure 6 is a flow diagram showing the steps preferably performed by the facility in order to add multiple-character words to the word list.
- Figure 7 is a flow diagram showing the step preferably performed by the facility in order to test the NextChar and CharPos conditions for a word candidate.
- Figure 8 is a flow diagram showing the steps preferably performed by the facility in order to determine whether the last character of the current word candidate overlaps with another word candidate that may be a word.
- Figure 9 is a flow diagram showing the steps preferably performed by the facility in order to add single-character words to the word list.
- Figure 10 is a flow diagram showing the steps preferably performed by the facility in order to assign probabilities to the lexical records generated from the words in the word list in accordance with a first approach.
- Figure 11 is a flow diagram showing the steps preferably performed by the facility in order to assign probabilities to the lexical records generated from the words in the word list in accordance with a second approach.
- Figure 12 is a parse tree diagram showing a parse tree generated by the parser representing the syntactic structure of the sample sentence.
- a word segmentation software facility (“the facility”) provides word segmentation for text in unsegmented languages such as Chinese by (1) evaluating the possible combinations of characters in an input sentence and discarding those unlikely to represent words in the input sentence, (2) looking up the remaining combinations of characters in a dictionary to determine whether they may constitute words, and (3) submitting the combinations of characters determined to be words to a natural language parser as alternative lexical records representing the input sentence.
- the parser generates a syntactic parse tree representing the syntactic structure of the input sentence, which contains only those lexical records representing the combinations of characters certified to be words in the input sentence.
- the facility weights, the lexical records so that longer combinations of characters, which more commonly represent the correct segmentation of a sentence than shorter combinations of characters, are considered by the parser before shorter combinations of characters.
- the facility adds to the dictionary, for each character occurring in the dictionary, (1) indications of all of the different combinations of word length and character position in which the word appears, and (2) indications of all of the characters that may follow this character when this character begins a word.
- the facility further adds (3) indications to multiple-character words of whether sub-words within the multiple- character words are viable and should be considered.
- the facility discards (1) combinations of characters in which any character is used in a word length/position combination not occurring in the dictionary, and (2) combinations of characters in which the second character is not listed as a possible second character of the first character.
- FIG. 1 is a high-level block diagram of the general -purpose computer system upon which the facility preferably executes.
- the computer system 100 contains a central processing unit (CPU) 110, input/output devices 120, and a computer memory (memory) 130.
- CPU central processing unit
- input/output devices 120 input/output devices
- memory memory
- a storage device 121 such as a hard disk drive
- a computer-readable media drive 122 which can be used to install software products, including the facility, which are provided on a computer-readable medium, such as a CD-ROM
- a network connection 123 through which the computer system 100 may communicate with other connected computer systems (not shown).
- the memory 130 preferably contains a word segmentation facility 131 for identifying individual words occurring in Chinese text, a syntactic parser 133 for generating a parse tree representing the syntactic structure of a sentence of natural language text from lexical records representing the words occurring in the natural language text, and a lexical knowledge base 132 for use by the parser in constructing lexical records for a parse tree and for use by the facility to identify words occurring in natural language text. While the facility is preferably implemented on a computer system configured as described above, those skilled in the art will recognize that it may also be implemented on computer systems having different configurations.
- Figure 2 is an overview flow diagram showing the two phases in which the facility preferably operates.
- step 201 as part of an initialization phase, the facility augments a lexical knowledge base to include information used by the facility to perform word segmentation.
- Step 201 is discussed in greater detail below in conjunction with Figure 3.
- the facility adds entries to the lexical knowledge base for the characters occurring in any word in the lexical knowledge base.
- the entry added for each character includes a CharPos attribute that indicates the different positions at which the character appears in words.
- the entry for each character further contains a NextChars attribute that indicates the set of characters that occur in the second position of words that begin with the current character.
- the facility also adds an IgnoreParts attribute to each word occurring in the lexical knowledge base that indicates whether the sequence of characters comprising the word should ever be considered to comprise smaller words that together make up the current word.
- step 201 the facility continues in step 202, ending the initialization phase and beginning the word segmentation phase.
- the facility uses the information added to the lexical knowledge base to perform word segmentation of sentences of Chinese text.
- step 202 the facility receives a sentence of Chinese text for word segmentation.
- step 203 the facility segments the received sentence into its constituent words. Step 203 is discussed in greater detail below in conjunction with Figure 5. Briefly, the facility looks up in the lexical knowledge base a small fraction of all the possible contiguous combinations of characters in the sentence. The facility then submits to a syntactic parser the looked-up combinations of characters that are indicated to be words by the lexical knowledge base. The parser, in determining the syntactic structure of the sentence, identifies the combinations of characters intended to comprise words in the sentence by its author. After step 203, the facility continues at step 202 to receive the next sentence for word segmentation.
- Figure 3 is a flow diagram showing the steps preferably performed by the facility in order to augment the lexical knowledge base in the initialization phase to include information used to perform word segmentation. These steps (a) add entries to the lexical knowledge base for the characters occurring in words in the lexical knowledge base; (b) add CharPos and NextChars attributes to the character entries in the lexical knowledge base; (c) add the IgnoreParts attribute to the entries for words in the lexical knowledge base.
- step 302 the facility loops through each word entry in the lexical knowledge base.
- step 302 the facility loops through each character position in the word. That is, for a word containing three characters, the facility loops through the first, second, and third characters of the word.
- step 303 if the character in the current character position has an entry in the lexical knowledge base, then the facility continues in step 305, else the facility continues in step 304.
- step 304 the facility adds an entry to the lexical knowledge base for the current character. After step 304, the facility continues in step 305.
- step 305 the facility adds an ordered pair to the CharPos attribute stored in the character's entry in the lexical knowledge base to indicate that the character may occur in the position in which it occurs in the current word.
- the ordered pair added has the form (position, length), where position is the position that the character occupies in the word and length is the number of characters in the word. For example, for the character " ⁇ " in the word "3 iJi , " the facility will add the ordered pair (1, 3) to the list of ordered pairs stored in the CharPos attribute in the lexical knowledge base entry for the character "3 .”
- the facility preferably does not add the ordered pair as described in step 305 if the ordered pair is already contained in the CharPos attribute for the current word.
- step 306 if additional characters remain in the current word to be processed, then the facility continues in step 302 to process the next character, else the facility continues in step 307.
- step 307 if the word is a single character word, then the facility continues in step 309, else the facility continues in step 308.
- step 308 the facility adds a character in the second position of the current word to the list of characters in the NextChars attribute in the lexical knowledge base record for the character in the first position of the current word. For example, for the word "3
- step 309 if the current word can contain other, smaller words, then the facility continues in step 311, else the facility continues in step 310.
- Step 309 is discussed in further detail below in conjunction with Figure 4. Briefly, the facility employs a number of heuristics to determine whether an occurrence of the sequence of characters that make up the current word may in some context make up two or more smaller words.
- step 310 the facility sets an IgnoreParts attribute for the word in the lexical knowledge base entry for the word. Setting the IgnoreParts attribute indicates that, when the facility encounters this word in a sentence of input text, it should not perform further steps to determine whether this word contains smaller words.
- step 312. because the current word can contain other words, the facility clears the IgnoreParts attribute for the word, so that the facility, when it encounters the word in a sentence of input text, proceeds to investigate whether the word contains smaller words.
- step 311 the facility continues in step 312.
- step 312 if additional words remain in the lexical knowledge base to be processed, then the facility continues in step 301 to process the next word, else these steps conclude.
- Table 5 Character Lexical Knowledge Base Entries It can be seen from Table 5, for instance, from the CharPos attribute of the character "B ⁇ ” that this character can appear as the first character of words that are 2, 3, or 4 characters long. It can further be seen from the NextChars attribute of the character "B ⁇ ” that, in words beginning with this character, the second character may be either "JL,” “5 ,” or “H .”
- Figure 4 is a flow diagram showing the steps preferably performed in order to determine whether a particular word can contain other, smaller words.
- step 401 if spaces and punctuation characters were removed from an English sentence, the sequence of characters "beat” could be interpreted either as the word “beat” or as the two words "be” and "at.”
- step 401 if the word contains four or more characters, then the facility continues in step 402 to return the result that the word cannot contain other words, else the facility continues in step 403.
- step 403 if all the characters in the word can constitute single-character words, then the facility continues in step 405, else the facility continues in step 404 to return the result that the word cannot contain other words.
- step 405 if the word contains a word frequently used as a derivational affix, that is, a prefix or a suffix, then the facility continues in step 406 to return the result that the word cannot contain other words, else the facility continues in step 407.
- step 407 if an adjacent pair of characters in the word are often divided when they appear adjacently in text of the language, then the facility continues in step 409 to return the result that the word can contain other words, else the facility continues in step 408 to return the result that the word cannot contain other words.
- Figure 5 is a flow diagram of the steps preferably performed by the facility in order to segment a sentence into its constituent words. These steps generate a word list identifying different words of the language that occur in the sentence, then submits the word list to a parser to identify the subset of words in the word list that were intended to comprise the sentence by its author.
- step 501 the facility adds to the word list multiple-character words occurring in the sentence.
- Step 501 is discussed in greater detail below in conjunction with Figure 6.
- step 502 the facility adds to the word list the single-character words occurring in the sentence.
- step 502 is discussed in greater detail below in conjunction with Figure 9.
- step 503 the facility generates lexical records used by the lexical parser for the words that have been added to the word list in steps 501 and 502.
- step 504 assigns probabilities to the lexical records. The probability of a lexical record reflects the likelihood that the lexical record will be part of a correct parse tree for the sentence, and is used by the parser to order the application of the lexical records in the parsing process.
- the parser applies the lexical records during the parsing process in decreasing order of their probabilities.
- Step 504 is discussed in greater detail below in conjunction with Figure 10.
- the facility utilizes the syntactic parser to parse the lexical records in order to produce a parse tree reflecting the syntactic structure of the sentence.
- This parse tree has a subset of the lexical records generated in step 503 as its leaves.
- the facility identifies as words of the sentence the words represented by the lexical records that are the leaves of the parse tree. After step 506, these steps conclude.
- Figure 6 is a flow diagram showing the steps preferably performed by the facility in order to add multiple-character words to the word list. These steps use a current position within the sentence in analyzing the sentence to identify multiple-character words. These steps further utilize the CharPos, NextChar, and IgnoreParts attributes added to the lexical knowledge base by the facility as shown in Figure 4. In accordance with a first preferred embodiment, the facility retrieves these attributes from a lexical knowledge base on an as-needed basis during the performance of the steps shown in Figure 6. In a second preferred embodiment, the values of the NextChar attributes and/or the CharPos attributes of the characters in the sentence are all pre-loaded before the performance of the steps shown in Figure 6.
- a 3-dimensional array is preferably stored in the memory that contains the value of the CharPos attribute for each character occurring in the sentence. This array indicates, for a character at a given position in the sentence, whether the character may be at a given position in a word of a given length. Caching the values of these attributes allows them to be officially accessed when performing the steps shown in Figure 6.
- step 601 the facility sets this position at the first character of the sentence.
- step 602-614 the facility continues to repeat steps 603-613 until the position has advanced to the end of the sentence.
- the facility loops through each word candidate that begins at the current position.
- the facility preferably begins with the word candidate that starts at the current position and is seven characters long, and, in each iteration, removes one character from the end of the word candidate until the word candidate is two characters long. If there are fewer than seven characters remaining in the sentence beginning from the current position, the facility preferably omits the iterations for the word candidates for which there are insufficient characters remaining in the sentence.
- the facility tests for the current word candidate conditions relating to the NextChar and CharPos attributes of the characters comprising the word candidate. Step 604 is discussed in greater detail below in conjunction with Figure 7.
- step 605 the facility looks up the word candidate in the lexical knowledge base to determine whether the word candidate is a word.
- the facility looks up the word candidate in the lexical knowledge base to determine whether the word candidate is a word.
- step 606 if the word candidate is a word, then the facility continues in step 607, else the facility continues in step 609.
- step 607 the facility adds the word candidate to the list of words occurring in the sentence.
- step 608 if the word candidate may contain other words, i. e. , if the IgnoreParts attribute for the word is clear, then the facility continues in step 609, else the facility continues in step 611.
- step 611 the word candidate may contain other words, i. e. , if the IgnoreParts attribute for the word is clear.
- step 609 if additional word candidates remain to processed, then the facility continues in step 603 to process the next word candidate, else the facility continues in step 610.
- step 610 the facility advances the current position one character toward the end of the sentence.
- step 614 the facility continues in step 614.
- step 611 if the last character of the word candidate overlaps with another word candidate that may also be a word, then the facility continues in step 613, else the facility continues in step 612. Step 611 is discussed in greater detail below in conjunction with Figure 8.
- step 612 the facility advances the position to the character in the sentence after the last character of the word candidate. After step 612, the facility continues in step 614.
- step 613 the facility advances the position to the last character of the current word candidate. After step 613, the facility continues in step 614.
- step 614 if the position is not at the end of the sentence, then the facility continues in step 602 to consider a new group of word candidates, else these steps conclude.
- Figure 7 is a flow diagram showing the step preferably performed by the facility in order to test the NextChar and CharPos conditions for a word candidate.
- step 701 if the second character of the word candidate is in the NextChar list of the first character of the word candidate, then the facility continues in step 703, else the facility continues in step 702 to return the result that the conditions are both satisfied.
- steps 703-706 the facility loops through each character position in the word candidate.
- step 704 if the ordered pair made up of the current position and the length of the word candidate is among the ordered pairs in the CharPos list for the character in the current character position, then the facility continues in step 706, else the facility continues in step 705 to return the result that the conditions are not both satisfied.
- step 706 if additional character positions remain in the word candidate to be processed, then the facility continues in step 703 to process the next character position in the word candidate, else the facility continues in step 707 to return the result that both conditions are satisfied by the word candidate.
- Figure 8 is a flow diagram showing the steps preferably performed by the facility in order to determine whether the last character of the current word candidate overlaps with another word candidate that may be a word.
- step 801 if the character after the word candidate is in the list of characters in the NextChar attribute for the last character of the word candidate, then the facility continues in step 803, else the facility continues in step 802 to return the result that there is no overlap.
- step 803 the facility looks up in the lexical knowledge base the word candidate without its last character in order to determine whether the word candidate without its last character is a word.
- step 804 if the word candidate without its last character is a word, then the facility continues in step 806 to return the result that there is overlap, else the facility continues in step 805 to return the result that there is no overlap.
- Table 7 indicates, for each of the 53 combinations of characters from the sample sentence considered by the facility: the result of the CharPos test, the result of the NextChars test, whether the facility looked up the word in the lexical knowledge base, and whether the lexical knowledge base indicated that the combination of characters is a word.
- Combination 12 is therefore looked up and determined to be a word. After processing combination 12, and determining how far to advance the current position, the facility determines that the IgnoreParts attribute of the word constituted by combination 12 is clear, and therefore advances the current position one character to the character "T" rather than to the character following combination 12.
- combinations 18, 24, 37, and 43 are words that have their IgnoreParts attribute set and do not overlap in their final characters with any word candidates that may be words. After processing each, therefore, the facility advances the current position to the character following the character combination in accordance with step 612, thereby omitting to process unnecessarily up to 41 additional combinations for each of these four combinations. It can further be seen that the IgnoreParts attributes of the words constituted by combinations 23 and 50 are clear. For this reason, the facility advances the current position only one character in accordance with step 610 after processing these combinations.
- the two-character combinations 30, 36, 47, and 52 are not determined by the facility to constitute words.
- the facility therefore advances the current position only one character after processing these combinations in accordance with step 610.
- the facility looks up only 14 of 112 possible combinations in the sample sentence. Of the 14 combinations looked up by the facility, nine are in fact real words.
- the word list contains the words constituted by combinations 6, 12, 18, 23, 24, 37, 43, 50, and 53.
- Table 8 Word List with Multiple-Character Words
- Figure 9 is a flow diagram showing the steps preferably performed by the facility in order to add single-character words to the word list.
- the facility loops through each character in the sentence, from the first character to the last character.
- the facility determines, based on its entry in the lexical knowledge base, whether the character comprises a single-character word, else the facility continues in step 906 without adding a character to the word list. If the character comprises a single-character word, then the facility continues in step 903, else the facility continues in step 906 without adding the character to the word list.
- step 903 if the character is contained in a word that may not contain other words, i.e., a word already on the word list has its IgnoreParts attribute set, then the facility continues in step 904, else the facility continues in step 905 to add the character to the word list.
- step 904 if the character is contained in a word on the word list that overlaps with another word on the word list, then the facility continues in step 906 without adding the character to the word list, else the facility continues in step 905.
- step 905 the facility adds the single-character word comprising the current character to the word list.
- step 906 if additional characters remain in the sentence to be processed, then the facility continues in step 901 to process the next character in the sentence, else these steps conclude.
- Table 9 below shows that, in performing the steps shown in Figure 9, the facility adds single-character words 54-61 to the word list.
- the facility After adding multiple- and single-character words to the word list and generating lexical records for those words, the facility assigns probabilities to the lexical records that is used by the parser to order the application over the lexical records in the parsing process.
- Figures 10 and 11, discussed below, show two alternative approaches used by the facility in order to assign probabilities to the lexical records.
- Figure 10 is a flow diagram showing the steps preferably performed by the facility in order to assign probabilities to the lexical records generated from the words in the word list in accordance with a first approach.
- the facility preferably ultimately sets the probability for each lexical record to either a high probability value that will cause the parser to consider the lexical record early during the parsing process, or to a low probability value, which will cause the parser to consider the lexical record later in the parsing process.
- steps 1001-1005 the facility loops through each word in the word list.
- step 1002 if the current word is contained in a larger word in the word list, then the facility continues in step 1004, else the facility continues in step 1003.
- step 1003 the facility sets the probability for the lexical record representing the word to the high probability value. After step 1003, the facility continues in step 1005. In step 1004, the facility sets the probability for the lexical records representing the word to the low probability value. After step 1004, the facility continues in step 1005. In step 1005, if additional words remain in the word list to be processed, then the facility continues in step 1001 to process the next word in the word list, else these steps conclude.
- Table 10 shows the probability values assigned to each word in the word list in accordance with steps shown in Figure 10. It can be seen by reviewing the probabilities that the facility assigns the high probability value to at least one word containing each character, so that at least one lexical record containing each character is considered early in the parsing process.
- Figure 11 is flow diagram showing the steps preferably performed by the facility in order to assign probabilities to the lexical records generated from the words in the word list in accordance with a second approach.
- the facility uses the word list to identify all the possible segmentations of the sentence comprised entirely of the words in the word list.
- the facility selects the one or more possible segmentations identified in step 1101 that contain the fewest words. If more than one of the possible segmentations has the minimum number of words, the facility selects each such possible segmentation. Table 11 below shows the possible segmentation generated from the word list shown in Table 9 having the fewest words (9).
- step 1103 the facility sets the probability for the lexical records of the words in the selected segmentation(s) to the high probability value.
- step 1104 the facility sets the probability for the lexical records of the words not in selected segmentation(s) to the low probability value. After step 1104, these steps conclude.
- Table 12 shows the probability values assigned to each word in the word list in accordance with steps shown in Figure 11. It can be seen by reviewing the probabilities that the facility assigns the high probability value to at least one word containing each character, so that at least one lexical record containing each character is considered early in the parsing process.
- Figure 12 is a parse tree diagram showing a parse tree generated by the parser representing the syntactic structure of the sample sentence. It can be seen that the parse tree is a hierarchical structure having a single sentence record 1231 as its head and having a number of lexical records 1201-1211 as its leaves. The parse tree further has intermediate syntactic records 1221-1227 that combine lexical records each representing a word into a larger syntactic structure representing one or more words.
- the prepositional phrase record 1223 combines a lexical record 1204 representing a preposition and lexical record 1206, representing a noun.
- the facility identifies the words represented by lexical records 1201-1211 in the parse tree as the words into which the sample sentence should be segmented. This parse tree may also be retained by the facility in order to perform additional natural language processing on the sentence.
Abstract
Description
Claims
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US23586 | 1979-03-26 | ||
US2358698A | 1998-02-13 | 1998-02-13 | |
PCT/IB1999/000320 WO1999041680A2 (en) | 1998-02-13 | 1999-01-13 | Segmentation of chinese text into words |
Publications (1)
Publication Number | Publication Date |
---|---|
EP1055182A2 true EP1055182A2 (en) | 2000-11-29 |
Family
ID=21816034
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP99902779A Withdrawn EP1055182A2 (en) | 1998-02-13 | 1999-01-13 | Segmentation of chinese text into words |
Country Status (4)
Country | Link |
---|---|
EP (1) | EP1055182A2 (en) |
JP (2) | JP4573432B2 (en) |
CN (1) | CN1114165C (en) |
WO (1) | WO1999041680A2 (en) |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6810375B1 (en) * | 2000-05-31 | 2004-10-26 | Hapax Limited | Method for segmentation of text |
WO2003003241A1 (en) * | 2001-06-29 | 2003-01-09 | Intel Corporation | Predictive cascading algorithm for multi-parser architecture |
FR2880708A1 (en) * | 2005-01-11 | 2006-07-14 | Vision Objects Sa | Term e.g. typed character, searching method for digital handwritten document, involves converting handwritten data into intermediate data, in intermediate format, in form of segmentation graph, and searching terms on intermediate data |
CN100424685C (en) * | 2005-09-08 | 2008-10-08 | 中国科学院自动化研究所 | Syntax analysis method and device for layering Chinese long sentences based on punctuation treatment |
US8310461B2 (en) * | 2010-05-13 | 2012-11-13 | Nuance Communications Inc. | Method and apparatus for on-top writing |
CN103177089A (en) * | 2013-03-08 | 2013-06-26 | 北京理工大学 | Sentence meaning composition relationship lamination identification method based on central blocks |
CN107748744B (en) * | 2017-10-31 | 2021-01-26 | 广东小天才科技有限公司 | Method and device for establishing drawing box knowledge base |
CN110955748B (en) * | 2018-09-26 | 2022-10-28 | 华硕电脑股份有限公司 | Semantic processing method, electronic device and non-transitory computer readable recording medium |
CN109670123B (en) * | 2018-12-28 | 2021-02-26 | 杭州迪普科技股份有限公司 | Data processing method and device |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2798931B2 (en) * | 1988-04-26 | 1998-09-17 | 健 楠井 | Chinese phonetic delimiter and phonetic kanji conversion |
US5448474A (en) * | 1993-03-03 | 1995-09-05 | International Business Machines Corporation | Method for isolation of Chinese words from connected Chinese text |
JPH08339383A (en) * | 1995-04-11 | 1996-12-24 | Ricoh Co Ltd | Document retrieving device and dictionary preparing device |
US5806021A (en) * | 1995-10-30 | 1998-09-08 | International Business Machines Corporation | Automatic segmentation of continuous text using statistical approaches |
-
1999
- 1999-01-13 EP EP99902779A patent/EP1055182A2/en not_active Withdrawn
- 1999-01-13 CN CN99802944A patent/CN1114165C/en not_active Expired - Fee Related
- 1999-01-13 JP JP2000531795A patent/JP4573432B2/en not_active Expired - Fee Related
- 1999-01-13 WO PCT/IB1999/000320 patent/WO1999041680A2/en not_active Application Discontinuation
-
2010
- 2010-02-23 JP JP2010037953A patent/JP5100770B2/en not_active Expired - Fee Related
Non-Patent Citations (1)
Title |
---|
See references of WO9941680A2 * |
Also Published As
Publication number | Publication date |
---|---|
WO1999041680A3 (en) | 1999-11-25 |
JP4573432B2 (en) | 2010-11-04 |
CN1114165C (en) | 2003-07-09 |
JP2002503849A (en) | 2002-02-05 |
JP2010157260A (en) | 2010-07-15 |
CN1290371A (en) | 2001-04-04 |
JP5100770B2 (en) | 2012-12-19 |
WO1999041680A2 (en) | 1999-08-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6640006B2 (en) | Word segmentation in chinese text | |
US6694055B2 (en) | Proper name identification in chinese | |
EP0907924B1 (en) | Identification of words in japanese text by a computer system | |
JP5100770B2 (en) | Word classification method in kanji sentences | |
JP4413349B2 (en) | Sample text based language identification method and computer system | |
US6470306B1 (en) | Automated translation of annotated text based on the determination of locations for inserting annotation tokens and linked ending, end-of-sentence or language tokens | |
US6816830B1 (en) | Finite state data structures with paths representing paired strings of tags and tag combinations | |
US7478033B2 (en) | Systems and methods for translating Chinese pinyin to Chinese characters | |
JP5379155B2 (en) | CJK name detection | |
JP2007265458A (en) | Method and computer for generating a plurality of compression options | |
Dahab et al. | A comparative study on Arabic stemmers | |
EP3598321A1 (en) | Method for parsing natural language text with constituent construction links | |
JP5107556B2 (en) | Improved Chinese-English translation tool | |
Qu et al. | Finding ideographic representations of Japanese names written in Latin script via language identification and corpus validation | |
Marcińczuk et al. | Statistical proper name recognition in Polish economic texts | |
Liang | Spell checkers and correctors: A unified treatment | |
WO1997048058A1 (en) | Automated translation of annotated text | |
WO1997048058A9 (en) | Automated translation of annotated text | |
JP4088171B2 (en) | Text analysis apparatus, method, program, and recording medium recording the program | |
EP1429257B1 (en) | Method and apparatus for recognizing multiword expressions | |
JP2022014400A (en) | Important technical term acquisition device, important technical term acquisition method, and program | |
JP4023384B2 (en) | Natural language translation method and apparatus and natural language translation program | |
JP2009009583A (en) | Method for segmenting non-segmented text using syntactic parse | |
HASANPOUR et al. | Design and implementation of a software system for detecting orthographical or morphological errors in persian words | |
Alhonen | Automatic Morphological Parsing of Chinese |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20000906 |
|
AK | Designated contracting states |
Kind code of ref document: A2 Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE |
|
RIN1 | Information on inventor provided before grant (corrected) |
Inventor name: JIANG, ZIXIN Inventor name: RICHARDSON, STEPHEN, D. Inventor name: WU, ANDI |
|
RIN1 | Information on inventor provided before grant (corrected) |
Inventor name: JIANG, ZIXIN Inventor name: RICHARDSON, STEPHEN, D. Inventor name: WU, ANDI |
|
17Q | First examination report despatched |
Effective date: 20010122 |
|
GRAG | Despatch of communication of intention to grant |
Free format text: ORIGINAL CODE: EPIDOS AGRA |
|
GRAG | Despatch of communication of intention to grant |
Free format text: ORIGINAL CODE: EPIDOS AGRA |
|
GRAH | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOS IGRA |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN |
|
18D | Application deemed to be withdrawn |
Effective date: 20020921 |