JP2002503849A - Word segmentation method in Kanji sentences - Google Patents

Abstract

(57) [Summary] The present invention provides a facility for selecting a combination of characters that may be a word from a character string in a natural language. This facility uses, for each of a plurality of characters, (a) the character that appears at the second position of the word beginning with the character, and (b) the indication of the position where the character appears within the word. For each successive combination of characters in the sequence, the facility determines whether the character appearing in the second position of the combination appears in a word beginning with the character appearing in the first position of the combination. . If so, the facility determines whether each character of the combination is indicated to appear in a word at the location where it appears in the combination. If so, the facility determines that the combination of characters may be a word. In some embodiments, the facility compares the character combination with a list of valid words to determine whether the character combination is a word.

Description

DETAILED DESCRIPTION OF THE INVENTION

TECHNICAL FIELD The present invention relates generally to the field of natural language processing, and more particularly to word segmentation (
word segmentation).

BACKGROUND OF THE INVENTION Word segmentation is the process of identifying the individual words that make up a language expression, such as a sentence. Word division is useful for checking spelling and grammar, synthesizing speech from sentences, and analyzing and understanding natural languages. These are all effects obtained by identifying individual words.

In the case of English sentences, it is rather simple to perform word division. That is, spaces and punctuation marks separate individual words in a sentence. Consider the English text in Table 1 below.

[0004]

[Table 1]

[0005] By identifying adjacent series and / or punctuation marks as terminators of words preceding the series of spaces, the English sentences in Table 1 can be simply partitioned as shown in Table 2 below. .

[0006]

[Table 2]

[0007] In Chinese sentences, word boundaries are implicit rather than explicit. Consider the text in Table 3 below. This means "the committee discussed this issue in Buenos Aires yesterday afternoon."

[0008]

[Table 3]

[0009] Despite the lack of punctuation and spaces in the text, Chinese readers recognize the text in Table 3 as consisting of words that are underlined in Table 4 below.

[0010]

[Table 4]

From the above example, it can be seen that Chinese word divisions cannot be done in the same way as English word divisions. Therefore, a highly accurate and efficient method of automatically classifying Chinese would have great utility.

SUMMARY OF THE INVENTION According to the present invention, a word segmentation software facility ("facility")
Performs a word segmentation operation on a sentence in a non-segmented language such as Chinese, (1)
Evaluate the possible combinations of characters in the input sentence, discard those that are unlikely to represent words in the input sentence, and (2) check the remaining character combinations in the dictionary to see if they can form a word (3) The combination of characters determined to be a word is submitted to the natural language parser as an alternative vocabulary record representing the input sentence. The parser generates a parse tree representing the syntax structure of the input sentence. It contains only vocabulary records that represent combinations of characters that have proven to be words in the input sentence. When submitting a vocabulary record to the parser, the facility weights the vocabulary record so that the parser considers long letter combinations before short letter combinations. This is because, in general, a combination of long characters often represents a correct section of a sentence more than a combination of short characters.

In order to easily discard combinations of characters that are unlikely to represent words in the input sentence, the facility provides, for each character that appears in the dictionary, (1) all combinations of different word lengths and character positions where words appear. And (2) when the character starts a word, add instructions to the dictionary for all characters that may follow the character. Further, the facility also adds (3) an instruction for the multi-character word as to whether or not the partial word can exist in the multi-character word and should be considered. When processing a sentence, the facility may include character combinations where any of the words are used in word length / position combinations that are not in the dictionary, and (2) the second character in which the second character may be the first character. Discard character combinations not listed as characters. Furthermore,
The facility also discards (3) combinations of characters that appear in words that do not consider partial words.

As described above, the facility minimizes the number of combinations of characters to be looked up in the dictionary, and differentiates between the results of the category choices each composed of valid words using the syntactic context of the sentence. .

DETAILED DESCRIPTION OF THE INVENTION The present invention performs word segmentation in Chinese sentences. In a preferred embodiment, the word segmentation software facility ("facility") evaluates possible combinations of characters in the input sentence when performing word segmentation of a sentence in a non-segmented language such as Chinese. To discard words that are unlikely to represent words in the input text,
2) Look up the remaining character combinations in the dictionary and determine if they can form a word. (3) Use the character combinations determined as words as natural language parser as alternative vocabulary records representing input sentences. Submit to The parser generates a parse tree representing the syntax structure of the input sentence. It contains only vocabulary records that represent combinations of characters that have proven to be words in the input sentence. When submitting a vocabulary record to the parser, the facility weights the vocabulary record so that the parser considers long letter combinations before short letter combinations. This is because, in general, a combination of long characters often represents a correct section of a sentence more than a combination of short characters.

In order to easily discard combinations of characters that are unlikely to represent words in the input sentence, the facility includes, for each character that appears in the dictionary, (1) all combinations of different word lengths and character positions where words appear. And (2) when this character is at the beginning of a word, add all the characters that may possibly follow this character to the dictionary. Further, the facility also adds (3) an instruction on whether or not a partial word can exist in the multi-character word and should be considered for the multi-character word. When processing a sentence, the facility can be: (1) a combination of letters used in word length / position combinations where any word is not in the dictionary; and (2) the second letter can be the first letter Discard any combination of characters not listed as a second character. Further, the facility also discards (3) combinations of characters that appear in words that do not consider partial words.

As described above, the facility minimizes the number of combinations of characters to be looked up in a dictionary, and uses the syntactic context of a sentence to differentiate between the results of alternative sections each composed of valid words. I do.

FIG. 1 is a high-level block diagram of a general-purpose computer system that the facility preferably executes. The computer system 100 includes a central processing unit (CP)
U) 110, input / output device 120, and computer memory (memory) 1
30 is built in. Between the input / output devices, a storage device 121, such as a hard disk drive, a computer readable medium provided on a computer readable medium, such as a CD-ROM, which can be used to install a software product including the facility. Possible media drive 122 and network connection 12 that allows the computer 100 to communicate with other connected computer systems (not shown)
There are three. The memory 130 includes a word segmentation facility 131 for identifying individual words appearing in the Chinese sentence, and a syntax parser for generating a parse tree representing the syntax structure of the sentence of the natural language sentence from a vocabulary record representing the word appearing in the natural language sentence. 133 and a vocabulary knowledge base 132 that the parser uses to build a vocabulary record for the parse tree and that the facility uses to identify words that appear in natural language sentences. The facility is preferably implemented on a computer system configured as described above, but those skilled in the art will recognize that the facility can be implemented on a computer system having a different configuration.

FIG. 2 is a schematic flow diagram illustrating two phases in which the facility preferably operates. In step 201, as part of the initialization phase, the facility augments the vocabulary knowledge base and includes information that the facility uses in performing word segmentation. Step 201 is discussed in further detail below in connection with FIG. Briefly, in step 201, the facility adds an entry to the vocabulary knowledge base for characters that occur in any of the words in the vocabulary knowledge base. The entry to add for each character includes a CharPos attribute that represents the different positions where the character appears in the word. In addition, the entry for each character also includes a NextChars attribute that indicates the set of characters that appear at the second position of the word starting with the current character.
Finally, the facility assigns an IgnorePas to each word that appears in the vocabulary knowledge base.
The rts attribute is also added. This indicates whether the character strings that make up the word should also be considered to make up smaller words that also make up the current word.

After step 201, the facility proceeds to step 202, ends the initialization phase, and starts the word segmentation phase. In the word segmentation phase, the facility performs word segmentation of Chinese sentences using information added to the vocabulary knowledge base. In step 202, the facility receives a Chinese sentence for word segmentation. In step 203, the facility classifies the received sentence into its constituent words. Step 203 is discussed in further detail below in connection with FIG. In short, the facility looks in the vocabulary knowledge base for small fragments of all possible consecutive combinations of characters in a sentence. The facility then submits the examined combination of characters indicated as words by the lexical knowledge base to the syntactic parser. When determining the syntactic structure of a sentence, the parser identifies combinations of characters that the author of the sentence intends to form words in the sentence. After step 203, the facility proceeds to step 202 to receive the next sentence for word segmentation.

FIG. 3 is a flow diagram illustrating the steps that the facility preferably performs to augment the vocabulary knowledge base in the initialization phase and include information used in performing word segmentation. These steps include (a) adding an entry to the vocabulary knowledge base for characters that appear in words in the vocabulary knowledge base, and (b) adding CharPos and NextChar to the entry for this character in the vocabulary knowledge base.
s attribute is added, and (c) the entry for the word in the vocabulary knowledge base is
Add a reParts attribute.

The facility performs steps 301 to 3 for each word entry in the vocabulary knowledge base.
Repeat the 12 loops. In step 302, the facility repeats the loop for each character position in the word. That is, for words containing three characters, the facility performs a loop on the first, second, and third characters of the word. In step 303, if the character at the current character position has an entry in the vocabulary knowledge base, the facility proceeds to step 305; otherwise, the facility proceeds to step 304. In step 304, the facility adds an entry for the current character to the vocabulary knowledge base. After step 304, the facility proceeds to step 305. In step 305, the facility stores the CharPos attribute stored in the entry of the character in the vocabulary knowledge base in an ordered pair (orde
red pair) to indicate that the character is likely to appear at the location where it appeared in the current word. The alignment pairs to be added are (position, le
ngth), where position is the position of the character in the word and length is the number of characters in the word. For example,

For the character “文字” in the word “と い う”, the facility adds the alignment pair (1, 3) to the list of alignment pairs stored in the CharPos attribute in the vocabulary knowledge base entry for the character “委”. . Preferably, if the alignment pair is already included in the CharPos attribute for the current word, step 30
As explained in 5, no alignment pairs are added. At step 306, if there are still characters remaining in the current word to be processed, the facility proceeds to step 302 to process the next character. Otherwise, the facility proceeds to step 307. At step 307, if the word is a single letter word, the facility proceeds to step 309; otherwise, the facility proceeds to step 308. In step 308, the facility converts the character at the second position of the current word to the NextCha in the vocabulary knowledge base record of the character at the first position of the current word.
Add to the character list in the rs attribute. For example,

In the word, the facility is a letter

Is added to the character list stored for the NextChars attribute of the character “C”. After step 308, the facility proceeds to step 309. In step 309, if the current word can include other smaller words, the facility proceeds to step 311; otherwise, the facility proceeds to step 310. Step 309 is discussed in further detail below in connection with FIG. In short, the facility uses a number of heuristics to determine, if there is a string that makes up the current word, in some context this string might make up more than one smaller word. A determination is made as to whether or not it is.

In step 310, the facility sets the WordParts attribute of the word in the vocabulary knowledge base entry for the word. Setting the IgnoreParts attribute means that if the facility finds this word in the text of the input sentence, it will not take any further steps to determine whether this word contains a smaller word. Step 31
After 0, the facility proceeds to step 312. In step 311, since the current word may include other words, the facility may generate an Ign for that word.
If the oreParts attribute is cleared and the word is found in the text of the input sentence, the facility will proceed to check whether the word contains a smaller word. After step 311, the facility proceeds to step 312. At step 312, if there are still words left in the vocabulary knowledge base to be processed, the facility proceeds to step 301 to process the next word. Otherwise, these steps end.

The facility performs the steps shown in FIG. 3 and returns the CharPos and Next
When the vocabulary knowledge base is augmented by assigning the Chars attribute to each character, as shown in Table 5 below, a character (Char) appearing in the sample sentence shown in Table 3 is used.
acter).

Table 5: Character Vocabulary Knowledge Base Entries

[Table 5]

From the table of FIG. 5, for example, from the CharPos attribute of the character
It can be seen that it may appear as the first character of a word that is three or four characters long. Furthermore, from the NextChars attribute of the character “Year”, it is also understood that in a word starting with this character, the second character may be any of “Child”, “Heaven” or “Night”.

FIG. 4 is a flow diagram illustrating the steps that are preferably performed to determine if a particular word may include other smaller words. As a similarity with English, when the spaces and punctuation marks are removed from the English sentence, "beat
Is a word "beat" or two words "be" and "at".
Can be interpreted as any of In step 401, if the word contains more than three letters, the facility proceeds to step 402, returning a result that this word is unlikely to contain other words. Otherwise, the facility proceeds to step 403. In step 403, if all the characters in the word can make up a single character word, the facility proceeds to step 405,
Otherwise, the facility proceeds to step 404, returning a result that the word cannot contain other words. In step 405, if the word contains a derived suffix, ie, a word that is frequently used as a prefix or suffix, the facility proceeds to step 406, returning a result that the word is unlikely to contain another word. Otherwise, the facility proceeds to step 407. If, in step 407, adjacent character pairs in the word are often split when they appear adjacent in the language sentence, the facility proceeds to step 409, where the result is that the word may include other words. return. Otherwise, the facility proceeds to step 408, returning a result that the word cannot contain other words.

The results of determining whether an individual word may include other smaller words are shown in Table 6 below.

Table 6: Vocabulary Knowledge Base Entry

[Table 6]

For example, from Table 6, it can be seen that the facility has determined that the word “Word” may not include other words, while “World” may include other words.

FIG. 5 is a flow diagram of the steps that the facility preferably performs to segment a sentence into its constituent words. These steps generate a word list that identifies the different language words that appear in the sentence, submit this word list to the parser, and identify the subset of words in the word list that the author tried to compose the sentence I do.

In step 501, the facility adds a multi-character word appearing in a sentence to a word list. Step 501 is discussed in further detail below in connection with FIG. In step 502, the facility adds single letter words that appear in the sentence to the word list. Step 502 is described in further detail below in connection with FIG. In step 503, the facility generates a vocabulary record. This is used by the vocabulary parser for the words added to the word list in steps 501 and 502. In step 504, the facility assigns probabilities to the vocabulary records. The vocabulary record probability reflects the likelihood that the vocabulary record is part of the correct parse tree of the sentence, and is used by the parser to direct the application of the vocabulary record in the parsing process. During the parsing process, the parser applies lexical records in order of decreasing probability. Step 50
4 is discussed in more detail below in connection with FIG. In step 505, the facility uses the syntactic parser to parse the vocabulary record and generate a parse tree that reflects the syntactic structure of the sentence. This parse tree is stored in step 503
Has a subset of the vocabulary records generated in the above as its leaves. In step 506, the facility identifies the word represented by the vocabulary record, which is a leaf of the parse tree, as a sentence word. After step 506, these steps end.

FIG. 6 is a flow diagram showing the steps that the facility preferably performs to add multi-character words to the word list. These steps use the current position in the sentence to analyze the sentence and identify multi-character words. In addition, these steps are based on the CharPo added by the facility to the vocabulary knowledge base, as shown in FIG.
Utilizes the s, NextChar, and IgnoreParts attributes. According to a first preferred embodiment, the facility retrieves these attributes from the vocabulary knowledge base as needed during the execution of the steps shown in FIG. In a second preferred embodiment, the values of the NextChar and / or CharPos attributes of the characters in the text are all pre-loaded before performing the steps shown in FIG. In the context of the second preferred embodiment, it is preferable to store in memory a three-dimensional array containing the value of the CharPos attribute for each character that appears in the text. This array indicates, for a character at a given position in a sentence, whether the character may be at a given position in a word of a given length. By caching the values of these attributes, they can be formally accessed when performing the steps shown in FIG.

In step 601, the facility sets this position to the first character of the sentence. In steps 602 to 614, the facility continues to repeat steps 603 to 613 until the position has reached the end of the sentence.

In steps 603 to 609, the facility repeats a loop for each word candidate starting from the current position. The facility starts with a word candidate that is 7 characters long starting from the current position, removing one character from the end of the word candidate at each iteration and continuing until the word candidate is 2 characters long. If less than seven characters remain in the sentence starting from the current position, the facility preferably skips repetition for word candidates for which there are not enough characters remaining in the sentence. In step 604, the facility determines the NextChar and Cha of the characters that make up the word candidate.
Check the condition of the current word candidate related to the rPos attribute. Step 604 is discussed in further detail below in connection with FIG. NextChar and Cha
If both rPos conditions are satisfied for the word candidate, the facility proceeds to step 605; otherwise, the facility proceeds to step 609. In step 605, the facility examines word candidates in the vocabulary knowledge base and determines whether the word candidates are words. In step 606,
If the word candidate is a word, the facility proceeds to step 607; otherwise, the facility proceeds to step 609. In step 607, the facility adds the word candidate to the list of words that appeared in the sentence. In step 608, if the word candidate may include another word, that is, if the IgnoreParts attribute of the word is clear, the facility proceeds to step 609; otherwise, the facility proceeds to step 609. Proceed to 611. In step 609, if there are still word candidates to be processed, the facility proceeds to step 603 to process the next word candidate. Otherwise, the facility proceeds to step 610. In step 610, the facility advances the current position by one character toward the end of the sentence. After step 610, the facility proceeds to step 614.

In step 611, if the last character of the word candidate overlaps with another word candidate that may also be a word, the facility proceeds to step 613; otherwise, the facility proceeds to step 612. Step 611 is described in FIG.
Discussed in more detail below in connection with In step 612, the facility advances the character after the last character of the word candidate in the sentence. After step 612, the facility proceeds to step 614. In step 613, the facility advances to the last character of the current word candidate. Step 613
After, the facility proceeds to step 614. In step 614, if the position is not at the end of the sentence, the facility proceeds to step 602 to consider a new set of word candidates. Otherwise, these steps end.

FIG. 7 is a flow diagram illustrating the steps that the facility preferably performs to check the NextChar and CharPos conditions for word candidates. In step 701, if the second character of the word candidate is in the NextChar list of the first character of the word candidate, the facility proceeds to step 703. Otherwise, the facility proceeds to step 702 and returns a result that both conditions have been met. In steps 703 to 706, the facility repeats the loop for each character position in the word candidate. In step 704, if the alignment pair consisting of the current position and length of the word candidate is among the alignment pairs in the CharPos list for the character at the current character position, the facility proceeds to step 70
6; otherwise, the facility proceeds to step 705, returning a result that both conditions are not met. At step 706, if there is still a character position to be processed in the word candidate, the facility proceeds to step 703 to process the next character position in the word candidate. Otherwise, the facility proceeds to step 707 and returns a result that the word candidate satisfies both conditions.

FIG. 8 is a flow diagram illustrating the steps preferably performed by the facility to determine whether the last character of the current word candidate overlaps another word candidate that may be a word. . In step 801, if the character after the word candidate is in the character list in the NextChar attribute for the last character of the word candidate, the facility proceeds to step 803. Otherwise, the facility proceeds to step 802 and returns a result of no overlap. Step 8
At 03, the facility examines the word candidates in the vocabulary knowledge base except for the last character, and determines whether the word candidate excluding the last character is a word. If, in step 804, the word candidate excluding the last character is a word, the facility proceeds to step 806, returning a result that there is an overlap. Otherwise, the facility proceeds to step 805 and returns a result of no duplication.

The execution of the steps shown in FIG. 6 for the above example is shown in Table 7 below.

Table 7: Character combinations considered

[Table 7]

Table 7 shows that for each of the 53 combinations of characters from the sample sentences examined by the facility, the results of the CharPos test, the results of the NextChars test, and the facility looking up the word in the vocabulary knowledge base And whether or not the vocabulary knowledge base indicates that the combination of characters is a word (is a word?).

It can be seen that combinations 1 to 4 failed the CharPos test (fail on). This is because the CharPos attribute of the character "last" does not include the alignment pair (1,7), (1,6), (1,5) or (1,4). On the other hand, in combinations 5 and 6, the CharPos test and NextCha
Both rs tests pass. Thus, the facility looks up combinations 5 and 6 in the vocabulary knowledge base and determines that combination 5 is not a word but combination 6 is a word. After processing combination 6 and deciding how far to advance from the current position, the facility determines that the IgnoreParts attribute is set, but that the word "Yearly" is a duplicate of a word candidate beginning with the letter "heaven". Is determined. Thus, the facility proceeds to the character “heaven” at the end of combination 6 according to step 613. In combinations 7-12, only combination 12 passed both the CharPos test and the NextChars test. Therefore, the combination 12 is examined and determined to be a word.
After processing the combination 12 and determining how much to advance the current position, the facility determines that the IngoreParts attribute of the word that the combination 12 comprises is clear, and thus the current position is determined by the combination 12 Advance one character to the character "down" instead of the character following.

Further, it can be seen that the combinations 18, 24, 37, and 43 have the IgnorParts attribute set and are words whose last characters do not overlap with any word candidates that may be words. Thus, after processing each, the facility unnecessarily 41 extra for each of these four combinations by advancing the current position to the character following the character combination in accordance with step 612. Processing of up to various combinations is omitted.

Further, it can be seen that the IgnoreParts attribute of the word formed by the combinations 23 and 50 is clear. Thus, after processing these combinations, the facility advances the current position by one character according to step 610.

Furthermore, it can be seen that the two letter combinations 30, 36, 47 and 52 did not determine that the facility constituted a word. Therefore, the facility
After processing these combinations according to step 610, the current position is advanced by one character. In the end, the facility looked at only 14 of the 112 possible combinations in the sample sentence. Of the 14 combinations that the facility looked at, nine were actual words.

As shown in Table 8, after the processing described in connection with Table 7, the word list includes words composed of combinations 6, 12, 18, 23, 24, 37, 43, 50, and 53 (Noun, noun, verb, pronoun, part of speech)
speech)).

Table 8: Word list of multi-character words

[Table 8]

FIG. 9 is a flow diagram illustrating the steps that the facility preferably performs to add a single character word to the word list. In steps 901 to 906, the facility repeats the loop from the first character to the last character for each character in the sentence. At step 902, the facility determines, based on its entry in the vocabulary knowledge base, whether the character forms a single character word, and if not, the facility adds the character to the word list. Instead, the process proceeds to step 906. If the character forms a single letter word, the facility proceeds to step 903; otherwise, the facility proceeds to step 9
Proceed to 06 and do not add any characters to the word list. In step 903, a word that is unlikely to contain another word, that is, the
If the word for which the arts attribute is set includes this character, the facility proceeds to step 904; otherwise, the facility proceeds to step 905 to add the character to the word list. In step 904, this character is
If it is contained within another word on the word list that overlaps with another word on the word list, the facility proceeds to step 906 and does not add this character to the word list. Otherwise, the facility proceeds to step 905. In step 905, the facility adds the single letter words that make up the current letter to the word list. In step 906, if characters to be processed still remain in the text,
The facility proceeds to step 901 and processes the next character in the sentence. Otherwise, these steps end.

Table 9 below shows the single letter words 54-61 that the facility added to the word list when performing the steps shown in FIG. 9 (nouns, morphemes).
), Noun (place qualifier) (noun (localizer), verb (verb), preposition (preposit
ion), adverb, function word, pronoun, part of speech of noun (classifier).

Table 9: Word list of single and multi-letter words

[Table 9]

After adding multi-letter and single-letter words to the word list and generating vocabulary records for these words, the facility assigns probabilities to the vocabulary records. This is used by the parser in ordering the application of lexical records in the parsing process. FIGS. 10 and 11, discussed below, show two alternative approaches that the facility uses to assign probabilities to vocabulary records.

FIG. 10 is a flow diagram showing the steps preferably performed by the facility to assign probabilities to vocabulary records generated from words in the word list according to the first approach. The facility ultimately determines the probability for each vocabulary record,
Preferably, it is set to either a high probability value that causes the parser to consider the vocabulary record early during the parsing process, or a low probability value that causes the parser to consider the vocabulary record late during the parsing process. In steps 1001 to 1005, the facility repeats the loop for each word in the word list. Step 1002
At, if the current word is included in a larger word in the word list, the facility proceeds to step 1004; otherwise, the facility proceeds to step 1
Proceed to 003. In step 1003, the facility sets the probability of the vocabulary record representing this word to a high probability value. After step 1003, the facility proceeds to step 1005. In step 1004, the facility sets the probability of the vocabulary record representing the word to a low probability value. Step 1
After 004, the facility proceeds to step 1005. In step 1005, if there is still a word to be processed in the word list, the facility proceeds to step 1001 to process the next word in the word list. Otherwise, these steps end.

Table 10 below shows the probability values assigned to each word in the word list according to the steps shown in FIG. By examining the probabilities, the facility has assigned a high probability value to at least one word containing each letter, so that at least one vocabulary record containing each letter is considered early in the parsing process. I understand.

Table 10: Word list with added probabilities

[Table 10]

FIG. 11 is a flow diagram showing the steps preferably performed by the facility to assign probabilities to vocabulary records generated from words in the word list according to the second technique. In step 1101, the facility uses the word list to identify all possible sections of the sentence that are entirely composed of the words in the word list. In step 1102, the facility includes step 1101
Select one or more sections having the smallest number of words from among the possible sections specified in. If there is more than one possible partition with the fewest words, the facility selects each such possible partition.

Table 11 below shows the minimum number of words (9) generated from the word list shown in Table 9.
) Indicates possible divisions.

[0060]

[Table 11]

In step 1103, the facility sets the probability of the vocabulary record of the word in the selected section (s) to a high probability value. In step 1104, the facility sets the probability of the vocabulary record for words not in the selected category (s) to a low probability value. After step 1104, these steps end.

Table 12 below shows the probability values assigned to each word in the word list according to the steps shown in FIG. By examining the probabilities, the facility has assigned a high probability value to at least one word containing each letter, so that at least one vocabulary record containing each letter is considered early in the parsing process. You can see that there is.

Table 12: Word list with added probabilities

[Table 12]

FIG. 12 is a parse tree diagram showing a parse tree generated by the parser and representing the syntax structure of the sample sentence. It can be seen that the parse tree has a hierarchical structure having a single sentence record 1231 as its head and many vocabulary records 1201 to 1211 as its leaves. In addition, the parse tree also has intermediate syntax records 1221-1227 that combine each vocabulary record representing a word into a larger syntax structure representing one or more words. For example, the preposition phrase record 1223 combines a vocabulary record 1204 representing a preposition and a vocabulary record 1206 representing a noun (noun). According to step 506 of FIG.
The facility specifies the words represented by the vocabulary records 1201 to 1211 in the parse tree as words that should be used to divide the sample sentences. The facility may hold this parse tree and perform further natural language processing on the sentence.

While the invention has been shown and described with reference to preferred embodiments, it will be appreciated by those skilled in the art that various changes or modifications can be made in form and detail without departing from the scope of the invention. Will be recognized. For example, even in languages other than Chinese, in order to perform word division, the above-mentioned facility features (spec
t) can be applied. Further, any suitable subset or superset of the techniques described herein may also be applied to perform word segmentation.

[Brief description of the drawings]

FIG. 1 is a high-level block diagram of a general-purpose computer system that the facility preferably executes.

FIG. 2 is a schematic flow diagram illustrating two phases in which the facility preferably operates.

FIG. 3 is a flow diagram showing the steps preferably performed by the facility to augment the vocabulary knowledge base in the initialization phase and include information used in performing word segmentation.

FIG. 4 is a flow diagram illustrating steps that are preferably performed to determine whether a particular word may include other smaller words.

FIG. 5 is a flow diagram of the steps preferably performed by the facility to segment a sentence into its constituent words.

FIG. 6 is a flow diagram illustrating the steps that the facility preferably performs to add multi-character words to the word list.

FIG. 7 is a flow diagram illustrating the steps that the facility preferably performs to check NexChar and CharPos conditions on word candidates.

FIG. 8 is a flow diagram showing the steps preferably performed by the facility to determine whether the last character of the current word candidate overlaps another word candidate that may be a word.

FIG. 9 is a flow diagram illustrating the steps that the facility preferably performs to add a single character word to the word list.

FIG. 10 is a flow diagram illustrating the steps preferably performed by the facility to assign probabilities to vocabulary records generated from words in the word list according to a first technique.

FIG. 11 is a flow diagram illustrating the steps preferably performed by the facility to assign probabilities to vocabulary records generated from words in the word list according to a second technique.

FIG. 12 is a parse tree diagram showing a parse tree generated by a parser, representing a syntax structure of a sample sentence.

[Procedure amendment]

[Submission date] March 13, 2001 (2001.3.13)

[Procedure amendment 1]

[Document name to be amended] Drawing

[Correction target item name] All figures

[Correction method] Change

[Correction contents]

FIG.

FIG. 2

FIG. 3

FIG. 4

FIG. 5

FIG. 6

FIG. 7

FIG. 8

FIG. 9

FIG. 10

FIG. 11

FIG.

────────────────────────────────────────────────── ─── Continuing the front page (72) Richardson, Steven Dee 98052, Washington, United States of America 98052, Redmond, Northeast One Handed and Thirty-Second Street 18028 (72) Inventor Chang, Chishin Washington, United States of America 98052, Redmond, One Hundred and Seventy Sixth Place Northeast 9307, Number 3 F-term (reference) 5B091 AA15 CA02 CA05 CC02 CC15

Claims (5)

[Claims] 1. A method for identifying words comprising a body of a natural language sentence in a computer system, wherein the body of the natural language sentence starts with a first character, ends with a last character, and includes the first and the last characters. Consisting of an aligned character string comprising an intermediate character selected between final characters, the method comprising: in the character string, a first character including the first character and the selected intermediate character.
Identifying a word; identifying a second word in the character string that includes the final character but not the selected intermediate character, and concatenates the first and second words to form the character string. And identifying a third word in the character string that includes the first character but does not include the selected intermediate character. In the character string, identifies the selected intermediate character and the last character. 4th including
Identifying a word and concatenating the third and fourth words to form the string; submitting the first, second, third and fourth words to a syntax parser; Generating a parse tree representing a structure, wherein the parse tree includes one of the first and second words or the third and fourth words; and wherein the parse tree includes the first and second words. Indicating that the first and second words constitute the body of the natural language sentence when including the second word, and when the parse tree includes the third and fourth words, indicating the third and fourth words. Indicating that four words make up the body of the natural language sentence. 2. The submitting step submits a supersequence of characters including the character string and constituting a sentence to the syntax parser, and generates a parse tree representing a syntax structure of the sentence. The method of claim 1, comprising a step. 3. For each of the plurality of characters, using the character appearing in a second position of the plurality of words beginning with the character and an indication of the position where the character appears in the word, For each of multiple consecutive combinations,
A computer-readable medium having content that causes a computer system to select a combination of characters that may be words from a natural language character string by performing the following steps: Making a determination as to whether the appearing character is indicated to appear in a word beginning with the character appearing in the first position of the combination; and determining whether the character appearing in a second position of the combination is the combination Is determined to appear in a word beginning with the letter that appears in the first position of the combination, it is indicated that each letter of the combination appears in a plurality of words at the location where it appears in the combination. Determining whether or not each character of the combination is In the position in which it appears in the allowed,
Determining that the combination of characters is likely to be a word if it is determined that the combination of characters is indicated to appear in a plurality of words. 4. The computer-readable medium of claim 1, further comprising the step of comparing the character combination with a word list to determine whether the character combination is a word. 5. A computer memory incorporating a word segmentation data structure for use in identifying individual words appearing in a natural language sentence, said data structure comprising: for each of a plurality of characters, beginning with said character Including identifying a character appearing in a second position of the word, including, for a word including the character, identifying a length of the word and a character position within the word occupied by the character; Computer memory, including an indication that the strings that make up a word may also make up a series of short words.