JP2000259673A - Method and device for dividing sentence to words - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device capable of dividing a sentence into words without depending upon a dictionary. SOLUTION: The sentence division device for dividing text data into words is provided with a text storage means 102 for storing text data, a word processing knowledge storage means 107 for storing pattern information for word extraction and sentence division and a word extraction means 103 for extracting a word from the text data stored in the means 102 by applying the pattern information. The device is also provided with a word storage means 104 for storing the extracted word, a word detection means 105 for detecting respective locations of the text data using the extracted word, and a sentence division means 106 for dividing the text data into words on the basis of the detected locations of the word on the text data and a new sentence dividing position found out by applying the pattern information of sentence division based on the locations of the word. Consequently a sentence can be divided into words without using a dictionary.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus in a preprocessing / analysis unit of a natural language processing system for performing machine translation, mass document search, automatic text summarization, etc. using an electronic computer. The word can be divided efficiently.

[0002]

2. Description of the Related Art In natural language processing systems such as machine translation and automatic summarization, it is necessary to analyze sentences as the first step. In particular, in agglutinated words such as Japanese, the sentence is not divided for each word, so the division into words is necessary as the first analysis.

In the document search system, for example, the word "Tokyo Metropolitan Assembly" in the character string "Tokyo Metropolitan Assembly of the Month" is extracted whether the full-text search is performed in "Tokyo" or the full-text search in "Kyoto". That said, in order to reduce noise in such searches, it is necessary to break sentences into words.

In such a process, a morphological analysis process (Japanese Patent Application Laid-Open No. 9-288673) is usually performed. In the morphological analysis, a word dictionary for analysis is prepared and the sentence is divided into words. The accuracy of the morphological analysis depends on how well the dictionary is prepared. A method of estimating and collecting words not included in the dictionary as unknown words (unregistered words) is proposed in Japanese Patent Application Laid-Open No. 9-288673, and the like. A method for collecting words and idioms from frequency is disclosed in
138801 publication.

[0005]

However, since a new word is always born in a language, a morphological analysis dictionary always requires maintenance. Also, the word usage may differ depending on the target document,
You have to adjust the dictionary each time you change the target document. No matter how much attention is paid, the possibility of encountering an unknown word in a morphological analysis, that is, a word not listed in a dictionary cannot be denied, and the appearance of the unknown word may significantly reduce the accuracy of the morphological analysis.

The present invention has been made to solve the problems of the prior art, and provides a word division method capable of dividing a sentence into words basically without depending on a dictionary, and implements the method. It is intended to provide a device.

[0007]

Therefore, in the word segmenting apparatus of the present invention, a text memory for storing natural language text data, and a word processing knowledge memory for storing pattern information for word extraction and word segmentation. Means, word extraction means for extracting words from text data stored in the text storage means by applying pattern information, word storage means for storing the extracted words, and text in which the extracted words are used There are provided word finding means for finding each position of data, and word dividing means for dividing text data into words.

In the word dividing method of the present invention, a word is extracted from text data by applying pattern information for word extraction, and each position of the text data where the extracted word is used is found. Text data is divided into words using position information and the like.

Therefore, a sentence can be divided into words without using a dictionary.

[0010]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS According to a first aspect of the present invention, in a word dividing apparatus for dividing text data into words, text storage means for storing natural language text data, and word extraction and word division are performed. Word processing knowledge storage means for storing pattern information, word extraction means for applying pattern information to extract words from text data stored in text storage means, and word storage means for storing extracted words A word finding means for finding each position in the text data where the extracted word is used, a position of the word on the text data which has been found so far, and a word division based on the position of this word. A word division unit that divides text data into words at a new word division position obtained by applying pattern information is provided. Chapter can be divided into words.

The invention according to claim 2 is characterized in that the text storage means stores natural language text data, the word processing knowledge storage means stores pattern information for performing word extraction and word division, and the text storage means. Word extraction means for repeatedly applying pattern information to the stored text data to repeatedly extract words from the text data;
A word storage means for storing the extracted words, a word finding means for finding each position of the text data in which the extracted words are used each time the word extracting means extracts the words, Word division means for dividing the text data into words based on the positions of the words in the extracted text data, and extracting words from the text data and text data in which the extracted words are used. By alternately repeating the process of finding a position, the number of words to be extracted is increased, such that a word is specified, and words before and after the word are specified.

According to a third aspect of the present invention, there is provided a word number determining means for managing the number of words to be extracted by the word extracting means, and the word extracting means increases the number of words under the management of the word number determining means. The word extraction is repeated until there is no longer. If the word extraction is repeated several times, the number of words does not increase.

According to a fourth aspect of the present invention, there is provided a text storage unit for storing natural language text data, a word processing knowledge storage unit for storing pattern information for performing word extraction and word division, and a text storage unit. Word extraction means for extracting words from text data by applying pattern information to the stored text data, word storage means for storing the extracted words, and each position of the text data in which the extracted words are used Word discovery means for discovering
Dividability calculating means for quantitatively calculating the possibility of division at the position found by the word finding means, and word dividing means for dividing text data into words based on the calculation result of the dividing possibility calculating means are provided. Therefore, it can be quantitatively determined whether or not to be divided into words.

According to a fifth aspect of the present invention, there is provided a numerical expression finding means for specifying a numerical expression from text data stored in the text storage means. Since there is a break in the middle and there is no break in the middle, it can be a reference for word segmentation like punctuation marks.

According to a sixth aspect of the present invention, the word extracting means extracts a word of a specific character type from the text data, and there is a high possibility that there is a word break at a character type change.

According to a seventh aspect of the present invention, the word extracting means comprises a kanji word extracting means for extracting a kanji word, an extracting means for extracting a katakana word, and an extracting means for extracting a hiragana word. In the extraction means of
Words can be extracted using rules suitable for the character type.

According to the present invention, a combination possibility calculating means for quantitatively calculating the combination possibility of words of different character types extracted by the word extracting means is provided, and a combination of different character types having a high possibility of combination is provided. Is treated as one word, for example, "object-oriented"
Can be processed as one word.

According to a ninth aspect of the present invention, there is provided a text storage unit for storing natural language text data, a word processing knowledge storage unit for storing pattern information for performing word extraction and word division, and a text storage unit. A partial character string extracting means for extracting a character string containing a specific character type from the stored text data, a sorting and aligning means for classifying and sorting the extracted character strings by the number of characters by focusing on the number of characters of the character type, and a sorting and sorting means by character number Word extracting means for extracting a character type word by applying pattern information to the extracted character string, word storing means for storing the extracted word, and each position of the text data in which the extracted word is used. A word finding means to be found, a division possibility calculating means for quantitatively calculating the division possibility at a position found by the word finding means,
Word division means for dividing text data into words based on the calculation result of the division possibility calculation means, and the efficiency is improved as compared with a case where pattern information is sequentially applied to text data to extract words. Words can be extracted in a targeted manner.

According to a tenth aspect of the present invention, the partial character string extracting means extracts a character string including a kanji, and the word extracting means extracts a kanji word. Can be extracted.

[0020] According to the eleventh aspect of the present invention, there is provided a reduced character string creating means for creating a character string of text data in which a character type other than hiragana is replaced with a symbol, and the partial character string extracting means is provided by the reduced character string creating means. A character string including hiragana is extracted from the created character string, and the word extracting means extracts a word of hiragana, so that hiragana words can be efficiently extracted.

According to a twelfth aspect of the present invention, in the word dividing method for dividing text data into words, words are extracted from the text data by applying pattern information for word extraction, and the extracted words are used. The position of each word in the text data is found, and the text data is converted into words by using the word positions on the found text data and the new word division positions obtained by applying word division pattern information based on the word positions. The sentence can be divided into words without using a dictionary.

According to a thirteenth aspect of the present invention, a word is extracted from text data by applying pattern information for word extraction, and each position of the text data in which the extracted word is used is found. A new word is extracted by applying pattern information based on the found word position, and after repeating this procedure, the text data is divided at the position of the word on the found text data, By repeating the word extraction and the word position finding, the number of extracted words can be increased.

According to a fourteenth aspect of the present invention, this repetition is repeated until a new word is not extracted. If the word extraction is repeated several times, the number of words does not increase.

According to a fifteenth aspect of the present invention, a word is extracted from text data by applying pattern information for word extraction, and each position of the text data in which the extracted word is used is found. Calculates the possibility of division at the determined position, and determines the position of the text data to be divided into words based on the calculated value,
It is possible to quantitatively determine whether it is appropriate to use the word division position.

According to the present invention, a numeral is found from text data, a numerical expression is identified from a character string before and after the numeral, and a position on the text data where the numerical expression appears is defined as a reference position for word division. Is to do
Numerical expressions, like punctuation marks, are effective as criteria for word segmentation.

According to a seventeenth aspect of the present invention, a character string including a specific character type is extracted from text data, and the extracted character string is classified and sorted according to the number of characters by focusing on the number of characters of the character type. In this method, words of this character type are extracted by applying pattern information for word extraction to character strings, and each position of text data in which the extracted words are used is found. Efficiency can be improved.

The invention according to claim 18 is to extract a character string including a kanji from text data to extract a kanji word, so that a kanji word can be efficiently extracted.

[0028] According to a nineteenth aspect of the present invention, a character string in which character types other than hiragana in the text data are replaced by symbols is created, a character string containing hiragana is extracted from this character string, and the extracted character string is converted to the hiragana character. It focuses on the number of characters, sorts and sorts by the number of characters, and applies pattern information for word extraction to each character string that is sorted by the number of characters to extract hiragana words. Efficient extraction becomes possible.

According to a twentieth aspect of the present invention, in the word division method for subdividing text data divided into words into words by using an updated word group, a word group used for word division of the text data; Extract the difference word from the updated word group, search the text data including the character string of this difference word, release the word division of the searched text data, play the text data before word division, and play back The word data is divided into words using the updated word group. Text data that has already been word-divided can be re-divided using the latest word.

According to a twenty-first aspect of the present invention, there is provided an information retrieval apparatus for retrieving text divided into words,
A search target text storage unit that stores text data of the search target text divided into words, an index creation unit that creates an index of the search target text,
Index storage means for storing the created index, search means for searching the search target text using the index, extracted word storage means for storing words used for word division of the search target text, and text data converted to words A word division device for dividing, a word comparison unit for comparing a word group used for word division of new text data by the word division device with a word group stored in the extracted word storage unit, and detecting the difference word And word division correcting means for controlling word division of the search target text, wherein the word division correction means performs word division on the search target text including the character string of the difference word through the word division device. The data of the search target text divided by the word of the information search device and the index for the word search are used. It is possible to update the door.

According to a twenty-second aspect of the present invention, the word segmenting apparatus according to any one of the first to eleventh aspects is used as the word segmenting apparatus. Word subdivision is performed.

According to a twenty-third aspect of the present invention, the search means comprises:
The search target text including the character string of the difference word is searched using the index, and the text including the character string of the difference word is searched using the index for full-text search.

According to a twenty-fourth aspect of the present invention, the word division correcting means compares the text data obtained by word division by the word division device with the data of the text to be searched before the word division, and finds a difference between the two. If the search target text data stored in the search target text storage unit is updated with the word-subdivided text data, and the index creation unit updates the search target text in the search target text storage unit In addition, the index for word search of the search target text is re-created, so that the search target text data and the word search index can be updated efficiently.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First Embodiment) As shown in FIG. 1, a character string dividing device according to a first embodiment is a text input means 10 for inputting text data to be processed in an electronic form.
1, text storage means 102 for temporarily storing the text data to be processed inputted from the text input means 101, the text and the attributes and marks of characters in the text in the course of processing, and knowledge of unique patterns in natural language, etc. , Word extracting means 103 for extracting words from the text, word storing means 104 for storing the words extracted by the word extracting means 103, and text in which the words stored in the word storing means 104 appear. Word finding means 105 for finding a place inside, word dividing means 106 for dividing the text stored in the text storage means 102 into words, word processing knowledge storage means 107 for storing word processing knowledge, A text output unit 108 for outputting a text as a processing result.

The word extracting means 103 and the word dividing means 105
Using the knowledge stored in the word processing knowledge storage means 107, a word is extracted from the text or the text is divided into words.

The word storage means 104 for storing the words extracted by the word extraction means 103 does not store anything in the initial state. The words to be stored can be searched at high speed using a trie structure or the like.

The word finding means 105 finds a place where the word stored in the word storage means 104 appears in the text,
Text storage means 102 with marks before and after the found word
Write in.

The word processing knowledge storage means 107 stores, as word processing knowledge, knowledge for word extraction, word division, and word extraction and division. These knowledges are unchanged from the initial state.

The device of the present invention is constituted by a computer. The text input unit 101 includes an input device such as a keyboard, an OCR input device, and the like, and the text output unit 108 includes an output device such as a display and a printer. The text storage unit 102 and the word processing knowledge storage unit 107 are set in a memory of a computer or a storage area of a hard disk device. The other means is constituted by a computer computing mechanism.

The operation of the word segmenting apparatus configured as described above will be described. FIG. 2 shows the overall flow.

Step 201: Data input from the text input means 101 is first stored in the text storage means 102.

Step 202: From this text,
The word is extracted by the word extracting means 103. Information of the word processing knowledge storage unit 107 is used for extraction. The extracted words are sequentially stored in the word storage means 104, and the extracted parts are marked in the corresponding parts of the text storage means 102.

Step 203: The word finding means 105 searches for whether the extracted word appears in a place other than the extracted word in the text, and if so, marks it at the corresponding place in the text storage means 102. Attach

Step 204: The word division means 106 divides the text data of the text storage means 102 into words based on the marks written up to the previous step and the information of the word processing knowledge storage means 107.

Step 205: Output the divided data.

Hereinafter, the details of the operation of the word extracting means 103 in the step 202 of FIG. 2 and the word extracting process will be described using Japanese as an example.

In the word extracting means 103, without using a dictionary,
Words are extracted from text only by analyzing the character pattern. In the case of Japanese, it is possible to find a pattern of a hiragana character string that is determined to be a case particle without parsing, and a word is found using this pattern. The pattern information used for this analysis is stored in the word processing knowledge storage unit 107.
Among those stored in the storage area, one used for extraction or for both extraction and division is used.

FIG. 3 shows an example of the pattern information stored in the word processing knowledge storage means 107. Pattern 3
01 is a pattern that is used for both extraction and division. For example, in the case of a character string “the child is,” there is a pattern between “the” and “child” and between “the child” and “wa,” It can be divided, indicating that "child" can be extracted as a word. Similarly, the pattern 302 of the extraction / divisional pattern indicates that division / extraction is possible in the same manner in the case of a character string such as “Deluxe furniture,” and the pattern 303 indicates that the child is “ In the case of the character string, it indicates that division and extraction are possible in the same manner.

The pattern 310 is a dividing pattern to be described later. For example, in the case of a character string "recording method", when a kanji character "recording" is recognized as a word, the pattern follows. The two-character kanji “method” indicates that it can be divided.

Here, “arbitrary hiragana”, “two kanji”
Is a pattern specification that matches the text string, "character string", "is a pattern specification that matches the character string itself, and" extractable part "can regard the character string that matches the pattern as a word The location “dividable” indicates a point that can be divided during the division processing.

The word extracting means 103 compares this pattern with the character string of the text, and if the pattern matches, extracts the word of the matched part and stores it in the word storage means.
Store in 104.

The application of the pattern basically tries all the patterns over the whole area of the text and finds a place where the pattern matches. Figure 4 shows an example of the basic algorithm
However, other methods may be used to find a suitable location.

Step 401: word processing knowledge storage means
Select the first pattern from 107. Note that only patterns that can be selected for word extraction or word extraction and division can be selected.

Step 402: The pointer indicating the processing location is moved to the beginning of the text storage means 102.

Step 403: Attempt to apply the pattern from the position of the pointer.

Step 404: If the application of the pattern is successful, Step 405: Process the character string of the successful part. Here, since the character string is determined to be a word, a mark is added to the found word portion of the text storage means 102, and the found word is stored in the word storage means 104.

Step 406: The pointer is moved to a position where the pattern can be applied next. Details will be described later.

Step 407: Determine whether the pointer indicates the end of the text, and if not, step 4
Returning to step 03, the processing is repeated. If it indicates the end of the text, go to step 408.

Step 408: It is determined whether or not the selected pattern is the last one in the word processing knowledge storage means 107. If it is the last one, the process ends. If not the last, proceed to step 409.

Step 409: Knowledge processing means for word processing
The next pattern is selected from among 107 and step 402
Return to and repeat the process.

A case where the character string of the example of FIG. 5 is analyzed using the pattern shown in the example of FIG. 3 will be described. When the pattern 301 shown in FIG. 3 which is a word extraction divisional combined pattern is selected, the character string “this device is
If the processing pointer is located at the "ko" part of
The string "this" matches the pattern "any hiragana"
Since the character string "device" matches the pattern "two Chinese characters" and the character string "wa," matches the pattern character string "wa," the entire pattern matches the character string, and the pattern application succeeds. . Therefore, it is determined that the part of the character string “apparatus” is a word, and the part is marked and stored in the word storage unit 104.

The movement of the pointer in step 406 will be described with reference to the example of FIG. Moving the pointer in step 406, whether successful or unsuccessful, does not necessarily mean moving to the next character. When the pattern 301 is applied to the character string “this device is” in FIG. 5, it is assumed that the first application starts from “ko” at character position 1. In this case, if the next application is started from the next character, "no" at character position 2, the character string "no" matches the pattern "arbitrary hiragana", and all the following matches. The application succeeds.

However, in this case, the same result is obtained regardless of whether the processing is started from the character position 1 or from the character position 2, so that the results are duplicated. In order to avoid this, the movement of the pointer must proceed to the character following the end of the first pattern “arbitrary hiragana” in the pattern 301. In the example of FIG. 5, if the first application is from character position 1, the next application will be attempted from character position 3 and on. By analyzing the pattern, if successful or unsuccessful, it may be possible to calculate in advance how much to try to apply each time.

Patterns for finding Japanese words include information on character types such as kanji and hiragana,
A character string that is considered to be a case particle such as “wa” or “ga” is used. The character type change is likely to be a word break, and the character string immediately before the character string such as “wa,” “ga,” is likely to be a noun word. In addition to the above example,
A kanji character string just before an assertive expression such as “is.” Or “is.” Is also likely to be a noun word. Such knowledge is stored in the word processing knowledge storage unit 107 in advance.

Hereinafter, the operation of the word finding means 105 in the word finding process in step 203 of FIG. 2 will be described in detail. This processing aims at finding a place where the word extracted in step 202 appears in a place other than the place where the word is extracted in the text. An example of the basic algorithm is shown in FIG. 6. Step 601: Move the processing pointer to the beginning of the text.

Step 602: It is checked whether a word starting from the position of the pointer exists in the word storage means 104. If there is, go to step 603; otherwise go to step 604.

Step 603: If there is such a word, a mark is written in the text storage means 102 for the character string recognized as a word. As in the example of FIG.
In the text, the word extracted by the word extraction process in step 203 is marked if the same word appears in a place other than the extracted place.

Step 604: Move the pointer to the next character.

Step 605: Determine whether the pointer indicates the end of the text, and if not, return to 602 to repeat the processing. If the pointer is at the end of the text, the process ends.

Since the word finding process in step 203 corresponds to the search for a word in the text, a search for searching the text for each word in the word storage means 104 instead of the algorithm in FIG. A method may be adopted.

Next, the operation of the word dividing means 106 in the word dividing process in step 204 of FIG. 2 will be described in detail. The operation of the word segmentation process of step 204 is the same as the operation of the word extraction process of step 202, that is, the algorithm of FIG. 4, but differs in the following three points. (1) In the word extraction process in step 202, words are extracted based on the pattern, and the words extracted in step 405 in FIG. In the word division processing in step 204, division into words is performed based on the pattern, and the division result is written in the text storage means 102. (2) The word extraction process in step 202 is performed for extracting, out of the information stored in the word processing knowledge storage 107,
Alternatively, one that is also used for extraction and division is used. In the word division processing in step 204, of the information stored in the word processing knowledge storage unit 107, information that is also used for division or for both extraction and division is used. (3) The word extraction process in step 202 uses only character types such as kanji and hiragana and a specific character string pattern such as “wa,” “ga,”, but the word segmentation process in step 204 has already been found. Information such as word positions can be used as patterns.

Assume that an attempt is made to apply the pattern 310 shown in the example of FIG. 3 to the character string of the example of FIG. It is assumed that the word “device” has already been stored in the word storage means 104 and has been marked by the word finding means 105. At this time, the character string “device” of the character string “the device itself” in the text data matches the pattern “kanji word”, the character string “single” matches the pattern “two kanji characters”, and the character string Since "wa" matches the pattern character string "wa", the entire pattern matches the character string, and the pattern application succeeds. Therefore, it is determined that the part of the character string “single” is a word, and that it can be divided before and after the word. In this way, the text data is divided into words based on the pattern information in the word processing knowledge storage unit 107 and the word information in the word storage unit 104.

As described above, the text data is divided into words in steps 202 to 204.

In this embodiment, word division in Japanese is taken as an example. However, the present invention can be similarly applied to Chinese, which is the same sticky word as Japanese.

As described above, in the present embodiment, the word extracted by the word extraction means 103 based on the information of the word processing knowledge storage means 107 is stored in the word storage means 104, and the word information and the word processing The word division means 106 divides the text into words using the information in the knowledge storage means 107, so that word division without using a dictionary can be performed, and the practical effect is large.

(Second Embodiment) In the character string dividing apparatus according to the second embodiment, the number of words to be extracted is increased by repeatedly performing word extraction on the same text.

For example, as shown in FIG. 11, if the word “device” can be extracted from the character string “this device is” in the first word extraction, the word “device” can be extracted in the second word extraction. Can be extracted as a word from the character string, and the next time,
The word "action" can be extracted from the character string. As described above, the number of words increases by repeating word extraction for the same text.

As shown in FIG. 9, the character string dividing apparatus manages the number of words stored in the word storage means 104 and repeats the word extraction until the increase in the number of words is not observed. Is provided. Other block configurations are the same as those of the first embodiment (FIG. 1). However, the word extraction means 103 has been extended to use a counter for repetition processing in order to perform the repetition processing, and the word storage means 104 has been discovered in any number of repetitions in order to perform the repetition processing. This is different from the first embodiment in that the word is expanded so that the word can be stored.

The word number judging means 109 is constituted by a calculation mechanism of a computer.

The operation of the word segmenting apparatus configured as described above will be described. FIG. 10 shows the overall flow.

Step 1001: text input means 101
Is first stored in the text storage means 102.

Step 1002: A variable N for counting repetitions is initialized to 1. Step 1003: The word finding means 105 returns N
-Search whether the word extracted first time appears in a place other than the extracted word in the text. If it appears, a mark is made at the corresponding location in the text storage means 102.
Step 1003 is the same as the word finding process in step 203 in the first embodiment (FIG. 2), but the word finding process in step 1003 is found one time before the repetition, that is, the (N−1) th time. The difference is that only words need to be processed. In the first process, that is, when N = 1 and N−1 = 0, no word is stored in the word storage unit 104, and thus this process ends without any word being found.

Step 1004: The word extracting means 103
A word is extracted from the text data in the text storage means 102. For the extraction, word information such as a word mark found up to the previous repetition, a word mark written in step 1003, and information in the word processing knowledge storage unit 107 are used. The extracted words are sequentially stored in the word storage means 104 together with the value of the variable N for counting repetitions. Further, for the extracted portion, a mark is given to the corresponding portion of the text storage means 102. The extraction method is the same as that of step 202 in FIG. 2 in the first embodiment, except that word information that has been found so far is used for extraction, and the number of repetitions N is stored together with the extracted words in the word storage means. The two points that are stored in 104 are different.

Step 1005: It is determined whether the word has been sufficiently extracted. If not, step 100
If it is determined to be sufficient, the process proceeds to step 1007. For the determination, the word number determination means 109 determines whether or not a new word has been extracted in the current repetition.
Judge based on the information in 4. If at least one new word has been extracted, it is determined that it is not sufficient, and step 100 is executed.
Proceed to 6.

Step 1006: The variable N for counting repetitions is incremented by 1, and the flow returns to step 1003 to perform repetitive processing.

Step 1007: The word dividing means 106
Based on the marks written up to the previous step and the information in the word processing knowledge storage unit 107, the text data in the text storage unit 102 is divided into words. The division strategy can be changed at any time by changing the contents of the word processing knowledge storage means 107, but basically the word is divided before and after the word mark.

Step 1008: Output the result and end.

The processing in step 1004 is similar to the word extraction processing in step 202 in FIG. 2 in the first embodiment.
Using the word information found up to the repetition,
The difference is that words are extracted.

The operation will be described with reference to the example shown in FIG. From the text data "this device is", the word "device" is N = 1st time
Is extracted in the same manner as in FIG. 5 in the first embodiment. When N = 2 times, the character string “single” is determined to be a word from the text data “this device alone,” as in FIG. 8 in the first embodiment. N = 3
In the same time, similarly, the character string “operation” is determined as a word and extracted from the text data “according to the single operation”. The N-th word extraction is performed while decomposing the compound word using the information of the words extracted up to the (N-1) -th time.

As described above, the steps 1001 to 1008 make it possible to more precisely extract words from text data and perform word division.

As described above, in this embodiment, the word extraction unit 103 extracts a word based on the information of the word processing knowledge storage unit 107, and stores the extracted word in the word storage unit 104. The extraction means 103 repeatedly performs word extraction using the word information, and determines that the word number determination means 109 has extracted a sufficient number of words, and then determines the word information and the information in the word processing knowledge storage means 107. Using the word division means 106
By dividing text into words, it becomes possible to perform word division without using a dictionary, and the practical effect is great.

(Third Embodiment) The character string dividing apparatus according to the third embodiment quantitatively determines the possibility of word division and determines the word division position from the value.

In addition, a point at which word division is possible is obtained based on the numbers used in the text data. For example, as shown in FIG. 15, when there is a character string “this amount is 15,000 tons in total”, by examining character strings before and after the numbers 1 and 5, the numerical expression word “ 15,000 tons in total ".

As shown in FIG. 12, this apparatus includes a numerical expression finding means 110 for finding a numerical expression in a text, and a word for quantitatively calculating the possibility of dividing a certain position of the text into words. And division possibility calculating means 111.

The word dividing means 106 divides the text into words based on the value obtained by the word dividing possibility calculating means 111. The word processing knowledge storage unit 107 stores knowledge used not only by the word extraction unit 103 and the word division unit 105 but also by the word division possibility calculation unit 111.

FIG. 16 shows an example of information stored in the word processing knowledge storage means 107. For example, the pattern “arbitrary hiragana” in the pattern 301 of FIG. 3 corresponds to the pattern “arbitrary hiragana” + (separation 30) + two characters of Kanji (extractable) + (separation 30) + “ha”. And the numerical value that indicates the possibility of separating “Kanji 2 characters” (separation 3
0) is added, and a numerical value (separation 30) indicating the possibility that “two kanji characters” and ““ wa, ”” can be separated is added. This numerical value can be either a positive value or a negative value. A positive value indicates that the image can be divided, and a negative value indicates that the image cannot be divided. Further, a pattern in which "(extractable)" is written indicates that a character string that matches the pattern can be recognized as a word and extracted.

The other structure is the same as that of the second embodiment (FIG. 9).

The numerical expression finding means 110 and the word division possibility calculating means 111 are constituted by a computer mechanism.

The operation of the word segmenting apparatus configured as described above will be described. FIG. 13 shows the entire flow.

Step 1301: text input means 101
Is first stored in the text storage means 102.

Step 1302: Numerical expression discovery means 110
Looks for a numeric expression throughout the text, and marks before and after the numeric expression is found. The numerical expression is a character string expressing a numerical value such as “100 yen”, “about two months ago”, “15,000 meters”. Details will be described later.

Step 1303: A variable N for counting repetitions is initialized to 1. Step 1304: The word finding means 105 repeats N
-Search whether the word extracted first time appears in a place other than the extracted word in the text. This process is the second
This is the same as step 1003 in the embodiment (FIG. 10).

Step 1305: The word extracting means 103
Based on the word position mark and the numerical expression mark found up to the previous step, and the information in the word processing knowledge storage unit 107, calculate the points that can be divided into words in the text data. Perform word extraction based on This process is the same as step 1004 in the second embodiment (FIG. 10), except that the information that word division is possible before and after the numerical expression found in step 1302 can be used. Numeric expressions are treated as one word.

Step 1306: It is determined whether or not the word has been sufficiently extracted. If not, step 130
The process proceeds to step 1308 if determined to be sufficient. For the determination, the word number determination means 109 determines whether or not a new word has been extracted in the current repetition.
Judge based on the information in 4. If at least one new word has been extracted, it is determined that the word is not sufficient, and step 130
Proceed to 7.

Step 1307: The variable N for counting repetitions is incremented by 1, and the process returns to step 1304 to perform repetitive processing.

Step 1308: Dividability calculating means 11
1 is the mark of the word written up to the previous step,
Based on the numerical expression mark and the information in the word processing knowledge storage unit 107, a point that can be divided into words in the text data is calculated. Then, the word dividing means 106 divides the text at a place where it is determined that the text can be divided. Details will be described later.

Step 1309: Output the result and end.

Hereinafter, the details of the operation of the numerical expression finding means 110 in the numerical expression finding process in step 1302 in FIG. 13 will be described with reference to Japanese as an example. FIG. 14 shows the algorithm.

Step 1401: The pointers P, Q, and R are positioned at the head of the text. The pointer P is used to record the processing position, the pointer Q is used to record the beginning of the numerical expression, and the pointer R is used to record the end of the numerical expression.

Step 1402: It is determined whether the position pointed by the pointer P is at the end of the text. If it is the last, it ends. If not, go to step 1403.

Step 1403: It is determined whether or not the character pointed by the pointer P is a character expressing a number. Characters that represent numbers include arithmetic numbers such as 1, 2, 3, 1,
There are two, three, ten, one hundred and one thousand Chinese numbers. If it is a character representing a number, go to step 1405; otherwise, go to step 1404.

Step 1404: The pointer P is advanced by one character, and the flow advances to step 1402.

Step 1405: The pointer Q is matched with the pointer P. Set the pointer R to the character following P.

Step 1406: At this point, the pointer Q
To the character of the pointer P are regarded as a number expression string. It is determined whether the character next to the pointer P, that is, the character at the position of P + 1 can be connected to the expression of the numbers from Q to P. In this case, the kanji numerals such as "520" and the algebraic numeral "23" do not follow, but the kanji numerals "billion", "million" and "one" can be connected. Examine these connectivity possibilities. If the connection can be made, go to step 1407.
If connection is not possible, go to 1408.

Step 1407: The pointers P and R are advanced one character at a time, and the process returns to step 1406 to repeat the same processing.

Step 1408: It is determined whether the character next to the pointer P, that is, the character at the position of P + 1 can be a unit of numerical value. If it can be a unit, go to step 1409; otherwise go to step 1414.

Step 1409: It is determined that the pointer R represents a unit, and the cursor moves to the next character of the character string.

Step 1410: It is determined whether the character string located before the position of the pointer Q is a number preposition. Numerical prepositions are "about", "approximately", "sum" and the like.
If it is determined to be a number preposition, go to step 1411; otherwise go to step 1412.

Step 1411: Move the pointer Q to the head of the found number preposition.

Step 1412: The character string from the pointer Q to the pointer R is determined to be a numerical expression. Then, the fact that the possibility of division is high before and after that and the possibility of division is low in the meantime are written in the text storage means 102. FIG. 15 is a schematic diagram showing an example of this division possibility. A positive value is written because the possibility of division before and after the numerical expression is high, and a negative value is written during the numerical expression because the possibility of division is low, that is, division is impossible. By making the absolute value of the negative value sufficiently large, even if a positive value is erroneously written in the calculation of the dividable point in step 1308, it is possible to prevent division between numerical expressions.

Step 1413: Move the pointer P to the position of the pointer R (substitute R for P), and
Return to 02.

Step 1414: Compare pointers Q and R. If RQ is 1, the found character string is a single character, and it is highly likely that it is not a numerical representation. So we do not judge this as a numerical representation. RQ is 1
If so, go to step 1415. Otherwise step 14
Go to 12.

Step 1415: Move the pointer P to the character next to Q, and return to step 1402.

As described above, "1" of the text character string "Ichiro Suzuki" is not determined to be a numerical expression (step 141).
4), the text character string "about 15,000 meters" can be determined to be a numerical expression.

Hereinafter, the details of the operations of the word division possibility calculation means 111 and the word division means 106 in the division possibility calculation processing and the word extraction processing in step 1308 of FIG. 13 will be described using Japanese as an example. FIG. 17 shows the algorithm.

Step 1701: division possibility calculating means 11
1 is for the word processing knowledge storage means 107,
Apply all patterns that can be used for word segmentation. This processing corresponds to step 20 in the second embodiment.
Apply the pattern in the same way as in step 4, but if the application is successful, the value of the separability written in the pattern is
Write it to the corresponding location in the text storage means 102. If a numerical value has already been written, a numerical value added to the written numerical value or a numerical value obtained by adding and averaging is written.

Step 1702: The word dividing means 106
The text is split where there is a separability value equal to or greater than a preset threshold value over the entire text.

FIG. 18 is a schematic diagram showing the relationship between the numerical value of the division possibility corresponding to the character string and the threshold value. For example, step 13
Regarding the word extracted at the Nth time in 05, the certainty as a word depends on the accuracy of the word extracted up to the N-1th time. That is, the first extracted word has the highest likelihood as a word, but thereafter, the accuracy of extraction gradually decreases. Therefore, if the knowledge applied in step 1701 reflects the fact that the greater the N at which the word is found, the smaller the probability that the word can be divided before and after the word is prepared, a portion with a small degree of accuracy is not divided. Conversely, even if the pattern has a small degree of accuracy, if a plurality of small degrees of accuracy are accumulated at one location, the total accuracy is increased and the pattern is divided. In this way, by calculating the word division based on the division possibility, it is possible to control high precision and fineness of division.

As described above, in this embodiment, the numerical expression finding means 110 specifies a numerical expression, and the word extracting means 103 extracts a word based on the information in the word processing knowledge storage means 107 and the position of the numerical expression. Extract and store the extracted words in word storage means 10
4 and the word extraction means 103 repeatedly performs word extraction using the word information, and after the word number determination means 109 determines that a sufficient number of words have been extracted, the word information and the word processing Using the information in the knowledge storage unit 107, the word division possibility calculation unit 111 calculates points that can be divided, and the word division unit 106 divides the text into words, thereby performing word division without using a dictionary. And its practical effect is great.

(Fourth Embodiment) The character dividing apparatus according to the fourth embodiment separates character types from text and separates words such as hiragana and katakana based on the character types.

This device is, as shown in FIG. 19, a kanji word extracting means for extracting kanji words from text data.
112, katakana word extracting means 113 for extracting katakana words from text data, and hiragana word extracting means 114 for extracting hiragana words from text data.

The word processing knowledge storage unit 107 classifies and stores knowledge for kanji words, katakana words, and hiragana words as word processing knowledge.

The other structure is similar to that of the third embodiment (FIG. 1).
No difference from 2). Further, the kanji word extracting means 112, the katakana word extracting means 113 and the hiragana word extracting means 114 are constituted by a computing mechanism of a computer.

The operation of the word segmenting apparatus thus configured will be described. FIG. 20 shows the entire flow.

Step 2001: Text input means 101
Is first stored in the text storage means 102.

Step 2002: Numerical expression discovery means 110
Searches for a numeric expression throughout the text, and if found, marks before and after it as a numeric expression.
This processing corresponds to step 1302 in the third embodiment.
Is the same as

Step 2003: Kanji word extracting means 112
Focuses only on the character string of the kanji part,
Extract Kanji words. Details will be described later.

Step 2004: Katakana word extraction means 11
In step 3, the katakana word is extracted by mainly processing only the character string of the katakana part. Details will be described later.

Step 2005: Hiragana word extraction means 11
4 extracts a hiragana word by mainly processing only the character string in the hiragana portion. Details will be described later.

Step 2006: The words extracted up to the previous step are stored in the word storage means 104. The word finding means 105 searches for a word that appears in the text for all the words extracted up to the previous step,
Mark all with words. Step 2007: The division possibility calculating means 111 calculates the word mark, the numerical expression mark, and the information in the word processing knowledge storage means 107 written up to the previous step.
Calculate points that can be divided into words in text data.
Then, at a place determined to be possible to divide, the word dividing means 10
6 split the text. This processing is the same as step 1308 in the third embodiment (FIG. 13).

Step 2008: Output the result and end.

Steps 2003 and 200
4. The order of the three steps of step 2005 does not matter. For example, in step 2004, the katakana word extraction process is executed in step 200.
It may be earlier than the three kanji word extraction process. However, it is generally easier to find katakana words and kanji words than to find hiragana words, and it is better to follow the hiragana word extraction process 2005.

Hereinafter, the kanji word extraction processing in step 2003 in FIG. 20 will be described. The kanji word extraction processing is basically performed by limiting the word extraction process in the second embodiment to kanji character strings, and the processing is the same. FIG. 21 shows the algorithm. Step 2101: N representing repetition is initialized to 1.

Step 2102: The word finding means 105
A search is made to see if the word extracted at the (N-1) th repetition appears in a place other than the extracted word in the text. If it appears, a mark is made at the corresponding location in the text storage means 102. The first process, ie, N = 1 and N−1 =
When the value is 0, no word is stored in the word storage means 104, so this process ends without finding any word.

Step 2103: Kanji word extracting means 112
Extracts a kanji word from the text data in the text storage means 102. For extraction, word information such as a mark attached to a word that has already been found, and information in the word processing knowledge storage unit 107 are used. A kanji character string is searched from the beginning of the text, and the word processing knowledge storage unit 107 is stored in that part.
Words are extracted by applying the kanji pattern from. The extracted words are sequentially stored in the word storage means 104 together with a variable N for counting repetitions. Further, for the extracted portion, a mark is given to the corresponding portion of the text storage means 102. The extraction method is the same as that of step 1305 in FIG. 13 in the third embodiment, except that pattern application can be omitted for parts other than kanji, and that only the kanji part is processed.

Step 2104: It is determined whether or not the kanji word has been sufficiently extracted. If not, step 2
If it is determined to be sufficient, the process is terminated. For the determination, the word number determination means 109 determines whether or not a new word has been extracted in this repetition, based on the information in the word storage means 104. If at least one new word has been extracted, it is determined that the word is not sufficient, and the process proceeds to step 2105.

Step 2105: The variable N for counting repetitions is incremented by 1, and the process returns to step 2102 to repeat.

The above is the kanji word extraction processing. The same processing is performed for katakana and hiragana, but the knowledge of the word processing knowledge storage means 107 used in each processing is limited to those for kanji, katakana, and hiragana. By specifying the application location of the pattern by the character type and selectively applying only the pattern suitable for that character type, the efficiency of pattern application can be improved, and depending on the character type, methods other than pattern application can be used. Can also be adopted.

As described above, in this embodiment, the numerical expression discovery means 110 detects a numerical expression, the kanji word extraction means 112 reads a kanji word, the katakana word extraction means 113 reads a katakana word, and the hiragana word extraction means 114 reads a hiragana word. Each word is efficiently extracted and stored in the word storage unit 104, and the word division possibility calculation unit 111 can divide the word using the word information, the information of the numerical expression, and the information of the word processing knowledge storage unit 107. By calculating points and dividing the text into words by the word dividing means 106, it becomes possible to perform word division without using a dictionary, and the practical effect is large.

(Fifth Embodiment) In the character dividing device according to the fifth embodiment, it is more appropriate to treat a combination of different character types as one word, such as a character string "object-oriented". Things are treated as one word.

As shown in FIG. 22, this apparatus includes word combination possibility calculation means 115 for examining the combination possibility of two adjacent words among the extracted words. Other configurations are the same as those of the fourth embodiment (FIG. 19). The word combination possibility calculation means 115 is constituted by a calculation mechanism of a computer.

The operation of the word segmenting apparatus thus configured will be described. FIG. 23 shows the overall flow. Steps 2301 to 230 of the processing flow
Steps 6 to 6 are performed in step 200 of the fourth embodiment (FIG. 20).
1 to step 2006.

The data input from the text input means 101 is first stored in the text storage means 102 (step 2301), and the numerical expression finding means 110 searches for the numerical expression over the entire text. Is marked as a numerical expression (step 230
2). The kanji word extracting means 112 extracts kanji words by mainly processing only the character string of the kanji part (step 2303). The katakana word extracting means 113 extracts katakana words by focusing only on the character string of the katakana part (step 2304). The hiragana word extracting means 114 extracts a hiragana word by mainly processing only the character string of the hiragana portion (step 2305).
The words extracted up to the previous step are stored in the word storage unit 104, and the word finding unit 105 searches for the word appearing in the text for all the words extracted up to the previous step, and marks the words in all of them. Attach (Step 230
6).

Step 2307: The word combination possibility calculation means 115 searches for a word having a high possibility of appearing adjacently among the extracted words. Then, the word division possibility of the adjacent part of those words is set to minus. Until step 2305, the processing was basically performed on the assumption that the word is cut off by the character type. Here, the possibility of combining them is examined, and the division when forming one word across the character type is performed. Recalculate the points. Details will be described later.

Step 2308: The word division possibility calculation means 111 executes the word mark written in the previous step, the numerical expression mark, the word combination possibility (negative value of division possibility), and the word processing Based on the information in the knowledge storage unit 107, a point in the text data that can be divided into words or a point that cannot be divided is calculated. Then, the word dividing means 106 divides the text at the place where it is determined that the text can be divided. This processing is performed in step 130 of the third embodiment (FIG. 13).
Same as 8.

Step 2309: Output the result and end.

Note that, as described in the fourth embodiment, steps 2303, 2304, and 2305
Does not matter.

Hereinafter, the word combination possibility determination processing in step 2307 of FIG. 23 will be described. FIG. 24 shows the algorithm.

Step 2401: One first word is extracted from the word storage means 104. Hereinafter, this word is referred to as a word t.

Step 2402: The word appearing after the word t and its frequency are calculated by scanning the text data, and the probability of appearing after the word t is calculated. At this time,
Words that appear later, such as punctuation and case particles, such as "ga", "wa", and "wo", are not counted. FIG. 25 shows a calculation example. FIG. 25 shows that when the word “object” is the word t, the following words include “model”, “directivity”, “relation”, “between”, etc., and the respective probabilities are 0.1, 0.5, 0.35. , 0.01.

Step 2403: It is determined whether or not there is a word having a probability equal to or greater than a predetermined threshold among the appearance probabilities of the respective words calculated in this way. If there is such a word, the process proceeds to step 2404; otherwise, the process proceeds to step 2409. In the example of FIG.
Assuming that the threshold is 0.3, it is determined that two words “directed” and “between” appear with a probability greater than or equal to the threshold.

Step 2404: A set of words appearing after the word t with a probability greater than or equal to the threshold is referred to as a set A. FIG.
In the example, two words “directed” and “between” are included in the set A.

Step 2405: One first word is extracted from the set A. This is called word a.

Step 2406: The word appearing before word a and its frequency are examined, and the probability of appearing before word a is calculated. At this time, punctuation marks and words considered to be case particles as words appearing before, that is, "ga", "wa", "wo", etc. are not taken into account. FIG. 26 and FIG. 27 are examples in which the appearance probabilities of words appearing before the respective words when “directivity” and “between” are selected as the word a are calculated.

Step 2407: Among the appearance probabilities of the respective words calculated in this way, there is a word having a probability equal to or more than a predetermined threshold, and it is determined whether or not the word t is included in the word. . If it is included, the process proceeds to step 2408; otherwise, the process proceeds to step 2409. In the example where the word a in FIG. 26 is “directional”,
It is determined that “object” appears before “directivity” with a probability equal to or greater than the threshold. In the example where the word a is “between” in FIG. 27, there is a word that appears before “between” with a probability greater than or equal to the threshold, but there is no “object” among them.

Step 2408: When the word t appears before the word a with a probability greater than or equal to the threshold value, the two are judged to have a high possibility of combination and are regarded as one word. Then, in the text storage unit 102, the word separability of a place where two words appear adjacent to each other is set to a negative value. FIG. 25 and FIG. 26
In the example, since the probability that “object” and “directivity” appear adjacent to each other is high, it can be determined that these two words are collectively a single word “object-oriented”. Therefore, as shown in FIG. 28, the word separability of a place where two words appear adjacent to each other is set to a negative value in the text storage means 102.

Step 2409: Determine whether word a is the last word in set A. Step 24 for the last word
Otherwise, go to step 2410.

Step 2410: One next word is taken out of the set A, and this is set as a word a. Step 24
Returning to step 06, the processing is repeated.

Step 2411: It is determined whether the word t is the last word in the word storage means 104. If the last word, end. Otherwise, go to step 2412.

Step 2412: The next word is fetched from the word storage means 104, this is used as word t, and the process returns to step 2402 to repeat the processing.

As described above, in this embodiment, the numerical expression finding means 110 is for numerical expression, the kanji word extracting means 112 is for kanji words, the katakana word extracting means 112 is for katakana words, and the hiragana word extracting means 114 is for hiragana. Each word is extracted and stored in the word storage means 104, and the word combination possibility calculation means 115
Calculates the likelihood of two words joining. Then, the word division possibility calculating means 111 calculates a dividable point using the word information obtained so far and the information of the word processing knowledge storage means 107, and the word dividing means 106 converts the text into words. To divide. By doing so, it becomes possible to perform word division without using a dictionary, and the practical effect is great.

(Sixth Embodiment) A character dividing device according to a sixth embodiment arranges kanji character strings included in text data according to the number of characters, and applies a common rule to kanji character strings having the same number of characters. , To efficiently extract kanji words.

As shown in FIG. 29, this apparatus includes a partial character string extracting means 116 for extracting a partial character string including a kanji character string and two characters before and after it from text data,
A text processing buffer 117 for storing the character strings extracted by the partial character string extraction means 116 while they are processed,
A character string classification and sorting means 118 for classifying and sorting the character strings in the text processing buffer 117. The kanji word extraction means 112 efficiently extracts kanji words from text data using the partial character string extraction means 116, the text processing buffer 117, and the character string classification and alignment means 118. Other configurations are the same as those of the fifth embodiment (FIG. 22).

The text processing buffer 117 is set in a memory of a computer or a storage area of a hard disk device, and the partial character string extracting means 116 and the character string sorting / aligning means 118 are constituted by a computer mechanism.

The operation of the word segmenting apparatus thus configured will be described. FIG. 30 shows the overall flow. Steps 3001 and 3002 of the processing flow
Are steps 2301, 23 of the fifth embodiment (FIG. 23).
Same as 02. Step 3004 to Step 30
09 is the same as Steps 2304 to 2309 of the fifth embodiment (FIG. 23). Hereinafter, the kanji word extraction processing by kanji character string extraction in step 3003 characterizing the sixth embodiment will be described in detail. The details of the algorithm are shown in FIG.

Step 3101: First, the partial character string extracting means 116 extracts the character string part of the kanji from the text storage means 102, and creates and stores a table of the character string in the text processing buffer 117. When extracting, N characters before and after the character are also extracted. This example will be described with reference to the drawings. FIG.
FIG. 33 shows a kanji character string extracted from the two example sentences together with two characters before and after the kanji character string. If the number N of characters before and after the kanji character string is set to a large value of 3 or more, detailed rules can be prepared in word extraction, and extraction accuracy improves, but the amount of calculation increases. On the other hand, if the number is set to 1 or less, detailed rules cannot be prepared, and the extraction accuracy decreases. Therefore, it is appropriate to set N to 2 in order to obtain appropriate accuracy with an appropriate amount of calculation.

Step 3102: The character strings extracted in this way are classified by the character string classification and alignment means 118 according to the length of the kanji character string.

Step 3103: The classified character string is
Further, the character string classification and sorting means 118 sorts the kanji character strings in dictionary order or character code order, and creates a table. If there is the same thing including the character string before and after, record the number of appearances of them and summarize them. The table of FIG. 34 is obtained by extracting and sorting two kanji character strings from the table of FIG. The table of FIG. 35 is obtained by extracting and arranging three kanji character strings from the table of FIG. Similarly, a table is created for four characters, five characters, etc., but M characters (M = 5,
Character strings longer than (6, 7, etc.) may be combined into one table.

Step 3104: The kanji word extracting means 112 extracts a kanji word from the two-character table and the three-character table among the tables thus created. In the extraction of words, words having punctuation before and after, and those surrounded by punctuation and case particles are regarded as words. These detailed rules are stored in the word processing knowledge storage unit 107. If the same kanji character string appears in the table, no duplicate processing is performed. For example, in the line 3502 in FIG. 35, it is determined that the character string “lower house” is surrounded by the case particle “no”, and if it is recognized as a word, the process in the line 3503 in FIG. 35 is not performed.
The extracted words are sequentially stored in the word storage unit 104.

Step 3105: Next, from the table of character strings of four or more characters, based on the word knowledge obtained from the table of two characters and three characters, the kanji word extraction means 112 performs word division of compound words, Extract words. This is because, for example, the word “lower house” is extracted from a three-character table, and the character string “lower house member” in the five-character table is surrounded by case particles and is assumed to be a noun. , "Lower of the lower house" is determined to be a compound noun composed of the known word "lower of the house" and an unknown word of "lower
By the principle that the word can be extracted. Such rules are stored in the word processing knowledge storage unit 107.

As described above, effective kanji word extraction can be realized. Note that the table of character strings extracted in step 3101 is similar to the N-gram statistical method in natural language,
Actually, instead of making a character string into a table, only a pointer may be made into a table. As shown in FIG. 36, only one character string is stored, and only the position and length of the extracted character string are recorded in the table. By doing so, the storage area can be reduced.

As described above, in this embodiment, the kanji character strings included in the text data are arranged according to the number of characters, and the same rules are applied to the kanji character strings having the same number of characters to extract the kanji words. are doing. Therefore, a kanji word can be extracted extremely efficiently as compared with a case where a word is extracted by applying a rule verbatim from the beginning of text data.

(Seventh Embodiment) The character dividing device according to the seventh embodiment sorts out hiragana character strings included in text data and extracts hiragana words.

In this apparatus, as shown in FIG. 37, a reduced character string creating means 120 for replacing a character string of text data to create a reduced character string, and a conversion table 119 for creating the reduced character string are used. Have. Degenerate string creation means 1
The degenerated character string created by 20 is a buffer 117 for text processing
The character string extraction means 116 extracts a partial character string including hiragana from the degenerated character string, and the character string classification and alignment means 118 classifies and sorts the partial character string, and extracts a hiragana word. The means 114 uses the partial character string extraction means 116, the text processing buffer 117, the character string classification and alignment means 118, the character string conversion table 119, and the degenerated character string creation means 120 to efficiently convert hiragana words from text data. To extract.

The character string conversion table 119 is set in a memory of a computer or a storage area of a hard disk device, and the degenerated character string creating means 120 is configured by a computer.

[0187] The operation of the word segmenting apparatus configured as described above will be described. FIG. 38 shows the overall flow. Steps 3801 to 380 of the processing flow
Step 4 is the same as steps 3001 to 3004 in the sixth embodiment (FIG. 30). Steps 3806 to 3809 are the same as steps 3006 to 3009 of the sixth embodiment (FIG. 30). Hereinafter, the hiragana word extraction processing of step 3805, which characterizes the seventh embodiment, by generating a reduced character string will be described in detail. FIG. 39 shows details of the algorithm.

Step 3901: First, the contracted character string creating means 120 creates a character string in which only the hiragana portion of the entire text is focused and portions other than the hiragana are contracted.
This is realized by replacing a part other than Hiragana with an appropriate symbol based on the character string conversion table 119. String conversion table
An example of 119 is shown in FIG. In this way, character strings other than hiragana are converted into one symbol. FIG. 41 shows a conversion example using the conversion table of FIG. In this example, punctuation is not converted.
Since the types of characters can be reduced by using characters other than hiragana as symbols in this manner, one Japanese character that normally needs to be expressed in 2 bytes or more can be expressed in 1 byte. Therefore, the entire storage area used in the hiragana processing can be reduced.

Step 3902: partial character string extracting means 11
6 extracts the character string portion of Hiragana from the degenerated character string, creates and stores a table of the character string in the text processing buffer 117. When extracting, one converted character before and after that character is also extracted. FIG. 42 is a table in which the hiragana character string is extracted from the example sentence of FIG. This table is similar to the table created in step 3101 of the sixth embodiment (FIG. 31).
The same processing can be performed by creating a table containing only pointers as shown in FIG.

Step 3903: String sorting / sorting means 11
8 classifies the character string extracted in this way by the length of the hiragana character string. This process is the same as step 3102 in the sixth embodiment (FIG. 31).

Step 3904: Character string classification and alignment means 11
8 further sorts the classified character strings in the dictionary order or the character code order of the hiragana character string portion and creates a table. If the same thing is included, including the preceding and following symbols, record the number of appearances and summarize them. This process is the same as step 3103 of the sixth embodiment (FIG. 31).

Step 3905: The hiragana word extracting means 113 extracts a hiragana word from the two-character table, the three-character table, and the four-character table among the tables thus created. Word extraction includes punctuation marks before and after,
A character string or the like appearing many times surrounded by punctuation marks and case particles is regarded as a word. A character string that appears many times after a single kanji character is determined to be a ending of a declinable word. These detailed rules are stored in the word processing knowledge storage unit 107. Of the character strings listed in the table, the same character string is reduced in duplicate processing. This is the same as step 3104 in the sixth embodiment (FIG. 31).

Thus, high-speed hiragana word extraction can be realized.

As described above, in this embodiment, for hiragana, the degenerated character string creating means 120 creates a degenerated character string from text data based on the character string conversion table 119, and the partial character string extracting means 116 The buffer 117 for character strings and the character string classification / alignment unit 118 extract the hiragana partial character strings from the degenerated character strings, and classify and sort them. The hiragana word extracting means 113 extracts hiragana words from the extracted hiragana words and stores them in the word storage means 104. Therefore, hiragana words can be extracted very efficiently.

(Eighth Embodiment) In an eighth embodiment, a process of reviewing word division of text will be described.

In the word segmentation method of the present invention, words are extracted from text data without using a dictionary. Therefore, words used for word segmentation of text are not fixed. In a changing document, extremely adaptive word extraction and word segmentation are possible, such that a character string that has not been extracted as a word is taken in as a new word as a result of analyzing the article on the next day.

On the other hand, if the text that has been divided into words in the past is redone using the currently recognized word, the text may be divided at a different place from the previous one. Maintenance is required from time to time.

In this embodiment, a process of reviewing word division of a text that has already been word-divided will be described.

FIG. 43 shows an information retrieval apparatus for performing a search using a word-segmented text as a search object, wherein the word division of the text to be searched can be reviewed.

As in the first embodiment (FIG. 1), this apparatus comprises a text input means 101, a text storage means 102, a word extraction means 103, a word storage means 104, a word discovery means 105,
A search target text storage unit 204 that includes a word division unit 106, a word processing knowledge storage unit 107, and a text output unit 108, and stores word-divided text (search target text). Extracted word storage means 207 for storing the used words, index creation means 201 for creating a search index for the search target text, index storage means 202 for storing the created index, Search means 203 for searching for contained text
A word comparison unit 206 that compares the word stored in the extracted word storage unit 207 with the latest word stored in the word storage unit 104; and a word division correction that controls the word division of the search target text again. Means 205.

In this apparatus, as described in the first embodiment (flow chart in FIG. 2), text data input from the text input means 101 is stored in the text storage means 102, and Using the information in the word processing knowledge storage unit 107, a word is extracted from this text, the extracted word is stored in the word storage unit 104, and a mark is added to the word extraction location of the text. The word finding means 105 searches whether or not the extracted word also appears in other places in the text, and if so, marks the corresponding part of the text.
The word dividing means 106 divides the text data into words based on the marks written in the text and the information in the word processing knowledge storage means 107.

The text data after the word division is stored in the search target text storage unit 204, and the words stored in the word storage unit 104 are stored in the extracted word storage unit 207.

The index creating means 201 creates an index for searching for a text to be searched using the word-divided text data.

Here, the index creation means 201
A case will be described in which an index capable of performing full-text search and word search is created. A full-text search is to search whether a character string of a search word is included in text data, while a word search is to search whether a search word is included as a word in text data. . For example, if the text is "Tokyo, ‥", a full-text search will find hits such as "Tokyo" and "Kyoto". If they don't match, they won't hit.

The text data is obtained by word division processing.
As shown in FIG. 45 (b), the data is converted into text data in which the beginning of a word is separated by a word start symbol "[" and the end of the word is separated by a word end symbol "]". Since “[” and “]” may actually appear in the text, a special character code or the like that does not appear in the text is used as a word start symbol or a word end symbol.

First, the index creating means 201 extracts an index character string having a character string length N (for example, N = 2) from the word-divided text data.
In the case of the text "Implementation of the present invention", "Original"
"Invention", "Ming", "Fruit", "Implementation", "Departure" and "Ha" can be extracted. An index is created with each of these character strings. In addition to recording each character string, document number, and character position, the index is added with word information indicating how the character string touches a word start symbol and a word end symbol.

As the word information, here, FIG.
As shown in FIG. 3, three information of whether the first character is the beginning of a word, the first character is at the end of the word, or the second character is at the end of the word is 1bi.
It is provided in the form of a flag of t.

The index created in this way is stored in the index storage means 202.

When performing a full-text search using this index, the character string of the search word is divided from the head for each character string length N (here, N = 2) of the index, and each character string is the first character string. Remembers the starting character from the character. When a search string is decomposed and referred to as a partial search string, for example, the search string "full-text search device"
Is divided into three partial search character strings of “full text”, “search”, and “device”, which are 0th, 2nd, and 4th from the first character, respectively.

At this time, if the search character string is not divisible by N, the search character string is divided so as to partially overlap, and the set of N characters is set so that the set of partial search character strings always covers the entire original search character string. Take out. For example, the word "search string"
The “search”, “character”, and “character string” may be the 0th, 2nd, and 3rd, respectively.

Next, the index is searched for the partial search character string, and the corresponding character string is extracted. Then, for the extracted index, the index search target document number and the position of the character string in the document are checked.
Evaluate continuity. If the search string is "full-text search device", when the "full-text", "search", and "device" have the same document number and the occurrence position of the character string "full-text" is the xth character, the occurrence of the character string "search" If the position is the (x + 2) th character and the appearance position of the character string “device” is the (x + 4) th character, it is determined that this document includes the character string “full text search device”.

On the other hand, when performing a word search using this index, the character string of the search word is divided into partial search character strings, and the index is searched for this partial search character string, as in the case of full-text search. And extract the corresponding character string.

At this time, the search character string is “full text search device”.
In the case of, for the first partial search string "full text", refer to the flag where the first character of the index is the beginning of the word, and if the flag is not set, even if the character string matches, Identify those that do not. Similarly, for the last partial search character string "device", refer to the flag indicating whether the second character of the index is at the end of the word. , And those that do not apply. For other partial search character strings (the character string "search" in this example), only the character match is checked, and the flag is not checked.

For the index thus extracted, the document number to be searched for the index and the position of the character string in the document are examined to evaluate the continuity. If the document numbers are the same and their appearance positions are continuous , It is determined that this document contains the word “full-text search device”.

In the word search, the index flag confirms that the first character of the search character string is the beginning of the word and that the last character of the search character string is the end of the word. By doing so, it is possible to search for a word that exactly matches the search word, and by checking only that the first character of the search string is the beginning of the word, it is possible to search for words that match exactly and You can search for exact and suffix words by only checking that the last character of the search string is at the end of the word, so that the first character of the search string is You can search only for words that start with a word by checking that it is the beginning of a word and the last character of the search string is not at the end of the word. The first character is not made at the beginning of the word, the last character of the search string by making sure that it is at the end of a word, it is possible to search only the words of the rear match.

The operation of reviewing the word division of the search target text already stored and correcting the index of the text is performed in the following procedure as shown in FIG.

First, the word comparison means 206 compares the word stored in the extracted word storage means 207 and the latest word stored in the word storage means 104, used for word division of the search target text. Then, a word (difference word) stored in the word storage unit 104 but not stored in the extracted word storage unit 207 is extracted (a). This difference word is passed to the search means 203, and the search means 203 performs a full-text search using the difference word as a search character string using the index of the search target text, and searches for a text including the difference word character string, The document number is transmitted to the word division correction means 205 (b).

The word division correcting means 205 reads out the search target text of the document number from the search target text storage means 204, deletes the word start symbol and the word end symbol contained in the text data, and deletes the words before the word division. The data is returned to the text data (c), and the text data is output to the text storage means 102 so that the text data is divided again.

[0219] Word division is performed on the text data stored in the text storage means 102 in the above-described procedure. At this time, the words already stored in the word storage means 104 are used for word division of the text data (d).

[0220] The word division correction means 205 compares the text data, again stored in the text storage means 102, into which the word has been divided and the original search target text data stored in the search target text storage means 204. If there is a difference, the search target text data in the search target text storage unit 204 is updated with the text data that has undergone word division again (e).

When the search target text data is updated, the index creation means 201 re-creates the index of the search target text (f), and the created index is stored in the index storage means 202 (g). .

The word division correction means 205 performs the processing of (c), (d), (e), (f), and (g) on the search target texts of all document numbers detected by (b). To control. Further, the word division correction unit 205 replaces the content of the extracted word storage unit 207 with the word stored in the word storage unit 104.

With such a series of processing, maintenance of the search target text is performed.

[0224] Here, the text data that has undergone word segmentation again is compared with the original text data to be searched stored in the search text storage means 204, and only when there is a difference, (E) the search target text data in the search target text storage unit 204 is updated, but if this comparison is omitted and the word division of the text data is performed again, the search is performed using the text data after the re-division. The search target text data in the target text storage unit 204 may be constantly updated.

Also, here, text data that has been word-divided by the character string dividing device of the first embodiment is described as an example, but the character string dividing devices of the second to seventh embodiments will be described. Can be applied in a similar manner.

[0226] Here, a case where full-text search and word search can be performed using one index has been described. However, the present invention can be applied to a case where a full-text search index and a word search index are separately provided. Is possible. In this case, the search target text having the difference word is searched using the full-text search index, and the word search index is updated using the subdivided text data.

[0227]

As is apparent from the above description, the word dividing apparatus and the word dividing method of the present invention can divide a text into words without using a dictionary.

For example, when performing a document search for a newspaper, a word is extracted using a newspaper. When performing a document search for a technical document, a word is extracted using a technical document. Words that match the target document can be extracted, and word division processing can be performed accurately.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a word segmentation device according to a first embodiment of the present invention;

FIG. 2 is a flowchart showing the operation of the word segmentation apparatus according to the first embodiment of the present invention;

FIG. 3 is a conceptual diagram showing an example of the structure of character processing pattern information of the word segmentation device according to the first embodiment of the present invention;

FIG. 4 is a flowchart showing an operation of applying a character processing pattern according to the first embodiment of the present invention;

FIG. 5 is a conceptual diagram showing a result of applying a character string pattern according to the first embodiment of the present invention;

FIG. 6 is a flowchart showing an operation of a word finding process according to the first embodiment of the present invention;

FIG. 7 is a conceptual diagram showing a word finding result according to the first embodiment of the present invention;

FIG. 8 is a conceptual diagram showing word division according to the result of applying a character string pattern in the first embodiment of the present invention;

FIG. 9 is a block diagram illustrating a configuration of a word segmentation device according to a second embodiment of the present invention;

FIG. 10 is a flowchart showing the operation of the word segmentation device according to the second embodiment of the present invention;

FIG. 11 is a conceptual diagram showing a repeated result of word extraction according to the second embodiment of the present invention;

FIG. 12 is a block diagram showing a configuration of a word segmentation device according to a third embodiment of the present invention;

FIG. 13 is a flowchart showing the operation of the word segmentation device according to the third embodiment of the present invention;

FIG. 14 is a flowchart showing an operation of a numerical expression discovery process according to the third embodiment of the present invention;

FIG. 15 is a conceptual diagram showing the possibility of word division of a character string in the vicinity of a numerical expression according to the third embodiment of the present invention;

FIG. 16 is a conceptual diagram of character processing knowledge information according to the third embodiment of the present invention;

FIG. 17 is a flowchart showing a word division processing operation according to the third embodiment of the present invention;

FIG. 18 is a conceptual diagram showing an example of the possibility of word division of a character string in the third embodiment of the present invention;

FIG. 19 is a block diagram showing a configuration of a word segmentation device according to a fourth embodiment of the present invention;

FIG. 20 is a flowchart showing the operation of the word segmentation device according to the fourth embodiment of the present invention;

FIG. 21 is a flowchart showing word extraction processing according to the fourth embodiment of the present invention;

FIG. 22 is a block diagram showing a configuration of a word segmentation device according to a fifth embodiment of the present invention;

FIG. 23 is a flowchart showing the operation of the word segmentation apparatus according to the fifth embodiment of the present invention;

FIG. 24 is a flowchart showing the operation of word combination possibility determination processing in the fifth embodiment of the present invention;

FIG. 25 is a diagram illustrating an example of calculating a probability of a word connected after a word in the word combination possibility determination process according to the fifth embodiment of the present invention.

FIG. 26 is a diagram illustrating one example of calculating a probability of a word connected before a word in the word combination possibility determination process according to the fifth embodiment of the present invention;

FIG. 27 is a diagram illustrating another example of calculating the probability of a word connected before a word in the word combination possibility determination process according to the fifth embodiment of the present invention.

FIG. 28 is a conceptual diagram showing an example of the possibility of word division of a character string in the fifth embodiment of the present invention;

FIG. 29 is a block diagram showing a configuration of a word segmentation device according to a sixth embodiment of the present invention;

FIG. 30 is a flowchart showing the operation of the word segmentation apparatus according to the sixth embodiment of the present invention;

FIG. 31 is a flowchart showing the operation of kanji word extraction processing by kanji character string extraction according to the sixth embodiment of the present invention;

FIG. 32 is an example sentence for explaining an operation example of kanji word extraction processing by kanji character string extraction according to the sixth embodiment of the present invention;

FIG. 33 is step 3 in the sixth embodiment of the present invention.
FIG. 7 is a diagram showing an example of a table of kanji character strings extracted by 101;

FIG. 34 is step 3 in the sixth embodiment of the present invention.
FIG. 9 is a diagram showing an example of a table of two-character kanji character strings sorted and sorted by 103;

FIG. 35: Step 3 in the sixth embodiment of the present invention.
FIG. 10 is a diagram showing an example of a table of three-character kanji character strings sorted and sorted by 103;

FIG. 36 is a conceptual diagram illustrating another method of realizing a kanji character string table according to the sixth embodiment of the present invention;

FIG. 37 is a block diagram showing a configuration of a word segmentation device according to a seventh embodiment of the present invention;

FIG. 38 is a flowchart showing the operation of the word segmentation apparatus according to the seventh embodiment of the present invention;

FIG. 39 is a flowchart showing the operation of hiragana word extraction processing by generating a degenerated character string in the seventh embodiment of the present invention;

FIG. 40 is a diagram showing an example of a character string conversion table according to the seventh embodiment of the present invention;

FIG. 41 is a diagram showing an example of generating a degenerated character string according to the seventh embodiment of the present invention;

FIG. 42 is a diagram showing an example of hiragana character string extraction according to the seventh embodiment of the present invention;

FIG. 43 is a block diagram showing a configuration of an information search device according to an eighth embodiment of the present invention;

FIG. 44 is a view showing the operation of the information search device according to the eighth embodiment of the present invention;

FIG. 45 is a diagram illustrating an index of the information search device according to the eighth embodiment of the present invention.

[Explanation of symbols]

 101 text input means 102 text storage means 103 word extraction means 104 word storage means 105 word discovery means 106 word division means 107 word processing knowledge storage means 108 text output means 109 word count determination means 110 numerical expression discovery means 111 word division possibility Calculation means 112 Kanji word extraction means 113 Katakana word extraction means 114 Hiragana word extraction means 115 Word merging possibility calculation means 116 Partial character string extraction means 117 Text processing buffer 118 Character string classification and alignment means 119 Character string conversion table 120 Degenerated character strings Creation means 201 Index creation means 202 Index storage means 203 Search means 204 Search target text storage means 205 Word division correction means 206 Word comparison means 207 Extracted word storage means

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Chuichi Kikuchi 1006 Kadoma Kadoma, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. Term (reference) 5B075 ND03 NS10 QS01 5B091 AA15 CA02

Claims (24)

[Claims] 1. A word dividing apparatus for dividing text data into words, a text storage means for storing natural language text data, and a word processing knowledge storage means for storing pattern information for word extraction and word division. A word extracting unit that extracts a word from text data stored in a text storage unit by applying the pattern information; a word storage unit that stores an extracted word; and text data using the extracted word. And a new word division position obtained by applying word division pattern information based on the position of the word on the text data that has been found so far and the position of the word. And a word dividing means for dividing text data into words. 2. A word dividing apparatus for dividing text data into words, comprising: a text storage means for storing natural language text data; and a word processing knowledge storage means for storing pattern information for performing word extraction and word division. Word extraction means for repeatedly applying the pattern information to text data stored in text storage means to repeatedly extract words from the text data; word storage means for storing the extracted words; Each time a word is extracted, a word finding means for finding each position of the text data in which the extracted word is used, and text data is found based on the position of the word on the text data found by the word finding means. A word dividing device, comprising: word dividing means for dividing into words. 3. The apparatus according to claim 1, further comprising a word number determining unit that manages the number of words to be extracted by the word extracting unit. 3. The method according to claim 2, wherein the extraction is repeated.
A word segmentation device according to item 1. 4. A word dividing apparatus for dividing text data into words, a text storage means for storing natural language text data, and a word processing knowledge storage means for storing pattern information for performing word extraction and word division. Word extracting means for extracting words from the text data by applying the pattern information to the text data stored in the text storing means; word storing means for storing the extracted words; Word finding means for finding each position of the text data, division possibility calculating means for quantitatively calculating the division possibility at the position found by the word finding means, and a calculation result of the division possibility calculating means And a word dividing means for dividing the text data into words based on the term. 5. The word segmenting apparatus according to claim 4, further comprising numerical expression finding means for specifying a numerical expression from text data stored in the text storage means. 6. The word segmenting apparatus according to claim 4, wherein the word extracting unit extracts a word of a specific character type from the text data. 7. The word according to claim 6, wherein the word extracting means comprises a kanji word extracting means for extracting a kanji word, an extracting means for extracting a katakana word, and an extracting means for extracting a hiragana word. Splitting device. 8. A combination possibility calculation means for quantitatively calculating the combination possibility of words of different character types extracted by the word extraction means, wherein a combination of different character types having a high possibility of combination is regarded as one word. The word segmenting device according to claim 7, wherein the word segmenting device is used. 9. A word dividing apparatus for dividing text data into words, a text storage means for storing natural language text data, and a word processing knowledge storage means for storing pattern information for performing word extraction and word division. A partial character string extracting means for extracting a character string containing a specific character type from text data stored in a text storage means; and a classification for classifying and sorting the extracted character string by the number of characters by focusing on the number of characters of the character type. Sorting means; word extracting means for applying the pattern information to the character strings arranged according to the number of characters to extract words of the character type; word storage means for storing the extracted words; Word-finding means for finding each position in text data in which is used, and division at the position found by the word-finding means A dividing potential calculation means for quantitatively calculating the word dividing device, characterized in that it comprises a word dividing means for dividing the word text data calculated based on the result of the division potential calculation means. 10. The word segmenting apparatus according to claim 9, wherein said partial character string extracting means extracts a character string including a kanji, and said word extracting means extracts a kanji word. 11. A reduced character string creating means for creating a character string of the text data in which a character type other than hiragana is replaced with a symbol, and the partial character string extracting means is created by the reduced character string creating means. 10. The word segmenting apparatus according to claim 9, wherein a character string including hiragana is extracted from the character string, and the word extracting means extracts a word of hiragana. 12. A word division method for dividing text data into words, wherein a word is extracted from the text data by applying pattern information for word extraction, and each position of the text data in which the extracted word is used is determined. Characterizing the text data into words at the position of the word on the found text data and the new word division position obtained by applying the word division pattern information based on the position of the word. How to split words. 13. A word division method for dividing text data into words, wherein a word is extracted from the text data by applying pattern information for word extraction, and each position of the text data in which the extracted word is used. After extracting the new word by applying pattern information based on the found word position, repeating this procedure, dividing the text data at the position of the word on the found text data Characteristic word segmentation method. 14. The word division method according to claim 13, wherein the repetition is repeated until a new word is not extracted. 15. A word division method for dividing text data into words, wherein a word is extracted from the text data by applying pattern information for word extraction, and each position of the text data in which the extracted word is used. A word division method comprising: calculating a division possibility at a found position; and quantitatively calculating a division possibility at the found position; and determining a position of text data to be divided into words based on the calculated value. 16. A word dividing method for dividing text data into words, wherein a numeral is found from the text data, a numerical expression is identified from character strings before and after the numeral, and a position in the text data where the numerical expression appears is represented by a word. A word division method characterized by a division reference position. 17. A word dividing method for dividing text data into words, wherein a character string including a specific character type is extracted from the text data, and the extracted character string is classified and sorted according to the number of characters by focusing on the number of characters of the character type. Applying pattern information for word extraction to each character string arranged according to the number of characters to extract a word of the character type, and finding each position of text data in which the extracted word is used. Characteristic word segmentation method. 18. The word dividing method according to claim 17, wherein a character string including a kanji is extracted from the text data to extract a kanji word. 19. A character string in which a character type other than hiragana in the text data is replaced with a symbol is created, a character string including hiragana is extracted from the character string, and the extracted character string is focused on the number of characters of the hiragana 18. The word division according to claim 17, wherein the words of the hiragana are extracted by classifying and arranging the characters by the number of characters, and applying pattern information for word extraction to each of the character strings arranged by the number of characters. Method. 20. A word division method for subdividing text data divided into words into words by using an updated word group, comprising: a word group used for word division of the text data; and the updated word group. , The text data including the character string of the difference word is searched, the word division of the searched text data is released, and the text data before word division is reproduced, and the reproduced text data is retrieved. Is divided using the updated word group. 21. An information retrieval apparatus for searching a text divided into words as a search target, wherein a search target text storage means for storing text data of the search target text divided into words, and an index of the search target text are created. Index creation means; index storage means for storing the created index; search means for searching for a search target text using the index; and extracted word storage means for storing words used for word division of the search target text. A word segmentation device that divides text data into words, and a word group used for word segmentation of new text data in the word segmentation device is compared with a word group saved in the extracted word storage unit. Word comparison means for detecting a difference word, and controlling word subdivision of the search target text Word division correction means, wherein the word division correction means controls word subdivision to be performed on the search target text including the character string of the difference word through the word division device. Search device. 22. The word division device according to claim 1, wherein
22. The information search device according to claim 21, comprising the word segmentation device according to claim 1. 23. The information search apparatus according to claim 21, wherein the search means searches for a search target text including the character string of the difference word using the index. 24. The word division correction unit compares text data word-subdivided by the word division device with data of a search target text before word subdivision, and when there is a difference between the two, When the data of the search target text stored in the search target text storage unit is updated with the word subdivided text data, and the index creation unit updates the search target text in the search target text storage unit 22. The information search device according to claim 21, wherein an index for word search of the search target text is re-created.