JPH09138801A - Character string extracting method and its system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To utilize extracted words for the syntax analyzation, etc., of the text by enabling the generation of a dictionary optimum for the text if registering the words as a dictionary. SOLUTION: The optimum consecutive character string is extracted 3 from the text 1 described in natural language and concerning a character adjacent to the consecutive character string, an appearing frequency appearing at the same time of the consecutive character string is investigated 4. Whether the character is provided or not with the high probability of being used integrally with the consecutive character string is objectively evaluated by means of a numerical value corresponding to this appearing frequency. When the frequency is high, the character string is recognized to be one group of words and phrases including the adjacent character. Words extracted in this way are registered as the dictionary to utilize for the syntax analyzation, etc., of the text.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a character string extracting method and system for automatically extracting a continuous character string such as a word or an idiom used in a natural language processing system for carrying out machine translation, information retrieval or the like.

[0002]

2. Description of the Related Art In a machine translation system that automatically translates a data-formatted document that can be used by an information processing device, or a system that searches the entire document and searches for documents using the corresponding keyword, A process of extracting a character string having a certain meaning existing in the document is required. Regarding the technical terms and other academic terms, etc., this character string may be widely and commonly used as a dictionary. However, in general literature, terms used in documents are not always the same, and a dictionary prepared in advance is often insufficient. Therefore, a technology has been developed to directly extract a character string having a predetermined meaning from a document to be processed ([Reference] IPSJ Research Report Vol.93, No.61 (93-NL-96- 1)). Here, a technique for extracting a continuous character string from a text described in natural language based on the length and appearance frequency of the character string is introduced.

[0003]

By the way, the above conventional system has the following problems to be solved. For example, consider a case where the character string “address” appears 100 times and the character string “dress” appears once in the target text. At this time, if the character string "address" is simply searched and the number of appearances is counted, it becomes 100 times. On the other hand, when the character string “dress” is counted, the number of appearances is 101 times. In such a case, it may be erroneous to judge whether "dress" should be recognized as a meaningful character string in the text or only by the appearance frequency. That is, by using such a technique for extracting continuous character strings, a meaningful group of character strings is actually selected, and by considering the strength of the group, a dictionary is created with these, and the text Analysis, translation processing, and various other processing become easier.

[0004]

The present invention employs the following structure to solve the above problems. <Structure 1> An arbitrary continuous character string is extracted from a text written in natural language, and the appearance frequency of a character adjacent to the continuous character string that appears at the same time as the continuous character string is compared with a preset threshold value. Characters having a higher appearance frequency than this threshold are used as a unit with the continuous character string, and are selected as a character string to be collectively recognized.

<Explanation> The text described in natural language may have no word separation as in Japanese.
The arbitrary character string may be any part of the text or may be composed of any number of characters. The character adjacent to the continuous character string may be the preceding character, the following character, or one or both of them. By investigating the frequency of appearing at the same time as the continuous character string, it is possible to objectively evaluate whether or not the character has a high probability of being used together with the continuous character string by a numerical value.
If the frequency is high, the character string including adjacent characters can be recognized as a group of words or idioms. The words thus extracted are registered as a dictionary and used for text parsing and the like. The threshold is empirically selected. For example, in the case of extracting only continuous character strings with a very strong connection, the threshold value may be set high, or various words may be extracted with a grade from a strong connection to a weak connection.

<Structure 2> An arbitrary continuous character string is extracted from a text written in natural language, and the appearance frequency of the character immediately before and after the continuous character string, which appears simultaneously with the continuous character string, is calculated. When a character having a higher appearance frequency than this threshold does not exist as compared with a preset threshold, the continuous character string is collectively selected as a character string to be recognized in the text.

<Explanation> The appearance frequency of the character immediately before the continuous character string can be expressed as a forward dispersion value, and the appearance frequency of the character immediately after it can be expressed as a backward dispersion value. In addition to this, the appearance frequency can be statistically calculated using a standard deviation or the like. The character string to be recognized collectively in the text is a continuous character string having a high appearance frequency in the text. This can constitute the dictionary that is best suited for parsing that text. Therefore, the dictionary can be optimized for any text without preparing the dictionary in advance.

<Structure 3> A continuous character string extraction unit for extracting an arbitrary continuous character string from a text written in natural language, and an appearance frequency of a character adjacent to the continuous character string, which appears at the same time as the continuous character string, An appearance frequency calculation unit that calculates,
This appearance frequency is compared with a preset threshold value, and a character whose appearance frequency is higher than this threshold value is used as a unit with a continuous character string, and an appearance frequency comparison unit that collectively selects the character string to be recognized. And a storage device that registers and stores the selected continuous character string.

<Explanation> Text input may be performed by using a keyboard, a floppy disk, or a transfer from another information processing apparatus. The number of thresholds may be one or plural. The continuous character string may be registered by directly inputting the extracted character string, or may be temporarily stored in a floppy disk or the like and then registered in a storage device of another information processing device.

[0010]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to specific examples. <Specific Example 1> FIG. 1 is a functional block diagram of a system according to the present invention. The system of the present invention is realized by the functional blocks as schematically shown in this figure. That is, this system includes a text input unit 2 for inputting a natural language text 1, a continuous character string extraction unit 3, an appearance frequency calculation unit 4, an appearance frequency comparison unit 5, a storage device 6, and the like. Text 1 in natural language from the text input section 2
Is input in a digitized state. Continuous character string extraction unit 3
Extracts consecutive character strings from this. Although the extraction method and the like will be described later, the appearance frequency calculation unit 4 calculates the appearance frequency of characters adjacent to the front and rear of this character string. Then, the appearance frequency comparison unit 5 compares it with a predetermined threshold value 7. For example, if you count the number of times a character that appears immediately before a certain continuous character string appears at the same time as that continuous character string, and then calculate the ratio with the number of times that that continuous character string appears,
The connection between the previous character and the continuous character string becomes clear.

The strength of the connection with the immediately preceding character is digitized in this way, and when the connection is strong, it is determined that the continuous character string and the immediately preceding character string are used together. By doing this, it is possible to determine how far the continuous character strings are used together. The character just before and the character immediately after are judged in this way, and a character string as a whole is extracted from the text. In this way, a continuous character string such as an important word or idiom to be recognized collectively can be extracted from the text, and a dictionary using this can be generated and used.

The method of the present invention will be described in more detail below. FIG. 2 is a block diagram embodying the system of the present invention. The system shown in the figure includes a storage device 6, an input / output device 11, and a processing device 14. Here, the storage device 6
Is composed of hardware having a function of storing the input text and the processing result of each stage. I / O device 11
Is composed of a keyboard and a display having functions of inputting text and displaying extraction results. Processor 14
Has a function of executing various processes for extracting a continuous character string. This is composed of a workstation and the like. The storage device 6 includes an input text storage unit 21 that stores the input text, a buffer 22 that stores the processing result of each stage, and an extracted character string storage table 23 that stores information on the extracted character string. The buffer 22 has a buffer X, a buffer Y, a buffer A, a buffer B, and a buffer C. All of these are set in an appropriate storage area on the hard disk or main storage device.

FIG. 3 is an explanatory diagram of an example of the extracted character string storage table. The extracted character string storage table 23 has an extracted character string storage unit 25, an appearance count storage unit 26, a front variance value storage unit 27, and a rear variance value storage unit 28. Also,
The input / output device 11 shown in FIG. 2 includes an input unit 12 and an output unit 1.
3 Here, the input unit 12 has a function of inputting text. The input unit 2 may be composed of, for example, a keyboard, or may be a device that accesses a text file stored in a storage device of a computer. The output unit 13 has a function of displaying the extraction result. The output unit 13 is composed of, for example, a display and a printer.

The processing unit 14 includes a character string cutout unit 15,
Sort processing unit 16, character string extraction unit 17, variance value calculation unit 1
8 The character string cutout unit 15 has a function of cutting out and generating an arbitrary character string from the input text. The sort processing unit 16 has a function of sorting a character string by an arbitrary key and storing it in a buffer. The character string extraction unit 17 has a function of comparing and collating two character strings, counting the number of matching characters from the beginning, and storing the partial character string, the subsequent character, and the start address of the character string in a buffer. . The distributed value calculation unit 18 has a function of calculating the degree of dispersion of characters that follow the extracted character string and storing the calculated degree in the extracted character string storage table 23.

The system configured as described above operates as follows. FIG. 4 is a flowchart of a character string extraction process for showing the process of the present invention. First, a text input process is executed (step S1). The input text is stored in the input text storage unit 21 of FIG.
This process is executed using the input unit 12. Next, the input text is read from the input text storage unit 21 (step S2), and a process of cutting out a character string starting from each character and ending at an arbitrary end is performed (step S2).
3). This processing is executed by the character string cutout unit 15. For example, if a character string starting from the first character to the n-th character and ending from the n-th character is cut out of a text composed of n words, this process cuts out n character strings. become.

Then, the cut out character strings are sorted by an arbitrary key (step S4) and stored in the buffer X (step S5). Dictionary order means alphabetical order or alphabetical order. In this specific example, the sorting key is a dictionary. This processing is executed by the sort processing unit 16. Next, each character string in the buffer X is compared with the character strings located immediately before and after each character string in the text, and the types and number of matching character strings and the characters following each extracted character string are compared. A process of storing the type and the number in the buffer Y is executed (step S6). This processing is executed by the character string extraction unit 17. next,
A process of calculating the degree of dispersion of the characters following the extracted character string is performed (step S7). This is calculated based on the number of appearances of the extracted character string, the type of the following character, and the number of appearances. The variance value is the minimum when the number of appearances is m and the succeeding character is always one type and appears m times.
Set the type so that it becomes the maximum when it is once each. This processing is based on the result of step S6, and the variance value calculation unit 1
8 is executed.

From this result, as shown in FIG. 3, the extracted character string is stored in the extracted character string storage unit 25, the number of appearances of the character string is stored in the appearance count storage unit 26, and the variance value is stored in the backward distributed value storage unit 28. Store (step S8). When this process ends, the characters of the text in the input text storage unit 21 are rearranged in reverse order (step S10), and the processes of steps S2 to S9 are executed again. The variance value of the subsequent character obtained as a result of this processing is stored in the forward variance value storage unit 27 of FIG. 3 (step S8). Since the character string itself extracted by the processing from the reverse order and the number of appearances thereof are the same as the processing result from the front, it is not necessary to store them again. The above is the processing for extracting the character string.

Next, the contents of the matching character string extraction processing in step 6 shown in FIG. 4 will be described in more detail. FIG. 5 is a flowchart showing this processing. The character string extraction processing unit includes a process of obtaining the maximum number of matching characters, a process of extracting a matching character string, and a process of totaling and storing the processing results in a buffer. In the process of obtaining the maximum number of matching characters, first, each character string in the buffer X is read (step S
1) The other character string stored in the same buffer X is compared with the character string stored immediately before, and the number of matching characters is stored in the buffer A (step S2). Next, the number of matching characters is stored in the buffer B by comparing with the character string stored immediately after that (step S3). Here, the buffer A and the buffer B are compared (step S4), and the larger character number is set as the maximum matching character number (steps S5 and S6).

Next, in the process of extracting the matching character string, 1 is set to the parameter n representing the number of characters (step S
7), n is compared from the maximum number of matching characters (step S
8) If it is less than that, the first character to the nth character and the (n + 1) th character of the character string are stored in the buffer C (step S).
9, S10). Next, n is incremented by 1 (step S11), and the process returns to step S8. This process is
Continue until is the maximum number of matching characters. When n becomes larger than the maximum number of matching characters, the same process is repeated for the next character string. When the contents of both the buffer A and the buffer B are 0, there is no matching character string in the text, so the character string is not extracted. Finally, the extracted character string, the number of appearances thereof, and the number of appearances of each succeeding character string are totalized from the contents of the buffer C (steps S12 and 13) and stored in the buffer Y (step S14).

Next, the processing steps of the present invention will be specifically described by using an actual case. First, it is assumed that the text as shown in FIG. 6 is input. Line feed is "CR" for convenience
It is represented by FIG. 7 shows the correspondence between each character of the input text and the character number. In the following, in order to clarify which character string is pointed out, the extracted character string is described in the form of "character string (character number)" with the character number of the start position. The text is input using the input unit 12 (step S1 in FIG. 4). The input text is sequentially cut out into a character string from the first word to an arbitrary end (step S3 in FIG. 4). Here, a line feed "CR" is set at an arbitrary end. In the example of FIG. 6, first, the character string from the first character “stock (1)” to “CR (5)” is cut out. Next, the character strings from "expression (2)" to "CR (5)" are cut out. This process is performed by adding "((3
2) ”to“ CR (33) ”are repeated.

FIG. 8 shows a character string cut out as a result of this processing. The character number is the number of the first character. Next, the cut-out character strings are sorted in the order of the dictionary and stored in the buffer X (step S5 in FIG. 4). FIG. 9 shows the contents of the buffer X after processing. Next, a process of extracting a matching character string is performed (step S6 in FIG. 4). Here, first, each character string of the buffer X and the buffer X
A process of comparing the character strings immediately before and immediately after stored together in the character string, counting the number of matching characters from the beginning of the character string, and obtaining the maximum number of matching characters is executed (step S1 in FIG. 5).
~ S4). For example, in the case of “market integration (29)” in FIG. 9, the character that matches the immediately preceding character string “market (23)” is “market”, the number of matching characters is 2, and the character string immediately after that is “citizen (8)”. Is a city and the number of matching characters is 1
Therefore, the larger number of characters, 2 is the maximum number of matching characters.

Next, a process of extracting a partial character string of the matching character string is performed (steps S5 to S11). First, n is set to 1 (step S7 in FIG. 5). Where n (= 1)
Is less than or equal to the maximum number of matching characters (= 2), so 1
The characters from the first character to the n (= 1) th character, that is, the “city” of the first character and the “field” of the n + 1 (= 2) th character are stored in the buffer C (steps S9 and S10 in FIG. 5). Then n is 1
One counts up to 2 (step S1 in FIG. 5).
1). Here again, n (= 2) is the maximum number of matching characters (= 2)
Since it becomes the following, the "market" from the first character to the n (= 2) th character of the character string and the "combination" of the n + 1 (= 3) th character are stored in the buffer C (steps S9 and S10 in FIG. 5). ).
Next, n is incremented by 1 to be 3 (step S11 in FIG. 5). At this time, since n (= 3) becomes larger than the maximum number of matching characters (= 2), the character string “market integration (2
9) ”, and the same process is performed on the next character string“ citizen (8) ”(step S2). This process is repeated until the last character string.

FIG. 10 shows the contents of the buffer C after this processing. Next, a process is performed in which the matching character string and the number of appearances thereof, the type of character following each character string, and each number of appearances are totaled and stored in the buffer 22 (step S1 in FIG. 5).
3, S14). FIG. 11 shows the contents of the buffer Y after this processing. Next, a process of calculating the degree of dispersion of the characters following the matching character string is performed (step S7 in FIG. 4). This processing is executed by the variance value calculation unit 18 using the result of step S14 in FIG. The degree of dispersion here is
We will use entropy to find the value. Entropy can be obtained by ^{_{Σ n i = 1 -Pi * ln}} Pi (Pi = i probability). For example, in the case of “city” in FIG.
The "city" appears 5 times, the "place" appears 3 times after the "city", and the "people" appears 2 times. At this time, the backward dispersion value of “city” is −3 / 5 * ln (3/5)
-2 / 5 * ln (2/5) ≠ 0.673012.

Next, the processing result is stored in the extracted character string storage table 23 (step S8 in FIG. 4). In the case of this example, “city” is stored in the extracted character string storage unit 25 of FIG. 3, “5” is stored in the appearance count storage unit 26, and “0.673012” is stored in the backward variance value storage unit 28. FIG. 12 shows the result of this processing. Note that "CR" is 2 after the character "Market".
There are times, but in this case "CR" represents the end, so
Treat as different characters.

When this processing is completed, the above-mentioned processing is executed again in reverse order from the end of the text. The variance value obtained as a result of processing from the reverse order is stored in the forward variance value storage unit 27 shown in FIG. 3 (step S8 in FIG. 4). FIG.
Shows an example of text in reverse order. Further, an example of the character string cutout process is shown in FIG. 14, an example of the buffer X after the sort process is shown in FIG. 15, an example of the buffer C after the character string extraction process is shown in FIG. 16, and a buffer Y after the character string extraction process is shown. FIG. 17 shows an example, and FIG. 18 shows an example of the extracted character string storage table after the distributed value calculation processing.

The forward variance value represents the strength at which the character string is cut off from the preceding character. The backward dispersion value represents the strength at which the character string is cut off from the character behind it. When the forward and backward variances are close to each other, such as "citizen" and "stock market", it indicates that the character strings tend to be a unit. Also, if the forward variance value is large and the backward variance value is small like "city", the character string tends to be followed by a specific character (string) like a prefix or modifier. Show. On the contrary, if the forward variance value is small and the backward variance value is large like “market”, it indicates that there is a tendency to accompany a specific character (string) before, like a prefix or noun. These tendencies become stronger as the variance value increases. The larger the input text, the more prominent it appears. By using this property, it is possible to order or extract the character strings in the extracted character string storage table according to the purpose by providing an appropriate threshold value.

As described above, according to this specific example, continuous character strings to be recognized collectively from the character strings appearing in the text are automatically and automatically independent of the length and frequency of the character string. Can be accurately extracted. For example, in a text, “A” is 100 before “Dress”.
When the character appears once, and the other characters appear once, the forward dispersion value of “dress” becomes a very small value. At this time,
If various characters appear before and after "address", the forward and backward dispersion values will be large, and even if the appearance frequency of "address" is less than "dress", extract "address". You can Since the character string extracted by this invention has a high appearance frequency and is an expression that does not depend on the structure of words, it can be used for a wide range of natural language processing, such as creating a technical term dictionary in machine translation and keyword extraction in information retrieval. It can be used in the field.

[Brief description of the drawings]

FIG. 1 is a functional block diagram of a system according to the present invention.

FIG. 2 is a block diagram embodying the system of the present invention.

FIG. 3 is an explanatory diagram of an example of an extracted character string storage table.

FIG. 4 is a flowchart of a character string extraction process.

FIG. 5 is a flowchart of matching character string extraction processing.

FIG. 6 is an explanatory diagram of an example of input text.

FIG. 7 is an explanatory diagram of correspondence between input text and character numbers.

FIG. 8 is an explanatory diagram of an example of character string cutout processing.

FIG. 9 is an explanatory diagram of contents of a buffer X after sorting processing.

FIG. 10 is an explanatory diagram of contents of the buffer C after the character string extraction processing.

FIG. 11 is an explanatory diagram of contents of a buffer Y after totalization.

FIG. 12 is an explanatory diagram of contents of an extracted character string storage table.

FIG. 13 is an explanatory diagram of correspondence between texts and character numbers rearranged in reverse order.

FIG. 14 is an explanatory diagram illustrating an example of character string cutout processing.

FIG. 15 is an explanatory diagram of contents of a buffer X after sorting processing.

FIG. 16 is an explanatory diagram of the contents of the buffer C after the character string extraction processing.

FIG. 17 is an explanatory diagram of contents of a buffer Y after totaling.

FIG. 18 is an explanatory diagram of contents of an extracted character string storage table.

[Explanation of symbols]

 1 Text 2 Text Input Section 3 Continuous Character String Extraction Section 4 Appearance Frequency Calculation Section 5 Appearance Frequency Comparison Section 6 Storage Device 7 Threshold

Claims (3)

[Claims] 1. An arbitrary continuous character string is extracted from text described in natural language, and the appearance frequency of characters adjacent to the continuous character string that appear at the same time as the continuous character string is set as a preset threshold value. In comparison, a character string extraction method characterized in that characters having a higher appearance frequency than this threshold value are selected as a character string to be collectively recognized as being used integrally with the continuous character string. 2. An arbitrary continuous character string is extracted from a text written in natural language, and the appearance frequency of the character immediately before and after the continuous character string that appears at the same time as the continuous character string is previously determined. Character string extraction characterized by selecting the continuous character string as a character string to be collectively recognized in the text when a character having a higher appearance frequency than the threshold value does not exist in comparison with the set threshold value Method. 3. A continuous character string extraction unit for extracting an arbitrary continuous character string from a text written in natural language, and for a character adjacent to the continuous character string, an appearance frequency that appears at the same time as the continuous character string. Characters to be collectively recognized as a character having a higher appearance frequency than this threshold is compared with the appearance frequency calculation unit that calculates the appearance frequency and a preset threshold value, as they are used integrally with the continuous character string. A character string extraction system comprising: an appearance frequency comparison unit for selecting a string; and a storage device for registering and storing the selected continuous character string.