JP2002197095A - Keyword extracting device and information retrieving device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract a keyword from a document without necessity of a dictionary. SOLUTION: A keyword extracting device includes a suffix file generating part 22 to receive a group of documents and to generate a suffix file to be described later from the group of documents, a suffix file storage part 24 to store the suffix file, a punctuating part 28 to receive an optional document to be included in the group of documents or a document in the same field as the group of documents and to punctuate the document at a break of a sentence such as punctuation marks, a score calculating part 26 to properly punctuate the sentence based on the suffix file and the sentence supplied from the punctuating part 28 and to calculate appearance frequency α, a degree β of concentration of appearance and weight, etc., to be described later, an operation result storage part 30 to store an operation result, a document separating part 32 to punctuate the document into candidates of the keyword based on the operation result and a narrowing part 34 to narrow down the candidates of the keyword.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a keyword extraction device and an information retrieval device, and more particularly to a keyword extraction device and an information retrieval device capable of extracting a keyword from a document group without requiring a dictionary.

[0002]

2. Description of the Related Art When arranging breaking news or news of the latest technical information, a keyword for specifying the content of an article is provided for search. If the task of assigning a keyword can be automated, the operation of a document to which no keyword has been assigned can be facilitated. The automatic keyword extraction that has been studied so far is performed by performing a morphological analysis using a dictionary, and then discriminating keywords based on part of speech information and frequency information.

[0003]

However, the method using a dictionary has a problem as information processing in the Internet age where new words are born every day. The reasons for this are that productivity must be low because keywords must be registered in the dictionary from the latest sentence that requires automated processing, and that there is no versatility for completely unknown terms that are not registered in the dictionary. Can be

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problem, and an object of the present invention is to provide a keyword extracting device capable of extracting a keyword from a document without requiring a dictionary.

It is another object of the present invention to provide an information retrieval apparatus capable of extracting a document related to a certain document from a group of documents without requiring a dictionary.

[0006]

According to an aspect of the present invention, there is provided a keyword extracting apparatus comprising: an appearance frequency calculating unit for obtaining an appearance frequency of a partial character string included in each document in the document group; A document input device that is connected to an appearance concentration calculation unit for calculating the occurrence concentration degree of the character string in the document group, and an appearance frequency calculation unit and an appearance concentration degree calculation unit, and based on the appearance frequency and the appearance concentration degree, And first keyword extracting means for extracting a keyword from the keyword.

A keyword is extracted based on the appearance frequency and appearance concentration of the partial character string. Therefore, keywords can be extracted from the document without requiring a dictionary.

Preferably, the first keyword extracting means includes: a document dividing means for dividing the input document into partial character strings;
A word likeness calculating means connected to the appearance frequency calculating means and the appearance concentration degree calculating means, for calculating the word likeness of the partial character string based on the appearance frequency and the appearance concentration degree;
A second keyword extracting unit connected to the word-likeness calculating unit for extracting a keyword from the document based on the total value of the word-likeness.

More preferably, the word-likeness calculating means is connected to the document dividing means, the appearance frequency calculating means, and the appearance concentration degree calculating means.
Means for calculating wordiness of the partial character string based on the length of the partial character string and the average size of the document is included.

[0010] As the average size of a document decreases, the degree of appearance concentration tends to approach zero. For this reason,
By changing the method of calculating the word-likeness at the average size of the document, it is possible to calculate the appropriate word-likeness even if the appearance frequency is reduced.

[0011] More preferably, the keyword extracting device is further connected to a second keyword extracting means, an appearance frequency calculating means, and an appearance concentration degree calculating means, and outputs the keyword extracted by the second keyword extracting means. ,
A narrowing-down unit for narrowing down based on the appearance concentration degree and the length of the partial character string is included.

[0012] More preferably, the keyword extracting apparatus further includes a separating unit for separating the input document by punctuation marks and supplying the document to the document dividing unit.

[0013] More preferably, the document dividing means includes a means for dividing the input document into partial character strings so that the first character does not start with a predetermined character.

[0014] More preferably, the document dividing means includes means for dividing the input document into partial character strings such that the length of the partial character string does not exceed a predetermined number of characters.

[0015] An information retrieval apparatus according to another aspect of the present invention comprises:
Appearance frequency calculation means for calculating the appearance frequency of the partial character string included in each document in the document group, and occurrence concentration calculation means for obtaining the appearance concentration degree of the partial character string in the document group And an appearance frequency calculation unit and an appearance concentration degree calculation unit,
Keyword extraction means for extracting a keyword from the input document based on the frequency of appearance and the degree of occurrence concentration, and each keyword extracted by the keyword extraction means for each document in the document group connected to the keyword extraction means A degree-of-match calculating means for calculating the degree of coincidence with, and a degree-of-similarity for each document in the document group based on the degree of similarity with the input document based on the degree of matching. A similarity calculating unit; and a unit connected to the similarity calculating unit, for extracting a document related to the input document from the document group based on the similarity.

A keyword is extracted from the input document without using a dictionary, and a document related to the keyword is extracted. Therefore, it is possible to extract a document related to the input document without requiring a dictionary.

[0017]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [Embodiment 1] Referring to FIG. 1, a keyword extracting apparatus according to an embodiment of the present invention
A suffix file creation unit 22 that receives a document group and creates a suffix file described later from the document group, and a suffix file storage unit 24 that is connected to the suffix file creation unit 22 and stores the suffix file created by the suffix file creation unit 22 And a document included in the document group or a document in the same field as the document group, and separates the document at a break of a sentence such as “,” or “.”, A suffix file storage unit 24 and a separation unit 28, based on the suffix file stored in the suffix file storage unit 24 and the text supplied from the delimiter unit 28, the text is appropriately divided, and calculation of an appearance frequency α, an appearance concentration β, and a weight, which will be described later, is performed. line The score calculation unit 26, a calculation result storage unit 30 connected to the score calculation unit 26 and storing the calculation result of the score calculation unit 26, and a calculation result storage unit 30 connected to the calculation result storage unit 30 and stored in the calculation result storage unit 30 A document dividing unit 32 that divides the document into keyword candidates based on the calculated result, and a document dividing unit 3
2 and a narrowing-down unit 34 for narrowing down keyword candidates and extracting keywords.

[Summary of the Present Invention] The present invention is characterized in that a statistic indicating the degree of appearance concentration is used in addition to the appearance frequency of a character string. In the keyword extraction according to the present invention, it is necessary to calculate the appearance concentration degree for all partial character strings. we,
A technique using a "suffix file" for determining the degree of occurrence concentration of character strings in many documents has already been established and is used.

[Principle of keyword extraction] The appearance concentration is ad
A statistic known as aptation, where "a word is 1
Probability that the same word appears again in the document under the condition that it appears in one document: P (two occurrences | one occurrence). " In order to estimate this probability, regarding the target character string x, “the number of documents including the character string x: df (x)” and “the number of documents including the character string x twice or more: df2 (x)” ". Then, the above-mentioned probability is estimated from the following equation (1) in consideration of Bayes rule. Here, N is the total number of documents.

[0020]

(Equation 1)

The arguments x of df (x) and df2 (x)
Can take all substrings. In a simple way, when dealing with large texts, either the amount of memory or the amount of computation is so large that it is impractical.

Therefore, a partial character string is extracted using a data structure known as a suffix file described later. The suffix file requires five times the memory space of the text, but the position of all substrings is l
og (n) (n: text size) can be specified by the amount of calculation.

For details on how to create and use suffix files, see "M. Yamamoto and KWChurch, Using Suffi.
x Arrays to Compute Term Frequency and Document Fr
equency for All Substrings in a Corpus, In proceed
ing of 6th Workshop on Very Large Corpora, Ed.Eug
ene Charniak, Motreal, pp. 28-37, 1998 ".

By using a suffix file, the number of times a certain character string appears in a document database can be obtained at high speed. The suffix file is realized by rearranging a character string of a part that can occur in all documents in the order of character codes and adding a serial number (suffix). The number of times tf that a character string appears in the document database can be obtained by calculating the number of character strings that match the character string in the suffix file.

Specifically, first, a minimum value min and a maximum value max of a suffix in which a character string matching a certain character string appears are obtained by a binary search method. If there is no matching character string, the number of times the character string appears in the document database is zero. If the minimum value min and the maximum value max of the suffix are obtained, the number of times tf that the character string appears can be obtained as tf = max−min + 1.

The documents in the document database are distinguished from each other by a document number, and the partial character string registered in the suffix file is given this document number. Thus, a document including a certain partial character string can be efficiently searched. In addition, the number of documents df including a certain partial character string is calculated by counting the number of duplicate document numbers,
It can be calculated by subtracting that number from tf. At this time, the number of documents including the partial character string twice or more is df2.

Here, df (x) / N, which is the estimated value (appearance frequency) of the appearance probability of the character string x, is α, and the estimated value of df2 (x) / df (x) for estimating the adaptation (appearance concentration) Degree) is β. Assuming that the occurrence of the character string follows a Poisson distribution, α and β have the same value. It can be observed that the value of β is large in an actual corpus, and the difference is particularly large in a character string recognized as a keyword compared to α.

As the appearance and concentration, here, an example of a partial character string x of the word “about a robot” and corresponding df, df2, α (= df / N) and β (= d
f2 / df) is shown in FIG. There are two things we can observe here. One is the character strings that make up the keyword.
This means that β is larger than α. This means
The word used as a keyword indicates that it often appears more than once in a document. This can also be easily confirmed in FIG. The other is that β becomes smaller beyond word boundaries. This is derived from the fact that certain words always appear as the same word, but the characters that follow can vary. This can be confirmed from the fact that, when the partial character string x changes from "robot" to "robot" in FIG.

The word is divided by estimating the likelihood (weight) of the character string based on the value of β. Weight (sc
ore) are classified as shown in FIG. here,
Character string x with too small df2, that is, df2 is 3
In the following cases (NO in S2), the weight is defined as low without being regarded as a word (S4). When the word is recognized as a word, that is, when df2 is greater than 3 and tf is equal to or less than N (YES in S2 and NO in S6), the value of β is estimated, and its logarithmic value is used as a weight (S8). However, the total number of appearances tf
Where N exceeds N, the wordiness is not correctly reflected in β. This means that particles, etc., do not become words even though they appear many times in sentences.
Therefore, it is assumed that the word-likeness is saturated where the condition of tf> N is satisfied (YES in S6), and the logarithm of the constant 0.5 is used as the weight (S10). For all string splits,
As described above, the weight is obtained, and the division in which the total sum of the weights of the respective character strings is maximized is obtained.

Here, several thresholds were used to determine the wordiness. This is an operation that can be performed on new documents, because unlike a dictionary, which must cover all words, it can obtain values from sample words. For example, the value of β is affected by the length of the document, and can be handled by determining this threshold.

FIGS. 4 and 5 show examples of division using this method and examples of division using morphological analysis. Although it is unnatural to divide particles and auxiliary verbs, it can be observed that keywords are accurately extracted.

[Selection of Keyword Candidates] When the appearance frequency α of the partial character string x is large, it indicates that the character string x is a character string that appears in a very large number of documents. Therefore, such a character string x has a low ability to identify a document. Conversely, when α is small, for example, a character string that appears only once is considered to be a very special character string that is rarely used. Such a string would not have the ability to indicate relevance to other documents. Therefore, a character string desirable as a keyword is a character string in which α is included within a certain range due to the nature of a word that can specify a document. Furthermore, to estimate whether a word is related to the content of the document,
The weight determined in consideration of the value of is used again. With respect to the range of the values of α and β, learning is performed using a corpus, and candidates for the range to be extracted are selected.

More specifically, keywords satisfying all the conditions shown in FIG. 6 are considered as keywords. That is, the value of α of the partial character string x is greater than 0.00005 and 0.1
If the weight of the partial character string x is greater than -1.0 and the length of the partial character string x is greater than 1 (YES in S12, YES in S14 and YE in S18)
S), and determine that the partial character string x is a keyword (S)
20). Otherwise, it is determined that it is not a keyword (S16). Note that len [x] is the length of the character string x. The values used here are not limited to these. For example, it can change depending on the target document group, and can also change from the viewpoint of adjusting the number of keywords to be extracted.

[Process of Suffix File Creation Unit 22] Referring to FIG.
Will be described. The suffix file creating unit 22 creates one long character string by connecting a plurality of documents forming a document group (S22).
For example, as shown in FIG.
bcabd ”.

A partial character string that can occur while shifting the start position one character at a time from the created character string is created. A serial number (suffix) is assigned to each partial character string (S24). When a partial character string and a suffix are created from the character string “abcabd”, the result is as shown in FIG.

A suffix file is created by rearranging the partial character strings in dictionary order (S26). A sequence of suffixes in a suffix file is called a suffix array. That is, a suffix file as shown in FIG. 8C is created. The created suffix file is stored in the suffix file storage unit 24.
Is stored. By using this suffix file, the appearance frequency and appearance concentration of all character strings in the original document group can be obtained with a small amount of calculation.

[Processing of Separator 28] Referring to FIG.
The processing of the separation unit 28 will be described.

The separator 28 opens the document (S
32), a buffer (not shown) prepared for temporarily storing the character string is cleared (S34). A character string is read from the document (S36). The character reading in S36 is performed from the first character of the document.
The next character is sequentially read every time the processing of step 6 is performed.

It is determined whether the read character is EOF (End Of File) (S38). The read character is EO
If it is F (YES in S38), the sentence stored in the buffer is output from the delimiter 28 to the score calculator 26 (S40), and the process ends.

If the read character is not EOF (S38)
NO), it is determined whether the read character is a delimiter such as "." Or "," (S42). If the read character is not a delimiter (NO in S42), the read one character is added to the buffer (S44). Then, S
Return to 36.

If the read character is a delimiter (YES in S42), the sentence stored in the buffer is output from the delimiter 28 to the score calculator 26 (S4).
6), clear the buffer (S48). Then, S3
Return to 6.

[Process of Score Calculation Unit 26] The process of the score calculation unit 26 will be described with reference to FIG. 10 and FIG.

Referring to FIG. 11, score calculation unit 26
The length of the input character string X input from the delimiter 28 is set to le
An array value having n_X and len_X elements
e, table and from are created (S52).
Here, table [i] indicates the optimum value (maximum value) of the weight from the first character to the i-th character. value [i] indicates the optimum value (maximum value) of the weight from the first character to the i-th character. “from [i]” indicates the start position of the segment where the weight at the i-th character is optimal (maximum).

The sequences value, table and fr
The element of m is initialized (S54). That is, the array ta
ble_min_core × len_
Initialize to X. Here, min_score is a predetermined constant, and here, min_score =
-10000. Also, all the elements of the array value are initialized to 0. Further, the k-th element is initialized to (k-1) for all elements of the array from.

Next, the counter i representing the character of interest in the input character string is set to 1 (S56). That is, the value of the counter i is set so as to indicate the head of the input character string.

Table [i] = min_score ×
If len_X, table [i] = 0 is set, otherwise, nothing is performed (S58).

The value of the counter j is set to i + 1 (S6
0). The character string from the i-th character to the j-th character of the character string X is x (S62). It is determined whether the first character of the character string x is "-" or "" (blank) (S64). If the first character of the character string x is not "-" or "" (S
NO at 64), tf, df and df2 of the character string x are calculated (S66). It is determined whether or not df is 1 or more, that is, whether or not the character string x has appeared in the document group (S68). Note that tf indicates the number of times the character string x appears in all documents.

If the character string x appears in the document group (YES in S68), the value of df2 becomes min_df2
It is determined whether it is greater than (S70). min_
df2 is a predetermined constant, and is set to 3 here.

If df2 is greater than min_df2 (YES in S70), it is determined whether tf is greater than the total number of documents N (S72). If tf is larger than N (YES in S72), the weight scor
e is obtained as log (saturation_score) (S74). Here, satatio
n_score is a predetermined constant, and is set to 0.5 here.

If tf is equal to or smaller than N (N in S72)
O), and the weight score is obtained as log (df2 / df) (S76).

When df2 is equal to or less than min_df2 (NO in S70), the weight score is min_scor.
e (S78).

After S74, S76 or S78, tab
It is determined whether le [j] is smaller than (score + table [i]) (S80). table
If [j] is smaller than (score + table [j]) (YES in S80), i is substituted for from [j] and (score + tabl) is assigned to table [j].
e [i]), and score [value] is assigned to value [j].
Is substituted (S82).

If the first character of the character string x is "-" or "" (YES in S64), if df is 0 (NO in S68), table [j] is set to (score +
table [i]) or more (NO in S80) or after the processing in S82, the value of the counter j is incremented by one (S84). Thereafter, it is determined whether j is larger than len_X (S86).

If j is equal to or less than len_X (NO in S86), the flow returns to S62. If j is larger than len_X (YES in S86), the value of the counter i is incremented by one (S88). Thereafter, it is determined whether i is greater than len_X (S90). i is len_
If X or less (NO in S90), the process returns to S58. i
Is larger than len_X (YE in S90)
S), the arrays from and value are stored in the operation result storage unit 30 (S92), and the processing in the score calculation unit 26 ends.

For example, an array "from" and "valu" obtained by dividing the character string "Year 2000 countermeasures"
The values of e and table are as shown in FIG. Thus, each word is separated as shown in FIG. The weight in parentheses indicates the weight of each word.

Referring to FIG. 14, a process for determining the number df of documents in which character string a appears and the number df2 of documents in which character string a appears twice or more (S6 in FIG. 10).
6) will be described. In this processing, in order to reduce the processing time for the same character string, the character string a and the calculated df and df2 are stored in a hash table for storing the number of documents (hereinafter, referred to as a “document number hash table”). Registration eliminates the need for recalculation. It is determined whether the character string a is registered in the document number hash table. If the character string a has been registered (YES in S101), the registered df
And df2 are obtained (S102).

If the character string a is not registered (S10
1 and NO), the character string a is searched in order from the beginning of the suffix file, and the suffix corresponding to the character string a found first is set to min (S103). Suffix m
If in cannot be obtained, that is, if the character string a is not included in the suffix file (YE in S104)
S), where the character string a does not appear in the document. For this reason, the values of df and df2 are set to 0 (S1
05).

When the suffix min is obtained, (S
NO in 104), the suffix corresponding to the character string a that appears last after the suffix min in the suffix file is set to max (S106). A character string whose suffix ranges from min to max matches the character string a. The number of different document numbers assigned to these character strings is obtained, and this number is set to df (S1
07). Also, referring to the document numbers assigned to these character strings, the number of documents having two or more identical document numbers is obtained, and the number is set to df2 (S10).
8).

After the processing of S98 or S95, the character string a and the number of documents df and df2 are registered in the document number hash table (S109). S
After the processing of 99 or the processing of S92, df and df2
Are returned as the number of documents in which the character string a appears and the number of documents in which the character string a appears twice or more (S110).

With reference to FIG. 15, a description will be given of a process (S66 in FIG. 10) for calculating the number of occurrences tf of the character string a in all documents.

The character string a is searched sequentially from the beginning of the suffix file, and the suffix of the character string a that appears first is set to min (S121). If the suffix min is not obtained, that is, the character string a
Is not included (YES in S122), 0 is substituted for tf (S123). When the suffix min is obtained (NO in S122), the suffix of the character string a that appears last in the suffix file is set to max (S124). tf is obtained according to the following equation (2) (S125).

Tf = max−min + 1 (2) After S123 or S125, tf is returned as the number of appearances of the character string a (S126).

[Process of Document Dividing Unit 32] The document dividing unit 32 divides an input document based on the arrays “from” and “value” stored in the operation result storage unit 30. That is, the document is divided such that the total value of the weights score when the document is divided is maximized.

[Processing of Narrowing Unit 34] The narrowing unit 34
The process shown in the flowchart of FIG. 6 described in [Selection of Keyword Candidates] is executed to narrow down keywords.

[Configuration Example of Keyword Extraction Apparatus 20] The above-described keyword extraction apparatus 20 can be realized by a computer. Referring to FIG. 16, keyword extracting device 20 includes a computer 41, a keyboard 45 and a mouse 46 for giving instructions to computer 41, a display 42 for displaying results and the like calculated by computer 41, and a computer 41. 41, a magnetic tape device 43 for reading a program executed by each of the programs, and a CD-ROM (Compact Disc-Read Only Mem
ory) device 47 and a communication modem 49.

The program of the keyword extracting device 20 is as follows:
The information is recorded on a magnetic tape 44 or a CD-ROM 48 which is a recording medium readable by the computer 41, and is read by a magnetic tape device 43 and a CD-ROM device 47, respectively. Alternatively, it is read by the communication modem 49 via the communication line.

Referring to FIG. 17, computer 41 includes:
A CPU (Central Processing Unit) 50 for executing a program read via the magnetic tape device 43, the CD-ROM device 47 or the communication modem 49, and other programs and data necessary for the operation of the computer 41 are stored. (Read Only Memory) 5 for
1 and a RAM (Random Access Memory) for storing a program, parameters at the time of program execution, a calculation result, and the like.
mory) 52 and a magnetic disk 53 for storing programs and data.

Magnetic tape device 43, CD-ROM device 4
7 or the program read by the communication modem 49 is executed by the CPU 50, and a keyword extraction process is executed.

The suffix file storage unit 24 and the operation result storage unit 30 are realized by the RAM 52 or the magnetic disk 53. Other components of the keyword extracting device 20 are realized by software executed by the CPU 50.

[Modification of Weight Expression for Character String x] FIG. 3
In S8 of FIG. 10 or in S76 of FIG.
og (df2 / df), but the frequency of appearance,
When the average size of the document is larger than 200 characters, considering the occurrence concentration, the partial character string length, and the average size of the document, log ｛(N / df) × (df2
/ Df) × len (x)}, and if the average document size is 200 characters or less, log
The weight may be obtained as {(N / df) × len (x)}. As each document becomes smaller, the degree of appearance concentration tends to approach zero. Therefore, by changing the calculation method of the weight based on the average size of the document, it is possible to calculate an appropriate weight even if the value of the appearance frequency becomes small.

As described above, according to the present embodiment, keywords can be extracted without requiring a dictionary in advance as in morphological analysis.

[Second Embodiment] An information retrieval apparatus according to the present embodiment is realized by a computer similar to that described in the first embodiment.

This embodiment relates to a method for calculating the similarity between character strings based on extracted keywords. It is assumed that the similarity between an input character string and a plurality of documents registered in a database is calculated. When searching for a match between strings, a method is used in which all documents in the database are searched efficiently for documents containing keywords extracted from the input string by using a suffix file. ing.

Collection of matching information is performed on the extracted keywords in the following manner. Find documents containing the keyword from the entire document database. Where the keyword appears in each of those documents, where the keyword appears in the input string,
The length of the keyword and the weight of the keyword are recorded as matching information.

Normally, the obtained coincidence information is not recorded and managed, and the weight is added as it is to calculate the similarity. However, by recording and managing this, not only a method of calculating the similarity by adding the weights of the matched keywords, but also it is applicable to many similarity calculation methods while maintaining high speed.

The similarity between the input character string and the document in the database is calculated by adding the weight given to the matching keyword.

In the present invention, the processing flow of the document search program based on the character string similarity calculated by extracting the keyword to be calculated is shown in FIGS.
This program searches the document database based on the input search text and searches for a plurality of documents having a high degree of similarity.

Referring to FIG. 18, a process of searching a document database based on a search sentence, selecting and outputting a document having a high similarity will be described.

First, in preparation for efficiently calculating the number of appearances of a certain character string, a suffix file (Suffix File) is created by integrating all documents included in the document database (S131).

Next, the search text is read into the character string X (S1).
32). The keyword extracted from the character string X is recorded in the keyword management table (S133).

Matching information is collected for each keyword recorded in the keyword management table and recorded in the matching information management table (S134). The matching information is information indicating the location of the keyword in the character string X, the location of the keyword in the document, the length of the keyword, and the weight of the keyword. In the matching information management table, matching information is recorded as a list for each document number. The process of S134 will be described later in detail.

A list of one document Y is extracted from the matching information management table (S135).

The similarity between the character string X and the document Y is calculated from the extracted list (S136). The process of S136 will be described later in detail.

A set of the obtained similarity and the document number is registered in the document management table (S13).
7).

It is determined whether similarities have been calculated for all lists recorded in the matching information management table (S138). If the similarities have not been calculated for all the lists (NO in S138), the process returns to S135.

If the similarities have been calculated for all the lists (YES in S138), the sets of the similarities and the document numbers in the document management table are rearranged in descending order of the similarities.

A document having a high similarity is output (S
140). Only one document may be output, or a predetermined number of documents may be output.
Alternatively, a document having a predetermined similarity or higher may be output.

Referring to FIG. 19, a process of collecting matching information between each keyword recorded in the keyword management table and each document in the document database and recording the information in the matching information management table (FIG. 18) S
134) will be described.

A certain keyword is selected from the keyword management table and set as a (S151). All locations where the keyword a appears in the document database are obtained, and the keywords are sorted in the order of appearance (S15).
2).

For each occurrence of keyword a, a document number containing keyword a is determined. At this time, since the keywords a are arranged in the order of appearance, the obtained document numbers are also arranged in ascending order (S153).

One of the appearance locations of the keyword a is selected in order from the front of the appearance location (S154). It is determined whether or not the appearance location of the selected keyword a is the forefront appearance location in the document including the keyword a (S155). That is, if the document at the selected appearance location is different from the document at the previous appearance location selected, it is the first appearance location in the selected document.
If the document at the selected appearance location is the same as the document at the previous appearance location selected, it is the second or later occurrence location in the selected document.

If it is determined that the keyword a appears first in the document (YES in S155), the keyword a appears in the input character string X (hereinafter referred to as "startX") a (where "startdo
c ”), the length of the keyword a (hereinafter“ terml ”)
and the weight of the keyword a (hereinafter, referred to as “score”) is recorded as a set in the matching information management table (S156).

Referring to FIG. 20, the matching information management table includes a list of matching information for each document information. Matching information 1 and 5 are recorded as a document number 0002, matching information 2, 3 and 6 are recorded as a document number 0100, and matching information 4 and 7 are recorded as a document number 0111 as a list. Each match information includes the keyword start in the input character string X.
X, startdoc, termlength and s
core is stored.

When the match information 8 relating to the document number 0002 is newly obtained, as shown in FIG. 20, the pointer pointing to the head of the list that has previously pointed to the match information 5 points to the match information 8 and the match information 8 , A pointer to the matching information 5 is set, and the matching information 8 is recorded at the head of the list of the document information 0002.

Referring to FIG. 19 again, after S156 or the appearance position of keyword a is 2 in the document.
If it is determined that it is the third or later (NO in S155),
It is determined whether all occurrence locations of the keyword a have been checked (S157).

If there is an appearance location that has not been checked (S157)
NO), and returns to S154. If all the appearance locations have been checked (YES in S157), it is determined whether or not matching information has been collected for all keywords in the keyword management table (S158). If there is a keyword for which matching information has not been collected (NO in S158), a keyword a that has not yet been selected
The process returns to S151 in order to read. If the matching information has been collected for all keywords (Y in S158)
ES), and returns the obtained matching information management table (S15)
9).

Referring to FIG. 21, a process of calculating the similarity between input text X and document Y by adding the weight of the matched character string using the list extracted from the matching information management table (S136 in FIG. 18) ) Will be described.

Similarity between X and Y (hereinafter referred to as “sim”)
Is initialized to 0 (S161). One piece of matching information is selected from the list related to Y recorded in the matching information management table, and is set as I (S162).

The score of the coincidence information I is added to sim (S163). It is determined whether all the pieces of matching information recorded in the list of matching information regarding document Y have been checked (S164). If there is unmatched matching information (NO in S164), the process returns to S162. If all matching information has been checked (YE in S164)
S), the obtained sim is returned as the similarity between the input text X and the document Y (S165).

As described above, according to the information search apparatus of the present embodiment, a document similar to the document input by the user can be searched from a database registered in advance. Therefore, for example, in a FAQ (Frequently Asked Questions) system or the like, when a user gives a question as an input sentence, an FAQ corresponding to the input sentence can be extracted.

The embodiments disclosed this time are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

[0102]

According to the present invention, keywords can be extracted from a document without requiring a dictionary.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a keyword extraction device according to Embodiment 1 of the present invention.

FIG. 2 is a diagram illustrating a partial character string x of the word “about a robot” and corresponding values of df, df2, α, and β.

FIG. 3 is a flowchart of a process for estimating wordiness (weight) of a character string.

FIG. 4 is a diagram illustrating an example in which a document is divided using the weight of a character string.

FIG. 5 is a diagram showing an example in which a document is divided by morphological analysis.

FIG. 6 is a flowchart of a process for extracting a keyword.

FIG. 7 is a flowchart of a process performed by a suffix file creation unit 22.

FIG. 8 is a diagram for explaining a suffix file.

FIG. 9 is a flowchart of a process performed by a separation unit.

FIG. 10 is a flowchart of a process executed by a score calculation unit 26.

11 is a diagram for explaining arrays value, table, and from used for weight calculation in the score calculation unit 26. FIG.

FIG. 12 is a diagram showing values of arrays from, value, and table when the character string “about the measure against the year 2000 problem” is divided.

FIG. 13 is a diagram showing an example in which each word is divided by weight.

FIG. 14 shows the number df of documents in which a character string a appears.
And the number d of documents in which the character string a appears twice or more
It is a flowchart of a process of obtaining f2.

FIG. 15 is a flowchart of a process for obtaining the number of times tf that a character string a appears in all documents.

FIG. 16 is an external view of a computer that realizes the keyword extracting device.

17 is a diagram illustrating a hardware configuration of the computer illustrated in FIG.

FIG. 18 is a flowchart of a process of searching a document database based on a search sentence, selecting and outputting a document having a high degree of similarity.

FIG. 19 is a flowchart of a process of collecting matching information between each keyword recorded in the keyword management table and each document in the document database, and recording the information in the matching information management table.

FIG. 20 is a diagram showing a configuration of a coincidence information management table.

FIG. 21 is a flowchart of a process of calculating the similarity between an input sentence X and a document Y by adding the weight of a matched character string using a list extracted from a matching information management table.

[Explanation of symbols]

Reference Signs List 20 Keyword extraction device, 22 Suffix file creation unit, 24 Suffix file storage unit, 26
Score calculation section, 28 delimiter section, 30 calculation result storage section, 32 document division section, 34 narrowing down section, 41
Computer, 42 display, 43 magnetic tape device, 44 magnetic tape, 45 keyboard, 46
Mouse, 47 CD-ROM device, 48 CD-RO
M, 49 communication modem, 50 CPU, 51 ROM,
52 RAM, 53 magnetic disk.

Continued on the front page (72) Inventor Yoshinori Takenami 1-3-1 Shimaya, Konohana-ku, Osaka Sumitomo Electric Industries, Ltd. Osaka Works (72) Inventor Masahiro Kishida 1-1-3 Shimaya, Konohana-ku, Osaka Sumitomo Electric Industry Co., Ltd. Osaka Works F-term (reference) 5B075 ND03 NK31 PP02 PP03 PP22 PQ02 PR04 PR06 QM08

Claims (8)

[Claims] An appearance frequency calculating means for obtaining an appearance frequency of a partial character string included in each document in the document group in the document group; and an appearance concentration degree of the partial character string in the document group. An appearance concentration degree calculating means for obtaining, and an appearance concentration degree calculating means connected to the appearance frequency calculation means and the appearance concentration degree calculation means, for extracting a keyword from an input document based on the appearance frequency and the appearance concentration degree. A keyword extracting device, comprising: one keyword extracting unit. 2. The method according to claim 1, wherein the first keyword extracting unit includes: a document dividing unit configured to divide the input document into partial character strings; and a document dividing unit, the appearance frequency calculating unit, and the appearance concentration calculating unit. Connected, based on the appearance frequency and the appearance concentration degree, word-likeness calculating means for calculating the word-likeness of the partial character string; connected to the word-likeness calculating means, based on the total value of the word-likeness 2. The keyword extracting apparatus according to claim 1, further comprising a second keyword extracting unit for extracting a keyword from the document. 3. The word likeness calculating means is connected to the document dividing means, the appearance frequency calculating means, and the appearance concentration degree calculating means, and includes the appearance frequency, the appearance concentration degree, the length of the partial character string, 3. The keyword extracting device according to claim 2, further comprising: means for calculating a wordiness of the partial character string based on an average size of the document. 4. The method further comprising the steps of: connecting the keyword extracted by the second keyword extraction unit to the second keyword extraction unit, the appearance frequency calculation unit, and the appearance concentration degree calculation unit; 3. The keyword extracting device according to claim 2, further comprising a narrowing unit for narrowing down based on an appearance concentration degree and a length of the partial character string. 5. The keyword extracting apparatus according to claim 2, further comprising a separating unit for separating the input document by punctuation marks and supplying the separated document to the document dividing unit. 6. The keyword extracting apparatus according to claim 2, wherein said document dividing means includes means for dividing an input document into partial character strings such that a leading character does not start with a predetermined character. apparatus. 7. The apparatus according to claim 2, wherein said document dividing means includes means for dividing an input document into partial character strings such that the length of the partial character string does not exceed a predetermined number of characters. The described keyword extraction device. 8. An appearance frequency calculating means for calculating an appearance frequency of a partial character string included in each document in the document group in the document group, and an appearance concentration degree of the partial character string in the document group. A keyword for extracting a keyword from an input document based on the frequency of appearance and the degree of concentration of appearance based on the frequency of appearance and the degree of concentration of appearance. Extraction means, connected to the keyword extraction means, and a coincidence calculation means for calculating the degree of coincidence of each document in the document group with each keyword extracted by the keyword extraction means; Means for connecting each document in the document group based on the degree of coincidence. Te,
A similarity calculating unit for calculating a similarity with the input document; and a similarity calculating unit connected to the similarity calculating unit. Means for extracting information.