JP2000076254A - Keyword extraction device, similar document retrieval device using the same, keyword extraction method and record medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To highly precisely extract a keyword considering respective documents in a data base from a text given as a keyword extraction object without executing a troublesome processing such as morpheme analysis on the respective documents in the data base. SOLUTION: A word extraction part 12b extracts a word from a keyword extraction object text. Intra-text appearing frequency is obtained at every extracted word and is stored in a word management table 13b. A word retrieval execution part 12c searches the full texts of the respective documents in a document data base storage part 11b at every extracted word. Intra-data base appearing frequency is obtained and is stored in the word management table 13b. A significance calculation part 12d calculates the significance of the respective words based on intra-text appearing frequency and intra-data base appearing frequency, which are stored in the word management table 13b. A keyword deciding part 12e decides a keyword based on the significance of the respective words.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a keyword extracting apparatus for extracting a word having high importance from a text given as a keyword extraction target as a keyword. The present invention relates to a keyword extraction device, a similar document search device using the keyword extraction device, a keyword extraction method, and a recording medium.

[0002]

2. Description of the Related Art In recent years, it has become important to extract a highly important word as a keyword as an index key for efficiently searching document data or from a given text to search for similar documents. .

Here, there are several methods for selecting a keyword by weighting each word extracted from a text given as a keyword extraction target. The simplest method is a method of referring only to the appearance frequency of each word in the keyword extraction target text. However, this is not accurate because it merely increases the importance of words that are frequently used in the text.

There is also a system that uses a keyword and an unnecessary word dictionary together with the frequency of appearance. This is a method in which words to be keywords or unnecessary words are registered in a dictionary in advance, and weighting is performed based on how many times the words registered in the dictionary appear in the keyword extraction target text.

When a plurality of document data are stored in the database, an appearance frequency table for each word appearing in the database is created in advance, and the importance of words having a low appearance frequency is set high. There is also a way to put it.

It is to be noted that a word having a low appearance frequency in the database is set to have a high importance because words frequently appearing in the database, for example, words such as "patent" and "invention" in a database relating to patent documents. This is because, for example, the evaluation value as a keyword is low, and conversely, words that do not appear frequently have a higher evaluation value as a keyword. Further, a scheme has been attempted in which syntax analysis is performed on a text from which a keyword is to be extracted, and a weight is determined based on a dependency relationship between words.

[0007]

As described above, conventionally, various methods for extracting a keyword have been considered. However, in the method of registering words and unnecessary words used as keywords in a dictionary, for example, not only is it difficult to create the dictionary, but also when the target text is a newspaper article with novelty, the dictionary is not registered. There is a problem that the content cannot keep up with the latest one.

In the method in which a word appearance frequency table is created for each document in the database in advance, a word is extracted by performing morphological analysis on all documents existing in the database, and the appearance frequency is calculated. It is necessary to carry out processing such as taking statistics of. For this reason, a large amount of time is required for processing, and furthermore, every time a document is added or updated in the database, the occurrence frequency table must be updated each time. In particular, when the update is frequently performed, this problem becomes remarkable. Further, there is a problem that a huge storage capacity is required to store the statistics of the appearance frequency.

Also, the method of determining the weight of a word using syntactic analysis has a problem that the accuracy of syntactic analysis itself does not reach a practical level at present, and there is no reference to a database. It is difficult to extract keywords with sufficient accuracy because information on the appearance of words in the past cannot be obtained.

The present invention has been made in view of the above points, and does not perform a complicated process such as morphological analysis on each document in a database, and performs a process of extracting a text provided as a keyword extraction target from a text in the database. It is an object of the present invention to provide a keyword extraction device capable of extracting a keyword in consideration of each document with high accuracy, a similar document search device using the keyword extraction device, a keyword extraction method, and a recording medium.

[0011]

A keyword extracting apparatus according to the present invention extracts words from a text given as a keyword extraction target by morphological analysis or the like, and for each extracted word, the frequency of occurrence in the text. And for each of the extracted words, a full-text search of each document in the database is performed to determine the frequency of occurrence in the database, and based on the frequency of occurrence in the text and the frequency of occurrence in the database. By calculating the degree of importance of each word, a word serving as a keyword of the text is determined.

According to such a configuration, the importance of a word is obtained by taking into account the frequency of appearance of the word in the database as well as the frequency of appearance of the word in the text to be extracted. Therefore, relative importance can be obtained, and words appropriate as keywords can be extracted based on the importance.

[0013] Further, since the word search in the database is performed by the full text search, the search processing can be speeded up, and the memory capacity required for the keyword extraction processing can be reduced.

By searching the database using the keyword words extracted from the keyword extraction target text as described above, a document similar to the text can be obtained as a similar document from the database.

[0015]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a keyword extracting device according to one embodiment of the present invention. This apparatus is constructed as a function of a similar document search apparatus on a computer having a general architecture.

As shown in FIG. 1, the apparatus comprises an external storage device 11, a control device 12, and a memory 13. The external storage device 11 includes, for example, a hard disk device or a DVD (Digital Video Disc) device. The external storage device 11 includes a keyword extraction target text storage unit 11a, a document database storage unit 11b, and an appearance position management table 11c.

Keyword extraction target text storage unit 11a
Stores text data given as a keyword extraction target. This text data is, for example, document data directly input and created by a user or document data received from the outside via a network, and here is different from each document data registered in advance in a database. I do. The present apparatus uses this text data as a keyword extraction target text, extracts a keyword word from the text,
Processing such as retrieving a document containing the keyword word from the database as a similar document is performed.

The document database storage section 11b is composed of a plurality of documents to be searched, as shown in FIG.
As shown in the drawing, one search target document includes a document information data portion 21 such as a creator name and a creation date and time, a text data portion 22 constituting the document, and a chart and image included in the document. It comprises a multimedia data section 23. The document database storage unit 11b
Each document to be searched stored in is assumed not to consist of documents belonging to various fields, but to have a certain degree of homogeneity that is included in specific large items such as economic fields or technical fields .

The appearance position management table 11c stores the appearance positions of words for each search target document stored in the document database storage unit 11b. The appearance position management table 11c will be described later with reference to FIG.

The control unit 12 controls the entire apparatus, reads a program recorded on a recording medium such as a magnetic disk, and executes various processes. As a processing unit realized by the program, a text input unit 12a, a word extraction unit 12b, a word search execution unit 1
2c, an importance calculation unit 12d, and a keyword determination unit 12e.

The memory 13 is composed of, for example, a RAM and stores necessary data in these processing units. Here, an input text storage buffer 13a, a word management table 13b, and a keyword storage buffer 13c are provided in the memory 13.

Here, in the control device 12, the text input unit 12 a stores a keyword extraction target text stored in the keyword extraction target text storage unit 11 a of the external storage device 11 in an input text storage buffer provided in the memory 13. 13a.

The word extraction unit 12b performs a morphological analysis on the text data stored in the input text storage buffer 13a to extract words, and registers the words in a word management table 13b provided in the memory 13. . that time,
For each word extracted from the text, the frequency of appearance in the text (the number of times the word appears in the text) is counted, and the total number is stored in the word management table 13b. FIG. 3 shows the structure of the word management table 13b.

The word management table 13b is a table for comparing, for each word extracted from the text, the frequency of occurrence in the text with the frequency of occurrence in the database.

As shown in FIG. 3, this word management table 13b
Is “ID” for storing the ID number assigned to each word.
A "number" column 31, an "entry word" column 32 for storing words extracted from the text, an "appearance frequency in text" column 33 for storing the frequency of occurrence of words in the text,
An “appearance frequency in database” column 34 for storing the appearance frequency of words in the database, and stores the word importance calculated based on the appearance frequency of words in the text and the appearance frequency of words in the database. And a “word importance” column 35 for performing the operation.

Returning to FIG. 1, the description will be continued. The word search execution unit 12c executes the processing for the document database storage unit 1 for each word stored in the "entry word" column 32 of the word management table 13b.
The frequency of appearance is counted for every document stored in 1b, and the total number is stored in the “frequency of appearance in database” column 34 of the word management table 13b.

The importance calculating section 12d includes a word management table 13b.
For each word stored in the “headword” column 32 of
The importance is calculated with reference to the “appearance frequency in text” column 33 and the “appearance frequency in database” column 34, and the calculation result is stored in the “word importance” column 35 of the word management table 13 b.

The keyword deciding section 12e sets the word management table 1
With reference to the “word importance” column 35 of 3b, a word having a high importance is selected based on a certain criterion, determined as a keyword of the text, and stored in a keyword storage buffer 13c in the memory 13. .

Next, the processing flow of the present apparatus will be described with reference to FIG. FIG. 5 is a flowchart showing the operation of the keyword extraction process in the embodiment. Under the control of the control device 12, first, the text input unit 12a is activated, and a process of storing text given as a keyword extraction target is performed (step S11). That is, the keyword extraction target text storage unit 1 of the external storage device 11
1a, the text data from which the keyword is to be extracted is read, and the input text storage buffer 13 of the memory 13 is read.
a.

Next, the word extracting unit 12b is activated, and a process of extracting words in the text is performed (step S12).
In other words, the text data stored in the input text storage buffer 13a is subjected to morphological analysis, and the words are sequentially extracted from the text based on the analysis result.

Here, of the words, nouns and sa-variable nouns are sorted, for example, in JIS code arrangement order, and
A D number is given, and stored in the “entry word” column 32 corresponding to the ID number of the word management table 13b, and a totaling result of the appearance frequency of each word in the text is stored in the “appearance frequency in text” column 33. Note that the sa-variant noun is a noun that becomes a verb when an auxiliary word such as “do” is attached, such as “store” or “designate”.

Next, the word search execution unit 1d is activated, and a search process for words in the database is performed (step S1).
3). Here, the “headword” column 3 of the word management table 13b
2 for each word stored in the document database storage unit 11b, character string collation is performed by text search on the text data of each search target document, and how many times these words appear in each search target document Count Aggregate this result for all search documents,
The total count is stored in the “appearance frequency in database” column 34 of the word management table 13b.

Next, the importance calculation unit 12d is activated, and the importance is calculated for each word (step S).
14). Here, as shown in Expression (1), each word in the word management table 13b has such a property that it increases as the frequency of appearance in the text increases, and decreases as the frequency of appearance in the database increases. The word importance (evaluation value) is calculated using the function.

Word importance = frequency of occurrence in text / frequency of appearance in database (1) The word importance thus obtained is stored for each word in the “word importance” column 35 of the word management table 13b. .

FIG. 4 shows a state in which data is stored in each column of the word management table 13b by the above processing. In the example of FIG. 4, words such as “stock price”, “New York”, “securities”, “market”, and “rise” are extracted from the keyword extraction target text and are sorted in the JIS code arrangement order. I have. In the JIS code array,
In this example, "New York" comes first because katakana precedes hiragana. In addition, the kanji is in the order of the reading kana.

Here, for example, the word "New York" appears four times in the text, and seven words appear in the database.
When it appears 0 times, the importance of “New York” is 4/70 = 0.057 according to the above equation (1).

Similarly, as shown in FIG. 4, the importance of the “stock price” is 12/220 = 0.0 since the appearance frequency in the text is 12 times and the appearance frequency in the database is 220 times.
54.

The importance of "market" is 6/180 = 0.033 because the appearance frequency in the text is 6 times and the appearance frequency in the database is 180 times. The importance of the “rise” is 3/24 = 0.125 since the appearance frequency in the text is 3 times and the appearance frequency in the database is 24 times.

The importance of “securities” is 5/160 = 0.031 because the appearance frequency in the text is 5 times and the appearance frequency in the database is 160 times. As described above, when the number of appearances in the text is large and the number of appearances in the database is small, the importance of the word becomes high. Conversely, if the number of occurrences in the text is small and the number of occurrences in the database is large, the importance of the word is low.

Next, based on the degree of importance obtained for each word, the keyword determining unit 12e is activated, and a keyword determining process is performed (step S15). Here, for example, the word importance is ranked at the top 40 of the word in the word management table 13b.
If the keyword is included in% or has a word importance of 0.05 or more, the keyword is determined and stored in the keyword storage buffer 13c in order.

In the example shown in FIG. 4, the condition for determining a keyword that is included in the top 40% corresponds to “rising” and “New York”, and those words are stored in the keyword storage buffer 13c as keywords of the text. Is done. Further, under the determination condition that the word importance is 0.05 or more, “rise”, “New York”, and “stock price” are respectively selected as keywords of the text, and are stored in the keyword storage buffer 13c.

It is assumed that the conditions for determining such a keyword can be arbitrarily set by the user. The keyword storage buffer 13c selects a word to be a keyword from the word management table 13b based on the set determination conditions.

After the keyword for the keyword extraction target text is obtained, various processes are executed by the upper module using the word (keyword) stored in the keyword storage buffer 13c.

For example, a process including retrieving a document containing all the words (keywords) stored in the keyword storage buffer 13c from the document database storage unit 11b and presenting the retrieved documents as similar documents of the text is performed. That is, in the example above, "rise", "New York",
When each word such as “stock price” is obtained as a keyword, a document having these three words is searched as a similar document from the database.

Here, the search condition includes all words stored in the keyword storage buffer 13c. However, for example, the search condition includes at least one word, or includes each word. There are various methods such as using a ratio as a search condition.

The search target is not limited to a document in the database, but may be an input text. That is, it is also possible to perform a process of searching for the corresponding text from the keyword extraction target text storage unit 11a using all the words (keywords) stored in the keyword storage buffer 13c.

As described above, the importance of a word is determined by taking into account the frequency of appearance of the word in the database as well as the frequency of appearance of the word in the text to be extracted. Therefore, relative importance can be obtained, and words appropriate as keywords can be extracted based on the importance.

Further, for each word extracted from the keyword extraction target text, all documents existing in the database are subjected to a full text search, and the frequency of occurrence of the word in the database is obtained. Therefore, when extracting a keyword, only the keyword extraction target text needs morphological analysis. Therefore, the processing can be speeded up and the memory capacity required for keyword extraction processing can be reduced as compared with a method in which morphological analysis is required for all documents in a database as in the related art. .

In the above embodiment, in step S13 of FIG. 5, the total number of occurrences of words extracted from the keyword extraction target text in the database is used as the value of the occurrence frequency in the database. Performs only processing to determine whether a word extracted from the extraction target text exists in each search target document in the database, and uses the total number of documents including the word in the database as a value of the occurrence frequency in the database You may do it.

In this case, for each word extracted from the target text for keyword extraction, the frequency increases when the appearance frequency in the text increases, and decreases when the total number of documents in the database including the word increases. The importance (evaluation value) of the word is calculated by using the function.

As described above, if the total number of applicable documents in the database is used as the frequency of occurrence in the database, it is not necessary to count the number of occurrences of each word, so that the processing speed is increased accordingly. There are advantages. However, the reliability is lower than the method using the total number of appearances of the word itself as the appearance frequency in the database.

In step S13, as a method of searching for a word at high speed, there is a method of creating an appearance position management table 11c as shown in FIG. 6 and referring to the table. The appearance position management table 11c stores, for each character code appearing in each document in the database, the document name where the character code appears and the appearance position in the document.

For example, when character codes such as “A” and “SA” appear sequentially from the beginning of the document 1, the document 1 and its appearance position are displayed in the column of the character code “A” in the appearance position management table 11c. Information (in this case, "1" indicating the head position)
Is stored, and the character code “sa” in the appearance position management table 11c is stored.
Column, information indicating the document 1 and its appearance position (here, 2
"2") indicating the number is stored.

By referring to the appearance position management table 11c, the appearance frequency of each word can be obtained without consulting the database. This makes it possible to speed up the process of calculating the importance.

Although the appearance position management table 11c needs to be created in advance, the appearance position management table 11c can be easily created only by checking the character code of each document from the beginning. Compared to the creation of a word table that requires morphological analysis for each document, it does not take much time and can be updated easily even when a new document is added.

For example, when calculating the word importance in step S14, the frequency of appearance in the text and the frequency of appearance in the database are multiplied by a coefficient, or their values are squared, or processing using a logarithm is performed. It may be performed.

In short, the present invention can be variously modified without departing from the gist thereof. Note that the method described in the above-described embodiment can be executed on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), a semiconductor memory, etc. It is also possible to write and apply it to various devices, or to transmit it via a communication medium and apply it to various devices. A computer that realizes the present apparatus reads the program recorded on the recording medium, and executes the above-described processing by controlling the operation of the program.

[0058]

As described above in detail, according to the present invention, the importance of a word is determined by taking into account the frequency of the word in the database as well as the frequency of the word in the text to be extracted. As a result, relative importance can be obtained, and words suitable as keywords can be extracted based on the importance.

Further, since the word search in the database is performed by the full-text search, the search processing can be speeded up and the memory capacity required for the keyword extraction processing can be reduced.

Further, by searching the database using the keyword words extracted from the keyword extraction target text, a document similar to the text can be obtained as a similar document from the database.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a keyword extraction device according to an embodiment of the present invention.

FIG. 2 is a diagram showing a configuration of a document database storage unit provided in the keyword extraction device.

FIG. 3 is a diagram showing a configuration of a word management table provided in the keyword extraction device.

FIG. 4 is a view showing a state where specific data is stored in the word management table.

FIG. 5 is a flowchart showing the operation of keyword extraction processing executed by the keyword extraction device.

FIG. 6 is a diagram showing a configuration of an appearance position management table provided in the keyword extraction device.

[Explanation of symbols]

 Reference Signs List 11 ... External storage device 11a ... Keyword extraction target text storage unit 11b ... Document database storage unit 11c ... Appearance position management table 12 ... Control device 12a ... Text input unit 12b ... Word extraction unit 12c ... Word search execution unit 12d ... Importance calculation Part 12e Keyword determination part 13 Memory 13a Input text storage buffer 13b Word management table 13c Keyword storage buffer 21 Document information data part 22 Text data part 23 Multimedia data part

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Yukio Nakamoto 1-3-3 Shinmachi, Ome-shi, Tokyo Toshiba Computer Engineering Co., Ltd. (72) Takuya Nishina 3-3-3 Shinmachi, Ome-shi, Tokyo No. 1 Toshiba Computer Engineering Co., Ltd. F-term (reference) 5B075 ND03 NK14 NK32 PP25 PQ36 PR06 QS01

Claims (8)

[Claims] 1. A database storing a plurality of documents, text acquisition means for acquiring a text given as a keyword extraction target, and words extracted from the text obtained by the text acquisition means; Full-text search for each document in the database for each word extracted by the word extraction means in the text for obtaining the frequency of appearance in the text for each word; A word search means in a database for determining an appearance frequency; and an appearance frequency in the text obtained by the word extraction means in the text and an appearance frequency in the database obtained by the word search means in the database. Importance calculating means for calculating the importance of a word; Keyword extracting device being characterized in that includes a keyword determining means for determining a word serving as a keyword of the text based on each word of importance obtained I. 2. The in-database word search means performs a full-text search of each document in the database for each word extracted by the in-text word extraction means, and calculates the total number of documents containing the word in the database. 3. The method according to claim 1, wherein the frequency is used as a frequency of occurrence in the data.
The described keyword extraction device. 3. The importance calculating means increases the frequency of appearance of each word extracted by the word extraction means in the text if the frequency of appearance in the text increases, and increases the frequency of appearance in the database. 2. The keyword extracting apparatus according to claim 1, wherein the importance of each word is calculated using a function having a property of decreasing each time. 4. The importance calculating means increases the frequency of occurrence of each word extracted by the word extracting means in the text, and increases the document containing the word in the database. 3. The keyword extracting apparatus according to claim 2, wherein the importance of each word is calculated using a function having a property of decreasing as the total number of keywords increases. 5. A table means for storing, for each character code appearing in the database, a document in which the character code appears and its appearance position, the word search means in the database refers to this table means. 2. The keyword extracting apparatus according to claim 1, wherein each of the words extracted by the in-text word extracting means is searched. 6. A database storing a plurality of documents, text acquisition means for acquiring text given as a keyword extraction target, words extracted from the text obtained by the text acquisition means, Full-text search for each document in the database for each word extracted by the word extraction means in the text for obtaining the frequency of appearance in the text for each word; A word search means in a database for determining an appearance frequency; and an appearance frequency in the text obtained by the word extraction means in the text and an appearance frequency in the database obtained by the word search means in the database. Importance calculating means for calculating the importance of a word; Keyword determining means for determining a word to be a keyword of the text based on the degree of importance of each word obtained as described above; and a document similar to the text based on the word determined by the keyword determining means. And a similar document search means for searching for similar documents from the same. 7. A keyword extracting method for extracting a keyword in consideration of each document in a database from a text given as a keyword extraction target, wherein a word is extracted from the text, and each extracted word is extracted. The full text search is performed on each document in the database for each of the extracted words, and the frequency of occurrence in the database is obtained. A keyword extraction method comprising: calculating importance of each word based on the frequency of appearance in a database; and determining a word to be a keyword of the text based on the importance of each word. 8. A recording medium storing a keyword extraction program for extracting a keyword in consideration of each document in a database from a text given as a keyword extraction target, extracting a word from the text, and extracting the word. A procedure for obtaining the frequency of appearance in the text for each of the words, and a procedure for performing a full-text search of each document in the database for each of the extracted words, and calculating the frequency of appearance in the database; Calculating the importance of each word based on the frequency of appearance in the text and the frequency of occurrence in the database; and determining the word to be a keyword of the text based on the importance of each word. And a computer-readable recording medium on which a program for causing a computer to execute the program is recorded.