JP3508312B2 - Keyword extraction device - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention creates an index for a document to be registered in advance in a full-text search, and inputs a document using the index when searching. The present invention relates to a technique for specifying a document by comparing a keyword and an index, and more particularly to a keyword extraction device for creating an index. 2. Description of the Related Art Conventionally, when a document is registered in a document search device, a word serving as a keyword of the document is extracted and an index (index) for the document is created. Regarding the technology for extracting keywords, for example, the following documents are disclosed. [1] Japanese Patent Application Laid-Open No. 61-151738 (Title of Invention: "Keyword Extraction Device") This document discloses extracting a keyword from a document using a keyword extraction rule consisting of a character type and the number of characters. ing. [2] Japanese Patent Application Laid-Open No. 5-006398 (Title of Invention "Document Registration Apparatus and Document Retrieval Apparatus") In this document, a document or a keyword is divided at a character type change point and a set of two consecutive characters Is coded as an index, and the hiragana section is not coded. [3] Japanese Patent Application Laid-Open No. 3-116375 (Title of Invention: "Information Retrieval Apparatus") This document uses a word dictionary to determine whether a keyword is a compound word or a simple word. It is disclosed that the keyword is divided into simple words constituting the keyword, and all the simple words are held as keywords. [4] Japanese Patent Application Laid-Open No. 5-81328 (Title of the Invention: "Keyword Automatic Input System") This document describes a method in which Japanese sentence data is divided into sentences and the sentence data is obtained. Japanese Patent Application Laid-Open No. H11-64300 discloses that part-of-speech analysis is performed to extract noun data, and index data is created by using the extracted noun data and data representing a location in a sentence in which the data is described. [0007] Japanese Patent Application Laid-Open No. 61-1517
In the technology described in Japanese Patent Publication No. 38, only words that match the keyword extraction rules are extracted, and therefore, when the extracted keywords are complex, the words that compose the keyword cannot be extracted. Also, it is not possible to extract readings and absorb changes in utilization. In the technique described in Japanese Patent Application Laid-Open No. 5-006398, the section of hiragana is not coded, but is coded in units of two characters, so that a keyword such as "reading" which combines hiragana and kanji can be extracted. Can not. Also, it is not possible to extract readings and absorb changes in utilization. In the technique described in Japanese Patent Application Laid-Open No. 3-116375, compound words must be registered in a dictionary, but this is practically difficult. It is not possible to extract readings and absorb changes in utilization. According to the technique described in Japanese Patent Application Laid-Open No. 5-81328, when a compound word is included in a clause, the compound word is extracted as one word, so that the words constituting the compound word cannot be extracted. In addition, the extracted keywords are limited to nouns, and the change in the extraction and utilization of reading cannot be absorbed. As described above, in the conventional technique, there are many limitations in extracting keywords, and it has been difficult to extract useful keywords for search without omission. The present invention solves the above-described problems of the conventional technology, and enables to completely extract keywords that may be specified in a search expression when creating an index for full-text search at the time of registration. It is intended for that purpose. [0013] A keyword extraction device according to the present invention includes an input storage means (11) for storing the contents of text of a document to be subjected to keyword extraction, and a Japanese phrase structure rule. A phrase candidate analyzing means (12) for analyzing a combination of satisfying words to extract a phrase candidate; a phrase candidate storing means (13) for storing a result analyzed and extracted by the phrase candidate analyzing means;
An analysis result extraction means (14) for extracting a combination of words having the minimum cost from the contents of the phrase candidate storage means, an analysis result storage means (15) for storing the analysis results extracted by the analysis result extraction means, and a phrase candidate An extraction condition storage means (16) for storing conditions for extracting a keyword from the contents of the storage means or the contents of the analysis result storage means, and a keyword matching the extraction condition storage means from the contents of the phrase candidate storage means. First keyword extracting means (17) for extracting as a keyword, second keyword extracting means (18) for extracting, from the contents of the analysis result storing means, those which match the conditions of the extraction condition storing means as keywords, and first keyword extracting means Extracted keyword storage means (19) for storing the extracted keywords and the keywords extracted by the second keyword extraction means; It includes those were. The analyzing means performs a morphological analysis on the text in the target document, and stores information on the analysis result in the analysis result storage means. The keyword extracting means extracts the keyword from the information obtained as a result of the analysis held by the analysis result storing means, using the condition for determining the keyword held in the condition storing means. The information on the analysis result includes not only the final result of the morphological analysis but also a phrase candidate which is an intermediate result of the morphological analysis, and a keyword can be extracted from the phrase candidate. For example, extract words that are elements of compound words obtained as an intermediate result of morphological analysis, as well as words that are included in katakana character strings, as well as compound words that are finally obtained as a result of morphological analysis be able to. Therefore, it is possible to extract keywords for full-text search without omission of search. In addition, the analysis result information includes not only word notation but also reading and part-of-speech information.From these readings and part-of-speech information, it is possible to return to the final form, extract readings, and extract keywords by part of speech. Sorting can be used as a condition for extracting a keyword. When a search is performed using the keywords extracted in this manner, the allowable range of setting the search keywords is widened, and the omission of the search is reduced. The outline of the operation of a specific mode of the present invention (claim 4) is as follows. The phrase candidate analysis means analyzes a combination of words satisfying the Japanese phrase structure rule with respect to the text content, and the phrase candidates obtained as a result of the analysis are stored in the phrase candidate storage means. The analysis result extraction means extracts a combination of words having the minimum cost from the contents of the phrase candidate storage means, and the result is stored in the analysis result storage means. The extraction condition storage means stores conditions for extracting a keyword. The first and second keyword extracting means extract, from the contents of the phrase candidate storing means and the analysis result storing means, those which meet the conditions of the extracting condition storing means as keywords, and the extracted key storing means stores the extracted keyword. Is stored. In the related art, a keyword is extracted from the final result of the morphological analysis. Therefore, in the case of a compound word, a word that is an element of the compound word cannot be used as a keyword. Since keywords are also extracted from the storage contents of the phrase candidate storage means, keywords for full-text search with few omissions in search can be extracted. FIG. 1 is a functional block diagram schematically showing a keyword extracting apparatus according to an embodiment of the present invention. This keyword extraction device includes an input storage unit 11, a phrase candidate analysis unit 12, a phrase candidate storage unit 13, an analysis result extraction unit 14,
It comprises an analysis result storage unit 15, an extraction condition storage unit 16, a first key extraction unit 17, a second key extraction unit 18, and an extraction key storage unit 19. The function of each element is as follows. The input storage means 11 stores the text content of the registered document. The phrase candidate analyzing means 12 analyzes and extracts a combination of words satisfying the Japanese phrase structure rule with reference to an analysis dictionary and a connection table described later. The phrase candidate storage unit 13 stores the phrase candidates obtained by the analysis result of the phrase candidate analysis unit 12. The analysis result extraction means 14
A combination of words having the minimum cost is extracted from the contents of the phrase candidate storage unit 13. The analysis result storage unit 15 stores the extraction result of the analysis result extraction unit 14. The extraction condition storage unit 16 stores conditions for extracting a key from the contents of the phrase candidate storage unit 13 and the analysis result storage unit 15. The first key extraction unit 17 extracts a key from the contents of the phrase candidate storage unit 13 and an analysis result that matches the condition of the extraction condition storage unit 16. The second key extracting unit 18 extracts a key from the analysis result that matches the condition of the extraction condition storing unit 16 from the content of the analysis result storing unit 15. The extracted key storage unit 19 stores the keys extracted by the first key extraction unit 17 and the second key extraction unit 18. The Japanese morphological analysis is a process of analyzing information on words and phrases from a Japanese sentence that is not separated. This embodiment is based on a morphological analysis called a minimum cost method. The minimum cost method is an extension of the minimum number of clauses method (an analysis method that gives priority to the one with the minimum number of clauses when there are multiple analysis results), and assigns costs to word candidates to minimize the overall cost. (Yoshimura / Hidaka / Yoshida: Algorithm of Japanese morphological analysis including unregistered words, Kyushu University Journal of Engineering, Vol. 55, No. 6, 1982). In addition,
The phrase number minimization method is equivalent to the cost minimization method when the cost of an independent word is set to 1 and the attached cost is set to 0. FIG. 2 is a diagram showing an example of a dictionary referred to by the phrase candidate analysis means 12 at the time of analysis. The dictionary stores information on the words that make up Japanese phrases. For each word, a heading 21, a part of speech 22, a reading 23, a cost 2
4. Other information 25 is stored. FIG. 3 is a diagram showing an example of a connection table referred to by the phrase candidate analysis means 12 at the time of analysis. The connection table is a two-dimensional array that defines whether two adjacent words can be connected using the part of speech information defined in the dictionary. The part of speech of the row represents the part of speech of the word to the left of the adjacent word, and the part of speech of the column represents the part of speech of the word to the right of the adjacent word. If the value of the array element is 1, connection is possible,
If the value is 0, it means that connection is impossible. The part of speech in the column is at the beginning of a virtual phrase, which is provided to determine whether a word can be the end of a phrase. In the embodiment, the analysis of the input sentence is performed in accordance with the morphological analysis algorithm based on the above-mentioned minimum cost method. In this algorithm, first, words included in an input sentence are cut out using a dictionary, connection check with the immediately preceding word candidate, determination of the end of a phrase, and cost update processing are performed to extract a phrase candidate. Next, an analysis result that minimizes the sum of costs from the beginning of the sentence is extracted from the phrase candidates. FIG. 4 is a flowchart showing the operation of the embodiment. In step S1, the text content to be registered is stored in the input storage unit 11. In step S2,
A sentence is extracted from the text stored in the input storage means 11 using clues and line feeds as clues. In step 3, it is determined whether a sentence has been extracted. If no text is extracted, all text in the text has been processed and the process ends. If the text is extracted, the process proceeds to step 4. In step 4, the sentence is analyzed to obtain a phrase candidate, and the result is stored in the phrase candidate storage unit 13. Step 4 corresponds to the first step of the minimum cost method. In step 5, the phrase structure with the minimum cost is extracted from the phrase candidate storage unit 13, and the extracted result is stored in the analysis result storage unit 15. Step 5 corresponds to the second step of the minimum cost method. In step 6, the second key extracting means 1
8 extracts a key that matches the extraction condition stored in the extraction condition storage unit 16 from the analysis result storage unit 15 and stores the key in the extraction key storage unit 19. In step 7, the first
The key extracting unit 17 extracts a key that matches the extraction condition stored in the extraction condition storing unit 16 from the phrase candidate storing unit 13 and stores the key in the extracted key storing unit 19. FIG. 5A shows the first key extracting means 17 and the second key extracting means 17.
FIG. 4 is an explanatory diagram showing an example of an extraction condition referred to by a key extraction unit 18. The part of speech 51 and the lower limit 52 of the number of characters indicating the condition of the word length of the part of speech are associated as the extraction condition.
In this example, as an extraction condition, a general noun having a length of two or more characters is extracted, and a proper noun having a length of one or more characters is extracted, and is not registered in a dictionary (for example, alphanumeric characters, katakana, etc.). The condition ()) is based on the condition that a character having a length of two or more characters is extracted. FIG. 5B is an example of a content showing a part of the content of the document to be registered. It is assumed that the target document has a sentence 53 “Write to a text file.” FIG. 5C illustrates a boundary position of a character in a sentence.
FIG. 14 is a diagram illustrating position information referred to when an analysis result and a range of an extraction key are represented by a start point and an end point. For example, the character string “text” exists in the range of 0 and 4. The operation of extracting a keyword from the text shown in FIG. 5B will be specifically described below. In step S1 of FIG. 4, a text including a sentence “Write to a text file.” Is stored in the input storage unit 11. In step S2, a sentence is extracted using clues ".", A line feed, and the end of the file as clues. In the following description, a description will be given of the processing from step S2 onward when a sentence “write to a text file” is extracted at a certain point in time. Since the text is extracted in step S3, the process proceeds to step S4. In step S4, the first step of the minimum cost method is executed. Each boundary position (0, 1, 2,... 1) shown in FIG.
Word candidates starting with 5) are found by searching a dictionary as illustrated in FIG. Also, continuous katakana and alphabets may be unregistered words, so continuous katakana is not appropriate in the part-of-speech information (not registered in the dictionary in the example).
And a cost (in the embodiment, 80) are added to extract the word as a noun word unregistered word, and treat it as if it had been registered in the dictionary. After obtaining the dictionary information, it checks whether it can be connected to the immediately preceding word candidate based on the dictionary contents, and if it can be connected, calculates the minimum value of the sum of the dictionary information and the cost from the beginning, and stores the phrase candidate storage means. 13 is stored. The character string at the beginning of the current sentence is "text ...", and dictionary information of a partial character string of the input character string such as "te", "text", "text", and "text" is shown in FIG. Search the dictionary and ask for it. As a result, "text" is searched. In addition, since the first character “te” is katakana, the subsequent katakana character string is also cut out. As a result, a “text file” is cut out. No in FIG.
The phrase candidates No. 1 and No. 4 are stored in the phrase candidate storage means 3. The parts of speech are nouns and out of the dictionary. Since the costs are 70 and 80, respectively, and the cost at the beginning of the sentence is 0, the total cost from the beginning of the sentence is 7
0 and 80. The phrase end information means that if it is 0, it cannot be the end of the phrase, and if it is 1 or more, it can be the end of the phrase. The next cutout position is the smallest position among the end positions of the extracted word candidates. In this case, since “text” and “text file” have been extracted, the minimum end position is 4. Accordingly, as shown in FIG. 5C, a phrase candidate is cut out for "in the file ...". In this case, a file (not registered in the dictionary) and a file (noun class / symbol stem) are cut out, and phrase candidates such as No. 2 and No. 3 in the phrase candidate storage unit 3 are stored. My own costs are 80 and 70. The sum of the costs from the beginning of the sentence is 150 or 140, which is the sum of the minimum value of the sum of the costs at the start position (in this case, the minimum value at the fourth position) 70 and the cost of itself. Thereafter, the processing proceeds in the same manner, and after the execution of step S4, the phrase candidates as shown in FIG. In the next step S5, the second step of the minimum cost method is executed. Phrase candidate storage means 13
, The phrase structure with the minimum cost is extracted, and the extracted result is stored in the analysis result storage unit 15. Specifically, from the last clause candidate, the one with the smallest total cost is extracted from the beginning, and the candidate clauses connectable to this clause structure are extracted from the end of the sentence toward the beginning of the sentence. Whether or not connection is possible is determined based on the fact that they are adjacent, that connection is possible due to cost, and whether or not a connection is defined in the connection table. In the contents of the phrase candidate storage means 13 shown in FIG. 6, the last phrase candidate is "." The starting position is 15, the own cost is 300, and the total cost is 445. The phrase candidate immediately before the connection is a phrase candidate whose end position is 15, the total cost is 145, and the connection is defined in the connection table, that is, the phrase candidate of No. 39 in FIG. In the same manner, FIG. 7A shows the result of extracting phrase candidates having the minimum cost from the sentence end to the sentence start. FIG.
The content of (a) is a so-called morphological analysis result. In the next step S6, the second key extraction means 18 matches the extraction condition stored in the extraction condition storage means 16 of FIG. 5A from the analysis result storage means 15 of FIG. 7A. The extracted key is extracted and stored in the extracted key storage unit 19. Also in this case, no duplicate extraction is performed for the notations already extracted. As a result of executing step S6, a "text file" is extracted, and the contents shown as No. 1 in FIG. 7B are registered in the extraction key storage means. . In the next step S7, the first key extracting means 17 converts the contents stored in the phrase candidate storing means 13 shown in FIG. 6 into the extracting conditions stored in the extracting condition storing means 16 shown in FIG. A matching keyword is extracted and stored in the extracted key storage unit 19. No. 2 to N in FIG. 7B
o6 is the content of the key extracted in step S7. Here, since the same notation is not redundantly extracted, the “files” of No. 2 and No. 3 in FIG.
Only one is extracted. After the process in step S7, the process returns to step S2, and the processes in steps S2 to S7 are repeated for the next sentence until a sentence cannot be extracted. In FIG. 4, by exchanging steps S6 and S7 so as not to extract a duplicate key,
Similar effects can be obtained. FIG. 7A shows the result of the morphological analysis of the sentence "Write to a text file." When the key is extracted from this analysis result under the extraction conditions shown in FIG. Only "text files" are extracted. When creating an index for full-text search using such a key, specify a noun search term such as "text", "file", "write", "contain" or "masu" and write it to a text file. "Could not be searched. According to the present embodiment, the key as shown in FIG. 7B is also extracted from the intermediate result of the analysis as shown in FIG. 6, so that the document including "Write to text file.""text"
Search can be performed from noun search words such as "file", "write", "contain", and "masu", and omission of search can be eliminated. The present invention can be implemented by modifying or substituting a part of the embodiment described above as follows. (1) Steps S6 and S in FIG.
In the extraction of the keyword 7, not only the notation of the keyword but also the reading is extracted. When the notation is in katakana, the reading can be obtained by code conversion, so that katakana may be used. If no katakana readings are extracted, and if the search word is hiragana, a katakana search word is also generated and searched. By configuring as such, “text file”, “text”, “file”, “write”, “include”
Keys such as "Masu", "Kakikomi" and "Komi" are extracted. As a result, the user can understand the reading at the time of retrieval but cannot specify the exact notation, and also can read the case where the notation varies depending on the presence or absence of the sentence in the registered document (writing, writing, writing). To search for a desired document. (2) The contents of the extraction condition storage means 16 are replaced with contents as shown in FIG. 8, for example. In FIG. 8, designation is made to extract parts of speech other than nounity. For example, a five-stage verb means ka line, ga line, sa line, ta line, ba line,
It means the stem of five-row utilization such as ma row, la row, na row, c row, a row, etc. By specifying the extraction conditions as shown in FIG. 8, from a sentence such as “read”, “read (noun)”, “read (ma line 5 dan)”, “contain (ma line 5 dan)” “read (ma line) (3) When the part of speech of the key to be extracted means a conjugative word, the key is extracted as a final form, and the one that changes rules (for example, a five-step verb or an adjective) Etc.), the final form can be obtained by connecting the final form of the conjugative word to the stem except for the conjugation endings. By extracting the final form of the conjugation word in this way, it is possible to perform a search that is not affected by the difference in conjugation, for example, by extracting the final form such as “read, read” and “read”. By reading, A document containing "and" read "can be retrieved from the search term" read ". Also, by performing the same key extraction processing for the search expression, the search term “read” can be extracted from the search expression including “read” and “read”, and the document including “read” and “read” can be extracted. Can be searched. According to the present invention, not only a keyword for full-text search is extracted from only the final result of morphological analysis, but also an intermediate result such as a candidate for a word included in an input sentence. The keyword for full-text search can be extracted from. For example, it is possible to extract not only compound words but also words included in compound words or words included in katakana character strings. In addition, it is possible to return to the final form, extract the pronunciation, and select the keywords to be extracted by the part of speech from the keyword reading and part-of-speech information obtained by performing morphological analysis. it can. As a result, it is possible to extract a keyword for full-text search that eliminates omission of search.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a functional block diagram illustrating a schematic configuration of a keyword extracting device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a dictionary referred to by the phrase candidate analysis means 2 at the time of analysis. FIG. 3 is an explanatory diagram showing an example of a connection table referred to by the phrase candidate analysis means 2 at the time of analysis. FIG. 4 is a flowchart showing an operation of the embodiment. FIG. 5A shows a first key extracting unit 7 in the embodiment.
FIG. 3B is a diagram showing an example of an extraction condition referred to by the second key extracting unit 8, FIG. 4B is a diagram showing a content example showing a part of the content of the registration target document in the embodiment, and FIG. It is the figure which illustrated the position. FIG. 6 is a diagram showing the contents of a phrase candidate storage unit for the example sentence shown in FIG. 5 (b). FIG. 7A is a diagram illustrating a morphological analysis result for an example sentence, and FIG. 7B is a diagram illustrating a result of extracting a keyword from an intermediate result of the morphological analysis. FIG. 8 is a diagram showing another example of an extraction condition. [Description of Signs] 11: Input storage means, 12: Clause candidate analysis means, 1
3 ... phrase candidate storage means 14 ... analysis result extraction means
15: analysis result storage means, 16: extraction condition storage means,
17 ... first key extracting means 18 ... second key extracting means 19 ... extracted key storage means

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-4-211868 (JP, A) JP-A-6-301722 (JP, A) JP-A-7-73200 (JP, A) Satoshi Ito et al. Keyword extraction method that can be optimized according to the purpose, IEICE Technical Report, Japan, The Institute of Electronics, Information and Communication Engineers, December 9, 1993, Vol. 93, No. 366, pp. 41-41. 46 pages, NL C93-53 (58) Field surveyed (Int. Cl. ⁷ , DB name) G06F 17/30 JICST file (JOIS)

Claims (1)

(57) [Claims] [Claim 1] Input storage means for storing the text content of a registered document, and a phrase for extracting a phrase candidate by analyzing a combination of words satisfying Japanese phrase structure rules Candidate analysis means; a phrase candidate storage means for storing results analyzed and extracted by the phrase candidate analysis means; and an analysis result extraction means for extracting a combination of words having a minimum cost from the contents of the phrase candidate storage means. An analysis result storage means for storing analysis results extracted by the analysis result extraction means; an extraction condition storage means for storing conditions for extracting keywords from the contents of the phrase candidate storage means or the contents of the analysis result storage means; First keyword extracting means for extracting a keyword that matches the condition of the extraction condition storing means from the contents of the candidate storing means as a keyword; A second keyword extracting unit that extracts a keyword that matches a condition of the condition storing unit as a keyword; and an extracted keyword storing unit that stores a keyword extracted by the first keyword extracting unit and a keyword extracted by the second keyword extracting unit. A keyword extracting device.