JP5308918B2 - Keyword extraction method, keyword extraction device, and keyword extraction program - Google Patents

Description

  The present invention relates to a technique for extracting a keyword that appropriately represents the contents of a document from an electronic document existing inside a computer.

  A Web search engine that collects electronic documents on the Web and enables users to search is an indispensable tool for acquiring information on the Internet. However, in recent years, the number of documents returned by the Web search engine has increased more and more, and it has become difficult to search for documents required by the user from the search results of the Web search engine.

  In view of this, various methods have been proposed in which an electronic document is analyzed to extract words (keywords) that express the contents of the document.

  One is a technique called “proprietary expression extraction”, which extracts a term by analyzing a document and assigns a type such as a person name, an organization name, and a place name to the extracted term. As a result, keywords for each type can be extracted and used for document analysis and search. This technique is described in Non-Patent Document 1.

  Another technique is called “noun phrase extraction”, which is based on patterns based on part-of-speech information and the distribution of morphemes that appear in the vicinity. It is a technique to extract as a keyword. This is different from “specific expression extraction” in that only keywords are extracted, but keywords that cannot be extracted by “specific expression extraction” may be extracted. This technique is described in Non-Patent Document 2.

David Nadeau, Satoshi Sekine, “A survey of named entity recognition and classification”, Journal of Linguistic Investments 30-1 2007. Megumi Ishii, Kazunari Watanabe, “Proposal and Evaluation of Search Interface Using Classification System and Noun Phrase”, Information Processing Society of Japan Research Report, HCI Vol. 2000 No. 12 Marius Pasca, “Acquisition of Categorized Named Entities for Web Search”, Proceedings of the 13th ACM Conference on Information and Knowledge Management 04 (KN). 137-145.

  However, in “Native Expression Extraction” of Non-Patent Document 1, it is necessary to prepare a large amount of training data manually created for each keyword type, and it is difficult to extract keywords in a wide range of fields other than the names of people, places, and organizations. There is a fear.

  In “Noun Phrase Extraction” of Non-Patent Document 2, basically, noun phrases are comprehensively extracted based on patterns, so keywords separated at unnatural positions or conversely unnaturally connected. Keywords may be extracted. In addition, it may be difficult to classify the keywords acquired here according to their types.

  The present invention has been made to solve such a problem, and an object of the present invention is to appropriately extract a keyword expressing the contents of an electronic document without using manual training data.

  Therefore, in order to solve the above-described problems, the present invention includes a search condition (query) input to a search engine including keywords delimited in units assumed to be appropriate by a person, and inputs the keywords to the search engine. The search result title and summary sentence use what is appropriate as an example in which the keyword is used.

  One aspect of the present invention is a method for extracting a keyword included in an electronic document by applying a model generated using a search engine log, wherein the list generation means acquires the query log acquired from the search engine. A first step of generating a keyword list by extracting a query that satisfies a certain condition, and a collecting unit obtains a search result of the keyword in the list from the search engine, A second step of collecting examples in which keywords are used in the title and the summary sentence, and a third step in which the model generation means generates the model based on the examples collected in the second step.

  Another aspect of the present invention is an apparatus that extracts a keyword included in an electronic document by applying a model generated using a search engine log, and analyzes a query log acquired from the search engine. A list generating means for extracting a query that satisfies a certain condition to generate a list of keywords, and obtaining a search result of the keyword in the list from the search engine, and using the keyword in the title and summary sentence of the search result Collecting means for collecting the examples to be used, and model generating means for generating the model based on the examples collected by the collecting means.

  The present invention may be provided as an aspect of a program that causes a computer to function as the keyword extraction device.

  According to the present invention, it is possible to appropriately extract a keyword expressing the content of an electronic document without using manual training data.

The block diagram of the keyword extraction apparatus which concerns on embodiment of this invention. Process flow for generating the same keyword extraction model. Classification example of usage example of the keyword.

  Embodiments of the present invention will be described below. According to the present invention, an extraction model for each keyword is generated based on a set of keywords obtained from a query log of a search engine and a set of titles and summary sentences of search results for each keyword.

  This extraction model shows patterns such as morphemes and parts of speech that are assumed to be generally included in each keyword and its neighboring words. By applying this extraction model to an arbitrary electronic document, appropriate keywords are extracted from the document.

<Example of device configuration>
As shown in FIG. 1, a keyword extraction apparatus 1 according to an embodiment of the present invention is connected to a search engine 2 through a network so as to be communicable.

  The search engine 2 is a normal Web search engine that searches an electronic document (Web page) published on the Web, and includes a query log 3 that records a query received from a user terminal (not shown) in time series. And search execution means 4 for searching for an electronic document corresponding to the query and returning it to the user terminal.

  The keyword extraction device 1 includes hardware resources of a normal computer, that is, a CPU (Central Processor Unit), a memory (RAM), a hard disk drive device, a communication interface, and the like. As a result of the cooperation between the hardware resource and the software, the keyword extraction device 1 is provided with a keyword list generation means 5, a keyword classification means 6, an example collection means 7, a model generation means 8, a keyword extraction model database 9, and a keyword extraction means. 10 is implemented.

  Among these, each said means 5-8 implements the model production | generation process which produces | generates the extraction model of a keyword. That is, the keyword list generation means 5 analyzes the query log 3 to acquire a query that satisfies a certain condition as a keyword, and generates a list of the keyword.

  The keyword classification means 6 analyzes the query log 3 and a large amount of language data (corpus) registered in the extraction device 1 in advance, and classifies each keyword in the list according to its type.

  The example collection unit 7 acquires a list of keywords classified by the type, accesses the search execution unit 4 based on each keyword, and acquires an example of each keyword from the title and summary sentence of the search result. .

  The model generation means 8 generates an extraction model for extracting each keyword based on the example. The extraction model generated here is stored in the keyword extraction model database 9. It is assumed that this database 9 is constructed on the hard disk drive device.

  The keyword extraction means 10 applies the extraction model stored in the keyword extraction model database 9 to an arbitrary electronic document, and performs a keyword extraction process for extracting a keyword expressing the content of the document. Hereinafter, specific contents of each process will be described.

<Model generation process>
First, the model generation process will be described in detail based on the process flow of FIG. This model generation process corresponds to the main process of the keyword extraction device 1.

  Here, it is assumed that the keyword list generation means 5 accesses the query log 3 via the communication interface and acquires the query log 3.

  Here, in the query log 3, a log of queries (search keywords) previously input from the user terminal to the search engine 2 is recorded. In this log, a combination of an input query and an input date and time is recorded in time series. An example of data stored in the query log 3 is shown in Table 1.

  S01: The keyword list generation means 5 analyzes the query log 3, extracts a query that satisfies a certain condition as a keyword, and generates a keyword list. Examples of conditions include “being used as a search condition at a certain frequency or more” and “having a certain number of documents as a search result”. The conditions to be used may be set in the program in advance according to the specifications.

  The keyword list thus generated and the query log 3 are transferred to the keyword classification means 6. At this time, the generated keyword list may be stored in the memory or the like.

  S02: The keyword classification means 6 classifies each keyword of the keyword list transferred in S01 according to the type (category) by analyzing the query log 3.

  Examples of the classification method include a method of manually classifying a predetermined type, or a method of finding a keyword of a specific type based on an example of a keyword manually classified as in Non-Patent Document 3. . At this time, a large amount of language data that can be searched by a computer, that is, “corpus” may be registered in the keyword extracting device 1 in advance and analyzed together with the query log 3.

  The list classified here is transferred to the example collecting means 7 via the keyword list generating means 5.

  S03: When the keyword list for each type is transferred in S02, the example collection unit 7 accesses the search execution unit 4 based on each keyword, and acquires an example of each keyword from the title and summary sentence of the search result. To do.

  That is, when the keyword list for each type is transferred, the example collection unit 7 transmits the keywords of the list to the search execution unit 4 via the communication interface.

  When the search execution means 4 receives the keyword, the search result document title and URL corresponding to the keyword, and a portion including the keyword in the document as an outline sentence are used as the summary sentence via the communication interface. It returns to the keyword extracting device 1.

  At this time, the search execution unit 4 may perform a new document search using the keyword received from the example collection unit 7 and return the title, URL, and summary text of the search result.

  Thus, the title, URL, and summary sentence sent back to the keyword extracting device 1 are transferred to the example collecting means 7. Here, the example collection means 7 acquires a keyword example from the transferred title and summary text.

  In the early Web search engines, the beginning part of the document was used as a summary sentence, but in the late 1990s, Google (registered trademark) began to present text around search keywords, which has become the current mainstream. Yes.

  S04: The example collecting means 7 classifies the acquired examples according to the type of keyword.

  Here, a classification example of the example is shown in FIG. Here, the examples of each keyword “XX large chapter”, “XX crown”, and “Kyoto XX cup” are classified into types of “race names”. The examples classified here are transferred to the model generation means 8.

S05: When the example classified in S04 is transferred, the model generation means 8 generates a model for extracting a keyword for each type. Examples of the features used for generating the model include the following examples.
1. 1. The constituent morphemes of the keyword 2. morphemes near the keyword Part of speech of the constituent morphemes of the keyword 4. Part of speech of the morpheme near the keyword A morpheme that appears in the context in which the keyword appears For example, in the example shown in FIG. A model such as “XX cup” including “,” or “(place name) XXX” including a place name such as “Kyoto” at the beginning is generated.

  In addition, when the feature 2 “morpheme near the keyword” is applied to the keyword “XX large chapter”, the morpheme near “XX large chapter” (here, “40th” or “( G2) ”) and the like, and models such as“ No. XX ”and“ XXX (G2) ”are generated. The generated model is stored in the keyword extraction model database 9.

<Keyword extraction process>
The keyword extraction means 10 extracts keywords from an arbitrary electronic document using a keyword extraction model stored in the keyword extraction model database 9.

  As a specific example of the extraction process, there is a method in which the entire document is searched by a character string search method such as pattern matching, and a character string corresponding to the model is extracted from the document as a keyword.

  The extracted keyword may be output to an output unit such as a display, or may be stored in a storage unit such as a database. Further, specific implementation forms of the model generation means 8, the keyword extraction model database 9, and the keyword extraction means 10 are described in terms of a support vector machine (SVM), a CRF (Conditional Random Field), It is conceivable to use various learning algorithms such as a decision tree.

  As described above, according to the keyword extracting device 1, the keyword of the natural unit is manually trained based on the keyword input to the search engine and the title and summary sentence information of the search result output from the search engine. Can be extracted for each type without using.

  The keyword extracted here is in line with the extraction model generated from the title and summary sentence of the search result, so it is considered appropriate as a unit of information and can be used for analysis of a text set.

  Further, since the extracted keyword is considered to appropriately express the contents of the document, it is expected that the search accuracy during the document search can be improved by using the keyword as a search index of the document.

  The present invention can also be configured as a program that causes a computer to function as part or all of the means 5 to 10 of the keyword extraction device 1. In this case, the computer executes all or part of the processing steps (S01 to S05) of the embodiment.

  This program can be provided through a network such as a website or e-mail. The program is stored in a recording medium such as a CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, MO, HDD, Blu-ray Disk (registered trademark). It is also possible to save and distribute. This recording medium is read using a recording medium driving device (such as an optical drive device), and the program code itself realizes the processing of the above-described embodiment, so that the recording medium also constitutes the present invention.

DESCRIPTION OF SYMBOLS 1 ... Keyword extraction apparatus 2 ... Search engine 3 ... Query log 4 ... Search execution means 5 ... Keyword list production | generation means 6 ... Keyword classification means 7 ... Example collection means 8 ... Model generation means 9 ... Keyword extraction model database 10 ... Keyword extraction means

Claims (3)

A method for extracting keywords contained in an electronic document by applying a model generated using a search engine log,
A first step of generating a list of keywords by analyzing a query log acquired from the search engine and extracting a query that satisfies a certain condition;
A second step of collecting a search result of a keyword in the list from the search engine and collecting an example in which the keyword is used in a title and a summary sentence of the search result;
A third step in which the model generating means generates the model based on the examples collected in the second step;
A fourth step in which the classifying means classifies the keywords in the list for each type, and extracts the keywords for each type;
A keyword extraction method characterized by comprising: A device that extracts a keyword included in an electronic document by applying a model generated using a log of a search engine,
A list generation unit that analyzes a query log acquired from the search engine, extracts a query that satisfies a certain condition, and generates a keyword list;
A collecting means for acquiring a search result of the keyword in the list from the search engine and collecting an example in which the keyword is used in a title and a summary sentence of the search result;
Model generation means for generating the model based on the examples collected by the collection means;
Classifying means for classifying keywords in the list for each type, and extracting keywords for each type;
A keyword extracting device comprising: A keyword extraction program for causing a computer to function as the keyword extraction device according to claim 2 .

Images