JP4148247B2 - Vocabulary acquisition method and apparatus, program, and computer-readable recording medium - Google Patents

Description

The present invention relates to a vocabulary acquisition method and apparatus, a program, and a computer-readable recording medium. In particular, in a computer network represented by the Internet, a vocabulary for acquiring a vocabulary from tagged text such as HTML, XML, and SGML. The present invention relates to an acquisition method and apparatus, a program, and a computer-readable recording medium.

  When searching for information in a computer network, the amount of search results often becomes large, and users of the search system perform a distant work of acquiring the information they really want from the search results obtained after searching by keyword. Is forced.

  For such problems, extract key terms in documents such as proper nouns from text information of search results, select terms that are considered important in the search results from those terms, and search There is a method of easily realizing efficient document search by presenting it together with the result.

  Thus, the user can narrow down the desired information without looking for the desired information while looking at the search results one by one or without performing a re-search by considering additional keywords.

  In order to realize this, a technique for identifying proper nouns in text is necessary as a basic technique.

  The simplest method is to create a dictionary manually and extract words that match the dictionary word from the text.

In addition, it does not have a specific dictionary, it manually identifies proper nouns that exist in the document in advance, creates extraction rules as morpheme (part of speech information) level patterns from the specified learning data, and is included in the learning data in advance There is also a technique that enables extraction of new words as well as new words (see, for example, Patent Document 1 and Non-Patent Document 1).
JP 2003-331254 A Sekine.S .: Named Entity: History and Future, http://cs.nyu.edu/sekinepaper/NEsurvey 200402.pdf

  However, the above prior art has the following problems.

  The method of manually creating a dictionary can reliably identify the corresponding part in the text, but the cost of updating the dictionary is very high, so it is realistic to collect dictionary words in a wide range of fields and attributes It is difficult.

  In addition, the method that uses learning data automatically extracts rules from the text for new vocabulary not found in the learning data in addition to the vocabulary existing in the learning data by automatically generating rules based on the learning data. Can do.

  However, on the other hand, the generation of extraction rules requires extraction of characteristic patterns from learning data. For example, data such as a person name or a place name in a newspaper article has a characteristic appearance pattern, so that it can be extracted with relatively high accuracy, but the extraction target is a general Web page or the like. In such a case, the expressions become diverse and it is not always possible to specify with high accuracy. Also, in the case of “store name”, “book title”, etc., in which the name itself is not a person or place name but a variety of expressions, patterning is similarly difficult and extraction of high-precision expressions is difficult.

  The present invention has been made in view of the above points, and in a text or a tagged document, data of a specific attribute (person name, book title, etc.) is extracted from text in a computer network based on a small number of data examples. A vocabulary acquisition method, apparatus, program, and program that can automatically acquire data by identifying patterns in which data examples appear, removing unnecessary words from the appearance frequency and distribution of keywords extracted by the patterns It is an object to provide a storage medium.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (claim 1) analyzes the electronic text including HTML and XML based keywords example of a specific attribute, a lexical acquisition method in an apparatus for acquiring vocabulary,
A text search step in which the text search means searches the text storage means for storing electronic text including HTML and XML based on a keyword example of a specific attribute input from the outside, and acquires the text including the keyword example ( Step 1) and
A keyword position specifying step (step 2) in which the keyword position specifying means specifies the appearance position of the keyword example from the searched text;
In the text acquired in the text search step, the keyword appearance pattern extraction means acquires characters one by one from the appearance position of each keyword in the keyword example toward the beginning of the text, compares the acquired characters, and the same If it is a character, the next character is acquired, and by repeating the comparison, a keyword appearance extraction step (step 3) for specifying an appearance pattern common to the keyword example;
A keyword candidate extraction step (step 4) in which the keyword candidate extraction means analyzes the text based on the appearance position of each keyword in the keyword example and the appearance pattern common to the keyword example, and extracts the keyword candidate from the text;
The keyword extraction means, for each keyword candidate keyword, the number N _A of text acquired in the text search step, the number N _A ^W of texts including the keyword candidate among the text acquired in the text search step, and the text evaluation value for each keyword from the number N ^W of the text stored in the storage means seeking ^{_{(N a W / N a)}} × log (N a W / N W), the keyword evaluation value is higher than a predetermined threshold value And a keyword extraction step (step 5) for extracting from the keyword candidates .

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (claim 2) analyzes the electronic text including HTML and XML based keywords example of a specific attribute, a lexical acquisition apparatus for acquiring vocabulary,
Text storage means 2 for storing electronic text including HTML and XML;
Text search means 3 for searching the text storage means 2 based on a keyword example of a specific attribute input from the outside, and obtaining text including the keyword example;
Keyword position specifying means 4 for specifying the appearance position of the keyword example from the searched text;
In the text acquired by the text search means 3, characters are acquired one by one from the appearance position of each keyword in the keyword example toward the beginning of the text, and the acquired characters are compared. A keyword appearance extracting means 5 for identifying an appearance pattern common to the keyword examples by acquiring characters and repeating the comparison ;
A keyword candidate extraction unit 6 that analyzes text based on the appearance position of each keyword in the keyword example and an appearance pattern common to the keyword example, and extracts keyword candidates from the text;
For each keyword candidate keyword, the number of texts N _A acquired by the text search unit, the number of texts N _A ^W including the keyword candidates among the texts acquired by the text search unit 3, and the text storage unit 2 evaluation value for each keyword from the number N ^W of stored electrical text ^{_{(N a W / N a)}} × sought ^{_{log (N a W / N W}} ), keywords higher keyword than the evaluation value is a predetermined threshold value And keyword extracting means 7 for extracting from candidates .

The present invention (Claim 3) is a vocabulary acquisition program for causing a computer to function as each means constituting the vocabulary acquisition apparatus according to claim 2 .

The present invention (Claim 4) is a computer-readable storage medium storing the vocabulary acquisition program according to Claim 3 .

  According to the present invention, based on a keyword having a small number of specific attributes, a rule and a keyword candidate appear with high accuracy by automatically extracting the position and pattern in which the keyword of the corresponding attribute appears, and using these two features. The keyword that matches the rule specified above is extracted from the text including a plurality of keywords specified in advance, and the final extraction candidate is extracted based on the appearance frequency and distribution of each keyword extracted here. By specifying a keyword, it is possible to acquire a vocabulary with high accuracy.

  By using the dictionary obtained by this vocabulary acquisition, it is possible to extract keywords having specific attributes from the text.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows a configuration of a vocabulary acquisition apparatus according to an embodiment of the present invention.

  The vocabulary acquisition device 10 shown in FIG. 1 is connected to an example data input device 20 for inputting a keyword having an attribute assumed to be extracted, and the Internet 30, and is connected to a text collection unit 1, a text storage unit 2, a text search unit 3, a keyword. The position specifying unit 4, the keyword appearance pattern extracting unit 5, the keyword candidate extracting unit 6, and the keyword extracting unit 7 are configured.

  The text collection unit 1 collects HTML or XML text data existing on the Internet 30 and stores it in the text storage unit 2. If there is already collected text on the local disk, the text is collected and stored in the text storage unit 2.

  The text storage unit 2 stores the text data collected by the text collection unit 1. The stored text data is text data in HTML format or text data with tags in XML format.

  The text search unit 3 searches the text data stored in the text storage unit 2 based on a keyword example that is input from the data example input device 20 and includes a proper noun such as a place or a person, an attribute, and the like. Identify text data that contains more than one.

  The keyword position specifying unit 4 analyzes text including a plurality of keywords included in the keyword example input from the data example input device 20 by the text search unit 3 and specifies a position where the keyword exists. An example of specifying the position is shown in FIG.

In the case of HTML shown in FIG. 4 (a), when “Koi Shi Kimi To” and “Mito Komon-sama” are given as keyword examples, the part including the keyword example is specified, and the HTML tag structure is a tree structure And the determination is made that “the corresponding data exists under the td tag, below the tr tag, below the BODY tag below the HTML tag”. This process is performed for each keyword. as a result,
"/ HTML / BODY / tr / td"
Is obtained.

In the case of HTML shown in FIG. 4 (b), when “Koi Shi Kimi” and “Mito Komon-sama” are given as keyword examples, the position where the keyword example exists is as follows:
"/ HTML / BODY /"
Is obtained. The appearance positions of the obtained keyword examples are transferred to the keyword candidate extraction unit 6.

  The keyword appearance pattern extraction unit 5 analyzes text including a plurality of keyword examples input from the data input device 20 searched by the text search unit 3, and specifies a pattern in which the keyword appears. For example, identify the parts that contain the keywords "Koshi to you" and "Mito Komon-sama", and for each keyword, get the characters one by one toward the beginning of the document, compare the keywords, and use the same characters. For example, the next pattern is obtained and the same pattern is extracted by repeating the comparison.

In FIG. 5 (a), when “Kisoshi-kun” and “Mito Komon-sama” are given as keyword examples,
"<Tr><td> ENTITY </ td><td>"
Is obtained. Note that the “ENTITY” part in the appearance pattern is a part that includes the vocabulary to be finally extracted.

In addition, in FIG. 5B, when “Koisushi-kun” and “Mito Komon-sama” are given as keyword examples,
“, ENTITY,”
Is obtained. The obtained appearance pattern of the keyword example is transferred to the keyword candidate extraction unit 6. Note that the “ENTITY” part in the appearance pattern is a part that includes the vocabulary to be finally extracted.

  The keyword candidate extraction unit 6 is a text searched by the text search unit 3 based on the position of the keyword example extracted by the keyword position specifying unit 4 and the appearance pattern of the keyword example obtained by the keyword appearance pattern extraction unit 5. To extract keyword candidates.

  The keyword extraction unit 7 specifies a keyword to be extracted from the keyword candidates extracted by the keyword candidate extraction unit 6 based on the keyword appearance frequency and distribution evaluation.

  As a criterion for evaluating keyword candidates, the following criterion α can be considered.

ここで、
・Ｎ_Ａは、テキスト検索部３によって特定された解析対象テキストの数を示す。
・Ｎ_Ａ ^ｗは、テキスト検索部３によって特定された解析対象テキスト中で現在評価するキーワードを含む文書の数を示す。
・Ｎ^ｗは、文書集合全体中で現在評価するキーワードｗを含む文書の数を示す。

here,
N _A indicates the number of analysis target texts specified by the text search unit 3.
N _A ^w indicates the number of documents including the keyword currently evaluated in the analysis target text specified by the text search unit 3.
^Nw indicates the number of documents including the keyword w currently evaluated in the entire document set.

  A keyword whose evaluation value obtained as described above is higher than a predetermined threshold is extracted as a keyword.

  Next, the operation in the above configuration will be described.

  The present invention is divided into two stages: a pre-processing stage for collecting data and an actual vocabulary acquisition stage.

  FIG. 6 is a flowchart of the preprocessing in the embodiment of the present invention.

  Step 101) The text collection unit 1 receives an input of a URL from an input device (not shown), collects the text while following a link starting from the URL, and stores the collected text in the text storage unit 2.

  Step 102) The text search unit 3 reads out the document from the text storage unit 2, analyzes it, and creates an index for specifying the document including the keyword specified from the input device (not shown).

  Next, processing for acquiring a vocabulary will be described.

  FIG. 7 is a vocabulary acquisition flowchart in one embodiment of the present invention.

  Step 201) The text search unit 3 receives an input of at least one keyword example as an example of the extracted vocabulary from the data example input device.

  Step 202) The text search unit 3 searches the text storage unit 2 based on the accepted keyword example, and acquires text including the keyword example.

  Step 203) The keyword position specifying unit 4 analyzes the text searched by the text search unit 3 one by one, and specifies the position where the keyword example input from the data example input device 20 appears for each individual text. .

  Step 204) The keyword appearance pattern extraction unit 4 analyzes the text searched by the text search unit 3 one by one, and for each text, shows an occurrence pattern in which the keyword example input from the data example input device 20 exists. Identify.

  Step 205) The keyword candidate extracting unit 6 extracts keyword candidates from the text searched in Step 202 based on the position information specified by the keyword position specifying unit 4 and the pattern specified by the keyword appearance pattern extracting unit 5. To do.

  Step 206) The keyword extraction unit 7 includes the number of texts to be analyzed searched by the text search unit 3, the number of documents including the keywords extracted by the keyword candidate extraction unit 6, and the entire extracted document set. An evaluation value for evaluating a keyword candidate is obtained using the number of documents including the keyword to be evaluated, and a keyword candidate having a threshold value equal to or higher than a preset threshold value is extracted as a keyword. A keyword whose evaluation value is less than a predetermined threshold is regarded as an unnecessary word and is deleted from the candidates. This threshold is set empirically. As another method, the appearance frequency of a keyword extracted from text is calculated, and if the frequency does not exceed a threshold value, it can be considered as an unnecessary word and deleted from candidates. Similarly, this threshold is set empirically.

  Thereafter, the keyword extracted by the keyword extraction unit 7 is stored in the storage means 40 such as a dictionary. In addition, you may make it display once on a display apparatus, before storing in a memory | storage means.

  The present invention is not limited to the above-described embodiments and examples, and various modifications and applications can be made within the scope of the claims.

  The present invention is applicable to information retrieval technology in a computer network.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a block diagram of the vocabulary acquisition apparatus in one embodiment of this invention. It is an example of operation | movement of the keyword position specific | specification part in one embodiment of this invention. It is an example of operation | movement of the keyword appearance pattern extraction part in one embodiment of this invention. It is a flowchart of the pre-process in one embodiment of this invention. It is a flowchart of the vocabulary acquisition in one embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Text collection part 2 Text storage means, Text storage part 3 Text search means, Text search part 4 Keyword position specification means, Keyword position specification part 5 Keyword appearance pattern extraction means, Keyword appearance pattern extraction part 6 Keyword candidate extraction means, keyword candidate Extraction unit 7 Keyword extraction means, keyword extraction unit 10 Vocabulary acquisition device 20 Data example input device 30 Internet 40 Storage device

Claims (4)

Keyword Examples of specific attributes analyzes electronic text including HTML or XML based, a lexical acquisition method in an apparatus for acquiring vocabulary,
A text search step in which text search means searches for text storage means for storing electronic text including HTML and XML based on a keyword example of a specific attribute input from the outside, and acquires text including the keyword example When,
A keyword position specifying unit for specifying an appearance position of the keyword example from the searched text;
The keyword appearance pattern extracting means obtains characters one by one from the appearance position of each keyword in the keyword example toward the beginning of the text in the text obtained in the text search step, and compares the obtained characters. If it is the same character, the next character is obtained, and by repeating the comparison, a keyword appearance extraction step for specifying an appearance pattern common to the keyword example;
A keyword candidate extracting unit that analyzes the text based on an appearance position of each keyword of the keyword example and an appearance pattern common to the keyword example, and extracts a keyword candidate from the text;
For each keyword candidate keyword , the keyword extraction means uses the number N _A of texts acquired in the text search step, and the number N _A ^{W of} texts including the keyword candidates among the texts acquired in the text search step. the calculated evaluation value for each keyword from the number N ^W of the text stored in the text storage means ^{_{(N a W / N a)}} × log (N a W / N W), the evaluation value is above a predetermined threshold value A keyword extraction step of extracting a higher keyword from the keyword candidates ;
A vocabulary acquisition method characterized by: Keyword Examples of specific attributes to analyze the electronic text, including HTML and XML based on, a lexical acquisition apparatus to acquire the vocabulary,
Text storage means for storing the electronic text containing the HTML or the XML,
A text search unit that searches the text storage unit based on a keyword example of a specific attribute input from the outside, and obtains text including the keyword example;
Keyword position specifying means for specifying the appearance position of the keyword example from the searched text;
In the text acquired by the text search means, characters are acquired one by one from the appearance position of each keyword in the keyword example toward the beginning of the text, and the acquired characters are compared with each other. A keyword appearance extraction means for identifying an appearance pattern common to the keyword example by repeating the comparison and
A keyword candidate extracting means for analyzing the text based on an appearance position of each keyword of the keyword example and an appearance pattern common to the keyword example, and extracting a keyword candidate from the text;
For each keyword candidate keyword, the number N _A of texts acquired by the text search unit, the number N _A ^{W of} texts including the keyword candidate among the texts acquired by the text search unit, the text storage unit evaluation value for each keyword from the number N ^W of the text stored in the ^{_{(N a W / N a)}} × sought ^{_{log (N a W / N W}} ), the higher keyword than the evaluation value is a predetermined threshold value Keyword extraction means for extracting from keyword candidates ;
A vocabulary acquisition device characterized by comprising: A vocabulary acquisition program for causing a computer to function as each means constituting the vocabulary acquisition apparatus according to claim 2 . A computer-readable storage medium storing the vocabulary acquisition program according to claim 3 .

Images