JP5559104B2 - Information extraction method, information extraction apparatus, and information extraction program - Google Patents

Description

  The present invention relates to a technique for extracting a text from a structured document such as a Web page.

  In recent years, it has become possible to use a large amount of digitized documents and send information to unspecified large numbers of people through a computer network such as the Internet. An electronic document expressed on a computer network uses an expression that makes use of its characteristics, and a Web page on the WWW (World Wide Web) often uses a structured document described in a markup language. ing.

  For example, a document called HTML (HyperText Markup Language) has a function of “hyperlink” for referring to a document described by another person existing on another computer as well as describing some information. Have. This is used when, for example, trusting another document to supplement the information described by itself or referring to a document having the same content.

  On the other hand, the information of the HTML file may describe information unrelated to the information to be originally expressed, such as an “advertisement” portion or a “menu” portion. Therefore, in search services and analysis of access logs to HTML files, it is necessary to appropriately extract information that the HTML file originally intended to express. The part extracted here is called “text”, and the extraction of “text” is called “text extraction”.

  As an existing technique for extracting a text by using HTML tag information described in an HTML file, there is “Web scraping (sometimes expressed simply as scraping)”. To do this scraping, specify which part of the HTML (XML) file should be extracted by XSLT (XML Style Language Transformations), that is, a simple language for converting a document written in XML into another XML document How to do is known. This method is shown in Non-Patent Document 1.

Author: Erik T. Ray, Translated by: Kazuhiko Yamamoto, Junji Nakahara, Masaru Kajiura, “Introductory XML”, Ohm Co., Ltd., September 28, 2001, p. 203-241 "Element-parent relationship / HTML basic course", "online", first edition release date May 18, 2002 Latest update date March 7, 2004 [Search July 13, 2011], Internet <URL: http : //www,scollabo.com/banban/lectur/ht6.html> "Block level elements and inline elements", "online", updated April 5, 2001, [searched July 13, 2011], Internet <URL: http://www.kanzaki.com/docs/html/element -level.html>

  However, the method of Non-Patent Document 1 can collect all HTML on the Internet, although it is possible to prepare “rules” written in XSLT according to the HTML file to be extracted if it is a decimal number. When there are a large number of HTML files such as, a rule for performing text extraction for each HTML file must be prepared in advance, which is very difficult.

  Also, once prepared text extraction rules are not guaranteed to continue to be used forever, and there is a possibility that the text cannot be extracted if the description content of the HTML file is changed.

  The present invention has been made in order to solve the above-described problems of the prior art, and it is an object of the present invention to perform text extraction from a structured document without depending on text extraction rules.

  Therefore, the present invention extracts the information portion existing in the reference source structured document from the reference destination structured document using the reference relationship of the structured document. In other words, by extracting the part having information existing in the structured document of the reference source (link source) from the structured document of the reference destination (link destination), information on unnecessary parts such as the advertisement part and the menu part is eliminated. And only the text is extracted.

  The information extraction method according to the present invention includes a link source information extraction step for extracting a link existing in each structured document and the link peripheral text information for the collected structured document group, and link source information A text extraction step for identifying a linked structured document based on the link extracted in the extracting step, and extracting a representative part of the linked structured document including link peripheral text information extracted in the step as a text And having.

  In addition, the information extraction apparatus of the present invention includes a link source information extraction unit that extracts a link existing in each structured document and the link peripheral text information with respect to the collected structured document group, and a link Based on the link extracted by the original information extraction unit, the structured document of the link destination is specified, and a representative part in the structured document of the link destination including the link peripheral text information extracted by the information extraction unit is extracted as the text. A text extraction unit.

  The present invention may be a program that causes a computer to function as the device. This program can be provided through a network or a recording medium.

  According to the present invention, text extraction can be performed from a structured document without depending on text extraction rules.

The block diagram of the information extraction device which concerns on embodiment of this invention. The flowchart which shows the process of the link source information extraction part. Example document containing reference information by hyperlink. (A) is an example of an HTML file (source) of document A in FIG. 3, and (b) is an example of an HTML file (source) of document B in FIG. The flowchart which shows the process of a text extraction part.

  Hereinafter, an information extraction apparatus according to an embodiment of the present invention will be described. This information extraction apparatus processes structured documents described in a markup language, mainly Web pages on the WWW, and performs text extraction using reference expressions.

  Here, a processing example based on a reference expression by a hyperlink of an HTML document (HTML document) will be described. That is, the text is extracted from the linked HTML file by using the linked HTML file and the linked HTML file. At this time, when the information described in the link source HTML file is extracted and the extracted information exists in the link destination HTML file, the corresponding tag (for example, all the extracted information from the link source file is included). Pick up the tag).

≪Configuration example≫
A configuration example of the information extraction apparatus will be described with reference to FIG. This information extraction apparatus 1 is configured by a computer, and is a normal computer hardware resource such as a CPU. A storage device such as a memory (RAM) or a hard disk drive device is provided.

  As a result of the cooperation between the hardware resource and the software resource (OS. Application, etc.), the information extraction apparatus 1 has the document set recording unit 2. Document set DB3. 3. Link source information extraction unit 4. Text extraction unit The output unit 6 is mounted. The recording unit 2 records information on a document to be processed input through an input unit (not shown), such as an HTML file, in the DB 3. This HTML file may be collected by search engine crawling or the like. Note that the DB 3 is constructed in the storage device.

  The information extraction unit 4 extracts a hyperlink that realizes a reference relationship existing in the HTML file of the document recorded in the DB 3 and a peripheral text of the link (hereinafter referred to as a link peripheral text). Here, a character string surrounded by hyperlinks may be extracted, or based on the hierarchical structure (parent-child relationship) of elements (tags) in HTML, the upper nth (n = positive integer) including the URL of the linked document You may extract the character string which exists in a block element.

  If the hyperlink that refers to the HTML file for the document recorded in the DB 3 is extracted by the information extracting unit 4, the body extracting unit 5 identifies the document as a linked document. The character string of the text information existing in the HTML file of the link destination document specified here is compared with the character string of the link peripheral text extracted by the information extraction unit 4, and a representative portion in the link destination document is determined. Extract as text. Here, the character string existing under the top N-th (N = positive integer) element (parent tag) including all the character strings extracted by the information extraction unit 4 is extracted.

  Therefore, information on unnecessary parts such as an advertisement part and a menu part is excluded, and only a part corresponding to the text is extracted. The extracted text is output through the output unit 6 as, for example, a part representing HTML for creating a search index (index) in a search engine. Hereinafter, the details of the processing of the information extraction device 1 will be described by roughly dividing the information extraction unit 4 and the text extraction unit 5.

<< Processing Example of Information Extraction Unit 4 >>
First, a processing example of the information extraction unit 4 will be described with reference to FIG. Here, the documents A to D in FIG. The documents A to D are HTML files provided on a site on the Internet where a product purchase service is performed. The document A relates to a document listing the products, and the documents B to D are products 1 to 3. It relates to a document explaining the details.

  In FIG. 3, the underline (underline) of the product names 1 to 3 of the document A is the document B. A hyperlink that realizes reference to C is shown, the hyperlink of product name 1 refers to document B, the hyperlink of product name 2 refers to document C, and the hyperlink of product name 3 refers to document D Yes. It is assumed that the HTML files of the documents A to D are recorded in the DB2.

First, when the process starts, the information extraction unit 4 acquires an HTML file to be processed from the DB 3 (S01). Here, as an example, an HTML file of document A is acquired. Thereafter, it is confirmed whether or not a hyperlink exists in the HTML file acquired in S01 (S02). If there is no hyperlink as a result of the confirmation, the processing of the information extraction unit 4 is terminated. On the other hand, if a hyperlink exists, the hyperlink and link peripheral text are extracted (S03), and the process ends. This extraction information is defined by either (1) or (2) below.
(1) Character string (character string including URL) enclosed by hyperlink (an HTML tag ANCHOR tag)
(2) Character string existing in the block element that includes the URL of the hyperlink destination document and appears in the top nth (n = positive integer) This definition (1) Selection of (2) or “value of n” The range of extraction information can be changed depending on For example, if the definition (1) is selected, the minimum range, that is, “hyperlink (a character string surrounded by the HTML tag of the HTML tag)” is extracted. Also, the definition (2) is selected and “n = maximum value ( The maximum range, that is, “the full text (the entire contents of the BODY tag of the HTML tag)” is extracted. Whether the HTML tag “<>” is included in the extracted character string It is possible to give these conditions.

  Here, as shown in Non-Patent Document 2, HTML elements (tags) are expressed in a hierarchical structure such that a certain element includes an element and the element includes another element. This hierarchical structure is generally expressed as a parent-child relationship with parent elements, child elements, grandchild elements, etc., and each element has a parent-child relationship. The top nth of definition (2) is the parent-child relationship of the elements Is shown.

  As the types of elements, as shown in Non-Patent Document 3, a block element that constitutes a block (heading, paragraph, etc.) on the display, and an inline element that appears to be continuous with the block element on the display, Exists. This will be described with reference to FIG. 4A. In the HTML file of document A, DIV corresponds to a block element, and SPAN and A (ANCHOR) correspond to inline elements.

  At this time, from the viewpoint of the A (ANCHOR) tag, the DIV tag with “id = TR” corresponds to the block element that appears first, and the DIV tag with “id = TABLE” appears as the block element that appears second. It corresponds to. It is assumed that the href attribute of the A (ANCHOR) tag specifies a link destination URL.

  Here, an example of extracting the document A from the HTML file will be described. As shown in FIG. 4A, since there are three A (ANCHOR) tags in the HTML file of document A, it is confirmed in S02 that three hyperlinks are buried. Each A (ANCHOR) tag refers to the documents B to D as indicated by the href attribute. Here, as an example, it is assumed that the extraction range from the link source document in S03 is set in the definition (2) and “n = 1” is set. It is also assumed that a condition “only human-readable text not including tags” is given.

  At this time, from the viewpoint of each A (ANCHOR tag), each DIV tag of “id = TR” corresponds to the block element that appears first in the top, so that the character strings surrounded by the subordinate SPAN tags are respectively extracted. Is done. Therefore, for the document B, the URL of the document B shown in the upper “DIV id = TR”, that is, the href attribute, and the link peripheral text of “product name 1” “price: 100 yen” “color: red” are extracted. Is done.

  For document C, the URL of document C shown in the middle “DIV id = TR”, that is, the href attribute, and the link peripheral text of “product name 2”, “price: 300 yen”, and “color: blue” are extracted. Is done.

  Further, for document D, the lower part of “DIV id = TR”, that is, the URL of document D shown in the href attribute and the link peripheral text of “product name 3”, “price: 300 yen”, and “color: yellow” are extracted. Is done. It is assumed that the URL of the extracted link destination document and link peripheral text are stored in the storage device.

≪Processing content of text extraction unit 5≫
Next, a processing example of the text extracting unit 5 will be described with reference to FIG. When the process is started, the text extracting unit 5 acquires an HTML file to be processed from the DB 2 (S11). Here, as an example, it is assumed that HTML files of documents B to D are acquired.

  It is confirmed whether or not a hyperlink referring to the acquired HTML file is extracted in S03 (S12). For this confirmation, the URL of the link destination document stored in the storage device may be used. As a result, if the presence of the hyperlink is not confirmed, the process is terminated, and the process returns to S11 to acquire the next HTML file.

  On the other hand, if the existence of the hyperlink is confirmed, the acquired HTML file is identified as the link destination document, the link peripheral text extracted in S03 is acquired from the storage device, and the link destination document including the link peripheral text is acquired. The representative part is extracted as the text (S13). At this time, it is necessary to have a condition as to how much of the linked document is a representative part, that is, a text. Basically, it may be a part including the character string of the link peripheral text extracted in S03, but if “all character strings” of the link destination HTML file is used, the data capacity increases due to unnecessary information. There is a fear.

  Therefore, as the range designation of the representative part extracted from the body, the parent element (tag) that includes all the character strings of the link peripheral text extracted in S03 and appears in the top Nth (N = positive integer) is used. Search and use the character string under the search as the text. The text extraction character string extracted from the HTML file can be changed by changing the text extraction condition, that is, the value of N. At this time, it is also possible to give a condition as to whether or not an HTML tag is included in the text.

  A processing example for the HTML file of document B will be described below. Here, the text extraction condition is set to “N = 1”, and the parent element that appears first is searched. In addition, the hyperlink extraction of S03 is confirmed in document B in S12, and the link peripheral text “product name 1” “price: 100 yen” “color: red” extracted in S03 is stored in the storage device. It shall be.

  At this time, in the HTML of document B, as shown in FIG. 4B, “product name 1” is sandwiched between SPAN tags of “id = name”, and “price: 100 yen” is “id = price”. “Color: Red” is sandwiched between SPAN tags of “id = color”, and a “DIV” tag is acquired as a search result of “the first top parent tag including all”. .

  Therefore, the “character string existing under the subordinate” corresponds to the “character string existing under the DIV tag”. Here, it is assumed that the condition “only human-readable text including no tags” is given. As a result, the character string “product name 1, photo, model number: 123456, price: 100 yen, color: red, outline: sold” is extracted as a representative part of the document B, that is, the text. At this time, if a condition for extraction including an HTML tag is given, a character string with a tag is extracted. The extracted text is output to the output unit 6 (S14), and the process ends. The text output to the output unit 6 is provided to a search engine or the like, and is used for grasping information originally intended to be expressed in the HTML file when creating a search index.

  As described above, according to the information extracting apparatus 1, the text can be extracted by eliminating the information of the unnecessary part such as the advertisement part and the menu part without depending on the conventional rule for extracting the text. Maintenance costs are reduced. In particular, it is also useful in that text extraction can be realized flexibly according to the update of HTML files generated by others on the Internet.

  In addition, this invention is not limited to the said embodiment, It can deform | transform and implement within the range described in each claim. For example, the definitions (1) and (2) in S03, the “n value”, and the text extraction condition “N value” in S13 may be different for each link source HTML file, and may be different for each link destination HTML file. Also good. That is, it may be the same for all documents to be processed, or may be individually selected and set for each Internet site.

  However, in order to unify the information extracted as the text, it is preferable that the definitions (1) and (2) are selected and the “n value” setting is the same in all link source HTML files. Similarly, it is preferable that the setting of the text extraction condition “value of N” is the same in all link destination HTML files.

  Further, the present invention is not limited to HTML documents, and structured documents described in other markup languages such as XML can be processed. In this case, “XML Linking Language (XML Linking Language)” that defines a link between XML documents (documents) may be used.

≪Programs≫
The present invention can also be configured as a document search program that causes a computer to function as some or all of the units 2 to 6 of the information extraction apparatus 1. According to this program, S01 to S03. It becomes possible to cause the computer to execute part or all of S11 to S14.

  The program can be provided through a network such as a website or e-mail. The program is stored in a recording medium such as a CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, MO, HDD, BD-ROM, BD-R, or BD-RE. It is also possible to record, save and distribute. This recording medium is read using a recording medium driving device, and the program code itself realizes the processing of the above embodiment, so that the recording medium also constitutes the present invention.

DESCRIPTION OF SYMBOLS 1 ... Information extraction apparatus 2 ... Document set recording part 3 ... Document set DB
4 ... Link source information extraction unit 5 ... Text extraction unit 6 ... Output unit

Claims (3)

An information extraction method of an apparatus for extracting a text from a structured document of a link destination based on a structured document in which a link realizing a reference relationship is expressed,
A link source information extracting step for extracting a link existing in each structured document and the link peripheral text information with respect to the collected structured document group;
Identifying a structured document destination based on the link extracted by the link source information extracting step, extracting the representative portion in the structured document landing including the link around the text information extracted by said step as the body And a text extraction step to
In the link source information extraction step, all of the tags sandwiched from the start point to the end point of the tag present in the upper nth (n = positive integer) block element from the link element including the URL of the linked structured document Extract the string,
In the body extracting step, a character string existing under the top Nth (N = positive integer) element including all the character strings extracted in the link source information extracting step is extracted as a representative part. Information extraction method. An information extraction device that extracts a text from a structured document of a link destination based on a structured document in which a link realizing a reference relationship is expressed,
A link source information extracting unit step for extracting a link existing in each structured document and the link peripheral text information with respect to the collected structured document group;
Identifying a structured document destination based on the link extracted by the link source information extracting step, extracting the representative portion in the structured document landing including the link around the text information extracted by said step as the body A text extraction unit to
The link source information extraction unit includes all the tags sandwiched from the start point to the end point of the tag existing in the upper nth (n = positive integer) block element from the link element including the URL of the linked structured document . Extract the string,
The text extraction unit extracts, as a representative part, a character string existing under the top Nth (N = positive integer) element including all the character strings extracted in the link source information extraction step. Information extraction device.   An information extraction program for causing a computer to function as each unit of the information extraction device according to claim 2.

Images