JP2013105364A - Document feature extraction device, document feature extraction method, and document feature extraction program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To appropriately extract a feature corresponding to a browser's browsing intention using a reference relationship between structured documents.SOLUTION: A browsing history recording unit 2 of a document feature extraction device 1 records a browsing history of each browser in a browsing history set DB 3. A feature extraction unit 4 extracts a link and related text of the link from a structured document as a link source included in the browsing history in the DB 3. Words are extracted from body text as a representative portion in a structured document as a link destination including the extracted information. A feature recalculation unit 5 calculates weighting for the extracted words. An output unit 6 outputs the extracted words in a priority order corresponding to the weighting.

Description

  The present invention relates to a technique for extracting features from a structured document such as a Web page.

  In recent years, it has become possible to use a large amount of electronic documents (so-called electronic documents: hereinafter abbreviated as documents) and to send information to unspecified large numbers of people through computer networks such as the Internet. Such a document expressed on a computer network uses an expression that makes use of its features, and many structured documents are described in a markup language on a Web page on the WWW (World Wide Web). It's being used.

  For example, a document called HTML (HyperText Markup Language) has a function of “hyperlink” for referring to a document described by another person existing on another computer as well as describing some information. Have. This is used when, for example, trusting another document to supplement the information described by itself or referring to a document having the same content.

  In addition, as one of Web services realized on the Internet, there is a service that provides an automatically edited Web page for a viewer. For example, a method is mainly adopted in which the user registers information that has been browsed in advance and provides the information to the user when information that matches the registration is collected.

  However, forcing registration of information limits the number of users, which is not realistic. Therefore, in the general Web site, the information browsed by the viewer is attempted to draw the viewer's intention from the browsing history based on the viewpoint of “selecting information desired by the viewer”. For example, a technique has been proposed in which words existing in each page are taken out for all the read documents, and importance is given to the taken-out words. Non-Patent Document 1 proposes a technique for recording the contents of browsing records when clipping (bookmarking) a certain individual.

  By extracting the “characteristics of the document based on the browsing history” of such a viewer, it is determined whether or not the document should be presented to the viewer by defining similarity to the document that the viewer has not browsed. Judgment can be made.

Shinsuke Kuroda, Shinsuke Nakajima, Kazutoshi Tsunoya, Katsumi Tanaka “Sharing and Searching User Activities in the Web Environment” 13th Data Engineering Workshop (DEWS2002) Proceedings, A2-4 (2002)

  However, in order to appropriately provide information browsed by the viewer, it is necessary to acquire the viewer's request in some form. As a simple method for this, “extraction of words expressing features based on browsing history” can be considered, but appropriate “word extraction” is generally difficult in many cases.

  For example, when trying to extract words in “all browsing records”, the word extraction target is “entire browsing document”, so word extraction from the “menu” portion and “advertisement” portion other than the body text in the HTML file is performed. In addition, there is a problem that an unnecessary word must be processed from the viewpoint of filling in an intention of browsing because unnecessary words are buried.

  One method for solving this problem is the extraction of partial document information centered on the “anchor text” of Non-Patent Document 1, but in this case, if the role is different for each link, the viewer's request is appropriate. It may be difficult to pump out.

  The present invention has been made to solve the above-described problems of the prior art, and more appropriately extracts features corresponding to the browsing intention of the viewer by using the reference relationship of the structured document. Is a solution issue.

  Therefore, the present invention extracts links existing in the structured document included in the browsing history and related text of the links. For example, one of the character strings existing in the upper nth (n = integer) block element including the character string surrounded by the link and the URL of the structured document of the link destination is extracted.

  A representative portion in the linked structured document including the character string of the extracted information is extracted as the text. For example, a character string existing under the top Nth (N = integer) element (tag) including all the extracted link character strings is extracted as the body text of the linked structured document. Words are extracted from the text as features by general natural language processing such as morphological analysis. As a result, it is possible to suppress extraction of unnecessary words and extract only information according to the intention of the viewer.

  Further, the total features are totaled from the features of each structured document extracted by the feature extraction means, and the features of each document are weighted and recalculated. For example, prioritize the “characteristic words” of each document by assigning weights to the words extracted from a document in consideration of the order in which the words extracted from previous browsing documents are clicked. It becomes possible.

  According to the present invention, it is possible to more appropriately extract a feature corresponding to a browsing intention of a viewer by using a reference relationship of a structured document.

1 is a block diagram showing a configuration of a document feature extraction apparatus according to an embodiment of the present invention. The processing flow figure of the same feature extraction part. Example document containing reference information by hyperlink. (A) is a source example of the HTML file (y) of the document A in FIG. 3, and (b) is a source example of the HTML file (x) of the document B in FIG. Browsing history example of documents containing reference information

  Hereinafter, a document feature extraction apparatus according to an embodiment of the present invention will be described. This feature extraction apparatus uses a structured document described in a markup language, mainly a Web page on the WWW, as a processing target, and extracts a document feature using a reference expression. The extracted document features are used for recommending information to the viewer.

  Here, a processing example for an HTML file will be described. That is, in consideration of the browsing history, the feature word of the link destination document is extracted by reflecting the portion having information existing in the link source document in the link destination document.

≪Device configuration≫
A configuration example of the feature extraction apparatus will be described with reference to FIG. The feature extraction apparatus 1 is constituted by a computer, and is a normal computer hardware resource such as a CPU. A storage device such as a memory (RAM) or a hard disk drive device is provided.

  As a result of the cooperation between the hardware resource and the software resource (OS. Application, etc.), the feature extraction apparatus 1 has the browsing history recording unit 2. Browsing history set DB3. Feature extraction unit 4. 4. Feature recalculation unit The output unit 6 is mounted. This DB3 is assumed to be constructed in the storage device.

  The recording unit 2 records, in the DB 3, a browsing history for each browsing person who constitutes a processing target, that is, a document set of browsed HTML files. Each browsing history is input to the feature extraction apparatus 1 through an input unit (not shown), and a log such as a search engine can be used. Specifically, the recording unit 2 performs “recording date” and “browsing document” for each viewer, and when the viewer clicks to facilitate the analysis processing of the feature extraction unit 4 and the feature recalculation unit 5. The anchor text, that is, the character string enclosed in the <a> tag (ANCHOR tag) of the HTML file is recorded.

  The feature extraction unit 4 uses an information extraction stage for extracting hyperlinks in the HTML file based on the browsing history of the DB 3 and a viewer from a linked HTML file included in the browsing history using the extracted hyperlink. The body extraction stage which extracts the body which wants is performed.

  The browsing history in each stage is basically limited. That is, it is not impossible to say “all” after the recording unit 2 records the browsing history in the DB 3, but it is preferable that the recording unit 2 is divided by a period such as “daily unit” or “time unit” or “search site”. It is assumed that “browsing history” in an arbitrary unit such as “from the start of searching until the browser is closed” is used. This selection is set according to the specifications of the apparatus. Hereinafter, the term “browsing history” is used to mean a browsing history of a set unit.

  For the processing of each stage, the information extraction method of Japanese Patent Application No. 2011-166460 filed earlier by the applicant of the present application is used. First, in the information extraction stage, a hyperlink that realizes a reference relationship existing in the HTML file of the link source document included in the browsing history and related text around the link (hereinafter referred to as link related text). Extract.

  To extract this hyperlink (HTML ANCHOR tag), the anchor text recorded in the browsing history may be used. Specifically, a character string surrounded by hyperlinks, that is, anchor text may be extracted, or the top nth including the URL of the linked document based on the hierarchical relationship (parent-child relationship) of elements (tags) in HTML It is assumed that a character string existing in the block element (n = positive integer) may be extracted.

  Next, in the text extraction stage, the character string of the text information existing in the linked document referred to by the hyperlink is compared with the character string of the extracted information in the information extracting stage, and a representative part in the linked document Is extracted as the text. Here, a character string existing under the top Nth (N = positive integer) element (parent tag) including all the character strings extracted in the information extraction stage is extracted.

  As a result, information on unnecessary parts such as an advertisement part and a menu part is excluded, and only a part corresponding to the text is extracted. The word is extracted as a feature word (characteristic word) by general natural language processing such as “morphological analysis” for the extracted text. As a result, unnecessary word extraction is suppressed, and only feature words that match the viewer's intention can be extracted.

  The feature recalculation unit 5 aggregates the feature quantities of the entire browsing history for the feature words extracted by the feature extraction unit 4, and calculates the weights of the document features of each HTML file. Here, the feature words extracted by the feature extraction unit 4 are weighted in consideration of the entire browsing history and the time axis of the number of documents having a click relationship.

  In the calculation, “words (feature words) of documents browsed in the past” and “number of occurrences of words (feature words) in the entire browsing history” are used. For example, when a large number of appearances appear in the past few clicks in the browsing history, the characteristic words existing in the document selected by the viewer can be more emphasized by setting the weight as “more characteristic words”.

  Moreover, either (a) when using the entire browsing history or (b) when using a specific document in the browsing history may be set. In this setting (a), it is desirable that words that are shared in the entire browsing history be “more characteristic feature words”. As a simple calculation method, it is conceivable to use a score obtained by summing up the number of appearances of feature words for each document in the browsing history.

  On the other hand, in the setting (b), it is desirable to calculate in such a direction that the characteristic words existing in the entire browsing history are excluded, in other words, the score becomes small. As a simple calculation method, the feature word score may be calculated based on Equation (1). Each feature word is weighted according to the calculated score.

  The output unit 6 outputs feature words in a priority order corresponding to the weighting by the feature recalculation unit 5. For example, it is possible to output the ranking according to the score. At this time, a feature word may be output for each document, or the output may be switched according to the settings (a) and (b). The output format may be any of monitor output, printer output, and output to other devices.

<< Processing Example of Feature Extraction Unit 4 >>
A processing example of the feature extraction unit 4 will be described with reference to FIG. Here, the documents A to D in FIG. 3 are used as browsing histories to be processed. The documents A to D are HTML files provided at a shopping site on the Internet. The document A relates to a document listing products such as product search results in the site, and the documents B to D are products 1 to 3. It relates to a document that explains the details of.

  In FIG. 3, the underline (underline) of the product names 1 to 3 of the document A indicates a hyperlink that realizes reference to the documents B to D, and the hyperlink of the product name 1 refers to the document B. The hyperlink refers to the document C, and the hyperlink of the product name 3 refers to the document D.

  S01: When the process is started, an HTML file (x) relating to the document to be processed is acquired from the browsing history of the DB 3 (S01). Here, as an example, an HTML file of document B is acquired.

  S02: It is confirmed whether the HTML file (y) immediately before reaching the HTML file (x) acquired in S01 exists in the browsing history of the DB3. As a result of the confirmation, if the HTML file (y) does not exist, the process ends. If it exists, the HTML file (y) is acquired from the browsing history of the DB 3 and the process proceeds to S03.

  Here, since document A refers to document B by a hyperlink, the HTML file corresponds to the immediately preceding HTML file in S02. Therefore, the HTML file of document A is acquired from the browsing history of DB3, and the process of S03 is started.

  S03: Hyperlinks and link-related text information are extracted from the HTML file (y) acquired in S02, and a representative part of the HTML file (x) is extracted as a text using the hyperlink and link-related text information. In S03, the information extraction stage and the text extraction stage described above are executed. Details of each stage will be described below.

Information extraction stage The information extraction stage extracts the extraction information defined in either of the following (1) and (2) from the HTML file (y) of the document A.
(1) Character string (character string including URL) enclosed by hyperlink (an HTML tag ANCHOR tag)
(2) A character string that includes the URL of the hyperlink destination document (document B) and that exists in the block element that appears in the top nth (n = positive integer). This extraction information (1) (2) The range of the extracted information can be changed depending on “value of n”. For example, if the definition (1) is selected, the minimum range, that is, “hyperlink (character string enclosed in an ANCHOR tag of HTML tag)” is extracted. On the other hand, if the definition (2) is selected and “n = maximum value (number of HTML element layers)”, the maximum range, that is, “the full text (all contents of the BODY tag of the HTML tag)” is extracted.

  Here, an HTML element (tag) is expressed in a hierarchical structure such that a certain element includes an element and the element includes another element. This hierarchical structure is generally expressed as a parent-child relationship with parent elements, child elements, grandchild elements, etc., and each element has a parent-child relationship. The top nth of definition (2) is the parent-child relationship of the elements Is shown.

  In addition, as types of elements, there are block elements constituting blocks on display (headings, paragraphs, etc.) and inline elements that appear to be continuous with block elements on display. This will be described with reference to FIG. 4A. In the HTML file (y) of document A, DIV corresponds to a block element, and SPAN and A (ANCHOR) correspond to inline elements.

  At this time, from the viewpoint of the A (ANCHOR) tag, the DIV tag with “id = TR” corresponds to the block element that appears first, and the DIV tag with “id = TABLE” appears as the block element that appears second. It corresponds to. It is assumed that the href attribute of the A (ANCHOR) tag specifies a link destination URL.

  An example of extracting the document A from the HTML file (y) will be described. Since the HTML file (y) of document A has three A (ANCHOR) tags as shown in FIG. 4A, it can be confirmed that three hyperlinks are buried. Each A (ANCHOR) tag refers to the documents B to D as shown in the respective href attributes. Here, as an example, it is assumed that the extraction range of the extraction information is set in the definition (2) and “n = 1”. It is also assumed that a condition “only human-readable text not including tags” is given.

  At this time, from the viewpoint of each A (ANCHOR tag), each DIV tag of “id = TR” corresponds to the block element that appears first in the top, so that the character strings surrounded by the subordinate SPAN tags are respectively extracted. Is done. For example, for the document B, the URL of the document B shown in the upper “DIV id = TR”, that is, the href attribute, and the link peripheral text of “product name 1” “price: 100 yen” “color: red” Extracted.

-Text Extraction Stage The text extraction stage extracts a link destination document including all character strings of extracted information in the information extraction stage, that is, a representative part of the HTML file (x) as a text. At this time, it is necessary to have a condition as to how much of the linked document is a representative part, that is, a text. Basically, any part including the character string of the extracted information at the information extraction stage may be used, but if “all character strings” in the HTML file (x) is used, there is a possibility of increasing the data capacity due to useless information. is there.

  Therefore, as a range specification of the representative part extracted from the text, a search is made for a parent element (tag) that includes all the character strings of the extracted information in the information extraction stage and appears in the top Nth (N = positive integer). , The character string existing under it is used as the text. The text extraction character string extracted from the HTML file can be changed by changing the text extraction condition, that is, the value of N.

  A processing example for the HTML file (x) of document B will be described below. Here, the text extraction condition is set to “N = 1”, and the parent element that appears first is searched. Further, it is assumed that extraction information of “product name 1”, “price: 100 yen”, and “color: red” extracted at the information extraction stage is stored in the storage device.

  At this time, in the HTML file (x) of document B, as shown in FIG. 4B, “product name 1” is sandwiched between SPAN tags of “id = name”, and “price: 100 yen” is “id = “Price” SPAN tag, “color: red” is sandwiched between “id = color” SPAN tags, and “DIV” tag as a search result of “top first parent tag including all” Is acquired.

  Therefore, the “character string existing under the subordinate” corresponds to the “character string existing under the DIV tag”. Here, it is assumed that the condition “only human-readable text including no tags” is given. As a result, the character string “product name 1, photo, model number: 123456, price: 100 yen, color: red, outline: sold” is extracted as a representative part of the document B, that is, the text.

  S04: A word is extracted as a feature word (characteristic word) by general natural language processing such as “morphological analysis” for the text extracted in S03, and the process is terminated.

<< Processing Example of Feature Recalculation Unit 5 >>
A processing example of the feature recalculation unit 5 will be described based on the browsing record of FIG. It is assumed that this browsing history is recorded in the DB 3 as the browsing history of the viewer A.

  In FIG. 5, document (a) shows a search page [http://a.co.jp/top.htm] of a shopping site on the Internet, and document (b) is a search from the search page of document (a). The search result page [http://a.co.jp/detail.html] according to the request is shown.

  Document (c) shows a detailed page [http://a.co.jp/detail.html] of product A referenced by the hyperlink of “product A” displayed in document (b). (D) shows a tanks page [http://a.co.jp/purchase.html] referred to by the “purchase” hyperlink displayed in the document (c).

  Therefore, according to the browsing history of FIG. 5, as shown by the arrow (1), the viewer A requests a search for “product A” on the search page of the shopping site on the Internet, that is, the document (a), and the document The search result page of (b) was browsed. After browsing, the hyperlink of “product A” displayed in the document (b) was clicked, and the document (c) related to the detailed page of the product A was displayed and browsed as shown by the arrow (2).

  In order to purchase the product A after browsing, the hyperlink “purchase” displayed in the document (c) is clicked, and as shown by the arrow (3), the tanks page where the product A is purchased, that is, The document (d) was displayed and browsed.

  The feature extraction unit 4 processes the browsing history documents (a) to (d) recorded in the DB 3 and uses the “link source-link destination” relationship as shown in S01 to S04. The feature words b) to (d) are extracted. Here, “product A” is extracted as a feature word from document (b), “product A”, “Y shirt”, “pink” are extracted as feature words from document (c), and “product A” is extracted from document (d). It is assumed that “purchase content” and “product A” are extracted as feature words.

  The feature recalculation unit 5 calculates weights for the feature words extracted for each document by the feature extraction unit 4. In this weighting calculation, as described above, “words (feature words) of documents browsed in the past” and “number of occurrences of words (feature words) in the entire history” are used. Here, as an example, the weight of the feature word of the document (c) is calculated by using “past history twice” as the “document browsed in the past”. This calculation example corresponds to the setting (b) because the number of appearances of feature words is counted using the documents (a) and (b).

  Then, when calculating the weighting for the feature word of the document (c), the weighting score of the feature word existing in the entire browsing history of FIG. 5 is calculated to be small. At this time, since the feature word “product A” of the document (c) exists in the documents (a) to (c), the weighting score is smaller than the other feature words “Y-shirt” and “pink”. Become.

  Specifically, for each feature word of the document (c), a weighting score is calculated by the equations (2) to (4). In equations (2) to (4), equation (1) is multiplied by the number of appearances of the feature word in document (c).

  The score of “Y shirt = 4. Pink = 4. Product A = 1” is calculated for each feature word of the document (c) by the equations (2) to (4). On the other hand, when the entire browsing history is used, that is, in the case of the setting (a), the number of appearances of each document (a) to (d) of the feature word is totaled to calculate a weighting score. For example, if the score is calculated by simply summing up the number of appearances, the score of “product A = 4. Y shirt = 1. Pink = 1” is calculated.

  Each feature word is weighted according to the calculated score. The feature words are output through the output unit 6 with the priority according to the weighting. Therefore, according to the feature extraction apparatus 1, it is possible to more appropriately extract the feature word corresponding to the browsing intention of the viewer by using the “link source-link destination” relationship of the document. This makes it possible to extract “characteristic words” that are basic data for recommending to the viewer based on the browsing history of the viewer.

  Note that the present invention is not limited to an HTML document, and a structured document described in another markup language such as XML can also be included in the browsing history. In this case, “XML Linking (XML Linking Language)” that defines a link between XML documents may be used in the process of the feature extraction unit 4.

≪Programs≫
The present invention can also be configured as a document feature extraction program that causes a computer to function as some or all of the units 2 to 6 of the feature extraction apparatus 1. According to this program, it is possible to cause a computer to execute part or all of the processing procedures of the respective units 2 to 6.

  The program can be provided through a network such as a website or e-mail. The program is stored in a recording medium such as a CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, MO, HDD, BD-ROM, BD-R, or BD-RE. It is also possible to record, save and distribute. This recording medium is read using a recording medium driving device, and the program code itself realizes the processing of the above embodiment, so that the recording medium also constitutes the present invention.

DESCRIPTION OF SYMBOLS 1 ... Document feature extraction apparatus 2 ... Browsing history recording part 3 ... Browsing history collection DB
4. Feature extraction unit (feature extraction means)
5. Feature recalculation unit (feature recalculation means)
6 ... Output section

Claims (3)

A document feature extraction device that extracts features of each structured document based on a browsing history of the structured document in which a link realizing a reference relationship is expressed,
The link and the related text of the link are extracted from the structured document of the link source included in the browsing history, the representative part in the structured document of the link destination including the extraction information is extracted as the body, and the extracted A feature extraction means for extracting features from the body;
A feature recalculation unit that aggregates the overall features from the features of each structured document extracted by the feature extraction unit and recalculates by weighting the features of each document;
A document feature extraction apparatus comprising: A document feature extraction method executed by an apparatus for extracting features of each structured document based on a browsing history of the structured document in which a link realizing a reference relationship is expressed,
The link and the related text of the link are extracted from the structured document of the link source included in the browsing history, the representative part in the structured document of the link destination including the extraction information is extracted as the body, and the extracted A feature extraction step for extracting features from the body;
A feature recalculation step of summing up the overall features from the features of each structured document extracted by the feature extraction means and weighting and recalculating the features of each document;
A document feature extraction method characterized by comprising:   A document feature extraction program for causing a computer to function as the document feature extraction apparatus according to claim 1.