JP5317638B2 - Web document main content extraction apparatus and program - Google Patents

Description

  The present invention relates to a Web document main content extraction apparatus and program, and more particularly, to a Web document main content extraction apparatus and program for automatically extracting a portion of a Web document that can be determined as main content.

In the conventional main content extraction method, an extraction rule is manually created from each Web document, and the main content is extracted based on the rule (see, for example, Non-Patent Document 1).
http://fm.goo.ne.jp/

  However, when searching for information on a Web document or recommending information from a Web document browsed by a user, if the main content part of the Web document is not extracted, what is the original main content such as a navigation link or an advertisement part? There was a problem that unrelated parts became noise. For this problem, conventionally, extraction rules are manually created for each Web document and main contents are extracted. However, it is difficult to manually create extraction rules for all Web documents. is there. Further, since the configuration of the Web document is updated every time the date passes, it is difficult to use the created rule permanently.

  The present invention has been made in view of the above points, and an object thereof is to provide a Web document main content extraction apparatus and program capable of automatically extracting main content without creating an extraction rule manually. To do.

  FIG. 1 is a principle configuration diagram of the present invention.

The present invention (Claim 1) is a Web document main content extracting apparatus for extracting main contents of a Web document,
When a Web document is input, the Web document is divided into segments based on a predetermined division rule, and is stored in the storage unit 160.
A feature amount extraction unit 130 that extracts a feature amount for main content determination for each segment divided by the document division unit 120 and stores the feature amount in the storage unit 160 for each segment;
Main content determination means 140 for determining whether the content is main content using a machine learning algorithm based on the feature amount for each segment;
A main content output unit 150 that combines the parts determined to be the main content by the main content determination unit 140 and outputs the result as main content;
I have a,
The feature quantity extraction means 130
Number of punctuation marks included in character strings, amount of character strings, amount of character strings not including character strings displayed on the Web, number of anchor links, average amount of anchor link character strings, anchors The total amount of link character strings, the maximum character string amount of anchor link destination URL, and a numerical value for identifying whether or not an anchor link related to an advertisement is included are extracted as feature amounts .
Further, the present invention (Claim 2) is characterized in that in the feature quantity extraction means 130,
The number of HTML tags related to text, the number of consecutive HTML tags related to text, the number of linked list tags, the ratio of the number of HTML tags related to text and the number of consecutive HTML tags related to text, and displayed on the web. One or more of the ratio of the number of character strings and the number of tags and the ratio of the number of anchor links and the number of link list tags are further extracted as feature quantities.
Further, according to the present invention (Claim 3), in the feature amount extraction unit 130,
Before performing extraction means, special characters used for displaying symbols used in HTML tags on a Web browser, and special character deletion means for deleting HTML special characters other than the special characters are included.

In addition, the present invention (Claim 4 ), when an advertisement target area exists in the input Web document, the advertisement target area extracting means for extracting the advertisement target area;
Noise removing means for removing tags and areas that cause noise from a Web document;
A dividing unit that divides the Web document output from the noise removing unit using a predetermined dividing rule and stores it in the storage unit.

Further, the present invention (Claim 5) comprises normalization means for normalizing the feature value issued extracted, the.

The present invention (Claim 6 ) is a Web document main content extraction program for causing a computer to function as each means constituting the Web document main content extraction apparatus according to any one of claims 1 to 5 .

  As described above, according to the present invention, it is possible to automatically extract main contents without creating a rule for manual extraction. In addition, since the main content extraction is fully automatic, it is possible to cope with changes in the content of the Web document.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The present invention first divides a Web document, then extracts feature amounts from information contained in the divided Web document, and determines whether or not the main content is the main content. Extract. As main feature amounts, class information and tag information used in text information, anchor link information, HTML, XHTML, and the like are used. In addition, as processing for improving accuracy, processing such as extraction of a section advertisement portion and removal of an advertisement is performed. Examples of main contents are shown in FIGS. In FIG. 2 and FIG. 3, the inside of the broken line is the main content.

  FIG. 4 shows the configuration of the Web document main content extraction apparatus according to the embodiment of the present invention.

  A Web document main content extraction apparatus 100 shown in FIG. 1 includes a Web document acquisition / input unit 110, a Web document division unit 120, a feature amount extraction unit 130, a main content determination unit 140, a main content output unit 150, and a storage unit 160. Is done.

<Web document acquisition / input unit>
The Web document acquisition / input unit 110 inputs a Web document (data) to be processed. The configuration of the Web document acquisition / input unit 110 is shown in FIG. The Web document acquisition / input unit 110 shown in the figure is the URL of the Web document from which the main content input by the user is to be extracted or the data input unit 111 for acquiring the file itself, and if the input is a URL, the URL is input. URL input unit 113 to acquire, Web document acquisition unit 114 to acquire the Web document of the URL destination, Web document file input unit 112 to acquire the Web document if it is the Web document itself, and the character code of the Web document Is composed of a document code conversion unit 115 for converting the data into UTF-8.

<Web document dividing unit 120>
The Web document dividing unit 120 divides the acquired document. FIG. 6 shows the configuration of the Web document dividing unit 120. The Web document dividing unit 120 includes an advertisement target area extracting unit 121, a tag / area removing unit 122 that causes noise, and a Web document dividing processing unit 123.

  First, in the Web document dividing unit 120, if there is a region including a content tag for Internet advertisement in the advertisement target region extracting unit 121, the region is extracted. Here, the Internet advertisement is a tag used by an advertising company such as google or overture to narrow down main contents for advertisement distribution. In the case of a google advertisement, the range from <!-google_ad_section_start-> to <!-google_ad_section_end-> corresponds to that area. These tags have a slightly different character string depending on the Web document or are displayed in uppercase letters. Use a regular expression that does not distinguish between uppercase and lowercase letters, or use a wildcard regular expression. Process to absorb the difference. Hereinafter, it is assumed that processing for absorbing differences in objects is performed in the description of processing using regular expressions.

  When there is an internet advertisement, the noise tag or area removal unit 122 (hereinafter referred to as “noise removal unit”) performs the process of extracting the area described above, and when there is no internet advertisement area, The first input Web document is processed. Next, the noise removing unit 122 performs a process of removing extra tags and regions and specific character strings. The tag or area to be removed may be a comment tag explaining the HTML of the Web document, JavaScript, or a form tag. The tags and areas to be removed are described below.

・ Comment tags that begin with "<!-" And end with "->";
-The area enclosed by the "</ script>" tag from the "<script>"tag;
・ A region surrounded by "</ style>" tags from "<style>"tags;
・ A region surrounded by "</ select>" tags from "<select>"tags;
・ "<Noscript>" area surrounded by "</ noscript>"tag;
・ A region enclosed by "</ form>" tag from "<form>"tag;
・ Consecutive blank character string (excluding single blank)
・ Consecutive tab strings (excluding single tabs)
The noise removing unit 122 removes the above tags, regions, and character strings using regular expressions. Since there may be cases where the alt attribute and class attribute exist in the tag, in such a case, division is performed using a regular expression that takes into account the tag including them (eg, <style class = "hoge">).

  The web document division processing unit 123 divides the web document. The division rule is to divide using the following tags.

・ <Div>
・ / Div
・ <Td>
・ </ Td>
Since there may be a case where an alt attribute or a class attribute is present in the tag, in such a case, division is performed using a regular expression that takes into account the tag including those (eg, <div class = "hoge">). Hereinafter, each of the divided Web documents is referred to as a “segment”, and feature amount extraction and determination as to whether or not the content is main content is performed for each segment.

  Each segment is stored in the storage unit 160.

<Feature Extraction Unit 130>
The feature amount extraction unit 130 extracts feature amounts from the segments stored in the storage unit 160, and determines the main content portion of the Web document. The configuration of the feature quantity extraction unit 130 is shown in FIG.

  The feature quantity extraction unit 130 shown in the figure includes an anchor link information feature quantity extraction unit 131, a tag information feature quantity extraction unit 132, a feature quantity normalization unit 134, and a character string feature quantity extraction unit (hereinafter referred to as a Web document). 133, a feature amount normalization unit 134, and a feature amount ratio feature amount extraction unit (hereinafter referred to as a “feature amount extraction processing unit”) 135.

<Anchor Link Information Feature Extraction Unit 131>
The anchor link information feature amount extraction unit 131 extracts a feature amount related to the anchor link from the segment output from the Web document dividing unit 120 and stored in the storage unit 160.

(1) Number of anchor links:
In a certain segment, there is a high possibility that a segment including many anchor links is not the main content. Therefore, the anchor link information feature amount extraction unit 131 uses the number of anchor links as the feature amount. Specifically, the number of anchor links represented by <a href=...> ... </a> tags is used as the feature amount.

  This feature amount is executed by a feature amount normalization unit 134, which will be described below, by performing a method of normalizing a character string amount to obtain a feature amount and a feature amount using an absolute value of the character string. Characteristic amount of character string. For example, when the maximum number of anchor links in all segments is “10” and the number of anchor links in a segment is “5”, the feature quantity of the number of anchor links in the segment is 0.5. It becomes. Since the anchor link tag may include a class attribute and an alt attribute, the number of anchor link tags is counted using a regular expression.

(2) Average amount of character string of each anchor link When the character string of each anchor link is large on average, the segment is likely to be a navigation link such as a related article. Also, if the average number of anchor link character strings is small, there is a high possibility that the keyword search link is included in the main content. Accordingly, the anchor link information feature amount extraction unit 131 uses the average amount of anchor link character strings included in the segment as a feature amount. The character string of the anchor link corresponds to the XXX part of <a href='...'...> XXXXX </a>. This feature amount is also executed in the feature amount normalization unit 134 described below by normalizing the amount of the character string to be a feature amount and using the absolute value of the character string as the feature amount, The final character string feature amount is stored in the memory 136.

(3) Total amount of character strings of all anchor links:
When the total amount of anchor link character strings included in a segment is large, there is a high possibility that the segment is a navigation link. Therefore, the anchor link information feature amount extraction unit 131 uses the total amount of anchor link character strings included in the segment as a feature amount. The character string of the anchor link corresponds to the XXX part of <a href='...'> XXXXX </a>. This feature is also executed in the feature amount normalization unit 134 described below by performing two methods of normalizing the amount of the character string to obtain the feature amount and using the absolute value of the character string as the feature amount. The character string amount is stored in the memory 136 as a feature amount.

(4) Amount of anchor link URL of maximum character string:
If the URL character string of the anchor link destination is very long, the segment is likely to be an advertisement. Therefore, the character string of the URL with the longest anchor link destination in the segment is used as the feature amount. The URL character string of the anchor link destination described here corresponds to the portion of ΔΔΔ in <a href='ΔΔΔ'...></a>. In the feature amount normalization unit 134, which will be described below, this feature amount is also subjected to a method of normalizing the amount of the character string to obtain the feature amount, and two as the feature amount using the absolute value of the character string. The characteristic amount of the character string is stored in the memory 136.

(5) Whether to include an anchor link related to the advertisement:
An anchor link including a URL related to an advertisement is likely to include a characteristic character string. For example, “adclick”, “adnet”, “banner”, and the like. Therefore, the anchor link information feature amount extraction unit 131 extracts a feature amount that sets the feature amount to 1 when there is an anchor including a URL including a character string that is likely to be an advertisement, and sets 0 when the anchor does not exist. And stored in the memory 136. Character strings that are likely to become advertisements are listed on sites such as Adblock plugin, which is an add-in for Firefox that removes advertisements.

<Tag information feature amount extraction unit 132>
The tag information feature amount extraction unit 132 extracts feature amounts relating to tag information such as HTML tags.

(1) Number of text HTML tags:
When there are many character strings displayed in a Web document in a certain segment, many HTML tags related to text are included. In CGMs such as blogs, there are few character strings displayed in Web documents, but there are many cases where users frequently use line feed tags. Therefore, the tag information feature amount extraction unit 132 uses the number of HTML tags related to text as a feature amount. In the feature amount normalization unit 134, which will be described below, this feature amount is also subjected to a method of normalizing the amount of the character string to obtain the feature amount, and two as the feature amount using the absolute value of the character string. The characteristic amount of the character string is stored in the memory 136.

  For example, if the maximum amount of HTML tags in all segments is “10” and the amount of HTML tags in a segment is “5”, the feature amount of the amount of HTML tags in that segment is “0”. .5 ". The text-type HTML tags used in the present embodiment target the following tags.

・ <P>
・ </ P>
・ <br>
・ </br>
・ <Font>
・ / Font
Since the size attribute and class attribute may exist in the tag, in that case, count using a regular expression that takes into account the tag including them (example: <font size = "+ 1">) .

(2) Number of consecutive occurrences of text-based HTML tags:
In a segment in which character strings displayed in a Web document are concentratedly described, there are many text-based tags, and at the same time, text-based HTML tags appear continuously. Here, “appears continuously” means that other HTML tags such as anchor links do not appear in between. Therefore, the tag information feature amount extraction unit 132 sets the number of continuous appearances of the text-type HMTL tag described in (1) as a feature amount. In the feature amount normalization unit 134, which will be described below, this feature amount is also subjected to a method of normalizing the amount of the character string to obtain the feature amount, and two as the feature amount using the absolute value of the character string. The characteristic amount of the character string is stored in the memory 136.

  For example, when the continuous amount of the maximum HTML tag in all segments is “10” and the continuous amount of HTML tags in a segment is “5”, the feature amount of the character string amount of the segment is “ 0.5 ".

(3) Number of link list tags:
When there are many link list tags in a certain segment, there are many anchor links such as navigation links in the segment, and there is a high possibility that the segment will not become main content. Therefore, the tag information feature amount extraction unit 132 sets the number of link list tags as a feature amount. This feature amount is also executed in the feature amount normalization unit 134 described below by normalizing the amount of the character string to be a feature amount and using the absolute value of the character string as the feature amount, The final character string feature amount is stored in the memory 136.

  For example, if the amount of the maximum link list tag in all the segments is “10” and the amount of the link list tag in a certain segment is “5”, the feature amount of the link list tag amount of that segment is “0.5”. The link list tag used in the present embodiment targets the following tags.

・ <Li>
・ <Ul>
・ <Dl>
・ <Dd>
・ <Ol>
Since there may be cases where the alt attribute and class attribute exist in the tag, in such a case, counting is performed using a regular expression that considers the tag including those (eg, <font class = "hoge">).

(4) Amount of character strings (including HTML tags) that do not include character strings displayed in the Web document:
If there are many character strings (including HTML tags) that are not displayed on the Web in a certain segment, there is a high possibility that the segment will not have main content for advertising. Therefore, the amount of character strings (including HTML tags) other than the character strings displayed in the Web document is used as the feature amount. This feature amount is also executed in the feature amount normalization unit 134 described below by normalizing the amount of the character string to be a feature amount and using the absolute value of the character string as the feature amount, The final character string feature amount is stored in the memory 136.

  For example, if the amount of character strings that are not displayed on the web in the segment where the amount of character strings that are not displayed on the maximum Web in all the segments is “100” is “50”, they are not displayed on the Web of that segment. The feature amount of the character string amount is “0.5”.

<Character String Feature Extraction Unit 133 Displayed in Web Document>
Below, the feature-value regarding the character string displayed with a web browser is described. The “character string” described here does not include a character string that is not displayed by a Web browser such as an HMTL tag.

(1) Amount of character string:
In general, the main content portion of a Web document includes many character strings as compared with a portion that is not the main content. The same can be said for a Web document with a small number of character strings as a whole. Therefore, the character string feature amount extraction unit 133 uses the number of character strings included in the divided Web document as a feature amount. Then, in the feature amount normalization unit 134 described below, a method of normalizing the amount of the character string to obtain the feature amount and two using the absolute value of the character string as the feature amount are executed. The character string amount is stored in the memory 136 as a feature amount.

<Feature amount normalization unit 134>
The feature amount normalization unit 134 normalizes each feature amount stored in the memory 136 by the following method.

(1) Character string:
a) A method of normalizing the amount of character string to obtain a feature amount:
Normalization is performed by setting the feature amount of the segment having the maximum character string amount to “1” in all segments. For example, if the maximum amount of character strings in all segments is “200” and the amount of character strings in a segment is “100”, the feature amount of the amount of character strings in that segment is “0 · 5 ". By performing such normalization, it is possible to extract main contents even in a Web document having a small character string as a whole.

b) A method of using the absolute value of the amount of character string as a feature amount:
The normalization method described above in a) and used as a feature amount is effective for a Web document with a small number of character strings as a whole. However, by performing normalization, the amount of character strings is large as a whole. Inconvenience occurs when the difference in the amount of character strings between segments of the main content portion is large. For example, if the maximum amount of character strings in all segments is “1000” and the amount of character strings in a segment is “100”, the feature amount of the amount of character strings in that segment is “0.1”. become. Although the amount of character strings should be large, such detriment is caused by normalization. Therefore, it is necessary to perform a technique for using the absolute value of the character string as a feature amount.

  Specifically, when a certain value is exceeded, a method of setting the character string feature amount to “1” is used. For example, when the amount of character string in a segment is “100”, the feature amount when the amount of character string is “5” or more is “1”, and the feature amount of character string is “10” or more. The feature amount is set to “1”, so that the feature amount of the character string “105” or more is “0”, and the feature amount of the character string “200” or more is “0”. create. In this way, by using the method of setting the feature value to “1” when a certain character string amount is exceeded, the absolute value of the character string amount is maintained with the maximum feature value being “1”. It can be an amount. Further, although the feature value interval of the absolute value of the character string amount in this embodiment is set to “5”, it is preferable to use an appropriate interval in this case. A method of setting the interval between feature quantities using a multiplier of 2 such as 8, 16, 32, and 64 is also conceivable. Similarly, the maximum x of the character string amount x or more is changed to an appropriate value in some cases. In order to reduce the amount of calculation in the main content determination unit 140, the value of x may be reduced.

(2) Number of punctuation marks:
Segments that tend to be noisy, such as Web advertisements, tend to have a small amount of punctuation, although the amount of character strings is large. Therefore, it is characterized by the number of punctuation marks. Specifically, the feature quantity normal lower part 134 includes “,”, “,”, “.”, “.”, “!”, “•”, “?”, “...” included in the character string in the segment. "Is counted as a feature quantity. This feature amount is also calculated in two ways: a feature amount by normalization described in terms of the amount of character string and a feature amount by absolute value. As the calculation method, the same method as that described in the character string section of (1) is used.

<Characteristic Ratio Ratio Feature Quantity Extraction Unit 135>
A characteristic feature ratio feature quantity extraction unit (hereinafter referred to as “ratio feature quantity extraction unit”) 135 is displayed as the aforementioned anchor link information feature quantity, tag information feature quantity, and Web document stored in the memory 136. A feature amount using a ratio between character string feature amounts is obtained.

(1) Ratio of the number of text tags and the number of consecutive occurrences of text tags:
A segment having a large number of text-type tags and a large number of consecutive appearances of text-type tags is highly likely to be the main content because the text is densely written. However, although there are many text-type tags, a segment with a small number of consecutive appearances of text-type tags can be said to have a high possibility that it is not the main content because the segment size is large. Therefore, the ratio feature quantity extraction unit 135 uses the ratio of the number of text tags and the number of consecutive appearances of text tags as a feature quantity.

  Specifically, a value with the number of text tags as the denominator and the number of consecutive occurrences of text tags as the numerator is used as the feature amount. Here, when the number of text tags is “0”, the denominator is “0”. In this case, the feature amount of the ratio between the number of text tags and the number of consecutive occurrences of text tags is Set to “0”. The feature amount normalization unit 134 also performs normalization of the feature amount in the same manner, and stores the feature amount as a final feature amount in the memory 136. The larger the feature amount, the higher the possibility that it is the main content.

(2) Ratio of character string and tag displayed on the web:
If there are many character strings displayed on the Web in a certain segment, there is a high possibility of becoming main content, but there may be many tags such as HTML tags in the same segment. In this case, as described in the section “Ratio between the number of text-type tags and the number of consecutive appearances of text-type tags” in (1) above, there is a possibility that the segment content is large and it is not the main content. Therefore, the ratio feature quantity extraction unit 135 copes with such a case by using the ratio between the character string displayed on the Web and the tag as the feature quantity.

  Specifically, a character string displayed on the Web is used as a numerator, and a value using the number of tags as a denominator is used as a feature amount. The larger the feature amount, the higher the possibility that it is the main content. This feature quantity is also normalized by the feature quantity normalization unit 134 and stored in the memory 136 as a final feature quantity. When the number of tags is “0”, the denominator is “0”, so the feature value is “1”.

(3) Ratio of the number of anchor links to the number of link list tags:
Within a certain segment, the more anchor links or link list tags, the more likely that segment is not the main content, but because these segments are large, these feature quantities will increase by chance. Is also possible. Therefore, the ratio feature quantity extraction unit 135 uses the ratio of the number of anchor links and the number of link list tags as a feature quantity.

  Specifically, the number of anchor links is used as the denominator, the number of link list tags is used as the numerator, and the feature amount is set. The larger the feature amount, the higher the number of links with respect to the area of the segment, and the higher the possibility that it is not the main content. This feature quantity is also normalized by the feature quantity normalization unit 134 and stored in the memory 136 as a final feature quantity. When the number of anchor links is “0”, the denominator is 0, so the feature value is “0”.

<Main content determination unit>
The main content determination unit 140 determines whether the content is main content by using the feature amount extracted by the feature amount extraction unit 130. The configuration of the main content determination unit 140 is shown in FIG.

  The main content determination unit 140 shown in the figure includes a feature amount input unit 141 that inputs the feature amount of the segment output from the feature amount extraction unit 130, and a text determination unit that determines whether text exists for each segment. 142 and a main content determination processing unit 143 that extracts a segment whose main content is based on a parameter for each feature amount.

  Hereinafter, a method for determining whether or not the content is the main content using the feature amount for each segment acquired from the feature amount extraction unit 130 will be described.

  FIG. 9 is a flowchart of the feature quantity parameter estimation method according to the embodiment of the present invention.

  First, training data is created for each segment from which a feature amount is extracted to determine whether or not the content is main content manually (steps 101, 102, and 103). Here, when there is no character string displayed on the Web, it is described that it is not regarded as main content. However, in the method using machine learning, it is effective for learning as a negative example. Such data is also adopted. Then, learning is performed using the feature amount of the segment, and a weight for each feature amount is calculated (step 104). When importance is attached to speed, learning is performed by the maximum entropy method, and when importance is attached to accuracy, learning is performed using a support vector machine using a second-order polynomial kernel to create a learning model (step 105). Then, using these learned parameters, the segment feature amount is input to the main content determination processing unit 143.

  The main content determination processing unit 143 determines whether or not the main content is based on the feature amount for each segment, and outputs only the main content to the main content output unit 150.

  Note that when the apparatus is incorporated in a user PC or the like, it is difficult to process using all the feature amounts, so the processing amount is reduced by narrowing down the feature amounts to be extracted. Here, the learning model by machine learning creates a learning model for each narrowed-down feature quantity model.

<Main content output unit 150>
The main content output unit 150 combines only segments determined to be main content by the learning device after the main content determination unit 140 determines whether or not the content is main content, and finally outputs the combined content.

  The configuration of the main content output unit 150 is shown in FIG. The main content output unit 150 shown in the figure includes a tagged text output unit 151, an untagged text output unit 152, and a data output unit 153. When it is desired to use this apparatus as pre-processing for information retrieval, a tag such as an HTML tag is left using the tagged text output unit 151 and output from the data output unit 153. On the other hand, when it is desired to analyze the content of the Web document for information recommendation or the like, tags such as HTML tags are deleted using the untagged text output unit 152 and output from the data output unit 153.

<Pre-processing for feature extraction>
In order to improve accuracy, a technique of removing unnecessary character strings and the like is effective as a pre-processing for extracting feature amounts in the feature amount extraction unit 130.

  By removing unnecessary character strings described below in advance, the determination accuracy of main contents is improved.

・ &Nbsp;
・ &Lt;
・ &Gt;
・ &Amp;
・ &Laquo;
・ &Raquo;
These character strings are special characters used when symbols used in HTML tags or the like are displayed on the Web browser. In addition, HTML special characters other than the special characters listed above are also subject to deletion. Special characters tend to be noise during learning because the amount of character strings is larger than the character strings actually displayed on the Web browser.

  As described above, the present invention is based on a technique for automatically extracting various statistical features and automatically extracting the main content portion of the Web document, so that the main process is performed from the Web document as a pre-processing of the information search technique and the information recommendation technique. Content can be extracted and analysis accuracy can be improved.

  In addition, the operation | movement of each component of the Web document main content extraction apparatus shown in FIG. 4 in the above embodiment is constructed as a program, and is installed and executed on a computer used as the Web document main content extraction apparatus. It is possible to distribute through a network.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention can be used as a pre-process for improving accuracy of a full-text search engine, a function capable of creating an RSS feed simply by specifying a URL, advertisement distribution in accordance with the main content of a Web document, and further a user's Web It can be used as a basic technique for analyzing a Web document, such as being used as preprocessing for browsing history analysis.

It is a principle block diagram of this invention. It is a main content example (1) in one embodiment of this invention. It is a main content example (2) in one embodiment of this invention. It is a block diagram of the Web document main content extraction device in one embodiment of the present invention. It is a block diagram of the Web document acquisition / input part in one embodiment of this invention. It is a block diagram of the Web document division part in one embodiment of this invention. It is a block diagram of the feature-value extraction part in one embodiment of this invention. It is a block diagram of the main content determination part in one embodiment of this invention. It is a flowchart of the parameter estimation method of the feature-value in one embodiment of this invention. It is a block diagram of the main content output part in one embodiment of this invention.

Explanation of symbols

100 Web document main content extraction device 110 Web document acquisition / input unit 111 Data input unit 112 Web document file input unit 113 URL input unit 114 Web document acquisition unit 115 Character code conversion unit 120 Document dividing means, Web document dividing unit 121 Region extraction unit 122 Tag that causes noise and region removal unit 123 Web document division processing unit 130 Feature amount extraction means, feature amount extraction unit 131 Anchor link information feature amount extraction unit 132 Tag information feature amount extraction unit 133 Displayed as a Web document Character string feature amount extraction unit 134 Feature amount normalization unit 135 Feature ratio ratio feature amount extraction unit 136 Memory 140 Main content determination means, main content determination unit 141 Feature amount input unit 142 Text determination unit 143 Main content determination processing unit 150 Main Content output means, main Content output unit 151 tagged text output unit 152 untagged text output unit 153 data output unit 160 storage unit, the storage unit

Claims (6)

A Web document main content extraction apparatus for extracting main contents of a Web document,
When a Web document is input, the Web document is divided into segments based on a predetermined division rule, and stored in a storage unit;
For each segment divided by the document dividing unit, a feature amount for main content determination is extracted and stored in the storage unit for each segment;
Main content determination means for determining whether or not the main content is based on the feature value for each segment using a machine learning algorithm;
A main content output unit that combines the parts determined to be the main content by the main content determination unit, and outputs the main content;
I have a,
The feature amount extraction means includes:
In the segmented segment, the number of punctuation included in the character string, the amount of the character string, the amount of the character string not including the character string displayed on the web, the number of anchor links, the average amount of the anchor link character string, A total amount of character strings of anchor links, a character string amount of a URL having the maximum anchor link destination, and a numerical value for identifying whether or not an anchor link related to an advertisement is included are extracted as feature amounts. Web document main content extraction device. The feature amount extraction means includes:
In the divided segment, the number of HTML tags related to text, the number of consecutive HTML tags related to text, the number of linked list tags, the ratio of the number of HTML tags related to text and the number of consecutive HTML tags related to text, displayed on the web One or more of the ratio of the number of character strings to be processed and the number of tags and the ratio of the number of anchor links to the number of link list tags are further extracted as feature quantities.
The Web document main content extraction device according to claim 1. The feature amount extraction means includes:
Before performing the extraction means, includes special characters used when displaying symbols used in HTML tags on a Web browser, and special character deletion means for deleting HTML special characters other than the special characters
The Web document main content extraction device according to claim 1 or 2 . The document dividing means includes
An advertising target area extracting means for extracting the advertising target area when the advertising target area exists in the input web document;
Noise removing means for removing a tag or region that causes noise from the Web document;
A dividing unit that divides the Web document output from the noise removing unit using the predetermined dividing rule and stores the divided document in the storage unit;
including
The Web document main content extraction device according to any one of claims 1 to 3 . Normalizing means for normalizing the extracted issued feature amount including,
The Web document main content extraction device according to any one of claims 1 to 3 . 6. A Web document main content extraction program for causing a computer to function as each means constituting the Web document main content extraction device according to claim 1 .

Images