JP2010204866A - Significant keyword extraction device, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To calculate significance unique to a keyword itself, and to robustly extract a significant keyword without being affected by its appearance frequency in a document. <P>SOLUTION: This significant keyword extraction device is configured to extract the significant content of a Web document, and to extract keyword candidates from the text in the document, and to calculate and rank the significance of the extracted keywords. The significant content is found on the basis of the quantity of characters, the quantity of punctuation marks, the quantity of tags, and the number of links or the like, and the significant part of the content is specified, and the significance of the keyword is determined from the link structure or the retrieval query of Wikipedia(R) only with respect to the significant part of the content. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an important keyword extraction apparatus, method, and program, and more particularly, to an important keyword extraction apparatus, method, and program for extracting an important keyword from text included in a document.

  The conventional important keyword extraction method has mainly been a method of extracting using “frequency” such as the frequency of appearance of keywords and the frequency of concatenation of morphemes constituting a compound word (for example, see Non-Patent Document 1).

Nakagawa Hiroshi, Mori Yasunori, Yumoto Yasuaki, “Terminology extraction based on appearance frequency and connection frequency. Natural language processing” Vol.10 No.1, pp.27-45, January 2003

  In the conventional important keyword extraction method, since a method using “frequency” such as a keyword appearance frequency and a connection frequency is mainstream, it is necessary to require a relatively large amount of document sets in advance. In addition, the method of determining that it is important just because the frequency is high often does not work well in some cases, and the results are not very good in terms of accuracy.

  The present invention has been made in view of the above points, and it is possible to calculate the intrinsic importance of the keyword itself, and to extract the important keyword robustly without depending on the appearance frequency in the document. An object is to provide an apparatus, a method, and a program.

  FIG. 1 is a principle configuration diagram of the present invention.

The present invention (Claim 1) is an important keyword extraction device for extracting an important keyword from text included in a Web document,
Main content extracting means 10 for acquiring a Web document, extracting a feature amount of the Web document, and extracting main content;
A document set having a unique headword in a document set represented by an online encyclopedia (for example, Wikipedia (registered trademark)), and a document set having a citation relationship or a reference relationship in the document set (hereinafter referred to as a dictionary document set) An important keyword candidate extraction means 2 for extracting an important keyword candidate from the main content with reference to the keyword dictionary 101 in which the keyword is registered as a dictionary for morphological analysis;
Appearance frequency calculation means 3 for weighting important keyword candidates in the Web document;
The importance level of the important keyword candidate is calculated so that the importance level of the keyword having a high degree of name recognition, high topicality, interesting contents, and various topics when talking about the keyword is high, and the first memory is calculated. Keyword importance calculation means 4 stored in the means 105;
Based on the importance of the important keyword candidate, the position information calculation means 5 for obtaining the appearance position keyword importance based on the position in the Web document and storing it in the second storage means 106;
Based on a value obtained by multiplying the importance of the important keyword candidate stored in the first storage unit 105 by the appearance position keyword importance stored in the second storage unit 106, a keyword set with importance is output. Keyword output means 6 for
The main content extracting means 10
Web document dividing means 12 for dividing the Web document into segments based on a predetermined division rule;
A feature amount extracting means 13 for extracting a feature amount for main content determination for each segment;
Main content determination means 14 for determining whether the content is main content using a machine learning algorithm based on a feature amount for each segment;
Including main content output means 15 for combining the parts determined to be main content and outputting them as main content;
The feature quantity extraction means 13 of the main content extraction means 10
A means for extracting a feature amount of a character string displayed in a Web document, a feature amount of tag information, and a feature amount of anchor link information, and using a ratio between each feature amount as a final feature amount;
The appearance frequency calculation means 3
Means for weighting important keyword candidates using WebIDF, which is a weight considering the probability of appearance in a set of Web documents collected by a search engine;
Or
Any of means for weighting the keyword candidates using BM25 that calculates the appearance frequency of the important keyword candidates in the Web document and weights each important keyword candidate based on the appearance frequency and the appearance distribution in the document set. Have

Further, according to the present invention (Claim 2), the appearance frequency calculation means 3 of Claim 1 described above is
Means for obtaining the importance of the important keyword candidate using the appearance frequency of the important keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result of the morpheme analysis means is included.

Further, according to the present invention (Claim 3), the keyword importance degree calculation means 4 of Claim 1 or 2 is
Means for obtaining importance levels of important keyword candidates using the link structure in the dictionary document set is included.

Further, according to the present invention (Claim 4), the keyword importance degree calculation means 4 of Claims 1 to 3 described above is
The keyword includes a means for obtaining the importance of the important keyword candidate by using a value that becomes a more important keyword as the keyword is frequently input to the search engine.

Further, according to the present invention (Claim 5), the keyword importance degree calculation means 4 of Claim 1 or 2 is
Means for obtaining the importance of the important keyword candidate based on the appearance position of the important keyword candidate in the Web document is included.

In the present invention (Claim 6), the keyword importance degree calculation means 4 of Claim 1
Using the link structure in the dictionary document set, a means for determining the importance of important keyword candidates,
Means for determining the importance of the important keyword candidate using the appearance frequency of the important keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result of the morpheme analysis means.

In the present invention (Claim 7), the keyword importance degree calculation means 4 of Claim 1
Means for determining the importance of the important keyword candidate using the appearance frequency of the important keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result of the morpheme analysis means;
Means for determining the importance of important keyword candidates by using a value such that the keyword that has been input to the search engine more frequently becomes a more important keyword.

In the present invention (Claim 8), the keyword importance degree calculation means 4 of Claim 1
Means for determining the importance of the important keyword candidate using the appearance frequency of the important keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result of the morpheme analysis means;
Means for determining the importance of the important keyword candidate based on the appearance position of the important keyword candidate in the document.

Further, according to the present invention (claim 9), the keyword importance calculation means 4 of claim 1
Means for determining the importance of the important keyword candidate using the appearance frequency of the important keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result of the morpheme analysis means;
Using the link structure in the dictionary document set, a means for determining the importance of important keyword candidates,
Means for determining the importance of important keyword candidates by using a value such that the keyword that has been input to the search engine more frequently becomes a more important keyword.

Further, according to the present invention (claim 10), the keyword importance calculation means 4 of claim 1
Means for determining the importance of the important keyword candidate using the appearance frequency of the important keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result of the morpheme analysis means;
Using the link structure in the dictionary document set, a means for determining the importance of important keyword candidates,
A keyword is used to determine the importance of potential keyword candidates by using a value that makes the keyword more important as the keyword is used more frequently.
Means for determining the importance of the important keyword candidate based on the appearance position of the important keyword candidate in the document.

  FIG. 2 is a diagram for explaining the principle of the present invention.

The present invention (Claim 11) is an important keyword extraction method for extracting an important keyword from text included in a Web document,
A main content extracting means for acquiring a Web document, extracting a feature amount of the Web document, and extracting the main content;
A document in which the important keyword candidate extracting means is a document having a unique headword in a document set represented by an online encyclopedia (for example, Wikipedia (registered trademark)) and having a citation relationship or a reference relationship in the document set An important keyword candidate extraction step (step 2) for extracting an important keyword candidate from main contents with reference to a keyword dictionary in which entry words of the set (hereinafter referred to as a dictionary document set) are registered as a dictionary for morphological analysis; ,
An appearance frequency calculating step (step 3) in which an appearance frequency calculating unit weights important keyword candidates in the Web document;
The keyword importance calculation means calculates the importance of important keyword candidates so that the importance of keywords having a high level of name recognition and high topicality, and various information that includes various topics when talking about keywords is increased. A keyword importance degree calculating step (step 4) to be stored in the storage means;
A position information calculation means for obtaining an appearance position keyword importance based on the position of the important keyword candidate in the Web document, and storing it in the second storage means (step 5);
The keyword output means adds a keyword with importance based on the value obtained by multiplying the importance of the important keyword candidate stored in the first storage means and the appearance position keyword importance stored in the second storage means. And a keyword output step (step 6) for outputting the set,
In the main content extraction step (step 1),
A Web document dividing step of dividing the Web document into segments based on a predetermined dividing rule;
A feature extraction step for extracting a feature for determining main content for each segment;
A main content determination step for determining whether or not the main content is based on a feature amount for each segment using a machine learning algorithm;
A main content output step of combining the parts determined to be the main content and outputting as a main content,
In the feature extraction step,
Extracting a feature amount of a character string displayed in a Web document, a feature amount of tag information, and a feature amount of anchor link information, and using a ratio between each feature amount as a final feature amount,
In the appearance frequency calculation step (step 3),
Weighting important keyword candidates using WebIDF, which is a weight considering the probability of appearance in a set of Web documents collected by a search engine;
Or
Obtaining an occurrence frequency of the important keyword candidate in the Web document, and weighting the keyword candidate using BM25 for weighting each important keyword candidate based on the occurrence frequency and the appearance distribution in the document set;
Do one of the following.

Further, the present invention (Claim 12), in the appearance frequency calculating step (Step 3) of Claim 11,
Using the appearance frequency of the important keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result of the morpheme analysis means, the importance of the important keyword candidate is obtained.

Further, according to the present invention (Claim 13), in the keyword importance calculation step (Step 4) of Claim 11 or 12,
Using the link structure in the dictionary document set, the importance of the important keyword candidate is obtained.

Further, the present invention (Claim 14) is a keyword importance calculation step (Step 4) according to Claims 11 to 13,
The importance of the important keyword candidate is obtained by using a value that becomes a more important keyword as the keyword is frequently input to the search engine.

Further, according to the present invention (Claim 15), in the keyword importance calculation step (Step 4) of Claim 11 or 12,
Based on the appearance position of the important keyword candidate in the Web document, the importance of the important keyword candidate is obtained.

  The present invention (Claim 16) is an important keyword extracting program for causing a computer to function as each means constituting the important keyword extracting apparatus according to any one of Claims 1 to 10.

  According to the present invention, important keywords in a document can be extracted, and can be applied to a document management system using them, advertisement distribution from important keywords, and the like. Furthermore, in the conventional technology, an important keyword is identified only from the frequency of occurrence of the keyword in the document. However, the link structure of the online encyclopedia (for example, Wikipedia (registered trademark)) and the number of actual search queries input By using it, it is possible to calculate the inherent importance of the keyword itself that could not be realized by the conventional technique, and it is possible to robustly extract important keywords that are not influenced by the appearance frequency in the document.

It is a principle block diagram of this invention. It is a figure for demonstrating the principle of this invention. It is a block diagram of the important keyword extraction apparatus in one embodiment of this invention. It is a figure which shows the outline | summary of the process of this invention. It is a block diagram of the main content extraction part in one embodiment of this invention. It is a block diagram of the Web document acquisition / input part in one embodiment of this invention. It is a block diagram of the Web document division part in one embodiment of this invention. It is a block diagram of the feature-value extraction part in one embodiment of this invention. It is a block diagram of the main content determination part in one embodiment of this invention. It is a block diagram of the main content output part in one embodiment of this invention. It is a flowchart of the parameter estimation method of the feature-value in one embodiment of this invention. It is a main content example (the 1) in one embodiment of this invention. It is a main content example (the 2) in one embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The present invention is a system for extracting important keywords in a document. In the prior art, it was necessary to prepare a document set having a relatively large amount in advance. However, the present invention proposes a method in which the accuracy is not easily lowered even when the number of documents is small.

  In the present invention, an important keyword is extracted using “a document set having a unique headword in the document set and a citation relationship or a reference relationship in the document set”. For example, the online encyclopedia “Wikipedia (registered trademark)”, “Hatena bookmark (registered trademark)”, “Mypedia (registered trademark)”, and other document sets correspond to this. The keyword candidates use document headwords in these document sets. By using a headword of a document set having a unique headword such as an encyclopedia, a compound word composed of a plurality of morphemes uniquely indicating an event can be cut out. Furthermore, since such headwords cover keywords that are generally considered important, it is possible to narrow down important keywords by using the headwords that are listed as keyword candidates.

  In the present embodiment, the target document set is “a document set having a unique headword in the document set and having a citation relationship or a reference relationship in the document set”. (Registered trademark) will be described as an example.

  FIG. 3 is a configuration diagram of the important keyword extracting apparatus according to the embodiment of the present invention.

  The important keyword extracting apparatus shown in FIG. 1 includes a document input unit 1 having a main content extracting unit 10, an important keyword candidate extracting unit 2, an appearance frequency calculating unit 3, a keyword importance calculating unit 4, a position information calculating unit 5, an important keyword. Output unit 6, keyword dictionary 101, WebIDF storage unit 102, keyword importance level 1 storage unit 103 that stores keyword importance levels from the link structure of Wikipedia (registered trademark) that has been obtained in advance, search that has been obtained in advance The keyword importance degree storage unit 104 stores a keyword importance level using the number of engine search query inputs, a unique importance level storage unit 105, and a position importance level storage unit 106.

  FIG. 4 is a diagram showing an outline of the processing of the present invention.

  Step 100) The main content extraction unit 10 of the document input unit 1 determines whether the input target document is a Web document. If the input target document is a Web document, the process proceeds to Step 200. If not, the process proceeds to Step 300. Transition.

  Step 200) If the document is a Web document, the main content is extracted (main content extraction process).

  Step 300) The important keyword candidate extraction unit 2 extracts keyword candidates from the text in the document (keyword candidate extraction process).

  Step 400) The appearance frequency calculating unit 3 obtains the appearance frequency of the keyword candidate appearing in the document, the keyword importance calculating unit 4 obtains the importance (unique importance) of the keyword candidate, and the position information calculating unit 5 The importance (position importance) is obtained from the position where the candidate appears in the document, and the important keyword output unit 6 orders the keywords based on the unique importance and the position importance (keyword importance calculation processing). .

  The above is the main flow of processing for extracting important keywords in a document.

[1] Extraction process of main content (step 200):
This process is a process performed by the main content extraction unit 10 of the document input unit 1.

  When the input document is a Web document, there is text unrelated to the content of the Web document, such as a navigation link and advertisement text. Therefore, it is necessary to extract the main content part from which these unnecessary texts are removed. Whether or not the document is a Web document is determined using the extension of the file.

[1-1] Process flow:
As shown in FIG. 5, the main content extraction unit 10 of the document input unit 1 includes a Web document acquisition / input unit 11, a Web document division unit 12, a feature amount extraction unit 13, a main content determination unit 14, and a main content output unit 15. Consists of Here, the feature amount extraction unit 13 and the main content determination unit 14 perform processing for each Web document divided by the Web document division unit 12.

  The Web document acquisition / input unit 11 inputs a Web document (data) to be processed. As shown in FIG. 6, the data input unit 111, the Web document file input unit 112, the URL input unit 113, and the Web document acquisition Part 114 and a character code conversion part 115.

  The Web document dividing unit 12 divides the acquired Web document, and includes an advertisement target area unit 121, a noise and tag removing unit 122, and a Web document dividing unit 123 as illustrated in FIG. 7. When the advertisement target area exists in the Web document, the advertisement target area 121 extracts the advertisement target area. The tag or area removing unit 122 that becomes noise removes the tag or area that becomes noise of the advertisement target area or the input Web document, and the Web document dividing unit 123 divides the document.

  The feature amount extraction unit 13 extracts a feature amount for main content determination for each Web document divided by the Web document division unit 12, and as shown in FIG. 8, an anchor link information feature amount extraction unit 131, a tag information feature amount extraction unit 132, a character string feature amount extraction unit 133 displayed in a Web document, a feature amount normalization unit 134, and a feature amount ratio feature amount extraction unit 135.

  The main content determination unit 14 determines whether or not the content is the main content from the feature amount extracted by the feature amount extraction unit 13. As shown in FIG. 9, the feature amount input unit 141, the text determination unit 142, the main content determination unit 14 The content determination unit 143 is configured.

  As shown in FIG. 10, the main content output unit 15 includes a tagged text output unit 151 that is used when tagged text is requested, an untagged text output unit 152 that is used when untagged text is requested, The data output unit 153 combines and outputs the segments determined to be the main content output from the tagged text output unit 151 or the untagged text output unit 152.

  The processing of the main content extraction unit 10 is roughly as follows.

  (1) The Web document acquisition / input unit 11 inputs a Web document (data) to be processed.

  (2) The Web document dividing unit 12 divides the Web document acquired in (1).

  The following processing is performed for each divided Web document.

  (3) The feature amount extraction unit 13 extracts feature amounts for main content determination.

  (4) The main content determination unit 14 determines whether the content is the main content from the feature amount extracted in (3).

  (5) The main content output unit 15 combines the parts determined as the main content in (4) to obtain the final output.

  The operation of the main content extraction unit 10 will be described in detail below.

[1-2] Input of Web document:
The input of the Web document is a process performed by the Web document acquisition / input unit 11.

  The data input unit 111 of the Web document acquisition / input unit 11 acquires the URL of the Web document from which the main content input by the user is extracted or the file itself. When the input is a URL, the URL input unit 113 and the Web document acquisition unit 114 acquire the URL destination Web document, and when the file is directly input, the Web document file input unit 112 acquires the file. The character code conversion unit 115 converts the character code of the Web document acquired from the Web document file input unit 112 and the Web document acquisition unit 114 into UTF-8 to be unified.

[1-3] Web document division:
The division of the Web document is a process performed by the Web document dividing unit 12.

  First, the advertisement target area extraction unit 121 of the Web document dividing unit 12 extracts an area including a content tag such as an Internet advertisement. Here, the Internet advertisement is a tag used by an advertising company such as google (registered trademark) or Overture (registered trademark) to narrow down main contents for advertisement distribution. In the case of google (registered trademark) advertisement, the range from <!-google_ad_section_start-> to <-google_ad_section_end-> corresponds to that area. These tags have a slightly different character string depending on the Web document or are displayed in uppercase letters, so there are some differences in character string notation by using regular expressions that are not case sensitive or using wildcard regular expressions. Process to absorb. Hereinafter, it is assumed that processing for absorbing a slight difference is performed in the description of processing using regular expressions.

  In the subsequent processing, when there is an internet advertisement area, the above-described area extraction process is performed. When there is no internet advertisement area, the first input Web document is processed.

  Next, the tag or area removing unit 122 that causes noise performs processing to remove an extra tag or area or a specific character string. The tag or area to be removed may be a comment tag explaining HTML of a Web document, JavaScript, or a form tag. The tags and areas to be removed are described below.

・ Comment tags that begin with "<!" And end with "->";
-The area enclosed by the "</ script>" tag from the "<script>"tag;
・ A region enclosed by "</ style>" tag from "<style>"tag;
・ A region enclosed by "</ select>" tag from "<select>"tag;
・ A region enclosed by "</ noscript>" tag from "<noscript>"tag;
・ A region enclosed by "</ form>" tag from "<form>"tag;
• Consecutive whitespace (excluding single whitespace);
・ Consecutive tab strings (excluding single tabs);
The tag or area removing unit 122 that causes noise removes the above tag, area, and character string using a regular expression. Since there may be cases where the alt attribute and class attribute exist in the tag, in such a case, division is performed using a regular expression that takes into account the tag including them (eg, <style class = "hoge">).

  Next, the Web document dividing unit 123 divides the Web document. The division rule is to divide using the following tags.

・ <Div>
・ </ Div>
・ <Td>
・ </ Td>
Since there may be cases where the alt attribute and class attribute exist in the tag, in such a case, division is performed using a regular expression that takes into account the tag including them (eg, <div class = "hoge".).

  Each of the Web documents divided by the Web document dividing unit 12 is called a “segment”, and these are stored in a memory (not shown). Thereafter, the processes of the feature amount extraction unit 13 and the main content determination unit 14 are performed for each segment.

[1-4] Feature amount extraction:
Here, the feature quantity extracted by the feature quantity extraction unit 13 will be described. The feature amount extraction unit 13 extracts feature amounts described below for each segment stored in a memory (not shown), and determines the main content portion of the Web document.

[1-4-1] Character String Displayed in Web Document The feature amount is a feature amount extracted by the character string feature amount extraction unit 133 displayed in the Web document. The character string described here does not include a character string that is not displayed by a Web browser such as an HTML tag.

  As a character string displayed in a Web document, there are the amount of character strings and the number of punctuation marks.

[1-4-1-1] Amount of character string:
In general, the main content portion of a Web document includes many character strings as compared with a portion that is not the main content. The same can be said for a Web document having a small number of character strings as a whole. Therefore, the character string feature amount extraction unit 133 displayed in the Web document uses the number of character strings included in the divided Web document as the feature amount. Then, the feature amount normalization unit 134 executes two methods, ie, a method for normalizing the amount of the character string and making it a feature, and a method for making the feature amount using the absolute value of the character string, thereby obtaining a final character string. The feature amount of

-Method to normalize the amount of character string and use it as a feature value:
The feature amount normalization unit 134 performs normalization by setting the feature amount of the segment having the maximum character string amount to 1 in all the segments. For example, when the maximum amount of character strings in all segments is 200 and the amount of character strings in a certain segment is 100, the feature amount of the amount of character strings in that segment is 0.5. By performing such normalization, it is possible to extract main contents even in a Web document having a small character string as a whole.

・ Method to make the feature value using the absolute value of the character string amount:
Although the above-described normalization and feature amount technique is effective for Web documents with a small number of character strings as a whole, normalization increases the amount of character strings as a whole, and the main content Inconvenience occurs when there is a large difference in the amount of character strings between partial segments. For example, when the maximum amount of character strings in all segments is 1000 and the amount of character strings in a certain segment is 100, the feature amount of the amount of character strings in that segment is 0.1. Although the amount of character strings should be large, such detriment is caused by normalization. Therefore, it is necessary to perform a technique for using the absolute value of the character string as a feature amount. Specifically, the feature quantity normalization unit 134 uses a technique in which the feature quantity of the character string is set to 1 when a certain value is exceeded. For example, when the amount of character strings in a segment is 100, the feature amount is 1 when the amount of character strings is 5 or more, the feature amount is 1 when the amount of character strings is 10 or more,. A feature quantity is created so that a feature quantity with a column quantity of 105 or more is 0,..., And a feature quantity with a string quantity of 200 or more is 0. In this way, by using a method of setting the feature amount to 1 when a certain character string amount is exceeded, the absolute value of the character string amount can be used as the feature amount while the maximum value of the feature amount remains 1. it can. In the example, the feature value interval of the absolute value of the character string amount is set to 5, but it is preferable to use an appropriate interval in some cases. A method of setting the interval between feature quantities using a multiplier of 2 such as 8, 16, 32, and 64 is also conceivable. Similarly, the maximum x of the character string amount x or more is changed to an appropriate value in some cases. In order to reduce the amount of calculation in main content determination, the value of x may be reduced.

  A case where there is no character string in the segment is also conceivable. In this case, it is possible to determine that the content is not the main content by the text determination unit 142 in the main content determination unit 14 without extracting the feature amount described below. Therefore, at the time of actual execution, the following feature amount extraction is not performed, and the corresponding segment is determined as non-main content. This process is not performed when the feature amount is learned.

[1-4-1-2] Number of punctuation marks:
Segments that tend to be noisy, such as Web advertisements, tend to have a large amount of character strings but a small number of punctuation marks. Therefore, it is characterized by the number of punctuation marks. Specifically, “,”, “,”, “.” Included in the character string in the segment. ",". ","! , “・”, “? ] And "..." are counted as feature quantities. This feature amount is also calculated in two ways: a feature amount by normalization described in terms of the amount of character string and a feature amount by absolute value. In the calculation method, the same method as described in the character string quantity of [1-4-1-1] is used.

[1-4-2] Tag Information Here, feature amounts related to tag information such as HTML tags handled by the tag information feature amount extraction unit 132 will be described. Tag information includes
・ Number of text-based HTML tags;
-Number of consecutive occurrences of text-based HTML tags;
The number of link list tags, the amount of character strings that do not include character strings displayed in the Web document;
There is.

[1-4-2-1] Number of text-type HTML tags:
When there are many character strings displayed in a Web document within a certain segment, many HTML tags relating to text are included. In CGM (Consumer Generated Media) such as blogs, there are few character strings displayed in Web documents, but there are many cases where users frequently use line feed tags. Therefore, the tag information feature amount extraction unit 132 uses the number of HTML tags related to the text as the feature amount. This feature amount is also normalized in the same manner as the feature amount used in the character string displayed in the Web document, and is stored as a final feature amount in a memory (not shown). For example, if the maximum HTML tag amount in all segments is 10 and the HTML tag amount in a segment is 5, the feature amount of the HTML tag amount in that segment is 0.5. The text-type HTML tags that are actually used target the following tags.

・ <P>
・ </ P>
・ <br>
・ </br>
・ <Font>
・ </ Font>
Since the size attribute and class attribute may exist in the tag, in that case, count using a regular expression that takes into account the tag including them (example: <font size = "+ 1">) .

[1-4-2-2] Number of consecutive occurrences of text-type HTML tags:
In a segment in which character strings displayed in a Web document are concentratedly described, there are many text-based tags, and at the same time, text-based HTML tags appear continuously. Here, appearing continuously means that HTML tags such as other anchor links do not appear in between. Therefore, the number of consecutive appearances of the text type HTML tag described in [1-4-2-1] is used as the feature amount. This feature amount is also normalized in the same manner as the feature amount used in the character string displayed in the Web document, and is stored as a final feature amount in a memory (not shown). For example, when the continuous amount of the maximum HTML tag in all segments is 10, and the continuous amount of HTML tags in a segment is 5, the feature amount of the character string amount of that segment is 0.5. .

[1-4-2-3] Number of link list tags:
When there are many link list tags in a segment, there are many anchor links such as navigation links in the segment, and there is a high possibility that the segment will not become main content. Therefore, the number of link list tags is used as a feature amount. This feature amount is also normalized in the same manner as the feature amount used in the character string displayed in the Web document, and is stored as a final feature amount in a memory (not shown). For example, if the maximum link list tag amount in all segments is 10 and the link list tag amount in a segment is 5, the feature amount of the link list tag amount in that segment is 0.5. It becomes. And the link list tag to be specifically used targets the following tags.

・ <Li>
・ <Ul>
・ <Dl>
・ <Dd>
・ <Ol>
Since there may be a case where an alt attribute or a class attribute exists in the tag, in that case, counting is performed using a regular expression that considers the tag including those (for example, <font class = "hoge").

[1-4-2-4] Amount of character strings (including HTML tags) that do not include character strings displayed in the Web document:
If there are many character strings (including HTML tags) that are not displayed on the Web in a certain segment, there is a high possibility that the segment is not the main content such as an advertisement. Therefore, the amount of character strings (including HTML tags) other than the character strings displayed on the Web document is used as the feature amount. This feature amount is also normalized in the same manner as the feature amount used in the character string displayed in the Web document, and is stored as a final feature amount in a memory (not shown). For example, if the amount of character strings that are not displayed on the maximum Web in all segments is 100 and the amount of character strings that are not displayed on the Web in a segment is 50, the character strings that are not displayed on the Web of the segment The feature amount of the quantity is 0.5.

[1-4-3] Anchor link information:
In the following, a method for extracting feature values related to anchor links handled by the anchor link information feature value extraction unit 131 of the feature value extraction unit 13 will be described. Anchor link information includes
・ Number of anchor links;
-Average amount of text for each anchor link;
-Total value of all anchor link strings;
The amount of anchor link URL of the maximum character string;
• include anchor links for advertisements;
There is.

[1-4-3-1] Number of anchor links:
In a certain segment, a segment containing a large number of anchor links is likely not to be main content. Therefore, the number of anchor links is used as a feature amount. Specifically, the number of anchor links represented by <a href=...> ... </a> tags is used as the feature amount. This feature quantity is also normalized by the feature quantity normalization unit 134 in the same manner as the feature quantity used in the character string displayed in the Web document, and is stored as a final feature quantity in a memory (not shown). To do. For example, if the maximum number of anchor links in all segments is 10, and the number of anchor links in a certain segment is 5, the feature quantity of the number of anchor links in that segment is 0.5. Since the anchor link tag may include a class attribute and an alt attribute, the number of anchor link tags is counted using a regular expression.

[1-4-3-2] Average amount of character string of each anchor link:
When the character string of each anchor link is large on average, the segment is likely to be a navigation link such as a related article. Also, if the average number of anchor link character strings is small, there is a high possibility that the keyword search link is included in the main content. Therefore, the average amount of anchor link character strings included in the segment is used as the feature amount. The character string of the anchor link corresponds to the XXX part of <a href='...'> XXXXX </a>. The feature amount normalization unit 134 similarly normalizes the feature amount used in the character string displayed in the Web document, and stores it in a memory (not shown) as a final feature amount.

[1-4-3-3] Total value of all anchor link character strings:
When the total amount of anchor link character strings included in a segment is large, there is a high possibility that the segment is a navigation link. Therefore, the total amount of anchor link character strings included in the segment is used as the feature amount. The character string of the anchor link corresponds to the xxx part of <a href='...'xxx </a>. The feature amount normalization unit 134 similarly normalizes the feature amount used in the character string displayed in the Web document, and stores it in a memory (not shown) as a final feature amount.

[1-4-3-4] Amount of anchor link URL of maximum character string:
If the URL character string of the anchor link destination is very long, the segment is likely to be an advertisement. Therefore, the character string of the URL with the longest anchor link destination in the segment is used as the feature amount. The URL character string of the anchor link destination described here corresponds to the portion of ΔΔΔ in <a href='ΔΔΔ'...></a>. The feature amount normalization unit 134 similarly normalizes the feature amount used in the character string displayed in the Web document, and stores it in a memory (not shown) as a final feature amount.

[1-4-3-5] Whether to include an anchor link related to the advertisement:
An anchor link including a URL related to an advertisement is likely to include a characteristic character string. For example, “adclick”, “adnet”, “banner”, and the like. Therefore, the anchor link information feature quantity extraction unit 131 extracts a feature quantity with a feature quantity of 1 when there is an anchor link including a URL including a character string that is likely to be an advertisement, and 0 when there is no such link. And stored in a memory (not shown). Character strings that are likely to become advertisements are listed on sites such as adblock plugin, which is an add-in for Firefox for removing advertisements, and so are used.

[1-4-4] Feature quantity ratio:
Here, the feature amount using the ratio between features described in [1-4-1] to [1-4-3] handled by the ratio feature amount extraction unit 135 will be described.

[1-4-4-1] Ratio of the number of text-type tags and the number of consecutive appearances of text-type tags:
A segment having a large number of text-type tags and a large number of consecutive appearances of text-type tags is likely to be the main content because the text is densely written. However, although there are a large number of text-type tags, a segment with a small number of consecutive occurrences of text-type tags can be said to have a high possibility that it is not the main content because the segment size is large. Therefore, the feature amount ratio feature amount extraction unit 135 uses the ratio of the number of text tags and the number of consecutive appearances of text tags as a feature amount. Specifically, a value having the number of text tags as a denominator and the number of consecutive text tags as a numerator is used as a feature amount. Here, when the number of text tags is 0, the denominator is 0. In this case, the feature amount of the ratio between the number of text tags and the number of consecutive appearances of text tags is 0. This feature amount is also normalized by the feature amount normalization unit 134 in the same manner as the feature amount used in the character string displayed in the Web document, and is stored as a final feature amount in a memory (not shown). The larger the feature amount, the higher the possibility that it is the main content.

[1-4-4-2] Ratio of character string and tag displayed on Web:
If there are many character strings displayed on the Web in a certain segment, there is a high possibility that it will be the main content, but there may be many tags such as HTML tags in the same segment. In this case, as described in [1-4-4-1] above, there is a possibility that the segment is not the main content only with a large segment size. Therefore, the feature amount ratio feature amount extraction unit 135 copes with this case by using the character string / tag ratio displayed on the Web as the feature amount. Specifically, a value having the number of character strings displayed on the Web as a numerator and the number of tags as a denominator is defined as a feature amount. The greater this feature, the more likely it is the main content. This feature quantity is also normalized by the feature quantity normalization unit 134 in the same manner as the feature quantity used in the character string displayed in the Web document, and is stored as a final feature quantity in a memory (not shown). To do. Since the denominator is 0 when the number of tags is 0, the feature amount is 1.

[1-4-4-3] Ratio of the number of anchor links to the number of link list tags:
Within a certain segment, the more anchor links or link list tags, the more likely that segment is not the main content, but because these segments are large, these feature quantities will increase by chance. Is also possible. Therefore, the feature quantity ratio feature quantity extraction unit 135 uses the ratio of the number of anchor links and the number of link list tags as the feature quantity. Specifically, the number of anchor links is used as the denominator, and the number of link list tags is used as the numerator, which is the feature amount. The larger the feature amount, the higher the number of links with respect to the area of the segment, and the higher the possibility that it is not the main content. This feature amount is also normalized by the feature amount normalization unit 134 in the same manner as the feature amount used in the character string displayed in the Web document, and is stored as a final feature amount in a memory (not shown). To do. When the number of anchor links is 0, the denominator is 0, so the feature amount is 0.

[1-5] Determination of main content:
Here, the main content determination unit 14 determines whether the content is the main content by using the feature amount described in [1-4] obtained by the feature amount extraction unit 13 and stored in a memory (not shown). A method to be performed using the above determination will be described. The determination is performed using a machine learning algorithm such as Support Vector Machine (SVM), maximum entropy method, naive Bayes method or the like.

  FIG. 11 is a flowchart of the feature quantity parameter estimation method according to the embodiment of the present invention. First, it is determined for each segment from which the feature amount is extracted whether it is the main content manually, and training data is created (steps 301 to 303). Here, in [1-4-1-1], it is described that if there is no character string displayed on the Web, it is not regarded as a main content. However, in the method using machine learning, learning is performed as a negative example. Since it is effective, such data is also adopted as training data. Then, learning is performed using the feature amount of the segment (step 304), and a weight for each feature amount is calculated (step 305). When speed is important, learning is performed by the maximum entropy method, and when accuracy is important, learning is performed using a Support Vector Machine using a second-order polynomial kernel. Then, using these learned parameters, the feature amount input unit 141 inputs the segment feature amount to the main content determination unit 143, and the main content determination unit 143 determines whether or not each segment is the main content.

[1-6] Output of main content part:
After determining whether or not the main content determination unit 14 described in [1-5] is the main content, the data output unit 153 of the main content output unit 15 determines the segment determined as the main content by the learning device. Are combined into the final output. Examples of output are shown in FIGS.

  Here, when it is desired to use this apparatus as pre-processing for information retrieval, the tag-attached text output unit 151 is used to leave and output tags such as HTML tags. Further, when it is desired to analyze the contents of a Web document by information recommendation or the like, tags such as HTML tags are deleted and output using the untagged text output unit 152.

[1-7] Processing for improving accuracy:
As a pre-process for extracting the feature value described in [1-4], a technique of removing unnecessary character strings and the like is effective. By removing unnecessary character strings described below in advance, the determination accuracy of main contents is improved.

・ &Nbsp;
・ &Lt;
・ &Gt;
・ &Amp;
・ &Laquo;
・ &Raquo;
These character strings are special characters used when symbols used in HTML tags or the like are displayed on the Web browser. In addition, HTML special characters other than the special characters listed above are also subject to deletion. Special characters tend to cause noise during learning because the amount of character strings is smaller than the actual displayed character strings.

[1-8] Selection of feature value for implementation reason:
When the main content extraction method is incorporated in a user PC or the like, it is difficult in terms of processing amount to perform processing using all the feature amounts described in [1-4]. Therefore, the processing amount is reduced by narrowing down the feature amount to be extracted. Here, the learning model by machine learning creates a learning model for each narrowed-down feature quantity model.

[2] Keyword candidate extraction process (step 300):
Below, the important keyword candidate extraction part 2 is demonstrated.

  Conventional general keyword extraction methods used morpheme analysis and specific expression extraction. However, morpheme analysis methods have problems related to compound noun extraction and important keyword narrowing down. In the extraction, since only a narrow range of keywords such as a person name, an organization name, and a place name can be extracted, keyword candidates that cover the user's interests cannot be extracted. Therefore, the present invention attempts to solve these problems by using, as keyword candidates, headwords of Wikipedia (registered trademark), which is a user-participation-type online encyclopedia. Since it is a headword of Wikipedia (registered trademark), it is organized systematically and has features that uniquely represent real-world events, so it can be said to be a language resource that covers a wide range of interests of users. In addition, the keywords listed in Wikipedia (registered trademark) are important, and since only keywords with abundant information on various topics are registered when talking about keywords, it is possible to narrow down important keywords. Become. As a specific keyword extraction method, a Wikipedia (registered trademark) entry word is registered in the keyword dictionary 101 as a dictionary for morphological analysis, and a keyword is extracted from a morphological analysis result using the keyword dictionary 101. When registering in the keyword dictionary 101, the dictionary is weighted so as to extract the longest keyword.

[3] Keyword importance calculation processing (step 400):
In [2] above, keyword candidates that are considered important are narrowed down using Wikipedia (registered trademark) headwords. In [3], the appearance frequency calculation unit 3 and the keyword importance calculation unit 4 A method for calculating the importance level of a keyword in order to position a more important keyword in the higher rank will be described.

[3-1] WebIDF:
It is assumed that the weight considering the appearance probability in a large-scale Web document collection collected by the search engine is WebIDF and stored in the WebIDF storage unit 102. Specifically, the appearance frequency calculation unit 3 inputs a keyword into the search engine, calculates an IDF value from the number of hits of the Web document obtained as a result, and sets this as WebIDF. Many keywords appearing in the Web document set are weights for determining that they are general keywords and not characteristic keywords. The WebIDF calculation formula for keyword k is shown below.

ここで、Ｎはキーワードの閾値で、ｎ_ｋはキーワードｋのヒット数である。ここでキーワードｋの検索エンジンのヒット数の閾値Ｎは、ｎ_ｋを降順にソートし、不必要なキーワードが少なくなってきた辺りのｎ_ｋの値を用いる。その理由として、ｎ_ｋが高ければ高いほど一般的なキーワードである可能性が高く、また、閾値Ｎを設定すると閾値Ｎ以上のｎ_ｋを持つキーワードｋのWebIDF値は常に負となり、常にスコアとして低い値となるためである。なお、Ｎの推奨値は20000000である。

Here, N is a threshold value of the keyword, and _nk is the number of hits of the keyword k. Here, for the threshold value N of the number of hits of the search engine for the keyword k, _nk is sorted in descending order, and the value of _nk around which unnecessary keywords are reduced is used. The reason is that the higher the n _k high general be a keyword, also, WebIDF value of the keyword k with the threshold value N or more n _k Setting the threshold N is always negative, as always score It is because it becomes a low value. The recommended value for N is 20000000.

[3-2] BM25:
Here, a keyword weighting method using the appearance frequency in the document and the appearance distribution in the document set by the keyword importance calculation unit 4 will be described. BM25 which is the existing technique is used for these weights.

ここで、ｄは文書であり、ｋは文書に含まれるキーワードである。そして、tf(d,k)はｄ内のキーワードｋの出現頻度、dlは文書長、avdlは文書集合内における平均文書長、Nは総文書数である。ｋ_１とｂは自由パラメータで、ｋ_１はtf(d,k)に関するパラメータ、ｂはdlに関するパラメータである。

Here, d is a document and k is a keyword included in the document. Tf (d, k) is the appearance frequency of the keyword k in d, dl is the document length, avdl is the average document length in the document set, and N is the total number of documents. k ₁ and b are free parameters, k ₁ is a parameter related to tf (d, k), and b is a parameter related to dl.

  If a document set cannot be prepared and keywords are to be extracted from one document, the following calculation formula is used.

なお、式に関するノーテーションは（２）式と同様である。

Note that the notation related to the equation is the same as that in the equation (2).

[3-3] Calculation of Keyword Importance of Title and Body:
In [3-2] above, the weighting method based on the appearance frequency using the BM25 by the appearance frequency calculation unit 3 is described. In this section, the keyword importance calculation unit 4 describes the calculation method of the keyword importance using the appearance frequency in the title of the document and the appearance frequency in the text.

  Specifically, the keyword importance calculation unit 4 calculates a score using a linear sum of the BM25 value calculated from the document title (or file name) and the BM25 value calculated from the text. The calculation formula is shown below.

ここで、BM25_allはタイトルと本文の出現頻度を用いたBM25スコアで、BM25_titleはタイトルでの出現頻度を用いたBM25スコア、BM25_bodyは本文でのBM25スコアとなる。αはパラメータである。αの推奨値は0.7とする。

Here, BM25 _all in the BM25 score using the frequency of occurrence of title and body, BM25 _title is BM25 score using the frequency of occurrence of the title, BM25 _body is the BM25 score in the text. α is a parameter. The recommended value for α is 0.7.

[3-4] Keyword importance calculation using proper nouns of morphemes:
In general news articles, proper nouns such as personal names and organization names are often described in abbreviated form after first being described in formal names. For example, the case where “Aso” is written after “Jiro Aso” is first written is applicable. When it is described in abbreviated form, there is a problem that a keyword related to the subject of an article may appear less frequently than other unimportant keywords. Therefore, we propose a simple method for dealing with abbreviations using morphological analysis results. A general morphological analyzer can output a personal name (last name, first name), a place name, and an organization name as parts of speech. Therefore, this problem is addressed by replacing the appearance frequency of the keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result with the appearance frequency of the morpheme. For example, when the keyword “Jiro Aso” appears once and the proper noun of the morpheme “Asou” appears three times, the frequency of “Jiro Aso” appears three times.

  On the contrary, since the place name is described without being omitted, there is a problem that the number of appearances of the keyword becomes too large. Therefore, we propose the number of appearances of place names nouns obtained from morphological analysis results. The calculation formulas of the method described above used in the keyword importance calculation unit 4 are shown below.

tf (d, k) = α _person・ match (d, k, m _k )
tf (d, k) = α location · match (d, k, m k) (5)
tf (d, k) = α _organization・ match (d, k, m _k )
Here, tf (d, k) is a score of the keyword k based on the appearance frequency in the document d, and α _person , α _location , and α _organization are coefficients relating to the person name, place name, and organization name, respectively. Match (d, k, m _k ) is an appearance frequency of the morpheme m _k included in the keyword k in the document d. The tf (d, k) calculated by the above formula (5) is used in the keyword importance calculation formula using the appearance frequency described in [3-2] and [3-3].

[3-5] Keyword Importance Calculation (Keyword Specific Importance):
In this section, the keyword importance calculation unit 4 calculates the importance of the keyword. The importance of a keyword is defined as the more important (the deeper content) the keyword is, which has a high degree of name recognition and high topicality and has various topics when talking about the keyword. The importance of this keyword is defined as the keyword Called inherent importance. The keyword unique importance calculated by the keyword importance calculation unit 4 is stored in the unique importance storage unit 105.

[3-5-1] Method using link structure in Wikipedia (registered trademark):
Below, the keyword importance calculation part 4 demonstrates the method of calculating keyword specific importance using the link structure in Wikipedia (trademark).

  In general Web document ranking methods such as HITS and PageRank, Web document ranking is performed using the link structure of Web pages. However, since a Wikipedia (registered trademark) document has one headword (keyword) per document, the document ranking obtained from the Wikipedia (registered trademark) link structure is regarded as a keyword ranking. Can be tricked. Therefore, in this apparatus, the keyword importance calculation unit 4 applies a ranking method specialized for Wikipedia (registered trademark), thereby using the keyword importance stored in the keyword importance 1 storage unit 103 to identify the keyword. The importance is calculated and stored in the unique importance storage unit 105. The HITS algorithm is used as the base algorithm.

  In the HITS algorithm, it is defined that all Web documents are composed of authority (content) and hub (link collection). And ranking web documents by repeatedly executing the hypothesis that the authority linked to many good hubs is the better authority, and the two hypotheses that the hub linked to many good authorities is the better hub . However, since the HITS algorithm is an algorithm that models the link structure of a Web document in the Web world, there is a slight difficulty when applied as it is to Wikipedia (registered trademark) with a very dense link structure. In view of this, this apparatus proposes a method corresponding to the characteristic structure of Wikipedia (registered trademark) and the dense link structure. Then, the ranking based on the authority value calculated from this algorithm is approximated to the attenuation function proposed by the present method to obtain the final keyword-specific importance.

● Consideration of text volume:
The headwords of Wikipedia (registered trademark) tend to have a larger amount of text description as headwords with higher name recognition and topicality. Therefore, when calculating the authority value, the weight (text (k)) that the document is more important as the text amount of the document itself is larger is considered.

● Ratio of own link and backlink:
In general, the headword of Wikipedia (registered trademark) has a larger number of self-linked and linked links as the famous keyword. However, keywords having a broad concept such as place names and genre names are keywords that are easy to quote, and thus tend to have an overwhelmingly larger number of linked links than the number of links. The normal HITS algorithm uses the hypothesis that the authority that is linked to many from a good hub is a good authority, but if there are an overwhelming number of linked links, these hypotheses are not expected to hold. On the other hand, headlines such as new actors and topic words, which have recently become popular, have a small number of citations because of their shallow days, but tend to have many links. Therefore, it is necessary to increase authority even with a small number of linked links. To solve these problems, flink (k) / blink (k) is taken into account when calculating the authority value. Here, flink (k) represents the number of own links included in the keyword k document, and blink (k) represents the number of linked links linked to the keyword k document.

● Handling of headwords that are not clearly calculated by authority:
In Wikipedia (registered trademark) headwords, there are headwords that clearly do not become authority, such as “˜year” and “˜list”. These headwords are prone to noise because they have many links themselves and many links. Therefore, this problem is addressed by not always changing the authority value of the headword that is clearly not authority.

● Hub average link quality:
Wikipedia (registered trademark) documents have many links, but there is a poor quality document as a hub. Therefore, by changing to the hypothesis that hubs with high average keyword authority are important, the weight to lower the hub value of low-quality documents as many hubs but own links

を考慮する。

Consider.

● Handling redirects:
A Wikipedia (registered trademark) document includes a redirect in order to eliminate the headword typographical error. For example, “ICHIRO” includes “SUZUKI ICHIRO” and “ICHIRO” redirect. Redirect not only has the effect of grouping differently-written keywords uniquely, but also has a great influence on the linked structure of the keywords. Also solve the problem.

  The final Wikipedia (registered trademark) ranking algorithm is defined by the following equation.

ここで、a(k)は、キーワードｋのauthority値で、h(k)はhub値である。そして、flink(k)はキーワードｋから自リンク数、blink(k)はキーワードｋからの被リンク数である。text(k)は、キーワードｋが見出し語になっているWikipedia（登録商標）文書の文字数（アンカーリンク対象の文字列は除く）であり、Ｋ'はキーワードｋが見出し語になっているWikipedia（登録商標）文書内に含まれるリンク数の総数となる。なお、式（６）は交互に繰り返し計算を行うものとし、１０回ほど繰り返し行うとよい。

Here, a (k) is the authority value of the keyword k, and h (k) is the hub value. Flink (k) is the number of links from the keyword k, and blink (k) is the number of links from the keyword k. text (k) is the number of characters of the Wikipedia (registered trademark) document in which the keyword k is the entry word (excluding the character string to be anchored), and K ′ is Wikipedia (the keyword k is the entry word). The total number of links included in the registered trademark document. It should be noted that Equation (6) is repeatedly calculated alternately, and is preferably repeated about 10 times.

  Then, the keywords are ranked in descending order using the authority value calculated by the above equation (6). Then, the score is approximated using the following keyword specific importance calculation formula to obtain keyword specific importance WKIS (k) using the link structure in Wikipedia (registered trademark), which is stored in the specific importance storage unit 105. .

ここで、y₁はキーワード固有重要度の上界であり、y₀はキーワード固有重要度の下界である。そして、k_rはキーワードｋのauthorityの値による順位、Ｋはキーワードの総数、ａはスコアの勾配係数で、ａの値が大きくなればなるほどスコアの勾配が急になる。本関数の特徴は、キーワードの候補数（ｘ軸の要素数）に左右されることなく、また、上界と下界を設定でき、またキーワードの候補数の上位１０％以内において、最大値と最小値の差が３０％〜７０％以内に収まる減衰曲線を描き、そしてパラメータ係数にて減衰度合いを整数値を用いることで容易に設定できる特徴をもつスコア関数である（勿論、減衰度合いの設定は実数値でも設定可能である）。y₁、y₀の推奨値は、それぞれ1，0.1で、勾配係数の推奨値は3から7である。

Here, y ₁ is the upper limit of keyword-specific importance, and y ₀ is the lower limit of keyword-specific importance. Then, k _r values by rank, K is the total number of keywords authority keyword k, a is a gradient coefficient of the score, is steep indeed score the greater the value of a is. The features of this function are not affected by the number of keyword candidates (number of elements on the x-axis), and the upper and lower bounds can be set, and the maximum and minimum values are within the top 10% of the number of keyword candidates. It is a score function having a feature that draws an attenuation curve in which the difference of values falls within 30% to 70%, and can easily set the attenuation degree by using an integer value by the parameter coefficient (of course, the setting of the attenuation degree is (A real value can also be set). The recommended values for y ₁ and y ₀ are 1, 0.1, respectively, and the recommended value for the gradient coefficient is 3 to 7.

  The keyword-specific importance calculated from the link structure in Wikipedia (registered trademark) is generally low in name because the important keywords in Wikipedia (registered trademark) in the online encyclopedia with user participation are ranked high. The key words are deep and important.

[3-5-2] A method using the search query input count of the search engine:
This method is a method in which a search query that is frequently input to an actual search engine is regarded as an important keyword. The number of search queries input is preferably large-scale data such as a search query input on a portal site. Here, it is assumed that the target search query is a keyword existing in Wikipedia (registered trademark). Similar to the method using the link structure in Wikipedia (registered trademark), this method also uses the keyword specific importance calculation formula (7) to approximate the score, and uses the search query as the keyword specific importance. QKIS (k) is stored in the unique importance storage unit 105.

ここで、y₁はキーワード固有重要度の上界であり、y₀はキーワード固有重要度の下界である。そして、k_rはキーワードｋの検索回数による順位、Kはキーワードの総数、aはスコアの勾配係数で、aの値が大きくなればなるほどスコアの勾配が急になる。本関数の特徴は、キーワードの候補数（ｘ軸の要素数）に左右されることなく、また、上界と下界を設定でき、またキーワードの候補数の上位１０％以内において、最大値と最小値の差が３０％〜７０％以内に収まる減衰曲線を描き、そしてパラメータ係数にて減衰度合いを整数値を用いることで容易に設定できる特徴をもつスコア関数である（勿論、減衰度合いの設定は実数値でも設定可能である）。

Here, y ₁ is the upper limit of keyword-specific importance, and y ₀ is the lower limit of keyword-specific importance. K _r is the ranking based on the number of searches for the keyword k, K is the total number of keywords, a is the score gradient coefficient, and the score gradient becomes steeper as the value of a increases. The features of this function are not affected by the number of keyword candidates (number of elements on the x-axis), and the upper and lower bounds can be set, and the maximum and minimum values are within the top 10% of the number of keyword candidates. It is a score function having a feature that draws an attenuation curve in which the difference of values falls within 30% to 70%, and can easily set the attenuation degree by using an integer value by the parameter coefficient (of course, the setting of the attenuation degree is (A real value can also be set).

Moreover, although it exists as a headword in Wikipedia (registered trademark), it may not exist in a search query of a search engine. If that is the score of the relevant keywords y _0. Recommended values for y ₁ and y ₀ are 1 and 0.1, respectively, and recommended values for the gradient coefficient are 3 to 7.

  The keyword-specific importance calculated from the search query input count of the search engine stored in the keyword importance 2 storage unit 104 is determined by the actual search query input count. Keywords tend to be higher.

[3-6] Final keyword-specific importance calculation:
In the above [3-5], the keyword-specific importance WKIS (k) calculated from the link structure in Wikipedia (registered trademark) and the keyword-specific importance QKIS (k) calculated from the search query input count of the search engine. Stated. However, QKIS (k) has a problem that when the search query data collection period is short, there is a problem that it is strongly influenced by topics that occurred on the Internet during that period, and there are also many life-time queries such as Internet site names. is there. Therefore, the keyword importance calculation unit 4 uses the linear sum of WKIS (k) and QKIS (k) stored in the unique importance storage unit 105 as the final keyword unique importance, so that the topicality is increased. An importance level is calculated that ranks keywords that are high and deep and easy to be input as a search query.

Keyword_score (k) = α ・ WKIS (k) + (1-α) ・ QKIS (k) (9)
Since it has been confirmed through experiments that the keyword importance of WKIS (k) is generally more accurate than QKIS (k), the value of α is adjusted in the range of 0.5 to 0.8.

[3-7] Keyword importance calculation at appearance position in document:
Below, the process in the position information calculation part 5 is demonstrated.

  In an article in which a general description such as a news article or a column is made, it is more likely that an important keyword is included as the head of the sentence comes. Therefore, the importance degree is calculated that the more important the keyword is at the head of the sentence. This method also calculates the keyword importance Pos (k) using the appearance position by using the keyword specific importance calculation formula (7) as in the method using the link structure in Wikipedia (registered trademark). And stored in the position importance storage unit 106.

ここで、y₁は出現位置を用いたキーワード重要度の上界であり、y₀は出現位置を用いたキーワード重要度の下界である。そしてk_pはキーワードｋの文位置（文の位置は、文の先頭から１文ずつ（句点までを１文とする）数え上げたものを用い、さらにそのキーワードにおいて、最も先頭の位置を用いる）、Pは最後尾の文の位置、aはスコアの勾配係数で、aの値が大きくなればなるほどスコアの勾配が急になる。本関数の特徴は、キーワードの候補数（ｘ軸の要素数）に左右されることなく、また、上界と下界を設定でき、またキーワードの候補数の上位１０％以内において、最大値と最小値の差が３０％〜７０％以内に収まる減衰曲線を描き、そしてパラメータ係数にて減衰度合いを整数値を用いることで容易に設定できる特徴をもつスコア関数である（勿論、減衰度合いの設定は実数値でも設定可能である）。y₁、y₀の推奨値は、それぞれ1、0.5で勾配係数の推奨値は1から5である。ブログ記事のような一般的な記述がなされていない文書を多く処理する場合には、y₀の値を大きくし、勾配係数の値も小さくするのがよい。

Here, y ₁ is the upper bound of the keyword importance using the appearance position, and y ₀ is the lower bound of the keyword importance using the appearance position. K _p is the sentence position of the keyword k (the sentence position is counted from the beginning of the sentence one sentence at a time (up to the punctuation is one sentence), and the first position in the keyword is used). P is the position of the last sentence, a is the slope coefficient of the score, and the higher the value of a, the steeper the slope of the score. The features of this function are not affected by the number of keyword candidates (number of elements on the x-axis), and the upper and lower bounds can be set, and the maximum and minimum values are within the top 10% of the number of keyword candidates. It is a score function having a feature that draws an attenuation curve in which the difference of values falls within 30% to 70%, and can easily set the attenuation degree by using an integer value by the parameter coefficient (of course, the setting of the attenuation degree is (A real value can also be set). The recommended values for y ₁ and y ₀ are ₁ and 0.5, respectively, and the recommended value for the gradient coefficient is 1 to 5. Documents general description has not been made, such as posts in the case of many processes, by increasing the value of y _0, it is preferable to reduce the value of the gradient coefficient.

[3-8] Final keyword importance in the document:
The operation of the important keyword output unit 6 will be described below.

  The final keyword importance FS (d, k) in the document d is obtained by the method described in the section [3-7] and is stored in the unique importance storage unit 105 and the position importance storage unit 106. It is calculated by multiplying each value. A higher FS (d, k) is considered as an important keyword.

FS (d, k) = BM25 _all (d, k), WebIDF (k), Keyword_score (k), Pos (k) (11)
FD (d, k) calculates the importance of keywords for all keyword candidates. If only a small number of keywords can be displayed by an application, etc., the top 3-5 keywords are important keywords in the document. Display as.

  The operation of each component of the above important keyword extracting device can be constructed as a program, installed in a computer used as the important keyword extracting device and executed, or distributed via a network.

  In addition, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and installed in a computer or distributed.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to a technique for extracting a keyword of a Web document, for example, pre-processing of information search, information recommendation, and the like.

DESCRIPTION OF SYMBOLS 1 Document input part 2 Important keyword candidate extraction means, Important keyword candidate extraction part 3 Appearance frequency calculation means, appearance frequency calculation part 4 Keyword importance degree calculation means, keyword importance degree calculation part 5 Position information calculation means, Position information calculation part 6 Keyword Output means, important keyword output section 10 main content extraction means, main content extraction section 11 Web document acquisition / input section 12 Web document division means, Web document division section 13 feature quantity extraction means, feature quantity extraction section 14 main content determination means, Main content determination unit 15 Main content output means, main content output unit 101 Keyword dictionary 102 WebIDF storage unit 103 Keyword importance level 1 storage unit 104 Keyword importance level 2 storage unit 105 First storage unit, unique importance level storage unit 106 Second Storage means, position importance storage unit 111 Data input unit 112 Web document file input unit 113 URL input unit 114 Web document acquisition unit 115 Character code conversion unit 121 Advertisement target region extraction unit 122 Tag and region removal unit 123 that cause noise Web document division unit 131 Anchor link information feature amount Extraction unit 132 Tag information feature amount extraction unit 133 Character string feature amount extraction unit displayed in Web document 134 Feature amount normalization unit 135 Feature amount ratio feature amount extraction unit 141 Feature amount input unit 142 Text determination unit 143 Main content Determination unit 151 Text output unit with tag 152 Text output unit without tag 153 Data output unit

Claims (16)

An important keyword extraction device that extracts important keywords from text included in a Web document,
Main content extracting means for acquiring the Web document, extracting a feature amount of the Web document, and extracting main content;
A morpheme is an entry word of a document set represented by an online encyclopedia that has a unique entry word in the document set and that has a citation relationship or a reference relationship in the document set (hereinafter referred to as a dictionary document set). An important keyword candidate extracting means for extracting an important keyword candidate from the main content with reference to a keyword dictionary registered as a dictionary for analysis;
Appearance frequency calculating means for weighting the important keyword candidates in the Web document;
The importance level of the important keyword candidate is calculated and stored in the first storage means so that the importance level of the keyword having a high degree of name recognition and high topicalness and various topics when talking about the keyword is high. Keyword importance calculation means,
Position information calculation means for obtaining an appearance position keyword importance based on the position in the Web document based on the importance of the important keyword candidate, and storing it in a second storage means;
Based on a value obtained by multiplying the importance of the important keyword candidate stored in the first storage means and the appearance position keyword importance stored in the second storage means, a keyword set with importance is obtained. Keyword output means for outputting,
The main content extracting means includes
Web document dividing means for dividing the Web document into segments based on a predetermined division rule;
Feature amount extraction means for extracting feature amounts for main content determination for each segment;
Main content determination means for determining whether the content is main content using a machine learning algorithm based on the feature amount for each segment;
Including main content output means for combining the parts determined to be main content and outputting as the main content;
The feature amount extraction means includes:
Means for extracting a feature amount of a character string displayed in the Web document, a feature amount of tag information, a feature amount of anchor link information, and using a ratio between the feature amounts as a final feature amount;
The appearance frequency calculating means includes
Means for weighting the important keyword candidates using WebIDF, which is a weight considering the appearance probability in a set of Web documents collected by a search engine;
Or
Means for weighting the keyword candidates using a BM 25 that obtains the appearance frequency of the important keyword candidates in the Web document, and weights each important keyword candidate based on the appearance frequency and the appearance distribution in the document set;
An important keyword extraction apparatus characterized by comprising any of the following. The appearance frequency calculating means includes
The important keyword extraction apparatus according to claim 1, further comprising means for obtaining an importance level of the important keyword candidate using an appearance frequency of the important keyword candidate including the morpheme proper noun obtained from the morpheme analysis result of the morpheme analysis unit. The keyword importance calculation means includes:
3. The important keyword extracting apparatus according to claim 1, further comprising means for obtaining an importance level of the important keyword candidate using a link structure in the dictionary document set. The keyword importance calculation means includes:
The important keyword according to any one of claims 1 to 3, further comprising means for determining the importance of the important keyword candidate by using a value such that the keyword that is input to the search engine more frequently becomes a more important keyword. Extraction device. The keyword importance calculation means includes:
The important keyword extraction apparatus according to claim 1, further comprising means for obtaining an importance level of the important keyword candidate based on an appearance position of the important keyword candidate in the Web document. The keyword importance calculation means includes:
Means for determining the importance of the important keyword candidate using the link structure in the dictionary document set;
The important keyword extraction according to claim 1, further comprising: means for obtaining an importance level of the important keyword candidate using the appearance frequency of the important keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result of the morpheme analysis unit. apparatus. The keyword importance calculation means includes:
Means for determining the importance of the important keyword candidate using the appearance frequency of the important keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result of the morpheme analysis means;
The important keyword extraction apparatus according to claim 1, further comprising: means for obtaining an importance level of the important keyword candidate using a value such that a keyword that is input to a search engine more frequently becomes a more important keyword. The keyword importance calculation means includes:
Means for determining the importance of the important keyword candidate using the appearance frequency of the important keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result of the morpheme analysis means;
The important keyword extraction apparatus according to claim 1, further comprising: means for obtaining an importance level of the important keyword candidate based on an appearance position of the important keyword candidate in the document. The keyword importance calculation means includes:
Means for determining the importance of the important keyword candidate using the appearance frequency of the important keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result of the morpheme analysis means;
Means for determining the importance of the important keyword candidate using the link structure in the dictionary document set;
The important keyword extraction apparatus according to claim 1, further comprising: means for obtaining an importance level of the important keyword candidate using a value such that a keyword that is input to a search engine more frequently becomes a more important keyword. The keyword importance calculation means includes:
Means for determining the importance of the important keyword candidate using the appearance frequency of the important keyword candidate including the proper noun of the morpheme obtained from the morpheme analysis result of the morpheme analysis means;
Means for determining the importance of the important keyword candidate using the link structure in the dictionary document set;
A keyword that uses a value that makes a keyword more frequently entered into a search engine and uses a value that makes the keyword more important.
The important keyword extraction apparatus according to claim 1, further comprising: means for obtaining an importance level of the important keyword candidate based on an appearance position of the important keyword candidate in the document. An important keyword extraction method for extracting an important keyword from text included in a Web document,
A main content extracting unit that acquires the Web document, extracts a feature amount of the Web document, and extracts the main content;
The important keyword candidate extraction means is a document set represented by an online encyclopedia that has a unique headword in the document set and has a citation relationship or a reference relationship in the document set (hereinafter referred to as a dictionary document set). An important keyword candidate extraction step for extracting an important keyword candidate from the main content with reference to a keyword dictionary registered as a dictionary for morphological analysis;
An appearance frequency calculating unit that weights the important keyword candidates in the Web document;
The keyword importance calculation means calculates the importance of the important keyword candidates so that the importance of keywords having a high level of name recognition and topicality, and abundant information that includes various topics when talking about keywords is increased. A keyword importance degree calculating step to be stored in one storage means;
A position information calculating means for obtaining an appearance position keyword importance based on the position of the important keyword candidate in the Web document, and storing it in a second storage means;
The keyword output means is based on a value obtained by multiplying the importance of the important keyword candidate stored in the first storage means and the appearance position keyword importance stored in the second storage means. A keyword output step for outputting a keyword set with
In the main content extraction step,
Dividing the Web document into segments based on a predetermined division rule;
Extract feature quantities for main content determination for each segment,
Determine whether or not the main content using a machine learning algorithm based on the feature amount of each segment,
Combining the portion determined to be the main content and outputting as the main content,
In the feature amount extraction step,
Extracting a feature amount of a character string displayed in the Web document, a feature amount of tag information, and a feature amount of anchor link information, and using the ratio between the feature amounts as a final feature amount,
In the appearance frequency calculating step,
Weighting the important keyword candidates using WebIDF, which is a weight considering the probability of appearance in a set of Web documents collected by a search engine;
Or
Obtaining an appearance frequency of the important keyword candidate in the Web document, and weighting the keyword candidate using BM25 for weighting each important keyword candidate based on the appearance frequency and an appearance distribution in the document set;
An important keyword extraction method characterized by performing any of the following. The appearance frequency calculating step includes:
The important keyword extraction method according to claim 11, wherein the importance of the important keyword candidate is obtained using the appearance frequency of the important keyword candidate including the morpheme proper noun obtained from the morpheme analysis result of the morpheme analysis unit. The keyword importance calculating step includes:
13. The important keyword extraction method according to claim 11 or 12, wherein the importance level of the important keyword candidate is obtained using a link structure in the dictionary document set. The keyword importance calculating step includes:
The important keyword extraction method according to claim 11, wherein the importance of the important keyword candidate is obtained by using a value such that the more frequently the keyword is input to the search engine, the more important the keyword is. The keyword importance calculating step includes:
13. The important keyword extraction method according to claim 11 or 12, wherein the importance level of the important keyword candidate is obtained based on the appearance position of the important keyword candidate in the Web document.   11. An important keyword extracting program for causing a computer to function as each means constituting the important keyword extracting apparatus according to claim 1.

Images