JP5718213B2 - Web page topic determination device, Web page topic determination method, and Web page topic determination program - Google Patents

Description

  The present invention relates to a technique for determining the topic of a Web page described in a hypertext description language such as HTML (Hyper Text Markup Language).

  As a technique for extracting a topic of an electronic document as well as a Web page, the topic extraction method of Patent Document 1 is known. Here, a noun is extracted as a feature word from text included in an electronic document, a web search is performed using the extracted feature word as a search word, and a phrase commonly appearing in the obtained search result is used as a topic.

  However, a search engine crawler program (spider, robot) that collects a set of Web pages including only a specific topic automatically collects Web pages by repeatedly following links based on information on hyperlinks in the Web pages. When trying to apply Patent Document 1 to topic determination, it is not possible to obtain text information of the page before accessing the linked Web page, and it is desired to collect only Web pages related to a specific topic. In some cases, there was a problem with collection efficiency.

  Therefore, Non-Patent Document 1 proposes a technique for determining a topic by using an available URL as an information source before accessing a linked Web page. Here, the URL is divided into character strings in units of processing (hereinafter referred to as tokens “token”) with symbols or the like, and further, partial character strings of tokens are extracted as feature amounts. With respect to this feature quantity, it is determined by a determiner whether or not it is a corresponding topic for which mechanical learning has been completed by a training example.

JP2009-15796A

Eda Baykan, Monika Henzinger, Ludmila Marian, Ingmar Weber `` Purely URL-based Topic Classification '' Proceedings of the 18th international conference on World wide web (WWW'09) .pp1109-1110 "Web convenient tool / URL encoding / decoding form-TAG index website", "online", "November 10, 2011 search", Internet <URL: http://www.tagindex.com/tool/url. html

  When using machine learning that uses a collection of correct answers collected in advance as learning data when determining the topic of a Web page, topic determination accuracy results in what is used for the quality of the correct set and the characteristics of the Web page. . In particular, in a situation where only the URL of a Web page can be used as an information source for determination, what kind of features are created from the URL is important.

  In Non-Patent Document 1, a token partial character string obtained from a URL is used as a feature in order to maximize the Web page that can be determined as described above. However, if the token is a character string that represents a word in the language used by the main viewer of the Web page according to the URL convention, it is characterized by a partial character string that is not desirable for the language. There was a risk of adversely affecting accuracy.

  For example, in the case of a Web page with a URL “http://example.co.jp/suitouchou/”, one of the tokens “suitouchou” of the URL corresponds to a “book” in Japanese. However, in Non-Patent Document 1, since a partial character string of a token is acquired as an alphabet without considering the language used, a partial character string such as “suit” may be used as a feature. This is the same notation as a word representing a suit for men's clothing in English, so it is determined that it is a fashion-related topic different from the original topic of the Web page to be processed.

  The present invention has been made in order to solve the above-described problems of the prior art, and has constructed a feature that takes into account the language used by the main viewer from the URL of the Web page, and has specialized in that language. The problem is to perform appropriate topic determination.

  Therefore, according to the present invention, in order to specify the language used by the main viewer of the Web page from the character configuration of the URL, the host use country is specified from the host name in the URL, and the main language in the use country is determined. For example, the main language can be specified using an official language dictionary created in advance.

  Also, feature amounts corresponding to the main language are extracted from each character string obtained by decomposing URL into arbitrary units. For example, if each character string can be converted into a character string corresponding to the language feature of the main language, the converted character string is extracted as a feature candidate. A partial character string can be acquired from each feature candidate, and the appearance frequency of the acquired partial character string can be extracted as a feature amount.

  By using the feature amount extracted here for topic determination, it is possible to determine the topic in consideration of the language used by the main viewer of the Web page. That is, it is possible to suppress extraction of feature quantities inappropriate as words in a language used on a Web page, and to prevent erroneous topic determination. As a topic determination method, a topic of a Web page may be determined from the feature amount using a determiner that has learned whether or not it belongs to a specific topic.

  According to the present invention, it is possible to construct a feature that takes into account the language used by the main viewer from the URL of the Web page, and it is possible to perform appropriate topic determination specialized for the language.

The block diagram of the topic determination apparatus of the web page which concerns on embodiment of this invention. The processing flow figure of the same language determination part. The processing flow figure of the same feature quantity extraction part. 4 is a processing example of the processing flow of FIG.

  A web page topic determination apparatus according to an embodiment of the present invention will be described below. This topic determination device specifies the language used by the main viewer from the character configuration of the URL, and extracts a feature amount corresponding to the determined usage language. The topic of the Web page is determined using the feature amount extracted here.

≪Configuration example≫
A configuration example of the topic determination device will be described with reference to FIG. Here, the topic determination device 1 is used in a crawler program (spider, robot, etc.) of a search engine that collects a set of Web pages including a specific topic.

  Specifically, the topic determination device 1 is configured as a server group of search engines, and is a normal computer hardware resource such as a CPU. A storage device such as a memory (RAM) or a hard disk drive device is provided. As a result of the cooperation between the hardware resource and the software resource (OS. Application, etc.), the topic determination device 1 has the input unit 10. Language determination unit 11. Feature amount extraction unit 12. Topic determination unit 13. The output unit 14 is mounted.

  The URL of each Web page collected by the crawler program is input to the input unit 10 as a topic determination target Web page. The URL input here is output to the language determination unit 11, and the topic determination of the Web page using only the URL as a material is started.

  That is, the language determination unit 11 receives the output information from the input unit 10, specifies the country of use of the host name from the host name in the URL, and further determines the main language in the country of use of the host name. This main language is estimated as the language used in the Web page, that is, the language used by the main viewer of the Web page. The main language and URL are output to the feature amount extraction unit 12.

  The feature quantity extraction unit 12 receives the output information from the language determination unit 11 and extracts the feature quantity from the URL in consideration of the language characteristics of the main language. Here, the URL is broken down into character strings in units of processing, and feature quantities corresponding to the main language are extracted from the character strings. At this time, if it can be converted into a character string corresponding to the language feature of the main language, the appearance frequency of each converted character string is extracted as a feature amount. For example, if Japanese is specified as the main language, it is possible to extract features by performing Romaji-kana conversion or Kanji conversion. The extracted feature amount is output to the topic determination unit 13.

  The topic determination unit 13 receives the output information from the feature amount extraction unit 12 and determines the topic of the Web page based on the feature amount. Here, a determiner that has learned in advance whether or not it belongs to a specific topic is used. The topic of the Web page is determined based on whether or not the feature value input to the determiner has a previously learned topic. The determination result is output to a search engine or the like through the output unit 14. Hereinafter, the processing content of each part 11-13 is explained in full detail.

<< Processing content of language determination unit 11 >>
The processing content of the language determination unit 11 will be described in detail based on FIG. Here, the language determination unit 11 acquires the host name (site name) in the URL of the input Web page. Assume that the processing of FIG. 2 is started after this acquisition. This process is performed for each URL.

  S01: It is determined whether or not a country code top level domain is included in the host name. As a result of the determination, if the domain is not included, the process proceeds to S02. If the domain is included, the country of use of the host name is specified based on the country code. The country of use of the host name specified here is determined as the target country of the Web page, and the process proceeds to S03. For example, if “.jp” is included in the host name, Japan is determined as the target country of the Web page.

  S02: Use of the host name by using a whois system for the host name (more precisely, the domain name in the host name), that is, a protocol for searching for the owner of the domain name on the Internet. Check if the country can be identified. As a result of the confirmation, if the country of use of the host name can be identified, the country of use is determined as the target country of the Web page, and the process proceeds to S03.

  S03: S01. The main language in the target country of the Web page determined in S02 is determined using an official language dictionary created in advance, and the process ends. This official language dictionary only needs to contain major languages by country. If a plurality of languages are listed in the main language, each language can be determined as the main language of the target country.

≪Processing content of feature quantity extraction unit 12≫
The processing content of the feature amount extraction unit 12 will be described in detail based on FIG. Here, the processing contents when Japanese is specified as the main language in the language determination unit 11 will be described. It is assumed that the process of the feature amount extraction unit 12 is also performed for each URL.

  S11: The inputted URL is decomposed into a plurality of tokens, that is, respective character strings to be processed, by delimiters such as symbols “.”, “-”, “/”. The processing after S12 is executed for each token.

  S12: Is each token decomposed in S11 subjected to percent encoding (Percent-Encode), that is, an encoding method in which the character code is expressed in hexadecimal and converted into the form of “% xx” (xx is hexadecimal)? Determine whether or not.

  Here, according to RFC3986 that defines the URL rule, characters other than ASCII and ASCII reserved characters in the URL are converted to “% xx”. For example, the character “A” written in “SHIFT_JIS” is converted into the form “% 82% a0”, and the character “I” is converted into the form “% 82% a2”. Based on such conversion notation, it is determined whether or not the token is percent-encoded.

  S13. S14: As a result of the determination in S11, decoding is executed for the token that has been percent-encoded. If the decoded character string contains katakana or kanji, hiragana conversion is performed and the converted character string is acquired (S13). The character string acquired here is registered as a feature candidate in a list not shown (S14).

  For the decoding, a general-purpose tool such as Non-Patent Document 2, for example, can be used, and the Kanji-Hiragana conversion may be performed using a previously prepared Han-Japanese dictionary.

  S15. S16: As a result of the determination in S11, for tokens that have not been percent-encoded (non-percent-encoded tokens), Roman character kana conversion is performed on the token character string. Here, it is confirmed whether or not the character string of the token can be converted into a hiragana character string, that is, whether or not it can be completely expressed in hiragana (S15).

  As a result of the confirmation, if the token character string can be expressed completely in hiragana, the converted hiragana character string is registered as a feature candidate in the list (S16). For example, if the character string of the token is “suitouchou”, the hiragana character string “Sato Ito” is registered as a feature candidate. On the other hand, since a character string that cannot be converted into hiragana cannot be completely expressed in hiragana by romaji-kana conversion, it is registered as a feature candidate in the list as an alphabetic character string (S16).

  S17: A partial character string set obtained by extracting partial character strings from all feature candidates registered in the list in S14 or S16 is acquired. Here, the appearance frequency (number of appearances) of each partial character string in the set is counted, and the quantity of the count result is extracted as a feature amount. After the extraction, the feature amount is output to the topic determination unit 13 and the process is terminated. After this output, the list is initialized and the processing of the next URL is started.

  S15. In S16, it is determined whether or not a character string of a non-percent-encoded token can be converted into a hiragana character string by romaji kana conversion, and a method of using the hiragana character string and the alphabet character string exclusively is shown. However, when it can be converted into a hiragana character string, both the original alphabetic character and the hiragana character string may be used as feature candidates.

  Further, S13 shows a method of converting hiragana to a token subjected to percent encoding when the decoded character string includes kanji or katakana. However, the decoded character string is used as it is. A method of registering as feature candidates may be used.

  Furthermore, character string length restrictions may be set for feature candidates and partial character strings of feature candidates, and a method of excluding a character string with an appearance frequency that is too large in advance as a stop character string should be adopted. You can also.

<< Processing Example of Feature Quantity Extraction Unit 12 >>
Hereinafter, a processing example of the feature amount extraction unit 12 will be described with reference to FIG. Here, the URL “http://www.example.co.jp/ichirei.html?category=%e3%82%b5%e3%83%b3%e3%83%97%e3%83%ab” is the input section 10 and the language determination unit 11 determines that Japanese is the main language. In addition, the restriction of the character string length “3 to 8” is set for the feature candidate and the partial character string of the feature candidate, and “www”. It is assumed that “html” is set in advance.

  First, when the URL and the main language are input to the feature amount extraction unit 12, tokenization of the URL is executed in S11. As a result of the tokenization, the URL is “www”. “Example”. “Co”. “Jp”. “Ichirei”. “Html”. “Category”. It is broken down into tokens of “% e3% 82% b5% e3% 83% b3% e3% 83% 97% e3% 83% ab”.

  Next, the token “example”. “Category”. “Ichirei” is determined to be non-percent encoding in S12, and the process proceeds to S15. Here token "example". Since “category” cannot be converted to hiragana by romaji kana conversion, it is registered as a feature candidate as an alphabetic character in S16. On the other hand, since the token “ichirei” can be expressed as “Ichirei” by converting the Roman character to Kana, the hiragana character string “Ichirei” is registered as a feature candidate in S16.

  The token “% e3% 82% b5% e3% 83% b3% e3% 83% 97% e3% 83% ab” is determined to have been percent-encoded in S12. After this determination, it is decoded into “sample” in S13, and further hiragana converted to “sample”. After this conversion, the hiragana character string “Sample” is registered as a feature candidate in S14. The token “www”. Since “html” corresponds to a stop character string, it is excluded from the processes of S12 to S16. Furthermore, the token “co”. “Jp” is also excluded from the restriction on the character string length.

  As a result of the processing of S11 to S16, “example”. “Ichirei”. “Category”. The character string “Sampuru” is registered as a feature candidate. Then, in S17, partial character strings are acquired from the character strings of the respective feature candidates within the range of length “3 to 8”, and the frequency of appearance of each partial character string is counted, for example, “exa: 1 ". “Xam: 1”. Extract “amp: 1” and so on. The feature quantity “1” indicates the number of appearances in the partial character string set.

≪Processing content of topic determination unit 13≫
Hereinafter, the processing content of the topic determination part 13 is explained in full detail. Specifically, the topic determination unit 13 receives the feature amount output from the feature amount extraction unit 12 and outputs the result of determining the topic of the Web page. The topic determination unit 13 needs to learn a determiner using machine learning on a topic to be determined in advance. Here, as an example, a case where “politics” is the subject of determination will be described.

  In this learning, it is necessary to prepare a Web page set related to “politics” and a Web page set not related to “politics” in advance. That is, the feature amount obtained from the URL group of the Web page set related to “politics” is used as a learning example of the determiner as a positive example in the binary determination of the feature amount obtained by the feature amount extraction unit 12. Similarly, the feature quantity obtained from the URL group of the web page set not related to “politics” is used as a negative example in the binary judgment of the feature quantity obtained by the feature quantity extraction unit 12.

  Whether or not the Web page to be processed has a topic related to “politics” by giving the feature quantity obtained by the feature quantity extraction unit 12 as an input to the discriminator that has already learned positive examples and negative examples. judge.

  This determination result is output to the search engine through the output unit 14 and used for a classification algorithm for constructing a full-text index. At this time, according to the topic determination device 1, the feature amount considering the feature of the main language is extracted through the processing of S 11 to S 17, so the feature in the partial character string that is not desirable as the language used by the main viewer of the Web page Extraction of the amount is prevented, and appropriate topic determination specialized in the language used can be performed.

  For example, regarding the URL “http://example.co.jp/suitouchou/”, the country code top-level domain “.jp” is included in the URL, so that Japanese is determined as the main language. Since the token “suitouchou” obtained by decomposing the URL can be expressed as “Suichou” by the Roman character Kana conversion in S15, the character string “Suichou” is registered as a feature candidate in S16.

  Therefore, unlike the non-patent document 1, the partial character string “suit” is not extracted as a feature amount, and the extraction of a feature amount inappropriate as a language of a Web page language (in this case, Japanese) is suppressed. This makes it possible to prevent erroneous topic determination.

  In addition, this invention is not limited to the said embodiment, It can deform | transform suitably and implement within the range described in each claim. For example, the present invention can be applied not only when the language determination unit 11 specifies Japanese but also when other foreign languages are specified. In this case, S13. S15 may be converted according to the specified foreign language.

  In addition, the processing content of the topic determination unit 13 indicates a binary determination as to whether or not it is related to “politics”, but prepares a set of Web pages related to a plurality of determination target topics (for example, sports and fashion) in advance. By preparing a binary determiner corresponding to each topic, it is possible to adopt a method for determining the topic having the highest classification certainty obtained from the determiner as the topic of the topic determination target Web page. Further, a method may be adopted in which a topic having a certainty level or more is given as metadata for a Web page to be processed without uniquely defining the topic.

≪Programs≫
The present invention may be configured as a topic determination program for a Web page that causes a computer to function as part or all of the units 10 to 14 of the topic determination device 1. According to this program, S01 to S03. It becomes possible to cause the computer to execute part or all of S11 to S17.

  The program can be provided through a network such as a website or e-mail. The program is stored in a recording medium such as a CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, MO, HDD, BD-ROM, BD-R, or BD-RE. It is also possible to record, save and distribute. This recording medium is read using a recording medium driving device, and the program code itself realizes the processing of the above embodiment, so that the recording medium also constitutes the present invention.

DESCRIPTION OF SYMBOLS 1 ... Web page topic determination apparatus 10 ... Input unit 11 ... Language determination unit (language determination unit)
12 ... feature quantity extraction unit (feature quantity extraction means)
13 ... Topic determination unit (topic determination means)
14 ... Output section

Claims (7)

A web page topic determination apparatus for determining a topic referred to by a web page based on a URL,
Language determination means for identifying the host user country from the host name in the URL and determining the main language in the user country;
Feature quantity extraction means for extracting feature quantities according to the main language specified by the language determination means from each character string obtained by decomposing URL into arbitrary units;
Topic determination means for determining the topic of a web page from the feature quantity extracted by the feature quantity extraction means using a determiner that has learned whether or not it belongs to a specific topic ,
The language determining means determines the main language of the user country based on an official language dictionary created in advance,
The feature amount extraction unit extracts each converted character string as a feature candidate if each of the character strings can be converted into a character string corresponding to a language feature of a main language,
A Web page topic determination device characterized in that a partial character string is obtained from each feature candidate, and an appearance frequency of each partial character string is extracted as a feature amount. If the determination of the language determination means is Japanese, the feature amount extraction means determines whether or not each character string is subjected to percent encoding,
Decode the character string that has been percent-encoded, and if the decoded character string contains katakana or kanji, the character string after hiragana conversion is used as a feature candidate.
2. The Web page according to claim 1, wherein a non-percent-encoded character string is subjected to Roman-kana conversion, and if the character string can be expressed completely in hiragana, the converted character string is used as a feature candidate . Topic determination device. The topic determination device for a Web page according to claim 1, wherein a feature candidate or a character string to be excluded from the partial character string can be set. A method for determining a topic of a Web page executed by an apparatus for determining a topic referred to by a Web page based on a URL,
A language determination step of identifying a host user country from a host name in a URL and determining a main language in the user country;
A feature amount extraction step of extracting a feature amount according to the main language specified by the language determination means from each character string obtained by decomposing the URL into arbitrary units;
A topic determination step of determining the topic of the web page from the feature amount extracted by the feature amount extraction means using a determiner that has learned whether or not it belongs to a specific topic ,
The language determination step includes a step of determining a main language of the user country based on an official language dictionary created in advance,
The feature amount extraction step includes extracting each converted character string as a feature candidate if each character string can be converted into a character string corresponding to a language feature of a main language;
Obtaining a partial character string from each feature candidate and extracting the appearance frequency of each partial character string as a feature amount; and
A method for determining the topic of a Web page, comprising: If the determination of the language determination means is Japanese, the feature amount extraction step includes a step of determining whether or not percent encoding is applied to each character string;
Perform decoding on the character string that has been subjected to percent encoding, and if the decoded character string contains katakana or kanji, the character string after hiragana conversion is used as a feature candidate,
Performing a romaji kana conversion on a non-percent encoded character string, and if the character string can be expressed completely in hiragana, the converted character string as a feature candidate;
The method for determining the topic of a Web page according to claim 4, comprising: 6. The topic of a Web page according to claim 4, wherein the feature quantity extraction step excludes a character string that matches a preset condition from the feature candidates or the partial character string. Judgment method. A Web page topic determination program for causing a computer to function as the Web page topic determination device according to any one of claims 1 to 3 .

Images