JP2005135139A - Structural analysis system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To check the appropriateness of web pages and a web site that is an aggregate of the web pages, ie., to see if items desired to be shown on the web pages or the web site are shown appropriately. <P>SOLUTION: This structural analysis system 10 for analyzing structures of web pages or the structure of a web site that is an aggregate of the web pages includes a page acquisition part 14 for acquiring the web pages from a web page DB 12 that stores the web pages; a page analysis part 16 for analyzing the structures of the web pages acquired to specify at least positions where the web pages are processed or character strings with predetermined roles and words constituting the character strings; and a word counting/comparing part 18 for counting the words that appear to constitute each character string and for storing and presenting the counting result. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a system for analyzing the structure of a web page or a website.

Since there are a remarkably large number of sites due to the spread of the Internet, a user often uses a site (search site) equipped with a search engine in order to access a desired site. Basically, when the user inputs a keyword related to desired information, the search engine of the search site can present the URL of the page of the site highly relevant to the keyword to the user.
In this search engine, it is said that the order of pages to be presented to the user is determined by determining the importance of the pages constituting the site (Non-Patent Document 1).
Satoshi Baba, “Google's Secret-PageRank Thorough Explanation”, [online], February 1, 2001, July 9, 2003 search, <URL: http://www.kusastro.kyoto-u.as .jp / ~ baba / wais / pagerank.html>

  For the site operator, a situation where more users browse the pages in the site is desirable. For this reason, in the search result of the search site, it is important that the page of the own site is positioned higher. For this reason, the operator needs to select a page configuration and a term so that the page of his / her site is positioned higher in the search result.

  Furthermore, it is still desirable for the site operator to know whether the usage of words in the web pages constituting the site appropriately represents the site. For example, if the page title is in line with the actual content of the page, it can be made clear that the page wants to convey.

However, conventionally, the importance of a web page could be calculated, but the appropriateness of the web page could not be evaluated.
The present invention relates to a structural analysis system capable of knowing the appropriateness of a web page and a website as a collection of web pages, that is, whether a web page or a matter to be represented on the website is appropriately represented on the web page. The purpose is to provide.

  An object of the present invention is a structural analysis system for analyzing the structure of a web page or a website as a collection of the web pages, and a page acquisition means for acquiring a web page from a web page database storing the web page; Analyzing the structure of the web page acquired by the page acquisition means, and specifying at least the position of the processing of the web page or the character string having a predetermined role and the words constituting the character string; and the character This is achieved by a structural analysis system characterized in that for each of the columns, the number of occurrences of words constituting the character string is counted, the counting result is stored, and counting means for presenting the counting result is provided.

  In addition, by integrating the counting results of the counting means of each web page, it is possible to know the structure of the website that is a collection of web pages.

In a preferred embodiment, the page structure analyzing means is configured to refer to a synonym list for each attribute of the web page and unify synonyms into a specific single word.
In a preferred embodiment, the page structure analyzing means specifies a page title, a link anchor text, and / or a character string in an emphasis tag and words constituting the character string. The counting means calculates, for example, the number of appearances and the appearance ratio of each word.

  In another preferred embodiment, there is further provided rank correlation calculating means for specifying a predetermined word in the character string and calculating a value representing rank correlation related to the word. The rank correlation calculation system may further be configured to calculate another index obtained by integrating the count result of the word by the counting means and a value representing the rank correlation.

  ADVANTAGE OF THE INVENTION According to this invention, the structure which can know the appropriateness of the website which is a web page and the aggregate of a web page, ie, the web page and the matter to express on a website are appropriately represented on the web page An analysis system can be provided.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. FIG. 1 is a block diagram showing an outline of a structural analysis system according to an embodiment of the present invention. As shown in FIG. 1, the structural analysis system 10 includes a web page DB 12 that stores web pages, a page acquisition unit 14 that acquires web pages in the web page DB 12, and a page analysis unit that analyzes the acquired web pages. 16 and a result storage for storing a word count / comparison unit 18 that performs processing necessary for structure evaluation, such as counting a predetermined word based on the analysis result, and a comparison result of the word count / comparison unit 18. Part 20.

  In this embodiment, for each web page, the page is analyzed, and the structure is evaluated based on the analysis result. However, the present invention is not limited to this, and a website composed of a plurality of web pages It is also possible to calculate the evaluation of each web page and obtain the evaluation of the site from the evaluation result.

  The structural analysis system according to the present embodiment can be realized by installing a necessary program in a personal computer, starting the program, and operating the personal computer. This program may be stored in a CD-ROM or DVD-ROM, or may be downloaded to a personal computer via a network.

  FIG. 2 is a flowchart showing an outline of processing executed in the structure evaluation system according to the present embodiment. As shown in FIG. 2, in the structure evaluation system 10, first, the page acquisition unit 14 acquires a predetermined web page from the web page DB 12 (step 201). Next, the page analysis unit 16 removes unnecessary tags (for example, ALT, <.A>, etc.) from the acquired web page, and then analyzes the page structure (step 202). In the analysis of the page structure, a character string corresponding to a page title, a link anchor text, an emphasis tag, and the like is specified.

  Next, morphological analysis is performed on each character string (step 203). In morphological analysis, a character string is decomposed for each part of speech. Thereafter, the page analysis unit 16 performs a stemming process to remove unnecessary parts of speech and to unify the search words that are the stems (step 204). For example, a synonym list may be provided for each theme (theme according to the attributes of the web page), and the process of unifying synonyms into a single representative word may be executed. After such processing, counting / comparison processing by the word count / comparison unit 18 is executed (step 205).

In the present embodiment, two processes are mainly executed in the counting / comparison process.
In the first process, words in a character string having a predetermined position or a predetermined role in the page structure are counted. This count value is used as the first index value. Further, in the second process, each word around the word having the predetermined position and role is examined, and the rank correlation with the words in other character strings including the word is taken, and based on the rank correlation. A second index value is calculated.

  In the present embodiment, in the web page, the page title, the link anchor text, and the words in the emphasis tag are counted in the first process. For example, the sum of the number of occurrences of each word in the page title, the link anchor text, and the emphasis tag, the appearance ratio of the word in all the words, and the like are calculated and stored in the result storage unit 20.

  Further, the second processing is realized by taking a rank correlation with respect to words around a predetermined word in the page title, the link anchor text, and the emphasis tag. Hereinafter, the second process will be described in more detail.

  FIG. 3 is a diagram illustrating an example of a web page. The title of this page 300 is “Mail order beauty-About dullness / Isize clean” (see reference numeral 301). Among them, the word count / comparison unit 18 focuses on the words “mail order” and “beauty”. This may be selected from the top of the word count results of the portion having a predetermined position or role in the first process.

  Next, another character string having the word “mail order” or the word “beauty” is selected. For example, a character string including the word “beauty” is indicated by reference numerals 302 to 305. For these character strings, the word count / comparison unit 18 acquires four words before and after the focused word. Of course, when the character string is short, four or less words (or zero words) may be specified. In the present embodiment, four words are specified before and after, but the number is not limited.

  FIG. 4 shows an example in which four words before and after the word “beauty” are specified for each character string. For example, as shown in FIG. 4B, for the character string 302, two words “beauty” appear, and therefore the preceding and following words are specified for each. In FIG. 4, a numerical value arranged above (or below) a word (morpheme) represents a distance from a specific word “beauty”, that is, a rank. 5A to 5D also show an example in which four words before and after the word “mail order” are specified for each character string.

  The word count / comparison unit 18 ranks each word from “beauty” in the page title shown in FIG. 4A and “beauty” in other character strings (see FIGS. 4B to 4E). A value representing the rank correlation is calculated with reference to the rank of each word from. Similarly, a value indicating the rank correlation is calculated for “mail order” shown in FIG. Rank correlation is a method for investigating the relationship between two sets of ranks. Spearman's rank correlation coefficient, Kendall's rank correlation coefficient, Goodman Kruskal's gamma (γ), etc. are known. It has been.

  FIG. 6 is a diagram showing the results of examining the ranks of “beauty” in the page title and “beauty” in other character strings. For example, in the character string indicated by reference numeral 302 in FIG. 4B, the character strings including “beauty” indicated by “A” are respectively “3”, “ 2 ”and“ 1 ”, and“ -1 ”,“ −2 ”,“ −3 ”and“ −4 ”are assigned to“ if ”,“ mail order ”,“ beauty ”and“ corner ”, respectively. Is ranked. The same applies to other character strings. In FIG. 5, the column indicated by “B” relates to a character string including “beauty” indicated by “B” in the character string denoted by reference numeral 302. The columns indicated by “C” to “D” relate to character strings including “beauty” indicated by “C” to “E” of reference numerals 303 to 305, respectively.

  The word count / comparison unit 18 ranks each word in the page title and each character string (referred to here as character strings “A” to “E”) based on the rank as described above. Find the correlation.

In the present embodiment, Goodman Kruskal gamma (γ) is used as a value representing the rank correlation. This corresponds to the two corresponding variables x: x1, x2, ..., xN
y: y1, y2, ..., yN
Between
P = number of data with the same order of x and y between (xi, yi) and (xj, yj) Q = x and y between (xi, yi) and (xj, yj) Given the number of data in the reverse order of
γ = (P−Q) / (P + Q)
Defined in This γ moves between −1 ≦ γ ≦ 1, and the correlation becomes strongest when γ = 1.

  In the example of FIG. 5, considering the word “mail order” in the page title, in the page title and the other character strings “A” to “E”, a total of six words including the word in the page title in the same direction. The word “mail order” has appeared (see character strings “A” to “C” and “E” in FIG. 5). When the word “dull” is considered, the word “dull” does not appear in the same direction in other character strings. Similarly, when the words “Isize” and “Beautiful” are considered, two words “Isize” and “Beautiful” appear in the page title and other character strings, respectively. Therefore, P is “6 + 2 + 2 = 10”.

On the other hand, for the word “dull”, two words “dull” appear in the opposite direction in the other character strings (see character strings “A” and “C”). Therefore, Q is “2”.
Therefore,

In the example of FIG. 6, the value γ representing the rank correlation is
γ = (10−2) / (10 + 2) = 8/12 = 2/3
It becomes.

FIG. 7 is a diagram showing the results of examining the rankings of “mail order” in the page title and “mail order” in other character strings. Here, as in the example of FIG. 6, the value γ representing the rank correlation can be calculated. In the example of FIG. 6, when the word “beauty” in the page title is considered, in the page title and the other character strings “A” to “D”, a total of six words “
“Beauty” has appeared (see character strings “A” to “D”). Similarly, two words “Isaizu” appear (see character string “B”). Therefore, P is “6 + 2 = 8”.

On the other hand, the word “beauty” appears in the opposite direction (see character string “B”), and the word “dull” appears in the opposite direction (character strings “A” and “C”). "reference). Therefore, Q is “1 + 2 = 3”.
Therefore, in the example of FIG. 7, the value γ representing the rank correlation is
γ = (8-3) / (8 + 3) = 5/11.

In the above example, for the words “mail order” and “beauty”, it is understood that “beauty” is more important as a word for the value γ representing the rank correlation, so that “beauty” is more important.
In addition to the page title, the word count / comparison unit 18 similarly calculates a value γ representing the rank correlation for the anchor text of the link and the word in the emphasis tag. This makes it possible to obtain an index (γ) indicating the importance of the word for each of the page title, the link anchor text, and the emphasis tag. The results of the first process and the second process are stored in the result storage unit 20.

  When processing the entire website or a part thereof, the above-described processing may be executed for each page and repeated.

  FIG. 8 is a diagram showing the result of the counting / comparison processing by the word count / comparison unit 18. This is a graph obtained by reading the processing result stored in the result storage unit 20 by the word count / comparison unit 18. Such a charted result may be displayed on the screen of the display device or may be output as a printed matter.

  In FIG. 8A, the number of constituent words, the total number of words, extracted keywords and the like are shown in addition to the values of items representing page attributes such as the URL of the page, response code, and server name. Further, in FIG. 8B, the words having the highest number of appearances as a result of the first processing are shown in the left and center columns. In this example, it can be seen that the number of appearances of the word “mail order” is the largest (number of appearances = 12), and the word “beauty” is next (number of appearances = 7). That is, from the viewpoint of the number of words, this web page can be considered to represent mainly “mail order”. On the other hand, the value γ representing the rank correlation is shown in the right column of FIG. Referring to γ, γ = 5/11 of the word “mail order”, while γ = 2/3 of the word “beauty”. Since “5/11 <2/3”, it is indicated that the method of the word “beauty” is more important than the word “mail order” from the viewpoint of rank correlation.

  In the example of FIG. 8, the result for a single web page is shown. However, for a website, it is possible to display the result of each web page and the result of integrating the web page results. is there. As a result of the integration, the total number of words of the entire web page, the total number of appearances, the appearance ratio, the sum of the rank correlation, the average, and the like can be displayed.

  Thus, according to the present embodiment, from both the viewpoint of the number of occurrences of words and the viewpoint of rank correlation, what the web page mainly represents or what is an important word in the web page. I can know.

Further, in the present embodiment, an index integrating the result of the first process and the result of the second process may be output. For example, as an integrated indicator for a word,
(Appearance ratio of the word) × f ₁ + (value indicating the rank correlation of the word) × f ₂
For f ₁ and f _2, an appropriate function or constant may be used.

  In this way, by referring to the index based on the first processing result (the number of words appearing or the ratio of appearance) and the index based on the second processing result (the value representing the word rank correlation), The creator of the website can know what the page is about. That is, it is possible to know whether the matter that the creator wants to express on the web page or the website appears appropriately on the actual page. For example, if the creator thinks that a web page of “mail order” has been created, and the index based on the first processing result and the index based on the second processing result are small, the created web page Can be considered lacking in appropriateness. In particular, since the index based on the second processing result is not materialized on the web page, the creator can know important words of the hidden web page by the processing in the present embodiment. .

The present invention is not limited to the above embodiments, and various modifications can be made within the scope of the invention described in the claims, and these are also included in the scope of the present invention. Needless to say.
For example, although the Goodman Kruskal gamma (γ) is used in the above-described embodiment, it goes without saying that other values indicating rank correlation may be used.

FIG. 1 is a block diagram showing an outline of a structural analysis system according to an embodiment of the present invention. FIG. 2 is a flowchart showing an outline of processing executed in the structure evaluation system according to the present embodiment. FIG. 3 is a diagram illustrating an example of a web page. FIG. 4 shows an example in which four words are specified before and after a character string in a web page. FIG. 5 is a diagram showing the results of examining the ranks of “beauty” in the page title and “beauty” in other character strings in the example of the web page in FIG. 3. FIG. 6 is a diagram showing the results of examining the ranks of “mail order” in the page title and “mail order” in other character strings in the example of the web page in FIG. FIG. 7 is a diagram showing the results of examining the ranking of “mail order” in the page title and “mail order” in other character strings in the web page examples of FIGS. 7 and 3. FIG. 8 is a diagram showing a presentation example of the processing result in the present embodiment.

Explanation of symbols

10 Structural analysis system 12 Web page DB
14 page acquisition unit 16 page analysis unit 18 word count / comparison unit 20 result storage unit

Claims (6)

A structural analysis system for analyzing the structure of a web page or a website that is a collection of web pages,
Page acquisition means for acquiring a web page from a web page database storing the web page;
Analyzing the structure of the web page acquired by the page acquisition means, and specifying at least the position of the processing of the web page or a character string having a predetermined role and the words constituting the character string;
A structure analysis system comprising: a counting unit that counts the number of appearances of words constituting the character string for each of the character strings, stores the counting result, and presents the counting result. 2. The structural analysis according to claim 1, wherein the page structure analysis means is configured to unify synonyms into a specific single word with reference to a synonym list for each attribute of the web page. system. 3. The structure according to claim 1, wherein the page structure analysis unit specifies a page title, a link anchor text, and / or a character string in an emphasis tag and a word constituting the character string. Analysis system. The structural analysis system according to any one of claims 1 to 3, wherein the counting means calculates an appearance number and an appearance ratio of each word. 5. The rank correlation calculating means for specifying a predetermined word in the character string and calculating a value representing rank correlation related to the word. Structural analysis system. 6. The structural analysis system according to claim 5, wherein the rank correlation calculation system further calculates another index obtained by integrating the count result of the word by the counting means and a value representing the rank correlation.

Images