JP2015053087A - Grouping device and element extraction device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract an element included in a page by specifying a personal area page of a specific user on a network.SOLUTION: In the case that a similarity between a first address key and a second address key is determined to be equal to or more than a threshold, a URL is associated as a specific address group (personal area specification). Further, dots are arranged on an imaged Web page, and hierarchical structures of elements including the arranged dots are integrated, or the like, to extract corresponding contents data (element extraction).

Description

  The present invention relates to a technique for analyzing a Web page on the Internet, and particularly to grouping of Web pages and division processing of each Web page.

  Conventionally, various methods have been considered for analyzing Web pages. For example, Patent Document 1 discloses a method of analyzing a URI included in an access log and grouping Web pages. Patent Document 2 discloses a technique for automatically extracting main contents.

JP 2010-123000 A JP 2010-117941 A

  However, the technique of Patent Document 1 groups Web pages for an access log, and the group to be grouped is a group of Web pages having a similar domain and a similar URI structure. Therefore, it does not specify a personal area. Further, the technique of Patent Document 1 proposes a method of dividing a domain part, a directory part, and a query part and calculating the similarity between feature vectors, and the similarity calculation algorithm is complicated.

  The technique of Patent Document 2 is to divide HTML data into segments based on a predetermined division rule, and it is necessary to determine a rule in advance for each target HTML data. Therefore, it is difficult to extract articles with high versatility, and the amount of processing for extracting articles is enormous.

  It is an object of the present invention to accurately analyze a subject person with respect to the specified personal area by (i) specifying the personal area. Another object of the present invention is to easily extract only important article parts by focusing only on the structure of the website by (ii) division processing.

(1) The grouping program of this invention is
A grouping program for grouping web page addresses,
Computer
Web page acquisition means for acquiring a Web page from a specific address;
First address key generating means for extracting a link from the web page and dividing the address of each extracted link with a delimiter to generate a first address key;
Second address key generation means for extracting a link from a Web page acquired from the link that generated the first address key, and generating a second address key by dividing the address of each extracted link by a delimiter ,
Matching the matching keys between the first address key and the second address key, counting the number of combinations in which the matching keys appear in the same order, and calculating the similarity based on the result Similarity calculation means,
Grouping means for associating the addresses of the links determined to have a similarity equal to or higher than a threshold as a specific address group;
It is made to function as.

  Thereby, it is possible to group and specify the personal area of the specific user, and to analyze the Web page group of the personal area.

(2) The grouping program of this invention is
The similarity calculation means includes:
Whether a key that matches one of the keys constituting the first address key exists in the keys constituting the second address key between the first address key and the second address key Or from the front,
If there is a corresponding key among the keys constituting the second address key, the keys match backward from the key next to the key detected as matching in the first and second address keys. Count the number of key combinations,
Calculating the ratio of the number of key combinations associated with the second address key to the total number of keys constituting the first address key as the similarity,
It is characterized by that.

  Thereby, based on the similarity calculated by collating the second address key from the first address key, the personal area of the specific user can be grouped and specified, and the Web page group of the personal area can be specified. Analysis.

(3) The grouping program of this invention is
Grouping search means for associating, as a specific address group, addresses of links whose similarity is equal to or greater than a threshold value from the specific address to a predetermined number of link hierarchies,
A link is extracted from the Web page acquired from the address associated as a specific address group by the grouping means, and a third address key is generated by dividing the address of each extracted link by a delimiter character, Grouping search means for counting the number of matching address keys between one address key and the third address key, and calculating the similarity based on the result,
It is characterized by comprising.

  Thereby, it is possible to group and specify the personal areas of specific users up to a predetermined number of link hierarchies.

(4) The grouping program of this invention is
The first address key generating means or the second address key generating means replaces only an identifier that identifies a user at the specific address before generating at least the first address key or the second address key. Delete addresses that are determined to be
It is characterized by that.

  Thereby, URLs of different users can be easily deleted.

(5) The grouping program of this invention is
It is determined that the first address key generation means or the second address key generation means matches the address registered as a deletion target before at least generating the first address key or the second address key. Delete address,
A grouping program characterized by that.

  As a result, URLs such as advertisement sites registered in advance can be easily deleted.

(6) The grouping program of the present invention is:
Address storage means for automatically storing the specific address every predetermined time by accessing a server;
It is characterized by having.

  As a result, it is possible to automatically store specific users who specify personal areas by grouping them.

(9) The element extraction program of the present invention is
An element extraction program for extracting a predetermined element from a web page,
Computer
Web page expansion means for expanding the Web page in the display area;
Coordinate acquisition means for acquiring coordinates for specifying the display range of each element;
An element selection means for arranging a plurality of points in the arrangement direction of the elements on the display area and selecting an element including the arranged points in the display range;
Element arrangement means for arranging the hierarchical structure of the selected elements in order;
Hierarchy key generation means for generating a hierarchy key by dividing each hierarchical structure of the element in units of tags,
Similarity calculation means for counting the number of matching hierarchical keys between adjacent elements and calculating the similarity based on the result.
Content data acquisition means for specifying a hierarchical structure of two or more adjacent elements based on the similarity and acquiring content data corresponding to the hierarchical structure from a Web page;
It is made to function as.

  As a result, it is possible to divide and analyze in units such as articles included in the Web page. For example, it can be used in a search system even in article units, or only a part of a Web page determined to be harmful is hidden, or it is used for processing such as extracting only writing related to a specific product and collecting product evaluations. be able to.

(10) The element extraction program of the present invention is
Computer, and
An average of the similarities of the hierarchical structures of adjacent elements is calculated in the upper direction to generate a similarity pyramid, whether or not the similarity is equal to or greater than a threshold value, and the base of the detected similarity Content data acquisition means for specifying a hierarchical structure that matches a predetermined rule among the hierarchical structures of elements included in the URL, and acquiring content data corresponding to the hierarchical structure from a Web page;
It is made to function as.

  As a result, the hierarchical structure of articles from which content data is to be extracted can be easily specified.

(11) The element extraction program of the present invention is
The content data acquisition means is
Of the elements included in the bottom of the similarity pyramid, determine whether adjacent elements are inclusive,
If adjacent elements are not in an inclusive relationship, get the corresponding content data for each hierarchical structure,
It is characterized by that.

  Thereby, the hierarchical structure of the article from which the content data is extracted can be specified accurately.

(12) The element extraction program of the present invention is
The element extraction means;
Of the elements included in the bottom of the similarity pyramid, determine whether the hierarchical structure of adjacent elements is inclusive,
When adjacent elements are in an inclusive relationship, if the text difference is less than or equal to the threshold value, the included lower hierarchical structure is deleted, and content data corresponding to the upper hierarchical structure included is acquired,
When adjacent elements are in an inclusion relationship, if the text difference exceeds a threshold value, the upper hierarchical structure to be included is deleted, and content data corresponding to the lower hierarchical structure to be included is acquired.
It is characterized by that.

  Thereby, it is possible to more accurately specify the hierarchical structure of articles from which content data is extracted.

(13) The element extraction program of the present invention
The element selection means arranges a plurality of points at equal intervals in a predetermined direction on the display area, and selects an element including the arranged points in a display range.
It is characterized by that.

  As a result, it is possible to acquire a hierarchical key necessary for specifying a hierarchical structure from which content data is to be extracted.

(14) The element extraction program of the present invention is
The element selecting means arranges a plurality of points on a straight line perpendicular to the predetermined direction on the display area, and selects an element that includes the most points arranged on the same straight line in the display range.
It is characterized by that.

  As a result, it is possible to acquire a hierarchical key necessary for specifying a hierarchical structure from which content data is to be extracted.

(15) The element extraction program of the present invention
The element extraction means stores the attribute obtained by matching the URL of the A tag included in the element or the natural language in association with the element,
It is characterized by that.

  Thereby, the attribute of the content data extracted from the web page can be distinguished and stored.

  In the present invention, the “program” is a concept including not only a program that can be directly executed by the CPU but also a source format program, a compressed program, an encrypted program, and the like.

  The “net site” is a concept including a homepage service such as an abbreviation profile, Mobage Town, GREE (both are trademarks) and other sites that can have a page for each individual.

It is a block diagram of the grouping apparatus 100 of this invention. It is a figure which shows the procedure collated between the 1st address key and the 2nd address key. 2 is a diagram illustrating a hardware configuration of a grouping apparatus 100. FIG. It is a flowchart which shows the process by the grouping program 32 (FIG. 3). It is a figure which shows the example of data of URL extracted from the top page of the acquired user A. FIG. It is a figure which shows the state which deleted the link which corresponds to URL registered into the advertisement URL list | wrist 36. FIG. It is a figure which shows the state which deleted the link which corresponds to a user filter rule. It is a figure which shows the state which produced | generated the 1st address key. It is a figure which shows the data example of URL of the link extracted from the web page of link (alpha) 3 shown in FIG. It is a figure which shows the state which deleted the link which corresponds to URL registered into the advertisement URL list | wrist 36. FIG. It is a figure which shows the state which deleted the link which corresponds to a user filter rule. It is a figure which shows the state which produced | generated the 2nd address key. It is a figure which shows the collation method of a key. It is a figure which shows the calculation method of the similarity, and the result of having calculated the similarity. It is a figure which shows the example of data memorize | stored linked | related as a specific address group. It is a block diagram of the element extraction apparatus 200 of this invention. 2 is a diagram illustrating a hardware configuration of an element extraction device 200. FIG. It is a flowchart which shows the process by the element extraction program 38 (FIG. 16). It is the HTML data (source code) (FIG. 18A) of the bulletin board before being converted into image data, and a display example (FIG. 18B) in which this is converted into image data and displayed on the screen. It is a figure which shows the method of calculating the coordinate of an element. It is a figure which shows the state (FIG. 20B) which has arrange | positioned the state (FIG. 20A) which has arrange | positioned the point on a Web page, and the hierarchical structure of the element containing the arranged point. It is a figure which shows the state which produced | generated the hierarchy key from each hierarchy structure. It is a figure which shows the method of calculating the similarity of the hierarchical structure between elements, and the result of calculating the similarity. It is a figure which shows the state which produced | generated the similar value pyramid. It is a figure which shows the state which detected the pyramid which makes a vertex the similarity degree more than a threshold value. It is a figure which shows the detail of the process (step S220 of FIG. 17) which specifies a hierarchical structure. It is a figure which shows the method (when a text difference is below a threshold value) which integrates | stacks the hierarchical structure of the element in an inclusive relationship. It is a figure which shows the method (when a text difference exceeds a threshold value) which integrates | stacks the hierarchical structure of the element in an inclusive relationship. It is a figure which shows the example of data of the element finally extracted, and its attribute. It is a figure which shows other embodiment. It is a figure which shows the example of application of this invention.

1. Significance of the present invention In recent years, various problems have been pointed out in the mobile Internet network site. The network patrol system is automated, the children's behavior on the Internet site is continuously monitored, and children with problems are accurately identified. It has become an issue to teach.

  However, in order to automate the net patrol, it is necessary to specify the personal area of each user from a huge amount of Web pages on the net. Further, when a single Web page includes writings of a plurality of people, in order to analyze the target person more accurately, it is necessary to extract contents from the Web page in units of articles, not in units of Web pages.

  The present invention contributes to the automation of the net patrol support system, and the grouping apparatus 100 (FIG. 1) of the present invention is used to specify the personal area of the user on the net from a huge number of Web pages. It is an effective means. In order to identify the personal area on the net, for example, by searching the same user's page along the link from the viewpoint of whether the URL (address) linked from the top page of each user is similar, The personal area of each user can be easily identified.

  The element extraction program (FIG. 15) of the present invention is an effective means for extracting written contents when a plurality of contents appear repeatedly on one Web page, such as a bulletin board. . In other words, even when various data such as writings by different users are mixed in one page, only a portion that is assumed to be important can be easily extracted.

1-1. Structure of Grouping Device 100 First, the grouping device 100 of the present invention will be described with reference to FIG. FIG. 1 is a block diagram of a grouping apparatus 100 of the present invention.

  As shown in FIG. 1, the grouping apparatus 100 of the present invention includes a web page acquisition unit 2, a first address key generation unit 4, a second address key generation unit 6, a similarity calculation unit 8, Address grouping means 10 and grouping search means 12 are provided.

  Web page acquisition means 2 (FIG. 1) acquires a Web page from a specific URL. For example, the top page of the user A registered on the net site is acquired.

  The first address key generation means 4 (FIG. 1) extracts only the links from the Web page (top page) acquired by the Web page acquisition means 2, and divides the URL of each extracted link with a delimiter character as the first. Generate an address key for.

  Further, the second address key generation means 6 (FIG. 1) extracts the link from the Web page acquired from the link that generated the first address key, and divides the URL of each extracted link with a delimiter. A second address key is generated. In this way, an address key is generated up to a predetermined link hierarchy.

  The similarity calculation means 8 (FIG. 1) matches between the first address key generated by the first address key generation means 4 and the second address key generated by the second address key generation means 6. Keys to be matched, the number of combinations having the same appearance order of the corresponding keys is counted, and the similarity is calculated based on the result.

  The procedure for collating between the first address key and the second address key will be described with reference to FIG.

  As shown in FIG. 2, the collation process includes a first address key generated from one URL “link URL01” extracted from the top page (http: top ·) of user A and the URL “link URL01”. And a second address key generated from each link (for example, “link URL 11”) extracted from the Web page (http: // link1 ·). A method for calculating the similarity based on the collation will be described later.

  The grouping means 10 (FIG. 1) selects the URLs of links (for example, “link URL01” and “link URL11”) for which the similarity between the first address key and the second address key is determined to be equal to or greater than a threshold value. , As a specific address group.

  The grouping search means 12 (FIG. 1) further associates URLs of links whose similarity is equal to or greater than a threshold value from a specific URL (for example, top page) to a predetermined number of link hierarchies as a specific address group. It is means of.

  For this purpose, as shown in FIG. 2, the URL associated with the grouping means 10 as a specific address group (for example, “link URL 11”) to the Web page (the number of hierarchies 2 links ahead from the top page http: // link2 To obtain a third address key by dividing the URL of each link (for example, “link URL 21”) by a delimiter character. Further, the number of matching address keys is counted between the first address key generated from the “link URL 01” and the third address key generated from the “link URL 21”, and the similarity is calculated based on the result. Is calculated. Similarly for the n link destination hierarchy, the first address key generated from the “link URL01” and the nth address key generated from the link extracted from the n link destination Web page are identical. The number of address keys to be counted is counted, a similarity is calculated based on the result, and URLs of links whose similarity is equal to or greater than a threshold are associated as a specific address group.

  As described above, the URL related to a specific user is grouped, so that the personal area of the specific user can be accurately analyzed.

1-2. Hardware Configuration of Grouping Device 100 FIG. 3 shows a hardware configuration of the grouping device 100. The grouping processing apparatus 100 is composed of a computer including the CPU 20, RAM 22, display 24, hard disk 26, keyboard / mouse 28, and recording medium drive 30 shown in FIG.

  The hard disk 26 of FIG. 3 records a grouping program 32 for specifying and grouping a personal area of a specific user, that is, a group of Web pages belonging to the specific user. The grouping program 32 is a computer that includes the CPU 20, RAM 22, display 24, hard disk 26, keyboard / mouse 28, and recording medium drive 30, and the Web page acquisition means 2, first address key generation means 4, and so on shown in FIG. This is a program that functions as the second address key generation means 6, similarity calculation means 8, address grouping means 10, and grouping search means 12.

  In addition, a user list 34 of the net site is stored in the hard disk 26 of FIG. For example, in the user list 34, URLs corresponding to the user's top page collected in advance from the network site every predetermined time are stored.

  Further, the hard disk 26 of FIG. 3 stores an advertisement URL list 36 used for deleting advertisement URLs. For example, the advertisement URL list 34 stores URLs of advertisements collected in advance from the Internet.

1-3. Processing by Grouping Program 32 FIG. 4 is a flowchart showing processing by the grouping program 32 (FIG. 3).

  When the grouping program 32 is activated, the CPU 20 refers to the user list 34 and acquires a Web page from the URL of a specific user (step S102). For example, the top page of user A is acquired.

  Further, the CPU 20 extracts a link from the acquired web page (step S104). Specifically, the URL of the link as shown in FIG. 5 is extracted by searching for a hyperlink (so-called A tag) included in the HTML data.

  Next, an advertisement filter process is performed to delete unnecessary links (step S106). That is, the CPU 20 refers to the advertisement URL list 36 and deletes the link whose URL matches the web page of the advertisement site from the links extracted in step S104. Thereby, it is possible to delete the Web page of the advertising site that is clearly unrelated to the user A. For example, as shown in FIG. 6, all the links α1 that match the URLs registered in the advertisement URL list 36 are deleted. Note that even if an item not registered in the advertisement URL list passes the advertisement filtering process, it is usually deleted in a deletion process (step S122 in FIG. 4) based on similarity described later.

  Further, user filter processing is performed to delete unnecessary links (step S108). The user filter process is to delete a URL link in which only an identifier for specifying a user is replaced for a specific URL. For example, when only “16378304” indicating the user ID in the URL “http://pr.cccboy.com/16378304” is different, it is another user's URL (top page), and the link of the URL It was decided to delete.

  Specifically, as in “http://pr.cccboy.com/*”, only the user ID portion of the top page URL is searched as a wild card (regular expression), and as a result, The detected link α2 (URL “http://pr.cccboy.com/12345678”) shown in FIG. 7 was deleted.

  After deleting unnecessary links in Step S106 and Step S108, the CPU 20 generates a first address key by dividing the URL with a delimiter for the remaining links (Step S110).

  Specifically, each key is generated by dividing the text of the URL of each link by a delimiter (such as a non-alphanumeric slash “/”, period “.”, Etc.) excluding the protocol name. For example, from the URL “http://bbs.cccboy.com/Guestbook/BBS/16378304/” of the link α3 shown in FIG. 8, six keys “bbs”, “cccboy”, “com” "," Guestbook "," BBS ", and" 16378304 "are generated. The CPU 20 stores these keys together with the arrangement order in the RAM 22 for collation processing to be described later. Further, the URL of each link shown in FIG. 8 that is the source for generating the first address key is stored in association with the address group belonging to the top page of user A (step S111).

  Next, the CPU 20 acquires a web page from the URL of each link that generated the first address key (step S112), and extracts a link from the web page (step S114). A link extracted from the Web page of the link α3 shown in FIG. 8 is shown in FIG.

  Furthermore, unnecessary URL deletion processing is performed in the same manner as in steps S106 and S108 (step S116). For example, as shown in FIG. 10, the CPU 20 deletes the links β1 and β2 whose URLs match the Web page of the advertisement site from the link extracted in step S114 (the same processing as step S106). Further, the CPU 20 performs user filter processing, and deletes URL links β3 to β5 obtained by replacing only the identifier for specifying the user in the specific URL as shown in FIG. 11 (the same processing as step S108).

  After deleting unnecessary links in step S116, the CPU 20 divides the URL into predetermined text units for the remaining links and generates a second address key, as in step S110 (step S118).

  As described above, each key is generated by dividing the URL of each link into text units by using a delimiter (such as a non-alphanumeric slash “/” or period “.”), Excluding the protocol name. . For example, the second address key for each link is generated from the URLs of the links β6 to β9 shown in FIG. 12 (see FIG. 12).

  In the example shown in FIG. 12, when the alphanumeric string continues after “=”, the alphanumeric string (such as “0” in links β7 to β9) is set not to be generated as a key. In general, after “=”, an ID for specifying a description such as a comment is entered. Even if the ID is a comment of the same person, the ID is different for each comment. Therefore, if these are used as a key for determination of the same person, it is considered to be an obstacle. Therefore, as a result, the second address keys of the links β7 to β9 are the same.

  Furthermore, the CPU 20 collates the keys constituting each of the first address key generated in step S110 and the second address key calculated in step S118, in the same order, and The number of matching key combinations is counted, and the similarity is calculated based on the result (step S120). In this embodiment, even if there are different keys in the middle, the keys match if they appear in the same order of appearance.

  A key verification method will be described with reference to FIG. As shown in pattern 1 in FIG. 13a, there are three first address keys {a, b, c} and five second address keys {a, d, b, e, c}. In some cases, first, it is collated from the front whether or not a key matching one of the keys constituting the first address key {a} exists in the key constituting the second address key. Thereby, first, the combination of {a} shown in FIG. 13a is detected.

  Further, the next key {b} constituting the first address key is present in the second address key, or a key that matches backward by collating with the key {d} of the second address key. Count the number of combinations. Thereby, the combination of {b} {c} shown in FIG. 13A is detected, and the number of combinations having the same appearance order of the matching keys is 3.

  In the case of pattern 2 with four first address keys {a, b, c, d}, the fourth address of the first address key is detected after the combination of the keys {c} is detected. Since the key {d} is detected backward from the fifth {c} of the second address key, the second key {d} of the second address key indicated by a dotted arrow in FIG. The combinations are not counted, and the number of combinations is 3 as in the case of the pattern 1.

The similarity can be calculated from the following formula based on the number of combinations. Note that the similarity Ext (wn _i , wn _ij ) in the following expression represents the similarity between the first address key wn _i and the second address key wn _ij .

Here, the denominator count (Element (wn _i )) in the equation represents the total number of keys Element (wn _i ) constituting the first address key. Also, the numerator count (LCS (Element (wn _i ), Element (wn _ij ))) of the formula includes a key Element (wn _i ) that constitutes a first address key and a key Element (that constitutes a second address key). wn _ij ) represents the number of matching key combinations in the same order.

From the above formula, for example, the similarity Ext (wn ₁ , wn ₁₁ ) between the first address key wn ₁ shown in FIG. 13b and the second address key wn ₁₁ of the link β6 is Ext (wn ₁ , wn ₁₁ ) = 0 0/6 = 0. Further, the similarity Ext (wn ₁ , wn ₁₂ ) between the first address key wn ₁ shown in FIG. 13b and the second address key wn ₁₂ of the link β7 is Ext (wn ₁ , wn ₁₂ ) = 5/6 = 0.83333 .... (Ext (wn ₁ , wn ₁₃ ), Ext (wn ₁ , wn ₁₄ ) are the same).

  The CPU 20 removes the link for which the similarity is determined to be less than or equal to the threshold (step S122). For example, the link β6 (FIG. 12) whose similarity is calculated as “0” in FIG. 13B is deleted. On the other hand, the CPU 20 associates and stores the URL as a specific address group for the link whose similarity is determined to be greater than or equal to a threshold (for example, “0.6”) (step S124). For example, the URLs of the links β7 to β9 (FIG. 12) whose similarity is calculated as “0.83333...” In FIG. 13B are associated and grouped as an address group belonging to the top page of the user A.

  The above process is repeated until a predetermined number of links (for example, 10 links) is reached from the starting URL (step S126). If the same article is handled as the next page, the number of links will be counted.

  FIG. 14 shows an example of data stored in association with a specific address group (for example, a Web page for a specific site user) by the above processing. In the grouped URL example shown in FIG. 14, the URL (FIG. 8) of the link from which the first discrimination key stored in step S111 is generated is added to the URL of the top page of user A, and step S122. The URL (FIG. 14) that was not deleted due to the similarity determination in FIG.

  When the personal area is all the URL of the top page with a character string added, it is possible to group the URLs of the personal area by searching whether or not the URL of the top page has a forward matching relationship. Is possible. However, when not all of the personal area is obtained by adding a character string to the URL of the top page (for example, when a different character string is inserted between the URLs of the top page), according to the present embodiment. The above processing is particularly effective.

1-4. Other Embodiments In the above-described embodiment, the number of link hierarchies is set to 1 or 2. However, the number of link hierarchies is not limited to this. .

  In the above embodiment, the first address key is generated from the URL of the link included in the top page (FIG. 2 and the like), but the first address key may be generated from the URL of the top page itself. In this case, the second address key may be generated from the URL of the link included in the top page.

  In the above embodiment, the total number of the first address keys is used as the denominator in the similarity value calculation formula. However, the present invention is not limited to this, and the total number of the second address keys may be used as the denominator. The larger one may be used as the denominator.

  In the above embodiment, the number of matching keys is collated from the front to the rear. However, the present invention is not limited to this, and the collation may be performed from the rear to the front.

  In the above embodiment, the number of matching keys is collated from the front to the rear. However, the present invention is not limited to this, and the collation may be stopped halfway. For example, when a key that does not match appears, the subsequent verification may not be performed.

  In the above embodiment, only the number of matching keys is considered in order to calculate the similarity, but the similarity may be calculated by giving a weight to a specific key. For example, a key generated from a domain name may be weighted more than other keys.

  In the above embodiment, the URL is targeted, but other addresses (for example, URN) may be targeted. Both URL and URN are included in the concept of URI. It is not necessary to provide URL deletion processing.

  As described above, the personal area of the specific user can be specified, but there may be cases where writing by a plurality of individuals exists in the same Web page. In such a case, more advanced analysis can be performed if it can be distinguished from the part written by other users. Processing for dividing a Web page in a specific unit will be described below.

2-1. Structure of element extraction device 200 (FIG. 15)
The element extraction apparatus 200 according to the present invention extracts a predetermined element from a Web page, in order to extract a predetermined page from the Web page 52, coordinate acquisition unit 54, element selection unit 56, element arrangement unit 58, and hierarchical key generation unit 60 shown in FIG. , Similarity calculation means 62 and content data acquisition means 64 are provided. By using these means, even when an enormous number of elements are included in a Web page, article extraction rules for extracting only important elements are automatically determined for each Web page. Highly versatile article extraction is possible.

  Web page expansion means 52 (FIG. 15) expands the Web page (HTML document) in the display area, and coordinate acquisition means 54 (FIG. 15) displays each element included in the displayed Web page (HTML document). Get the coordinates that specify the range. Here, “element” means an element of an HTML document having a display position.

  The element selection means 56 (FIG. 15) arranges a plurality of points in the element arrangement direction on the display area, and selects an element including the arranged points in the display range. The element arrangement means 58 (FIG. 15) arranges the hierarchical structure of the selected elements in order. The hierarchy key generating means 60 (FIG. 15) generates a hierarchy key by dividing each hierarchical structure of the element in units of tags.

  The similarity calculation means 62 (FIG. 15) counts the number of matching hierarchical keys between adjacent elements, and calculates the similarity based on the result. The content data acquisition unit 64 (FIG. 15) specifies the hierarchical structure of two or more adjacent elements based on the similarity, and acquires content data corresponding to the hierarchical structure from the Web page.

2-2. Hardware Configuration of Element Extraction Device 200 FIG. 16 shows a hardware configuration of the element extraction device 200. The element extraction device 200 is configured by a computer including the CPU 20, RAM 22, display 24, hard disk 26, keyboard / mouse 28, and recording medium drive 30 shown in FIG.

  An element extraction program 38 is recorded on the hard disk 26 of FIG. The element extraction program 38 is a computer that includes the CPU 20, RAM 22, display 24, hard disk 26, keyboard / mouse 28, and recording medium drive 30. The page extraction means 52, coordinate acquisition means 54, and element selection means 56 shown in FIG. , Element arrangement means 58, hierarchical key generation means 60, similarity calculation means 62, and content data acquisition means 64.

  In addition, a browser 40 is stored in the hard disk 26 of FIG. The browser 40 virtually expands a Web page (HTML document) in a display area (for example, vertical 2040 px × horizontal 1200 px) inside the computer.

  Further, the element coordinate DB 42 is stored in the hard disk 26 of FIG. In the element coordinate DB 42, coordinates indicating the display range of the element in the display area of the browser 40 (for example, the upper left and lower right coordinates of each element when the X and Y coordinates are set on the display area) are accumulated.

2-3. Element Extraction Processing Flow FIG. 17 is a flowchart showing processing by the element extraction program 38 (FIG. 16). In the following example, a case will be described in which text is extracted in units of contents written by the user when a plurality of people have written on the bulletin board page.

  When the element extraction program 38 is activated, the browser 40 (FIG. 16) acquires an HTML document to be extracted from the Web, and arranges the elements on the display area (that is, converts them into image data) (step S202). ). FIG. 18 shows HTML data (source code) (FIG. 18A) of the bulletin board before being converted into image data, and a display example (FIG. 18B) in which this is converted into image data and displayed on the screen.

  Further, the CPU 20 acquires coordinates (for example, upper left and lower right) indicating the display area of each element on the image data, and stores them in the element coordinate DB 42 (FIG. 17) (step S204). FIG. 19 shows a method for calculating the coordinates of an element. For example, in FIG. 19, the upper left coordinates (X4, Y4) indicating the display range of the element El4 displaying “Yamada Hanako” are X 4 = x1 + x2 + x3 + x4, Y 4 = y1 + y2 + y3 + y4 Calculated. The lower right coordinates can be obtained by adding the height and width of each element to the upper left coordinates. Even if the image as shown in FIG. 19 is not actually displayed on the screen, the above processing can be performed inside the computer (that is, without being displayed on the display 24 in FIG. 16).

  Next, the process proceeds to element selection processing. First, the CPU 20 sets a virtual straight line in the vertical direction (longitudinal direction) with respect to the length direction (width direction) of the element (step S206). In the example shown in FIG. 20, one straight line L1 is set at the center. This is because there is a high possibility that important elements are arranged in the center, particularly in a prof.

  Further, the CPU 20 plots points at predetermined intervals on the set straight line (step S208). In the example shown in FIG. 20, the points are plotted at regular intervals (for example, about 10 px to 20 px). The CPU 20 refers to the element coordinate DB 42 (FIG. 17), extracts elements that include the plotted points within the display range, and arranges them in order (step S210).

  The CPU 20 divides the hierarchical structure of the extracted elements into tag units, and extracts a hierarchical key (step S212). For example, taking the third point P3 from the top of FIG. 20 as an example, the extraction of the hierarchy key is described as follows. The third point P3 corresponds to “Friends Nara ~ yo” on the 12th line in the HTML document of FIG. Search for a pair of tags (such as <body> and </ body>, which indicates the start and end of control) described between these “friends”, and use this as a hierarchical key .

  A pair of tags sandwiching "Friends ~~" are the <body> on the second line and the second </ body> on the second line, the <form> on the seventh line and the </ form> on the 15th line, 8 <Table> and 14th line </ table>, 8th line <tr> and 14th line </ tr>, 8th line <td> and 14th line </ td>, The <div> on the 9th line and the </ div> on the 12th line. Therefore, body, form, table, tr, td, and div are extracted as hierarchical keys. FIG. 21 shows an example of a hierarchical key generated by dividing the hierarchical structure of each element in units of tags.

After extracting the hierarchy key, the similarity of the hierarchy key between adjacent elements is calculated based on the formula shown in FIG. 22 (the same as the formula shown in FIG. 13b) (step S214). FIG. 22 shows an example of the result of calculating the similarity. In this embodiment, the denominator count (Element (wn _i )) of the equation is the larger number of adjacent hierarchical keys.

  Next, the CPU 20 generates a similar value pyramid as shown in FIG. 23 (step S216). In order to generate the similarity value pyramid shown in FIG. 23, it is only necessary to calculate the average of two adjacent degrees of similarity and stack them up.

  The CPU 20 searches the similarity value pyramid for a pyramid having a similarity equal to or higher than a threshold (for example, 0.7) as a vertex (step S218). At this time, search is made so that the bases of the pyramids do not overlap. For example, when a pyramid indicated by a dotted line in FIG. 24 is detected, a search is performed for a pyramid having N1 or N2 as a base so as not to include the bottom of the pyramid. In FIG. 24, the direction of search can be performed from the top to the bottom, from the bottom left to the right, or from the bottom right to the left. If the similarity pyramid does not include any similarity above the threshold value, all similarities are searched and the process is terminated.

  Further, the following integration process is executed on each searched pyramid to identify the hierarchical structure that is the target of element extraction (step S220). Details of the process of specifying the hierarchical structure will be described with reference to FIG.

  First, the CPU 20 determines an inclusion relationship for the bottom of the searched target pyramid (step S2221). Whether or not there is an inclusion relationship can be determined by comparing the coordinates of each element stored in the element coordinate DB 42 or by referring to the relationship of the hierarchical structure of tags between elements. It should be noted that “/ body / form / table / tr / td” and “/ body / form / table / tr / td / div” having different hierarchical structures included in the bottom of the pyramid indicated by the dotted line in FIG. 24 are inclusive relations. It is in.

  If the elements are not in an inclusion relationship (No in step S2221), each element is specified as an extraction target (step S2222). For example, the element “/ body / div” in the first column and the element “/ body / form / table / tr / td” in the second column shown in FIG. 21 are not in an inclusive relationship.

  If it is determined that the elements are in an inclusive relationship (Yes in step S2221), the difference in the number of characters in the text included in each element is a threshold value (for example, the difference is 0.8 of the number of text characters included in the upper layer element). In the following cases (Yes in step S2223), the included element is specified as an extraction target (step S2224). That is, the higher hierarchical structure (hierarchical structure corresponding to data2 shown in FIG. 26) is specified as an extraction target. As a result, the lower hierarchical structure (corresponding to data1 shown in FIG. 26) is integrated into the upper hierarchical structure (corresponding to data2 shown in FIG. 26).

  On the other hand, when the text difference in each element exceeds the threshold value (No in step S2223), the included element is specified as an extraction target (step S2225). That is, the lower hierarchical structure is specified as an extraction target.

  For example, if the hierarchical structure “/ body / div” (corresponding to data3) and “/ body / form / table / tr / td / div” (corresponding to data1) shown in FIG. Since the threshold value is exceeded, the lower hierarchical structure (corresponding to data1 in FIG. 26) is specified as the extraction target. In this case, it is highly likely that the upper hierarchy is not the main element itself but an upper element obtained by collecting a plurality of main elements. As a result, the upper hierarchical structure (corresponding to data3 shown in FIG. 27) is integrated into the lower hierarchical structure (corresponding to data1 in FIG. 26).

  As described above, the tag to be extracted is identified. Then, the CPU 20 determines whether or not a plurality of hierarchical structures to be extracted are included in the Web page, and extracts content data of the corresponding hierarchical structure from the HTML document of the Web page (step S226). In addition, when a plurality of pyramids are searched in step S218, if the hierarchical structure integrated in each pyramid is the same, only the same thing needs to be extracted, and when the hierarchical structures are different, What is necessary is just to extract data about each.

  After extracting the content data of the corresponding hierarchical structure from the HTML document, the CPU 20 further discriminates the attribute of the extracted content data by A tag or the like (step S228), and stores the attribute together with the extracted data (step S228). S230).

  FIG. 28 is a diagram showing an example of data finally extracted and its attributes. In the example shown in FIG. 28, text data “Taro Yamada”, “Prof. Mitayo!”, And “Naruto Friends” extracted from data1 shown in FIG. 26 are stored as content data. Note that all of data1 shown in FIG. 26 including the tag may be stored as content data.

  In the example shown in FIG. 28, the URL “http://pr.cccboy.com/0123456” obtained from the A tag portion data4 included in data1 shown in FIG. 26 is stored in association with the data attribute. Yes. Thus, if sorting is performed by the URL “http://pr.cccboy.com/0123456”, only the data of the top page URL of Taro Yamada can be acquired.

  Not only the URL of the A tag as described above but also an attribute obtained by natural language matching may be stored in association with the element. Matching by natural language processing is, for example, a method of preparing a dictionary (such as a female term dictionary) for determining an attribute in advance and determining whether or not to attach the attribute depending on the degree of matching (reference) Document “Research on Extracting Organizational Relationships Using Web Link Structure Analysis and Natural Language Processing”, Transactions of Information Processing Society of Japan, June 2006 issue). Element attributes include, for example, the name of the element creator, the element creator, the creation date, the gender, age, and organization of the element creator, as well as the topics described in the element (specific products, services, etc.) Etc.).

  As described above, the display position information of each tag (element) and content data such as articles appear repeatedly in the web page from the web page, (ii) occupy most of the web page, (iii) the center Based on the characteristics such as existing in the part, the possibility of the article part is specified, the extraction rule of the article part is generated by looking at the similarity of the hierarchical structure (XPATH) before and after the extraction, and the rule Article parts can be automatically extracted based on this.

2-4. Other Embodiments In the above embodiment, the case where write data is extracted from the bulletin board page has been described as an example. However, the present invention is not limited to this, and may be used for extracting other elements (such as news articles). Good.

  In the above embodiment, only one virtual line for placing points is set at the center (FIG. 20), but the present invention is not limited to this, and a plurality of virtual lines may be set to place points. Good. For example, in the dot arrangement example shown in FIG. 29, five straight lines are set, and the first straight line is set at the intermediate position between the maximum value and the minimum value of the x-coordinates of the elements included in the displayed Web page. Further, two straight lines offset from the straight line by a predetermined width on both sides at equal intervals are further set. In this case, in step S210 shown in FIG. 17, it is sufficient to extract an element including the most five points plotted on the same y coordinate.

  In the above embodiment, the points are arranged at equal intervals along the virtual line (FIG. 20), but the present invention is not limited to this, and the points may be arranged at different intervals.

  In the above embodiment, the similarity pyramid is generated in advance in step S216. However, the present invention is not limited to this, and the hierarchical structure may be searched without generating the similarity pyramid in advance.

  In the above embodiment, the similarity pyramid is generated in step S216. However, the hierarchical structure may be searched without generating the similarity pyramid. For example, when hierarchical structures having the same degree of similarity are not adjacent to each other but are separated by one, a method of integrating the hierarchical structure between them into the hierarchical structures on both sides is conceivable.

  In the above embodiment, the case where the element to be extracted is text has been described. However, the element to be extracted may be an image, a graph, or a moving image.

  In the above embodiment, the HTML document is displayed internally using a virtual browser. However, the HTML document may be displayed on the screen.

3. Application example to data analysis (i) Specify a personal area or generate a separation plane by sampling using Support Vector Machine (SVM) from words extracted from the divided data. By calculating the distance between the words used on the page and the separation plane, it is possible to calculate the harmful degree of the target user.

  “Harmfulness” is an index that quantifies the degree to which a target person has an adverse effect on other users. The explanation is divided into the following stages: (a) Advance preparation, (b) Hazard determination, and (c) Hazard calculation.

(A) Advance preparation First, select those who are considered to be typical harmful and harmless from users on the Internet. On that basis, frequent words in the profile / diary are extracted for a plurality of harmful persons. Similarly, frequent words in the profile / diary are extracted for a plurality of harmless persons. Then, the harmful person and the harmless person are plotted on a multi-dimensional space (shown in FIG. 30A) around the number of occurrences of all the frequent words. At that time, the plot is made by distinguishing harmful persons from harmless persons. In this multidimensional space, the separation plane is determined using the SVM technique.

(B) Harmful / Harmless Judgment Next, the number of words contained in the subject's prof / diary is counted and plotted on the multidimensional space. A harmful or harmless user can be determined by which side of the separation plane the plotted subject (indicated by Δ) is. In the embodiment, not all of the subjects plotted on the harmful side are determined to be harmful, but are determined to be harmful only when the distance from the separation plane is more than a threshold. If the distance from the separation plane is within the threshold, it is treated as harmless.

(C) Calculation of hazard level Finally, the hazard level is calculated from the following formula for the subject who is determined to be harmful.

Harmfulness = “distance from separation plane” × “number of delinquent dictionary words”
Here, the delinquency dictionary is a dictionary in which words that are considered to be related to delinquency are registered in advance. Count the number of delinquent words in the subject's profile / diary.

(Ii) In addition, when the relationship of the target user is updated or when a new user is discovered, if the risk is updated for the data identified or divided as a personal area, Can be tracked. The “risk level” is an index that quantifies how much the target person is exposed to danger from other users.

  The risk level of each user can be calculated based on the above-described degree of harm. Specifically, the risk level of the target person is obtained by calculating a value obtained by multiplying the harm degree of the linked user by the familiarity, and totaling each user linked directly or indirectly. The closeness between users can be calculated from a link relationship (FIG. 30B) or a relationship by natural language processing.

  The harmfulness level of the subject A in FIG. 30B can be obtained, for example, by summing up the values obtained by (B harmfulness × AB new density) + (D harmfulness × AD new density). In addition, the relationship with the subject C with low intimacy is not considered.

(Iii) In addition, the present invention can also be used in fields such as search by date and user in a search system, filtering (filtering processing only for articles judged to be harmful), and efficient collection of word-of-mouth articles. it can.

Claims (17)

A grouping program for grouping web page addresses,
Computer
Web page acquisition means for acquiring a Web page from a specific address;
First address key generating means for extracting a link from the web page and dividing the address of each extracted link with a delimiter to generate a first address key;
Second address key generation means for extracting a link from a Web page acquired from the link that generated the first address key, and generating a second address key by dividing the address of each extracted link by a delimiter ,
Matching the matching keys between the first address key and the second address key, counting the number of combinations in which the matching keys appear in the same order, and calculating the similarity based on the result Similarity calculation means,
Grouping means for associating the addresses of the links determined to have a similarity equal to or higher than a threshold as a specific address group;
A grouping program characterized by functioning as The grouping program of claim 1,
The similarity calculation means includes:
Whether a key that matches one of the keys constituting the first address key exists in the keys constituting the second address key between the first address key and the second address key Or from the front,
If there is a corresponding key among the keys constituting the second address key, the keys match backward from the key next to the key detected as matching in the first and second address keys. Count the number of key combinations,
Calculating the ratio of the number of key combinations associated with the second address key to the total number of keys constituting the first address key as the similarity,
A grouping program characterized by that. The grouping program according to claim 1 or 2, further comprising:
Grouping search means for associating, as a specific address group, addresses of links whose similarity is equal to or greater than a threshold value from the specific address to a predetermined number of link layers,
A link is extracted from the Web page acquired from the address associated as a specific address group by the grouping means, and a third address key is generated by dividing the address of each extracted link by a delimiter character, Grouping search means for counting the number of matching address keys between one address key and the third address key, and calculating the similarity based on the result,
A grouping program characterized by comprising: In the grouping program in any one of Claims 1-3,
The first address key generating means or the second address key generating means replaces only an identifier that identifies a user at the specific address before generating at least the first address key or the second address key. Delete addresses that are determined to be
A grouping program characterized by that. In the grouping program in any one of Claims 1-4,
It is determined that the first address key generation means or the second address key generation means matches the address registered as a deletion target before at least generating the first address key or the second address key. Delete address,
A grouping program characterized by that. In the grouping program in any one of Claims 1-5, Furthermore,
Address storage means for automatically storing the specific address every predetermined time by accessing a server;
A grouping program characterized by comprising: A grouping device for grouping web page addresses,
Web page acquisition means for acquiring a Web page from a specific address;
First address key generating means for extracting a link from the web page and dividing the address of each extracted link by a delimiter to generate a first address key;
Second address key generating means for extracting a link from a Web page corresponding to the link that generated the first address key, and generating a second address key by dividing the address of each extracted link by a delimiter character; ,
Matching the matching keys between the first address key and the second address key, counting the number of combinations in which the matching keys appear in the same order, and calculating the similarity based on the result Similarity calculation means,
Grouping means for associating addresses of links whose similarity is equal to or greater than a threshold as a specific address group;
A grouping device characterized by comprising: A grouping method for grouping addresses of Web pages by a computer, wherein the computer
Get a web page from a specific address,
Extracting a link from the web page, dividing the address of each extracted link with a delimiter to generate a first address key;
Extracting a link from the web page corresponding to the link that generated the first address key, dividing the address of each extracted link with a delimiter to generate a second address key;
Matching the matching keys between the first address key and the second address key, counting the number of combinations in which the matching keys appear in the same order, and calculating the similarity based on the result And
Associating addresses of links whose similarity is equal to or greater than a threshold as a specific address group;
A grouping method characterized by An element extraction program for extracting a predetermined element from a web page,
Computer
Web page expansion means for expanding the Web page in the display area;
Coordinate acquisition means for acquiring coordinates for specifying the display range of each element;
An element selection means for arranging a plurality of points in the arrangement direction of the elements on the display area and selecting an element including the arranged points in the display range;
Element arrangement means for arranging the hierarchical structure of the selected elements in order;
Hierarchy key generation means for generating a hierarchy key by dividing each hierarchical structure of the element in units of tags,
Similarity calculation means for counting the number of matching hierarchical keys between adjacent elements and calculating the similarity based on the result.
Content data acquisition means for specifying a hierarchical structure of two or more adjacent elements based on the similarity and acquiring content data corresponding to the hierarchical structure from a Web page;
Element extraction program characterized by functioning as In the element extraction program of Claim 9,
Computer, and
An average of the similarities of the hierarchical structures of adjacent elements is calculated in the upper direction to generate a similarity pyramid, whether or not the similarity is equal to or greater than a threshold value, and the base of the detected similarity Content data acquisition means for specifying a hierarchical structure that matches a predetermined rule among the hierarchical structures of elements included in the URL, and acquiring content data corresponding to the hierarchical structure from a Web page;
Element extraction program characterized by functioning as In the element extraction program of Claim 9 or Claim 10,
The content data acquisition means includes
Of the elements included in the bottom of the similarity pyramid, determine whether adjacent elements are inclusive,
If adjacent elements are not in an inclusive relationship, get the corresponding content data for each hierarchical structure,
An element extraction program characterized by that. In the element extraction program in any one of Claims 9-11,
The element extraction means includes
Of the elements included in the bottom of the similarity pyramid, determine whether the hierarchical structure of adjacent elements is inclusive,
When adjacent elements are in an inclusive relationship, if the text difference is less than or equal to the threshold value, the included lower hierarchical structure is deleted, and content data corresponding to the upper hierarchical structure included is acquired,
When adjacent elements are in an inclusion relationship, if the text difference exceeds a threshold value, the upper hierarchical structure to be included is deleted, and content data corresponding to the lower hierarchical structure to be included is acquired.
An element extraction program characterized by that. In the element extraction program in any one of Claims 9-12,
The element selection means arranges a plurality of points at equal intervals in a predetermined direction on the display area, and selects an element including the arranged points in a display range.
An element extraction program characterized by that. In the element extraction program in any one of Claims 9-13,
The element selecting means arranges a plurality of points on a straight line perpendicular to the predetermined direction on the display area, and selects an element that includes the most points arranged on the same straight line in the display range.
An element extraction program characterized by that. In the element extraction program in any one of Claims 9-14,
The element extraction means stores the attribute obtained by matching the URL of the A tag included in the element or the natural language in association with the element,
An element extraction program characterized by that. An element extraction device for extracting a predetermined element from a web page,
Web page expansion means for expanding the Web page in the display area;
Coordinate acquisition means for acquiring coordinates for specifying the display range of each element;
An element selection means for arranging a plurality of points in the arrangement direction of the elements on the display area and selecting an element including the arranged points in the display range;
Element arrangement means for arranging the hierarchical structure of the selected elements in order;
Hierarchy key generation means for generating a hierarchy key by dividing each hierarchical structure of the element in units of tags,
Similarity calculation means for counting the number of matching hierarchical keys between adjacent elements and calculating the similarity based on the result.
Content data acquisition means for specifying a hierarchical structure of two or more adjacent elements based on the similarity and acquiring content data corresponding to the hierarchical structure from a Web page;
An element extraction device characterized by comprising: An element extraction method for extracting a predetermined element from a Web page by a computer, wherein the computer
Expand the web page in the display area,
Get the coordinates that specify the display range of each element,
A plurality of points are arranged in the element arrangement direction on the display area, and an element including the arranged points in the display range is selected.
Arrange the hierarchical structure of the selected elements in order,
A hierarchical key is generated by dividing each hierarchical structure of the element in units of tags,
Count the number of matching hierarchical keys between adjacent elements, calculate the similarity based on the result,
Identifying a hierarchical structure of two or more adjacent elements based on the similarity, and acquiring content data corresponding to the hierarchical structure from a Web page;
Element extraction method characterized by

Images