JP5174385B2 - Duplicate Web site dynamic detection device - Google Patents

Description

  The present invention relates to a duplicate web site dynamic detection apparatus that dynamically performs web site duplication determination following web crawling.

  A duplicate Web site (mirror site) is a set of Web sites that provide Web pages that have the same content but differ only in the site name (host name) portion of a URL (Uniform Resource Locator). The reason for the occurrence of the duplicate Web site is logical by copying multiple host names to DNS (Domain Name System) in order to improve the ranking of physical copies for the purpose of load distribution and backup, or search engines. The act of making it appear to many sites is various.

  By detecting duplicate Web sites, it is expected to reduce duplication from the search engine results, improve the efficiency of information collection by Web crawling, and improve the cache hit rate in proxy servers and browsers.

  In the conventional duplicate Web site detection apparatus, a set of Web pages collected by Web crawling is collectively analyzed, and based on the commonality of the path name portion of the URL character string and the hash value of the Web page content, There is one that determines whether or not the above plural sites are duplicate Web sites (for example, see Patent Document 1).

  Also, URL character string similarity, IP address similarity, link URL commonality, etc. are collectively analyzed, and if these indices are higher than the reference value, the two sites are duplicate Web sites. There are some which are determined to be present (see, for example, Patent Document 2).

JP 2006-215735 A JP 2004-264926 A

However, the prior art has the following problems.
The conventional duplicate website detection method is based on the premise that information of each web page is given at once, and it is impossible to immediately determine whether an unknown website discovered during web crawling is a duplicate website. . Therefore, if the site is a duplicate website, a large number of web pages having the same content as other websites will be acquired, resulting in a decrease in the efficiency of web crawling and the statistical nature of the collected results. Will deviate from.

  On the other hand, a method is also conceivable in which only a small number of web pages are acquired from newly discovered websites, and the crawling is terminated once to detect duplicate websites. However, since duplicate Web site detection requires analysis of the entire Web page information, the execution time (that is, Web crawling stop time) increases, making it difficult to capture rapidly changing Web information in a timely manner. End up.

  The present invention has been made to solve the above-described problems, and it is possible to dynamically determine the duplication of websites following web crawling, and to efficiently collect web information and remove unnecessary information. An object is to obtain a duplicate Web site dynamic detection apparatus.

The duplicate Web site dynamic detection apparatus according to the present invention receives the URL and content of a Web page discovered during Web crawling, and associates the site name and path name extracted from the URL with the content feature amount calculated from the content At the same time, it is stored in the storage unit as Web page state information, and a set of site names in which the set of content feature values and path names match is associated with the content feature values and path names and stored as duplicate Web site candidate information in the storage unit. And a path having only one content feature amount for a plurality of site names from all path names and content feature amounts corresponding to each site name for each site name included in the set of site names. Number of names (number of hits) and multiple content features for multiple site names Path name determined number (misses) of hits and misses is a determining means for detecting a duplicate Web site set a set of site name when in the predetermined range, accepting means, and the site name, the same Among the number of paths corresponding to the number of different path names having the same site name and the path name counted as the number of paths, the same combination of the path name and the content feature amount exists in different site names. The number of duplicate paths corresponding to the number of path names is associated and further stored in the storage unit as Web site status information, and the determination means duplicates the number of paths corresponding to each site name included in the set of site names. When the ratio of the number of paths to the number of paths is within a predetermined range, a duplicate Web site set is detected .

  According to the present invention, a site having a possibility of a duplicate website by maintaining duplicate website candidate information in which a set of site names in which a set of content feature quantity and path name match is associated with the content feature quantity and path name. By making it possible to grasp a set at any time and performing duplication determination based on the correspondence between content feature quantities and path names, web site duplication determination is performed dynamically following web crawling, and Web information It is possible to obtain a duplicate Web site dynamic detection device that enables efficient collection and removal of unnecessary information.

  Hereinafter, a preferred embodiment of a duplicate Web site dynamic detection apparatus according to the present invention will be described with reference to the drawings.

Embodiment 1 FIG.
FIG. 1 is a configuration diagram of a duplicate Web site dynamic detection apparatus according to Embodiment 1 of the present invention. The duplicate Web site dynamic detection apparatus according to the first embodiment includes an accepting unit 1, a storage unit 2, a determining unit 6, and an inquiry unit 7.

  Further, the storage unit 2 stores Web page status information 3, duplicate Web site candidate information 4, and Web site status information 5. Further, the duplicate Web site dynamic detection apparatus configured as described above is connected to the Web crawler 8.

First, the function of each component will be described.
The accepting unit 1 receives the URL and content of the Web page from the Web crawler 8, and updates the Web page state information 3, the duplicate Web site candidate information 4, and the Web site state information 5 stored in the storage unit 2. In addition, when the website state information 5 satisfies a certain condition, the reception unit 1 starts the operation of the determination unit 6 by passing a set of website names.

  The determination unit 6 receives a set of website names from the reception unit 1 and refers to the Web page state information 3 stored in the storage unit 2 to determine whether the website set is a duplicate website. . When the determination unit 6 determines that the Web site set is a duplicate Web site, the determination unit 6 updates the Web site state information 5 stored in the storage unit 2.

  The inquiry means 7 receives the website name from the web crawler 8, refers to the website status information 5 stored in the storage unit 2, and the website name is an alias (non-canonical name) of the duplicate website. Is converted to a canonical name, and the converted canonical Web site name is returned to the Web crawler 8.

Next, various types of information stored in the storage unit 2 will be described with reference to FIGS.
FIG. 2 is a diagram showing details of the Web page state information 3 stored in the storage unit 2 according to Embodiment 1 of the present invention. The web page state information 3 includes a site name 31, a path name 32, and a content hash value 33. The site name 31 and the path name 32 represent the site name (host name) part and path name part of the URL character string, respectively.

  Each entry of the Web page status information 3 has a one-to-one correspondence with the Web page, and is uniquely identified by a set of a site name 31 and a path name 32 corresponding to the URL of the Web page. When a Web page with the same URL is received a plurality of times from the Web crawler 8, the same entry is overwritten on the Web page status information 3.

  Further, it is assumed that the Web page state information 3 is configured so that a plurality of entries having values designated by the site name 31 can be efficiently searched. For this purpose, for example, a known B-tree may be used to maintain each entry in an aligned state based on the set of the site name 31 and the path name 32.

  The content hash value 33 is a value obtained by applying a hash function to the entire content data of the Web page. As the hash function used here, one having a probability that the same hash value corresponds to different content data is so low as to be practically negligible is suitable. For example, known MD5 or SHA-1 can be used. .

  Next, FIG. 3 is a diagram showing details of the duplicate Web site candidate information 4 stored in the storage unit 2 according to Embodiment 1 of the present invention. The duplicate Web site candidate information 4 includes a content hash value 41, a path name 42, and a candidate site set 43. Each entry of the duplicate Web site candidate information 4 is uniquely identified by a set of a content hash value 41 and a path name 42.

  The duplicate Web site candidate information 4 shown in FIG. 3 rearranges the entries of the Web page state information 3 based on the set of the content hash value 33 and the path name 32 shown in FIG. The path name is 42. Further, the duplicate Web site candidate information 4 shown in FIG. 3 stores a plurality of entry site names 31 having the same content hash value 33 and path name 32 combination in FIG. doing.

  Next, FIG. 4 is a diagram showing details of the Web site state information 5 stored in the storage unit 2 according to Embodiment 1 of the present invention. The website status information 5 includes a site name 51, a path number 52, a duplicate path number 53, and a canonical name 54. Each entry of the Web site status information 5 is uniquely identified by a site name 51. The number of paths 52 is the number of the site names 31 in the entry of the Web page status information 3 in FIG. 2 that matches the site name 51 in FIG. 4, and represents the number of Web pages acquired from the Web site. .

  Further, the number 53 of duplicate paths indicates that the site name 31 in the entry of the Web page status information 3 in FIG. 2 matches the site name 51 in FIG. 4 and the path name 32 and content in the previous FIG. The set of hash values 33 is the number of items that are not unique in the entire Web page state information 3. That is, the number 53 of duplicate paths corresponds to the number of path names in which the same combination as the combination of the path name and the content hash value exists in different site names among the path names counted as the number of paths. Of the duplicate Web site candidate information 4 entries, the number of candidate site sets 43 includes the site name 51 and includes one or more site names different from the site name 51.

  The canonical name 54 is set to the name of the representative site of the duplicate website set in the duplicate website set. In the representative source website itself or a website that does not belong to the duplicate website set, the canonical name 54 is an empty string.

  Each means and each information indicated by reference numerals 1 to 7 can be realized by a general computer having a CPU, a memory, a magnetic disk device, and a communication interface. In this case, the reception unit 1, the determination unit 6, and the inquiry unit 7 are realized as programs to be executed by the CPU, and the storage unit 2 is realized as a magnetic disk device.

Next, a series of operations of each means will be described using a flowchart.
First, the operation of the accepting unit 1 will be described. FIG. 5 is a flowchart showing details of the operation of the accepting unit 1 according to Embodiment 1 of the present invention. First, in step S51, the accepting unit 1 receives a URL character string and content data of a Web page from the Web crawler 8, cuts out a site name and a path name from the URL character string, and sets them as an input site name and an input path name, respectively. . Further, the accepting unit 1 applies a hash function to the entire content data to convert it into a hash value, and generates an input content hash value.

  Next, in step S52, the accepting unit 1 checks whether there is an entry of the website state information 5 corresponding to the input site name. If not, the accepting unit 1 inserts a new entry into the Web site status information 5, sets the site name 51 as the input site name, sets the number of paths 52 and the number of duplicate paths 53 to 0, and creates the canonical name 54. Is set to an empty string.

  Next, in step S53, the accepting unit 1 checks whether there is an entry of the Web page state information 3 corresponding to the input site name and the input path name. And when it exists, it progresses to step S54, and when it does not exist, it progresses to step S55.

  When the process proceeds to step S54, the accepting unit 1 compares the value of the content hash value 33 in the entry of the Web page state information 3 corresponding to the input site name and the input path name with the input content hash value. If the two match, the operation ends. If not, the process proceeds to step S56.

  On the other hand, when the process proceeds to step S55 based on the determination at the previous step S53, the accepting unit 1 inserts a new entry into the web page state information 3, and stores the site name 31, the path name 32, and the content hash value 33. Set the input site name, input path name, and input content hash value respectively. Further, the accepting unit 1 adds 1 to the value of the number of paths 52 in the entry of the website state information 5 corresponding to the input site name, and then proceeds to step S58.

  When the process proceeds to step S56 as a result of the determination in the previous step S54, the accepting unit 1 executes a content hash value deletion process to be described later on the content hash value 33 of the existing entry of the Web page state information 3. Further, in the subsequent step S57, the accepting unit 1 updates the content hash value 33 of the entry by setting the input content hash value, and then proceeds to step S58.

  Next, in step S58, the accepting unit 1 executes content hash value insertion processing described later on the input site name, input path name, and input content hash value, and ends the operation.

  Next, the detailed operation of the content hash value deletion process in step S56 of FIG. 5 will be described. FIG. 6 is a flowchart showing details of the content hash value deletion processing in Embodiment 1 of the present invention.

  First, in step S61, the accepting unit 1 searches the duplicate Web site candidate information 4 entry corresponding to the deletion target content hash value and the deletion target path name (equal to the input path name), and the candidate site set 43 of the entry. Branches according to the number of element sites included in. If the number of element sites is equal to 1, the process proceeds to step S62. If the number of element sites is equal to 2, the process proceeds to step S63. Otherwise, the process proceeds to step S64.

  If the process proceeds to step S62 based on the determination in step S61, the accepting unit 1 deletes the entry searched in step S61 from the duplicate Web site candidate information 4, and the process ends.

  If the process proceeds to step S63 by the determination in step S61, the accepting unit 1 performs Web corresponding to each of the two site names stored in the candidate site set 43 of the entry searched in step S61. The number 53 of duplicate paths in the entry of the site status information 5 is decreased by 1, and the process proceeds to step S65.

  If the process proceeds to step S64 based on the determination in step S61, the accepting unit 1 sets the number of duplicate paths 53 of the entry in the website state information 5 corresponding to the deletion target site name (equal to the input site name). Decrease by 1 and proceed to Step S65.

  In step S65, the accepting unit 1 removes the deletion target site name from the candidate site set 43 of the entry searched in the previous step S61, and the process ends.

  Through the operations in steps S61 to S65 described above, the duplicate website candidate information 4 and the website state information 5 are set to the state before the deletion target content hash value is inserted, and the content hash value deletion process is completed.

  Next, the detailed operation of the content hash value insertion process in step S58 of FIG. 5 will be described. FIG. 7 is a flowchart showing details of the operation of the content hash value insertion process according to Embodiment 1 of the present invention.

  First, in step S71, the accepting unit 1 searches for an entry of the duplicate Web site candidate information 4 corresponding to the insertion target content hash value (equal to the input content hash value) and the insertion target path name (equal to the input path name). And conditional branching according to the result. If there is no entry, the process proceeds to step S72. If there is an entry and the number of element sites in the candidate site set 43 is equal to 1, the process proceeds to step S73. Otherwise, the process proceeds to step S74.

  When the process proceeds to step S72 based on the determination at the previous step S71, the accepting unit 1 inserts a new entry into the duplicate Web site candidate information 4, sets the content hash value to be inserted into the content hash value 41, and the path name 42. Is set to the insertion target path name, and the candidate site set 43 is set to the insertion target site name (equal to the input site name).

  If the process proceeds to step S73 based on the determination in step S71, the accepting unit 1 corresponds to the single element site name of the candidate site set 43 of the duplicate Web site candidate information 4 entry searched in step S71. The entry of the Web site status information 5 is searched, 1 is added to the number 53 of duplicate paths of the entry, and the process proceeds to step S74.

  Next, in step S74, the accepting unit 1 searches for an entry of the website status information 5 corresponding to the insertion target site name, adds 1 to the duplicate path number 53 of the entry, and duplicate web searched in step S71. The insertion target site name is added to the candidate site set 43 of the entry of the site candidate information 4.

  Further, in step S75, the accepting unit 1 searches for an entry of the Web site state information 5 corresponding to each element site name in the candidate site set 43, and the value of the number of duplicate paths 53 in each entry and the number of duplicate paths 53. A ratio for the number of passes 52 is obtained and compared with a threshold value for each. If the two values are equal to or larger than the threshold value in all entries, the process proceeds to step S76, and if not, the process ends. However, in the determination of step S75, the entry whose canonical name 54 is not an empty character string is ignored, and the process ends if the canonical name 54 has one or less entries of the empty character string.

  When the process proceeds to step S76 as a result of the previous determination in step S75, the accepting means 1 passes the name of the canonical site 54 of the corresponding website status information 5 among the plurality of site names in the candidate site set 43. Then, the operation of the judging means 6 is started and finished.

  Here, for example, the threshold value in step S75 is 3 for the number 53 of overlapping paths and 0.4 for the ratio of the number 53 of overlapping paths to the number 52 of paths. The purpose of step S75 is to avoid determination on a candidate site set that is clearly determined not to be a duplicate Web site by the determination means 6.

  Next, the operation of the determination unit 6 will be described. FIG. 8 is a flowchart showing details of the operation of the determination means 6 in Embodiment 1 of the present invention. This flowchart is started in step S76 of FIG. Then, in step S81, the determination means 6 uses the path name 32 and the content hash value for the entry of the Web page status information 3 that matches any of the site names belonging to the candidate site set received by the site name 31. The appearance frequency is counted for each set of 33 values.

  In subsequent step S82, the determination means 6 counts the path name type and the path name type (miss) corresponding to two or more content hash values, and calculates the ratio of the latter as the miss rate. . If the miss rate is greater than or equal to the first predetermined threshold, the determination unit 6 determines that the determination result is false and ends the process. On the other hand, if the miss rate is less than the first predetermined threshold, the process proceeds to step S83.

  Next, in step S83, the determination means 6 has an appearance frequency (number of sites) of 2 or more and a certain ratio or more of the number of candidate site set elements for a path name corresponding to one type of content hash value. Count the number of hits (hits) and calculate the number of hits. Then, when the number of hits is less than the second predetermined threshold, the determination unit 6 determines that the determination result is false and ends the process.

  In the determination in step S83, the determination unit 6 can also use the path name corresponding to one type of content hash value as a hit only on condition that the appearance frequency (the number of sites) is a predetermined number or more.

  On the other hand, even when the ratio of the number of hits to the type of path name (hit rate) is less than the third predetermined threshold, the determination unit 6 determines that the determination result is false and ends the process. If the number of hits is equal to or greater than the second predetermined threshold and the hit rate is equal to or greater than the third predetermined threshold, the determination unit 6 determines that the determination result is true and proceeds to step S84.

  Here, the second predetermined threshold for the number of hits and the third predetermined threshold for the hit rate correspond to the threshold for the number of overlapping paths and the threshold for the ratio of the number of overlapping paths to the number of paths in step S75 of FIG. And the same value is used (for example, 3 and 0.4 can be used similarly to the description in step S75).

  Next, in step S84, the determination unit 6 selects one representative site name from the candidate site set, and displays the representative name in the canonical name 54 of the entry of the website state information 5 corresponding to the site name other than the representative source. Set the site name of and exit.

  Here, when selecting a representative from the candidate site set, a score is assigned to the site name of each element, and the one with the highest rank (the one with the smallest value) is selected. FIG. 9 is a diagram showing an example of scores for site names in Embodiment 1 of the present invention. As shown in FIG. 9, a site name score defined in advance by the site name character string length and the number of domain levels is prepared as a calculation formula according to the character string pattern of the site name, and this score calculation formula is used as a regular site. Used for name selection.

  More specifically, the character string pattern of the entire site name is collated in order of application priority, and the site name score calculation formula of the first matching row is used. However, the number of domain levels is the number of domain name elements separated by “.”. For site names with the same score, the one that comes first in the order of character strings is given priority.

  FIG. 10 is a conceptual diagram showing the meaning of the operation of the determination means 6 in Embodiment 1 of the present invention. As shown in FIG. 10, a series of operations according to the flowchart of FIG. 8 is performed by hitting each column in a matrix in which the site name 31 is associated with each row and the path name 32 is associated with each column and the content hash value 33 is arranged. It is equivalent to making a mistake determination and counting each number.

  The portion where the content hash value is N / A indicates that the Web page corresponding to the site name and the path name has not been received from the Web crawler 8. Since the Web crawler 8 generally collects Web pages while following links between Web pages, even an existing Web page may not be accessed.

  In the example of FIG. 10, the path name “/” is a hit because all content hash values are equal, type = 1, and appearance frequency = 3. For the path name “/links.html”, the site aaa. bbb. ccc and zzz. www. Since there is no value corresponding to aaa, the type = 1 and the appearance frequency = 1 are regarded as neither a hit nor a miss. Furthermore, for the path name “/news.html”, the type of content hash value = 2, which is a mistake.

  Next, the operation of the inquiry means 7 will be described. FIG. 11 is a flowchart showing details of the operation of the inquiry means 7 in the first embodiment of the present invention. First, in step S111, the inquiry unit 7 receives an inquiry from the Web crawler 8, and sets the site name to be inquired as a result site name.

  Next, in step S112, the inquiry unit 7 searches for an entry of the website state information 5 corresponding to the result site name, and if there is no entry or the canonical name 54 of the entry is an empty character string, The process proceeds to step S114. On the other hand, if the canonical name 54 of the entry is not an empty character string but an entry exists, the inquiry means 7 determines that the result site name is a non-representative source in a duplicate Web site and the site name needs to be converted. Then, the process proceeds to step S113.

  In step S113, the inquiry unit 7 sets the canonical name 54 of the entry as the result site name, and returns to step S112. Here, the process of step S112 is repeated because there is a possibility that a representative source of a certain duplicate website is later included as a non-representative source in another duplicate website.

  On the other hand, when the process proceeds to step S114 based on the determination at the previous step S112, the inquiry unit 7 returns the value of the result site name to the web crawler 8 and ends.

  When the web crawler 8 extracts a link from the web page or before starting the download of the web page, the web crawler 8 extracts the site name from the URL and replaces the original site name with the result obtained by passing to the inquiry means 7. As a result, it is possible to avoid access to non-representative sources of duplicate Web sites.

  As described above, according to the first embodiment, the reception unit that sequentially receives Web pages from the Web crawler is provided, so that it is possible to keep track of a set of sites that may be duplicate Web sites while maintaining the duplicate Web site candidate information. In addition, it is possible to control the execution timing of the overlap determination by the determination means while maintaining the website state information. As a result, it is possible to realize dynamic duplicate Web site detection following Web crawling.

  Further, inquiry means for inquiring the duplicate Web site detection result is provided. As a result, it is possible to provide the Web crawler with a means for avoiding Web page collection from other than the representative of the duplicate Web site.

  In the first embodiment described above, the content hash value, which is an example of the content feature amount, is a value obtained by applying the one-way hash function to the entire content data. However, another calculation method may be used. For example, when the content is described in HTML, a value obtained by applying a one-way hash function to the remaining text data from which HTML tags, comments, scripts, and styles are removed is the content that is the content feature amount. The hash value does not affect the overall configuration or operation. By applying such a content hash value, it is possible to ignore variable elements such as advertisements included in the Web page, and there is an effect that more duplicate Web sites can be detected.

  In the first embodiment described above, the case where the regular name of the site name is stored and managed as the website state information together with the information on the number of paths and the number of duplicate paths has been described. However, apart from managing the number of paths and the number of duplicate paths, it is also possible to extract the legitimate site name using the inquiry means by managing the website name related information by associating the legitimate site name with the remaining site name. It becomes possible.

Embodiment 2. FIG.
In the first embodiment, since the candidate site set 43 of the duplicate website candidate information 4 is maintained for all input path names, even if there is a bias in the collection range for each website, the duplicate website is not detected. I was trying to prevent. However, on the other hand, there is a problem that the data amount of the duplicate Web site candidate information 4 becomes large and the update load becomes high.

  Therefore, in the second embodiment, a case will be described in which only a part of the input path name is reflected in the duplicate website candidate information 4 and the website state information 5. More specifically, if the input path name does not match the specific pattern, it is assumed that the accepting unit 1 does not execute the latter half of step S55 in FIG. 5, the steps S56 and S58.

  As the specific pattern of the input path name, for example, the input path name is matched with a pattern including “index” as a partial character string. Thereby, the candidate site set 43 can be maintained only for the general URL as the entrance of the Web site, and the data amount of the duplicate Web site candidate information 4 can be greatly reduced. Note that the Web page state information 3 includes information on all path names, and thus does not affect the operation of the determination unit 6.

  Further, as another example of the specific pattern of the input path name, it may be matched with one including only one “/”. As a result, only the top-level path name of the directory is reflected. In general, a link of a Web page tends to indicate a path name in the upper directory, so that it is possible to prevent a candidate site set from being overlooked by assuming a specific character string.

  Note that the update load can be reduced based on the value of the number 52 of paths in the website status information 5 in addition to limiting the input path name to a specific pattern. That is, after the number of paths 52 reaches a certain value, the latter half of step S55, step S56 and step S58 are not executed regardless of the input path name. This is equivalent to restricting the input path name not in terms of value but in first-come-first-served basis, and can solve the problem of oversight that cannot be avoided in the pattern-based method.

  As described above, according to the second embodiment, the input path name can be reflected in the duplicate Web site candidate information 4 and the Web site state information 5 only for the input path name having the specific pattern. As a result, the candidate site set can be maintained only for general URLs, and the data amount of the duplicate Web site candidate information can be greatly reduced.

  Further, the input path name is reflected in the duplicate website candidate information 4 and the website status information 5 only before the number of paths corresponding to the input path name reaches a certain value, not the specific pattern of the input path name. be able to. As a result, the input path name can be limited not in terms of value but in the order of arrival, and the overlooked problem that cannot be avoided by the method based on the specific pattern can be solved.

Embodiment 3 FIG.
In the third embodiment, in addition to the first embodiment, considering the similarity of the site names in the candidate site set, the allowable range of content hash value mismatches for the candidate site set consisting of similar site names A case will be described in which it is easy to detect a duplicate Web site.

  FIG. 12 is a flowchart showing details of the operation of the determination means 6 in Embodiment 3 of the present invention. In the third embodiment, the determination unit 6 performs the operation shown in the flowchart of FIG. 12 prior to step S81 of FIG. 8 described in the first embodiment.

  First, in step S121, the determination unit 6 obtains the number of domain levels (corresponding to the number of components delimited by “.”) For each site name of the candidate site set, and sets the minimum value as the minimum domain level number. For example, when the site name is xxx.yyy.zzz, the number of domain levels is 3.

  Next, in step S122, the determination unit 6 obtains how many site names are included in each site name component (for example, “xxx”, “yyy”, and “zzz”), and sets candidate site sets. Count the number of sites included in site names above a certain percentage of the number of frequent domain levels.

  Next, in step S123, the determination means 6 compares the number of frequent domain levels with the minimum number of domain levels, and ends if the number of frequent domain levels is smaller than the minimum number of domain levels. On the other hand, if the frequent domain level number is greater than or equal to the minimum domain level number, the process proceeds to step S124. In step S124, the determination unit 6 multiplies the threshold of the miss rate by a coefficient proportional to the logarithm of the number of elements of the candidate site set, and multiplies the threshold of the hit number and hit rate by the reciprocal of the coefficient. For example, log (number of sites) × 4.5 is used as the coefficient.

  In this way, by performing the series of pre-processing shown in FIG. 12, the determination means 6 can perform duplication determination using information on the similarity of site names. In particular, when the site names in the candidate site set have high similarity, it is possible to prevent the detection of a large-scale duplicate site from being missed by expanding the threshold range and preparing conditions that make it easy to determine that there is a duplicate.

  As described above, according to the third embodiment, it is possible to change the threshold used for the duplication determination according to the similarity by using the information related to the similarity between the site names. As a result, it is possible to prevent omission of detection of a large-scale duplicate site having high similarity.

Embodiment 4 FIG.
In the fourth embodiment, in addition to the first embodiment, a case will be described in which detection is possible even when Web sites are duplicated in directory units.

  In the fourth embodiment, the accepting unit 1 generates the input site name and the input path name from the received URL in step S51 of FIG. In addition to the path name, generate a pseudo site name that concatenates the upper directory name of the path name to the site name, and a pseudo path name that consists of the remaining path name parts that are not concatenated with the upper directory name. .

  That is, for the URL “http: //aaa.bbb.ccc/˜user1/diary.html”, not only the site name “aaa.bbb.ccc” and path name “/˜user1/diary.html” but also pseudo A typical site name “aaa.bbb.ccc / ˜user1” and a pseudo path name “/diary.html” are generated. Other operations are the same as those in the first embodiment.

  In this way, by performing duplication determination in consideration of a pseudo site name and a pseudo path name, even if a part of the web site is duplicated, it is possible to detect the web crawling. It is possible to prevent a decrease in efficiency.

  As described above, according to the fourth embodiment, when the input site name and the input path name are generated from the received URL, the pseudo site name and the pseudo path name are also generated and used for duplication determination. We are using. As a result, even when a part of the website overlaps, it can be reliably detected.

Embodiment 5 FIG.
In the fifth embodiment, a case will be described in which path names that are likely to be erroneously determined are not used for duplicate determination.

  FIG. 13 is a diagram showing excluded path name information 9 newly stored in the storage unit 2 in Embodiment 5 of the present invention. This excluded path name information 9 is a set of excluded path names 91.

  The determination unit 6 ignores the case where the path name 32 matches the excluded path name 91 when searching for the entry of the Web page state information 3 in step S81 of FIG. 8 described in the first embodiment. To.

  In addition, when the determination result in step S82 or step S83 in FIG. 8 is false, the determination unit 6 has a site name that appears among the hit path names that is equal to or greater than a certain threshold (within a second predetermined range). Equivalent) and add it to the excluded path name information 9.

  In this manner, by providing the excluded path name information 9, the influence of a path name (for example, a manual page of the Web server software) that may be common even if the website is completely unrelated is eliminated, and an error is caused. Therefore, it can be determined that the site is a duplicate site.

  As described above, according to the fifth embodiment, duplication determination processing can be performed in consideration of an excluded path name. As a result, it is possible to prevent a web site that is completely unrelated but has a common path name from being erroneously determined as a duplicate site, and to improve the determination accuracy.

It is a block diagram of the duplicate Web site dynamic detection apparatus in Embodiment 1 of this invention. It is the figure which showed the detail of the web page state information stored in the memory | storage part in Embodiment 1 of this invention. It is the figure which showed the detail of the duplication Web site candidate information stored in the memory | storage part in Embodiment 1 of this invention. It is the figure which showed the detail of the website status information stored in the memory | storage part in Embodiment 1 of this invention. It is a flowchart which shows the detail of operation | movement of the reception means in Embodiment 1 of this invention. It is a flowchart which shows the detail of operation | movement of the content hash value deletion process in Embodiment 1 of this invention. It is a flowchart which shows the detail of operation | movement of the content hash value insertion process in Embodiment 1 of this invention. It is a flowchart which shows the detail of operation | movement of the determination means in Embodiment 1 of this invention. It is the figure which showed the example of the score with respect to the site name in Embodiment 1 of this invention. It is a conceptual diagram which shows the meaning of operation | movement of the determination means in Embodiment 1 of this invention. It is a flowchart which shows the detail of operation | movement of the inquiry means in Embodiment 1 of this invention. It is a flowchart which shows the detail of operation | movement of the determination means in Embodiment 3 of this invention. In Embodiment 5 of this invention, it is the figure which showed the exclusion path name information newly stored in a memory | storage part.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Accepting means, 2 Storage unit, 3 Web page state information, 31 Site name, 32 Path name, 33 Content hash value (content feature amount), 4 Duplicate website candidate information, 41 Content hash value, 42 Path name, 43 candidate Site set, 5 Web site status information, 51 Site name, 52 Path count, 53 Duplicate path count, 54 Regular name, 6 Judgment means, 7 Inquiry means, 8 Web crawler, 9 Excluded path name information, 91 Excluded path name.

Claims (12)

The URL and content of the Web page discovered during Web crawling are received, and the site name and path name extracted from the URL are associated with the content feature amount calculated from the content and stored in the storage unit as Web page state information And a receiving unit that stores a set of site names in which the set of the content feature quantity and the path name match with each other in association with the content feature quantity and the path name, and stores it as duplicate Web site candidate information in the storage unit;
For each site name included in the set of site names, the number of path names having only one content feature amount for a plurality of site names from all path names and content feature amounts corresponding to each site name ( Number of path names having a plurality of content feature quantities for a plurality of site names (number of misses), and when the number of hits and the number of misses are within a predetermined range, Determination means for detecting as a duplicate Web site set ,
The reception means includes a site name, a path number corresponding to the number of different path names having the same site name, and a set of a path name and a content feature amount among the path names counted as the path number. The same combination is further stored in the storage unit as Web site state information in association with the number of duplicate paths corresponding to the number of path names existing in different site names,
The determination means determines the number of duplicate Web site sets when both the number of paths corresponding to each site name included in the set of site names and the ratio of the number of paths to the number of duplicate paths are within a predetermined range. A duplicate Web site dynamic detection apparatus that performs detection. In the duplicate Web site dynamic detection device according to claim 1,
The determination means selects one site name included in the detected duplicate Web site set as a normal site name, associates the remaining site name with the normal site name, and stores the storage unit as Web site name related information. To remember further,
When a site name of a web page discovered during web crawling is received, and a regular site name associated with the site name is stored in the website name related information in the storage unit, the site name A duplicate Web site dynamic detection apparatus further comprising inquiry means for outputting the corresponding regular site name. In the duplicate Web site dynamic detection device according to claim 2,
As a pre-processing for detecting the duplicate Web site set, the determination means stores a regular site name corresponding to each site name included in the site name set in the Web site name related information in the storage unit. If there is, the duplicate Web site dynamic detection apparatus, wherein a site name corresponding to the regular site name is removed from the set of site names. In the duplicate Web site dynamic detection device according to claim 2 or 3,
The determination means selects the site name with the highest score from the site names included in the duplicate Web site set according to a score calculation formula defined in advance by the character string pattern of the site name and the number of domain levels. A duplicate Web site dynamic detection apparatus characterized by being selected as: The duplicate Web site dynamic detection device according to any one of claims 1 to 4,
The determination unit counts the number of path names having only one content feature amount as a hit number for a plurality of site names that are equal to or greater than a predetermined ratio of the number of all site names. Detection device. In the duplicate Web site dynamic detection device according to any one of claims 1 to 5,
The duplicate Web site dynamic detection apparatus, wherein the reception unit calculates the content feature amount by applying a one-way hash function to the received content data. In the duplicate Web site dynamic detection device according to any one of claims 1 to 5,
The receiving means calculates the content feature amount by applying a one-way hash function to the remaining data obtained by removing HTML tags, HTML comments, scripts, and styles from the received content data. Web site dynamic detection device. In the duplicate Web site dynamic detection device according to claim 1 ,
If the path name does not match a predetermined character string pattern, the accepting unit does not store the path name as the duplicate Web site candidate information in the storage unit, and counts the path name and the number of duplicate paths. A duplicate Web site dynamic detection device characterized in that it is excluded from the above. In the duplicate Web site dynamic detection device according to claim 1 ,
When the site name of a web page discovered during web crawling matches a specific site name whose number of passes has reached a predetermined value, the accepting unit includes the duplicate web site candidate information. The path name corresponding to the site name of the Web page discovered during the Web crawling is not stored as the element of the set of site names, and the path number corresponding to the site name of the Web page discovered during the Web crawling and the number of duplicate paths A duplicate Web site dynamic detection device, characterized in that it is excluded from the counting target. In the duplicate Web site dynamic detection device according to any one of claims 1 to 9 ,
The determination means determines the similarity of each site name by dividing each site name into components for each site name included in the set of site names, and If it is determined that a site name having a similarity equal to or greater than a predetermined value is included, the predetermined range for the number of hits and the number of misses when detecting the duplicate Web site set is changed, and the similarity A duplicate Web site dynamic detection apparatus characterized by facilitating detection of a high site name as a duplicate Web site. The duplicate Web site dynamic detection device according to any one of claims 1 to 10 ,
When the receiving unit extracts a site name and a path name from the received URL, a pseudo site name obtained by concatenating the site name with an upper directory name of the path name and the upper directory name in the path name are A duplicate Web characterized in that a pseudo path name consisting of the remaining parts that are not connected is generated, and the pseudo site name and the pseudo path name are included in the site name and path name, respectively. Site dynamic detection device. The duplicate Web site dynamic detection device according to any one of claims 1 to 11 ,
The determination means excludes a path name included in the set of site names if the number of hits obtained for the set of site names not detected as a duplicate Web site set is within the second predetermined range. If the path name of the Web page discovered during Web crawling matches the path name stored as the excluded path name, the path name is detected when a duplicate Web site is detected. A duplicate Web site dynamic detection apparatus characterized by not being referred to.

Images