KR101057538B1 - Web page index update method and system - Google Patents

Abstract

According to an aspect of the present invention, a web page index updating method capable of detecting a search ineligible web page by estimating a residence time of a web page includes estimating a web page residence time of a search user; Selecting a search ineligible web page candidate group from the web pages using the estimated dwell time; And determining whether the candidate group is a search ineligible web page.

Index, Dead Page, Retention Time

Description

Method and System for Updating Web Page Index

The present invention relates to an index update, and more particularly, to a web page index update method by detecting dead pages.

Due to the development and spread of the Internet, various services using the Internet are provided, and a representative example thereof is a search service. When the user inputs a word or a combination of words to be searched for as a query, the search engine may search the document for a search result document corresponding to the input query (eg, a web page, an article, or a search query input from the user). An image having a file name including the corresponding search query).

In order to provide such a search service, an index for a web page must be established. However, numerous web pages are repeatedly created and changed on the Internet. Therefore, the problem of collection is very important from the standpoint of providing a search service for web pages. In the past, how much collection of newly created web pages influenced the performance of web robots, and recently, how much quality web pages are collected is important.

In this case, it is important to collect newly generated web pages quickly in order to collect high quality web pages, but there is a problem that the existing index needs to be updated when the collected web pages change. In other words, if you don't update the existing index, the index of web pages that existed in the past for a certain period of time becomes historical data, so it can't reflect the index of the changed web pages, providing accurate search results to users. There is a problem that can not be.

In particular, if you use a search engine, you will often find that the page exists in the search results, but when you click on it, you cannot access the page or the post is deleted. This happens because the search engine doesn't recognize this, even though documents that were normal at the time the crawler collected them were either removed from the server or deleted by someone over time. As time goes by, the search ineligible pages that no longer exist at the time of the previous collection are called dead pages or soft dead pages, and the more such pages, the lower the user's search satisfaction. have.

On the other hand, search engines should exclude these pages from the index to prevent users from seeing them. If billions of documents are collected by the crawler, the crawler will revisit all pages to find out if they are ineligible for search. There is a difficult problem in reality.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problem, and it is a technical object of the present invention to provide an index updating method and system capable of detecting a search ineligible page by using a residence time of a web page.

Another object of the present invention is to provide an index updating method and system that can determine whether a detected search invalid page candidate group corresponds to an actual search invalid page and delete the index from an index.

Web page index update method according to an aspect of the present invention for achieving the above object comprises the steps of estimating the residence time of the web page of the search user; Selecting a search ineligible web page candidate group from the web pages using the estimated dwell time; And determining whether the candidate group is a search ineligible web page.

In this case, estimating the dwell time may include estimating the dwell time using at least one of a session ID and a page ID of the web page.

In particular, the time of stay is estimated by using the time difference of click time for different web pages having the same page ID, and the time between click time and search time of different web pages having the same session ID. The difference is estimated using the residence time.

The selecting of the search ineligible web page candidate group may include selecting, as the candidate group, at least one of a web page having an effective number of clicks for estimating the dwell time or more and a web page having an average of the estimated dwell time less than or equal to the reference time. It is characterized by selecting.

In this case, the determining whether the candidate group is a search ineligible web page may include re-visiting the search ineligible web page candidate group and determining whether the candidate group is a search ineligible web page.

The determining of whether the candidate group is a search ineligible web page may include determining whether the search ineligible web page candidate group is a search ineligible web page corresponding to a preset classification.

In another embodiment, the determining whether the candidate group is a search ineligible web page may include determining whether the search ineligible web page candidate group corresponds to any one of an inaccessible page, an inability to service page, and a spam page. do.

In one embodiment, the inaccessible page is characterized in that any one of the page can not be accessed by the error on the network or the error on the web server, the unserviceable page is a non-disclosure of the post, terms of service or policy violation and post It is characterized in that any one of the pages impossible to provide a search service by the deletion of the spam page, the spam page is characterized in that any one of advertising, information insufficient page and commercial site.

On the other hand, the web page index updating method further comprises the step of deleting the web page determined to be the search ineligible web page from the index. In an embodiment, the web page determined as the search ineligible web page is deleted from an index using a unique key value of the web page, wherein the unique key value is a URL or a document. It is characterized in that the ID (document ID).

Web page index update system according to another aspect of the present invention for achieving the above object is a dwell time estimator for estimating the dwell time of the web page of the search user; A candidate group selecting unit which selects a search ineligible web page candidate group from the web pages using the estimated residence time; And a determination unit determining whether the candidate group is a search ineligible web page.

As described above, according to the present invention, it is possible to quickly delete an index for a web page that is deleted or cannot be searched by selecting a search ineligible web page candidate group using the residence time.

In addition, according to the present invention, it is possible to provide a high-quality search service to the user by deleting the index by re-visiting the web page belonging to the search-ineligible web page candidate group.

In addition, according to the present invention, it is possible to easily detect spam pages as well as dead pages or soft dead pages.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic block diagram of a web page index update system according to an embodiment of the present invention. As illustrated, the index update system 100 includes a residence time estimator 120, a candidate group selector 140, a determiner 160, and an index deleter 180.

The dwell time estimator 120 estimates the dwell time of the web page of the search user, and includes a first dwell time estimator 122 and a second dwell time estimator 124.

Here, the dwell time means a time spent by the search user on the web page. In addition, the reason for using such a dwell time in the present invention is that a document with a long dwell time generally has a high probability of being a high quality document, but when the dwell time is short, users may show that the dissatisfaction is somewhat dissatisfied with the result of the page. This is because it is effective to use the dwell time for detecting unsuitable web pages. Therefore, the present invention can determine whether the search ineligible web page is directly or indirectly using the click information and the web page dwell time information of the search service users, rather than the detection of the search ineligible page through the revisit of all web pages that are practically impossible. It is.

In one embodiment, the dwell time estimator 120 may estimate the dwell time using at least one of a session ID and a page ID of the web page.

Here, Session ID is a unique value given when a user uses a search service. The session ID is mainly issued through a server based on an IP address to distinguish users. In addition, the page ID is a value that is updated each time a user enters a specific search word and searches for the page ID. Therefore, if the search user has the same session ID, the user may be the same, and if the page ID is newly generated, it may mean that the search is newly started.

The first dwell time estimator 122 estimates the dwell time using the time difference between click times for different web pages having the same page ID, and the second dwell time estimator 124 has the same session ID. The time difference between the click time and the search time of different web pages is estimated.

An example of estimating the dwell time by the dwell time estimator 120 is illustrated in FIG. 2. As shown, when looking at the two page ID and five click time during the search time by the same session ID, it can be seen that one user makes two searches and five clicks in total. Here, the dwell time estimating unit according to the present invention may estimate the dwell time of the N th clicked page by using the time difference existing between the N th click and the (N + 1) th click of the user. Therefore, 9 seconds, the time difference between the first click time (23:22:23 seconds) and the second click time (23:22:32 seconds), can be estimated as the residence time of the first clicked web page. .

In addition, the dwell time estimator may estimate the dwell time by using the time difference between the click time and the search time during the same session ID. Therefore, the time difference between the time of the second click and the time of the third click is 15 seconds, but the page ID shows that the user searched by new query input at 23:22:37. The residence time of the first clicked page can be estimated as 5 seconds, which is the time difference between the second search time (23:22:37 seconds) and the second click time (23:22:32 seconds).

On the other hand, since there is no search or click information for the last click, the dwell time cannot be estimated, so it can be excluded at all or adjusted using the average value of the dwell time of the user.

Referring back to FIG. 1, the candidate group selecting unit 140 selects a search ineligible web page candidate group from the web pages using the residence time estimated by the residence time estimating unit 120.

In an embodiment, the candidate group selecting unit 140 may include a web page having a valid number of clicks for estimating a dwell time or more and a web page having an average of dwell time estimated by the dwell time estimator 120. At least one may be selected as a search ineligible web page candidate group. In a preferred embodiment, the candidate group selecting unit 140 may select a web page for which the effective click number for estimating the dwell time is greater than or equal to the reference number and the average of the dwell time is less than or equal to the reference time as the ineligible web page candidate group. Here, the effective number of clicks does not mean the total number of clicks on the page, but the number of clicks that can estimate the residence time.

In other words, the candidate group selecting unit according to an embodiment of the present invention selects the candidate group of the search ineligible web page in advance by using the residence time because it is practically impossible to revisit all of the search ineligible web pages.

An example in which the candidate group selection group is selected as a search ineligible web page candidate group is shown in FIG. 3. If the number of valid clicks is 2 or more and the average residence time is 10 seconds or less, the web page corresponding to the illustrated URL may belong to the search ineligible web page candidate group.

Therefore, the candidate group selecting unit according to an embodiment of the present invention may increase or decrease the size of the candidate group by adjusting the reference number of effective clicks and the reference time of the average residence time.

Referring back to FIG. 1, the determination unit 160 determines whether the search ineligible web page candidate group is a search ineligible web page corresponding to a preset classification. In one embodiment, the determination unit 160 may revisit the search ineligible web page candidate group to determine whether the candidate group is a search ineligible web page.

According to an embodiment, the preset classification may include at least one of an inaccessible page, an inoperable page, and a spam page. In other words, the determination unit determines whether the search ineligible web page candidate group corresponds to any one of an inaccessible page, an inoperable page, and a spam page included in the preset classification.

An example of a criterion for determining whether or not the search incompatible web page candidate group corresponds to a search incompatible web page is shown in FIG. 4 by the determination unit. As shown, a search ineligible web page may be classified into a dead page, a soft dead page, and a spam page, and each search ineligible web page is classified in detail.

In one embodiment, the inaccessible page may be either a page that is inaccessible due to an error on a network or an error on a web server. Here, such an inaccessible page may be called a dead page. For example, a 500 error, a 404 error, and a 403 error, which are DNS errors or HTTP errors, may correspond to this. An example of a web page determined to be a dead page by the determination unit is illustrated in FIG. 5, and the illustrated web page is a dead page that cannot be accessed due to a 404 error that is an HTTP error.

In one embodiment, the non-service page may be any one of the pages that cannot be provided by the search service due to the post's non-disclosure, violation of terms of service or policy, and deletion of the post. Here, such an unserviceable page may be referred to as a soft dead page. An example of a web page determined to be a soft dead page by the determination unit is shown in FIG. 6, and it can be seen that the illustrated web page is a soft dead page in which a search service cannot be provided as a non-existent web page.

In one embodiment, the spam pages may be any of advertising, information insufficient pages, and commercial sites. An example of a web page determined to be a spam page by the determination unit is shown in FIG. 7. It can be seen that the illustrated web page is an spam page which is not suitable to be provided as a search result as an advertisement page.

In other words, the determination unit 140 according to an embodiment of the present invention does not re-visit all web pages to determine whether they are search ineligible web pages, but actually revisits web pages belonging to a search ineligible web page candidate group. It is to determine whether it is a dead page, soft dead page, or spam page.

Referring back to FIG. 1, the index deleting unit 180 deletes a web page determined by the determination unit 160 as an ineligible web page from the index. In one embodiment, the index deleting unit 180 may delete a web page determined to be a search ineligible web page using a unique key value of the web page, where the unique key value is a UEL of the corresponding web page. (URL) or document ID.

In other words, the index deleting unit 180 according to an embodiment of the present invention performs an index update by deleting, from the index, a web page that is actually determined to be a search ineligible web page by the determination unit among the search ineligible web page candidate groups. can do.

Hereinafter, a method of updating a web page index according to an embodiment of the present invention will be described with reference to FIG. 8.

First, a web page residence time of a search user is estimated (S810). Here, the dwell time means a time spent by the search user on the web page. In addition, the reason for using such a dwell time in the present invention is that a document with a long dwell time generally has a high probability of being a high quality document, but when the dwell time is short, users may show that the dissatisfaction is somewhat dissatisfied with the result of the page. This is because it is effective to use the dwell time for detecting unsuitable web pages. Therefore, the present invention determines whether the search ineligible web page is directly or indirectly using the click information and the web page residence time information of the search service users, rather than the detection of the search ineligible page through the revisit of all web pages that are practically impossible. You can do it.

In one embodiment, the method of estimating the dwell time may estimate the dwell time using at least one of a session ID and a page ID of the web page.

Here, Session ID is a unique value given when a user uses a search service. The session ID is mainly issued through a server based on an IP address to distinguish users. In addition, the page ID is a value that is updated each time a user enters a specific search word and searches for the page ID. Therefore, if the search user has the same session ID, the user may be the same, and if the page ID is newly generated, it may mean that the search is newly started.

In another embodiment, the method of estimating the dwell time estimates the dwell time using the time difference between click times for different web pages having the same page ID, and click time of different web pages having the same session ID. Estimation of residence time using time difference between and search time.

As a result of the estimation, a candidate group for ineligible search web pages is selected from the web pages using the estimated residence time (S830).

In one embodiment, a method for selecting a search ineligible web page candidate group may include search ineligible for at least one of a web page having an effective number of clicks for estimating a dwell time or more, and a web page having an average of an estimated dwell time or less. Web page candidates are selected. In a preferred embodiment, a method of selecting a search ineligible web page candidate group includes searching for at least one web page candidate group having at least a valid number of clicks for estimating a dwell time and having an average of dwell time or less. Can be selected. Here, the effective number of clicks does not mean the total number of clicks on the page, but the number of clicks that can estimate the residence time.

In other words, in the web page index updating method according to an embodiment of the present invention, since it is practically impossible to revisit all of the search ineligible web pages, the candidate group of the search ineligible web pages is previously selected using the residence time, and the selected candidate group is selected. Revisit only web pages belonging to.

After selecting the candidate group, it is determined whether the candidate group is a search ineligible web page (S850). In this case, the method of determining whether the candidate group is a search ineligible web page may revisit the search ineligible web page candidate group and determine whether the candidate group is a search ineligible web page.

In one embodiment, the method for determining whether the candidate group is a search ineligible web page may determine whether the search ineligible web page candidate group corresponds to at least one of an inaccessible page, an inability to service page, and a spam page. In other words, the web page index update method according to an embodiment of the present invention is to determine whether the search-ineligible web page candidate group corresponds to any one of an inaccessible page, an inoperable page, and a spam page included in a preset classification. .

In one embodiment, the inaccessible page may be either a page that is inaccessible due to an error on a network or an error on a web server. Here, such an inaccessible page may be called a dead page. For example, a 500 error, a 404 error, and a 403 error, which are DNS errors or HTTP errors, may correspond to this. In addition, the unavailability page may be any one of pages that cannot be provided by the search service due to the posting of the posting, the violation of the terms of service or policy, and the deletion of the posting. Here, such an unserviceable page may be referred to as a soft dead page. The spam pages may be any of advertising, information insufficient pages, and commercial sites.

In other words, the web page index updating method according to an embodiment of the present invention does not revisit all web pages to determine whether they are search ineligible web pages, but actually revisits web pages belonging to a search ineligible web page candidate group. It is to determine whether it is a dead page, soft dead page, or spam page.

Finally, the web page determined to be a search ineligible web page is deleted from the index (S870). In one embodiment, the method for deleting a web page determined to be a search ineligible web page from an index may delete the web page determined to be a search ineligible web page using a unique key value of the web page, where unique One key value may be a URL or a document ID of a corresponding web page.

In other words, the web page index updating method according to an embodiment of the present invention may perform an index update by deleting a web page from the index, which is actually determined to be a search ineligible web page by the determination unit among the search ineligible web page candidate groups. Can be.

On the other hand, the above-described web page index updating method is implemented in the form of program instructions that can be executed by various computer means can be recorded in a computer-readable recording medium. In this case, the computer-readable recording medium may include program instructions, data files, data structures, and the like, alone or in combination. On the other hand, the program instructions recorded on the recording medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software.

The computer-readable recording medium includes a magnetic recording medium such as a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical medium such as a CD-ROM and a DVD, a magnetic disk such as a floppy disk, A magneto-optical media, and a hardware device specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. The recording medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, or the like.

In addition, program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

On the other hand, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without changing the technical spirit or essential features.

Therefore, it is to be understood that the embodiments described above are exemplary in all respects and not restrictive. The scope of the present invention is shown by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

1 is a schematic block diagram of a web page index update system according to an embodiment of the present invention.

2 is a diagram illustrating an example of estimating residence time.

3 is a diagram illustrating an example of being selected as a search ineligible web page candidate group.

4 is a diagram illustrating an example of a criterion for determining whether a search ineligible web page candidate group corresponds to a search ineligible web page.

5 is a diagram illustrating an example of a web page determined to be a dead page.

6 is a diagram illustrating an example of a web page determined to be a soft dead page.

7 is a diagram illustrating an example of a web page determined to be a spam page.

8 is a flowchart showing a web page index updating method according to an embodiment of the present invention.

<Explanation of symbols for the main parts of the drawings>

100: index update system 120: residence time estimation unit

122: first residence time estimation unit 124: second residence time estimation unit

140: candidate group selection unit 160: judgment unit

180: delete index

Claims (27)

A web page index updating method performed by a web page index updating system including a residence time estimating unit, a candidate group selecting unit, and a determining unit, Estimating, by the dwell time estimator, a web page dwell time of a search user by considering a page ID; Selecting, by the candidate group selecting unit, a search ineligible web page candidate group from the web pages using click information on the search user's web page and the estimated residence time; And Re-visiting the search ineligible web page candidate group to determine whether the determination unit is a search ineligible web page; Estimating the residence time, Estimating the dwell time of the Nth clicked page using the time difference between the Nth click time and the N + 1th click time when the page IDs at the Nth click time and the N + 1th click time are the same; or If the page ID at the Nth click time and the N + 1th click time are different, the time difference between the Nth click time and the search time of the N + 1th clicked page by entering a new search term is used to determine the Estimating residence time Web page index update method comprising the. The method of claim 1, wherein estimating the residence time And estimating the dwell time using a session ID instead of a page ID of the web page. The method of claim 1, wherein the page ID is The web page index updating method, characterized in that the value is updated each time a search term is entered and searched. delete The method of claim 1, wherein selecting the search ineligible web page candidate group comprises: The candidate group includes at least one of web pages having a valid number of clicks, which is the number of clicks used for estimating the dwell time, among the total number of clicks of a page associated with the same page ID, and the average of the estimated dwell times is less than or equal to the reference time. Web page index update method, characterized in that the selection. delete delete The method of claim 1, Determining whether the candidate group is a search ineligible web page Revisiting the search ineligible web page candidate group to determine whether the search ineligible web page candidate group corresponds to any one of an inaccessible page, an inoperable page, and a spam page; The inaccessible page is The page cannot be accessed due to an error on the network or a web server due to an HTTP error or a DNS error. The out of service page is (1) a page displaying at least one of the text, images, pop-ups, redirects, and home movements of the post; (2) a page displaying at least one of the restricted or bad posts in violation of the Terms of Service or Policy; and (3 ) Any one of the pages where at least one of the text, image, pop-up, redirect, window closed, window exit, move backward, and move home is displayed, the page cannot be provided. The spam page is (1) a short article of a preset length, an advertising page with a link, (2) a video deletion, an insufficient information page with fishing articles, and (3) a commercial site associated with an advertising site, adult site, or peer-to-peer business Web page index update method, characterized in that. delete delete delete The method of claim 1, The web page index updating system further includes an index deleting unit, The index deleting unit may further include deleting a web page determined as the search ineligible web page from the index. 13. The method of claim 12, Deleting the web page from the index, And deleting the web page determined as the search ineligible web page from the index by using a URL that is a unique key value of the web page. The method of claim 13, Deleting the web page from the index, Method of updating the web page index, characterized in that deleted from the index using a document ID (document ID) instead of the URL (URL). 15. A recording medium having recorded thereon a program for performing the method according to any one of claims 1 to 3, 5, 8 and 12 to 14. In the web page index updating system for performing a web page index updating method, A dwell time estimator for estimating dwell time of a web page of a search user in consideration of a page ID; A candidate group selecting unit which selects a search ineligible web page candidate group from the web pages by using the click time on the web page of the search user and the estimated dwell time; And Determination unit for re-visiting a search ineligible web page candidate group selected by the candidate group selecting unit and determining whether the search group is a search ineligible web page. Including, The residence time estimation unit, If the page IDs at the Nth click time and the N + 1th click time are the same, the first dwell time is estimated using the time difference between the Nth click time and the N + 1th click time. Estimator; And If the page ID at the Nth click time and the N + 1th click time are different, the time difference between the Nth click time and the search time of the N + 1th clicked page by entering a new search term is used to determine the Second dwell time estimator estimating dwell time Web page index update system, characterized in that it comprises a. The method of claim 16, wherein the residence time estimating unit And estimating the dwell time using a session ID instead of a page ID of the web page. The method of claim 16, wherein the page ID is Web page index updating system, characterized in that the value is updated each time a search term is input. delete The method of claim 16, wherein the candidate group selection unit The candidate group includes at least one of web pages having a valid number of clicks, which is the number of clicks used for estimating the dwell time, among the total number of clicks of a page associated with the same page ID, and the average of the estimated dwell times is less than or equal to the reference time. Web page index update system, characterized in that the selection. delete The method of claim 16, wherein the determining unit Revisiting the search ineligible web page candidate group to determine whether the search ineligible web page candidate group corresponds to any one of an inaccessible page, an inability to service page, and a spam page included in a preset classification; The inaccessible page is The page cannot be accessed due to an error on the network or a web server due to an HTTP error or a DNS error. The out of service page is (1) a page displaying at least one of the text, images, pop-ups, redirects, and home movements of the post; (2) a page displaying at least one of the restricted or bad posts in violation of the Terms of Service or Policy; and (3 ) This page cannot be provided with a search service, which is one of pages displaying at least one of a phrase, an image, a pop-up, a redirect, a window closed, a window closed, a move backward, and a home move, The spam page is (1) a short article of a preset length, an advertising page with a link, (2) a video deletion, an insufficient information page with fishing articles, and (3) a commercial site associated with an advertising site, adult site, or peer-to-peer business Web page index update system, characterized in that. delete delete delete The method of claim 16, And an index deleting unit which deletes a web page determined as the search ineligible web page from the index. The method of claim 26, The index deleting unit, And deleting a web page determined as the search ineligible web page from the index by using a URL or document ID, which is a unique key value of the web page.

Images