KR100975510B1 - Method and System for Updating Web Page Index - Google Patents

Abstract

According to an aspect of the present invention, a method of updating a web page index according to an aspect of the present invention, which may determine a revisit priority through prediction of a change degree and a change frequency of a web page, includes: predicting at least one of a change degree and a change frequency of the web page; And determining the revisit priority of the web page using at least one of the predicted change degree and the change frequency.

Index, Update, Return, Hamming Distance

Description

Method and System for Updating Web Page Index

The present invention relates to an index update, and more particularly, to an index update method through revisit of a web page.

Due to the development and spread of the Internet, various services using the Internet are provided, and a representative example thereof is a search service. When the user inputs a word or a combination of words to be searched for as a query, the search engine may search the document for a search result document corresponding to the input query (eg, a web page, an article, or a search query input from the user). An image having a file name including the corresponding search query).

In order to provide such a search service, an index for a web page must be established. However, numerous web pages are repeatedly created and changed on the Internet. Therefore, the problem of collection is very important from the standpoint of providing a search service for web pages. In the past, how much collection of newly created web pages influenced the performance of web robots, and recently, how much quality web pages are collected is important. In this case, it is important to collect newly generated web pages quickly in order to collect high quality web pages, but there is a problem that the existing index needs to be updated when the collected web pages change. In other words, if you don't update the existing index, the index of web pages that existed in the past for a certain period of time becomes historical data, so it can't reflect the index of the changed web pages, providing accurate search results to users. There is a problem that can not be.

On the other hand, if the change frequency is too high, the web page will already be changed to other content during the index update, and the index update will not be effective. For certain web pages, it will change frequently over time. If not, even if the index is updated, there is little problem to the search service.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems, and it is an object of the present invention to provide an index updating method and system capable of updating an index through revisit of an existing web page.

Another object of the present invention is to provide an index update method and system capable of updating an index according to a priority determined through a change degree and a change frequency prediction.

Web page index update method according to an aspect of the present invention for achieving the above object comprises the steps of predicting at least one of the degree and frequency of change of the web page; And determining the revisit priority of the web page using at least one of the predicted change degree and the change frequency.

In one embodiment, the degree of change and the frequency of change is at least one of a dynamic feature including information related to the change history of the web page or a static feature including information extracted from the web page. It is characterized by being predicted using one. In this case, the static feature includes at least one of information on the length of the body of the web page and the number of anchor text, the dynamic feature of the information using a Hamming Distance (hamming distance) for the hash value of the web page It characterized in that it comprises at least one.

In another embodiment, the change degree and the change frequency are predicted using the chunk extracted from the web page. On the other hand, the degree of change and the frequency of change is characterized by being predicted using a logistic regression (Logistic Regression).

In one embodiment, predicting the degree of change and the frequency of change comprises: building a change history of the web page; And predicting the degree of change and the frequency of change using the change history and the logistic regression model.

In this case, the change history is constructed using a hash value calculated using at least one of information about an image URL, anchor text, and body text extracted from the web page. .

The building of the change history may include calculating a hamming distance using a hash value of the web page; And generating at least one of an average of the Hamming distance, a standard deviation, a weighted average, and a ratio of the Hamming distance of zero. On the other hand, the change history is characterized by being built using the deep hash (SIMHASH) value of the web page.

In an embodiment, the determining of the priority may include determining the priority after performing normalization on the predicted change frequency, and in another embodiment, determining the priority. The priority is determined by using the elapsed time since the last visit to the web page.

The web page updating method may further include updating the index of the web page by re-visiting the web page according to the priority.

Web page index update system according to another aspect of the present invention for achieving the above object is a prediction unit for predicting at least one of the degree and frequency of change of the web page; And a priority determiner configured to determine revisiting priority of the web page using at least one of the predicted change degree and the change frequency.

As described above, according to the present invention, by re-visiting the existing web page and updating the index, it is possible to quickly provide the user with a search result for the changed web page.

In addition, according to the present invention, it is possible to provide a high-quality search service to the user by updating the index by re-visiting the web page according to the revisit priority determined by predicting the degree and frequency of change of the web page.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic block diagram of a web page index update system according to an embodiment of the present invention. As shown, the index update system 100 includes a predictor 120, a priority determiner 140, and an index updater 160.

The predictor 100 predicts at least one of a change degree and a change frequency of the web page, and includes a change history construction unit 122 and a regression model predictor 124.

The index update system according to an embodiment of the present invention predicts how the change and frequency of change of the web page will change in the future based on the past visit history of the web page, and basically visits the web page more than once. Assume that Here, the prediction of change degree refers to predicting how much the web page changes, and the prediction of change frequency refers to predicting how often the web page changes.

In one embodiment, the predictor 120 may include a dynamic feature including information related to the change history and the change frequency of the web page or a static feature including information extracted from the web page. ) Can be predicted using at least one of In this case, the dynamic feature may include at least one of information using a Hamming Distance for the hash value of the web page, and the static feature may include at least one of information about the length of the web page body and the number of anchor texts. It may include.

Here, the Hamming distance is the number of different characters at the same position in two strings of the same length. Therefore, a large hamming distance between the hash value of the past web page and the present web page may mean that the difference between the two hash values is large, that is, the difference between the past web page and the current web page may be large. have.

Examples of dynamic and static features used in the predictor 120 are shown in FIG. 2. Each of the features shown has values for the corresponding information, and features starting with "C", such as features 2 through 7, 16 and 17, can have real values, and numbers 1 and 8 Features starting with "D", such as features 15 through 15, may have a value of zero or one. In addition, the values of the static features are updated to the existing values every time a new value is created, but the values of the dynamic features can be stored and managed with the existing values.

As shown, the static feature is the type of parser like "DParserType", the length of the body like "CBodyLength", the number of anchor text like "CAnchorCount", the number of images like "CImageCount", the "CPageRank" Page rank (PageRank) as shown, number of slashes in URL as shown as "CUrlDepth", length of the URL as shown as "CUrlLength", domain type as shown in features 8 to 3, presence or absence of URL parameter as shown as "DHasParameter", "DIsRootPage It may include information about whether the URL is a "/" ending, such as "," and other information that can be extracted from the web page in addition to the information shown may be included in the static feature.

Also, as shown, the dynamic feature is the mean of Hamming Distance, such as "CHDAverage", the standard deviation of Hamming Distance, such as "CHDDeviation", the weighted average of Hamming Distance, such as "HDWeightedAverage", " UnChangedRate "may include information such as the ratio of the hamming distance to the total number of visits to zero, that is, the rate at which the web page does not change. The dynamic feature may include other information, not limited to the illustrated information.

In another embodiment, the prediction unit 120 may predict the degree of change and the frequency of change by using the chunk extracted from the web page. First, let's look at chunks. To crawl a web page, you can go through a parser, such as an HTML parser, to get the contents of that web page. In this case, the chunk may be extracted from the body of the web page, where chunk refers to each chunk when the body is divided into at least one chunk.

Meanwhile, the prediction unit 120 may predict the degree of change and the frequency of change using the Shingle method using the above-described chunk. Here, the phrase means a set of consecutive chunks appearing in the document.

For example, the string "a rose is a rose is a rose" may be divided into "a, rose, is, a, rose, is, a, rose". Here, assuming that a shingle consists of four words in chunks, the given string is "(a, roes, is, a), (rose, is, a, rose), (is, a, rose, is), (a , rose, is, a), (rose, is, a, rose) ", and if you remove duplicates," (a, roes, is, a), (rose, is, a The words "rose," (is, a, rose, is) "will be the final result.

In this case, assuming that there are two documents, the similarity of the two documents may be calculated as the ratio of the same number of articles to the total number of articles in which the duplicate articles of the two documents are removed. For example, assuming that each document has 10 articles and the same article has 5 articles, the similarity of the two documents may be calculated as "5/15".

Therefore, the prediction unit according to an embodiment of the present invention extracts chunks from the past web page and the current web page, for example, by a predetermined word consecutively, and uses the extracted chunks to determine the ratio of the same words to the total number of words. By calculating, you can predict the frequency and degree of change in the web page.

In one embodiment, the prediction unit 120 may predict the degree of change and the frequency of change by using a logistic regression. Here, the logistic regression model is a model that is usually used to predict the probability of occurrence of an event in statistics and predicts the probability of occurrence of an event by receiving various features. The prediction unit 120 according to an embodiment of the present invention uses the logistic regression model to input information using the above-described dynamic feature, static feature, and chunk as input values, and changes frequency related to how often a particular web page changes. You can estimate the degree of change associated with how much change. For example, the predictor 120 may predict the degree of change and the frequency of change using the 18 features shown in FIG. 2 as input values.

Referring back to FIG. 1, the change history builder 122 builds a change history of a web page. As shown in FIG. 3, the extractor 1220, the hash operator 1222, and the hamming distance calculator ( 1224 and a change history generator 1226. Here, the change history means a process of changing from a previous web page to a current web page of the web page.

The extractor 1220 extracts at least one of information about an image URL, anchor text, and body text from a web page. For example, the extracting unit may crawl a web page using a web robot, and then extract information such as an image URL, anchor text, and text through the crawled document through an HTML parser. Here, the anchor text is represented by underlined words or phrases on the web, and when the anchor text is clicked, it refers to text that is linked and moved to another screen or page.

The hash calculator 1222 calculates a hash value using at least one of information about an image URL, an anchor text, and a body extracted from the extractor 1220. In one embodiment of the present invention, the hash value may be a deep hash value (SIMHASH) value, which is only one embodiment of the present invention, and other hash values may be used.

Here, the SIMHASH value is a function value of the hash function using the algorithm introduced in the paper "Moses S. Charikar, Similarity estimation techniques from rounding algorithms, 2002", and serves as a fingerprint of the web page. In addition, the deep hash value is a hexadecimal value composed of 8 bytes, that is, 64 bits, and can be regarded as a value obtained by compressing the contents of the web page.

The hamming distance calculator 1224 calculates a hamming distance by using the hash value of the web page calculated by the hash calculator 1222. Here, the hamming distance means the number of different bits in two bit strings. This is to determine how much the web page has changed by using the hamming distance by comparing the past and present deep hash values serving as fingerprints of the web page.

For example, if you have a web page named A, the web page crawled a month ago was "1083c8b0e2e33dd8", and a month later, the web page crawled was still "1083c8b0e2e33dd8". No change has occurred. On the other hand, if the deep hash value checked after one month is equal to "1083c8b1f2e335d0", this may mean that the web page has changed in a month.

An example in which the hamming distance is calculated by using the deep hash value by the hamming distance calculator 1224 is illustrated in FIG. 4. As shown, since the value of the deep hash 1 and the value of the deep hash 2 differ from each other by three bits, it can be seen that the Hamming distance is "3". In this case, since the deep hash value is composed of 64 bits, the hamming distance has an integer from "0" to "64", and the larger the hamming distance, the more the two deep hash values are different. Means a lot to fly.

Referring back to FIG. 3, the change history generator 1226 generates at least one of a mean, standard deviation, weighted average, and zero hamming distance calculated by the hamming distance calculator 1224. Here, the average and standard deviation of the hamming distance may be generated using a commonly used average and standard deviation calculation method.

On the other hand, the weighted average of the hamming distance means the average value normalized by dividing the hamming distance by the elapsed time after obtaining the elapsed time using the time from the time of visiting the web page to the current visit in the past. Therefore, even if the Hamming distance is the same, if the elapsed time is short, the weighted average of the Hamming distance may have a higher value.

In addition, the rate of zero hamming distance may be obtained as a ratio of the number of times the hamming distance is zero to the total number of visits of the web page. In other words, since the hamming distance of 0 means that the web page has not changed, the ratio of the hamming distance of 0 can be obtained by dividing the number of times that the web page has not changed by the total number of visits.

Meanwhile, the change history generator 1226 according to an embodiment of the present invention maintains and manages the visit history of the web page from the current time point to the past N times for the web page, thereby averaging the hamming distance as the change history of the web page. , Standard deviation, weighted mean, and ratio with zero Hamming distance.

Referring back to FIG. 1, the regression model prediction unit 124 predicts the degree of change and the frequency of change using the change history established by the logistic regression model and the change history builder 122. The logistic regression model uses the sampled data to obtain the regression coefficient that shows the best performance using the degree of change and the frequency of change of the corresponding web page. same.

는 로지스틱 회귀 모형의 입력값으로, 도 2에 도시된 18개의 피쳐 값이 입력값이 될 수 있으며, 동적 피쳐로 해밍 거리의 평균, 표준편차, 가중 평균 및 해밍 거리가 0인 비율 등의 정보 등도 입력 값이 될 수 있다. 또한,

는 회귀 계수로서 회귀 모형 예측부(124)에 의해 가장 좋은 성능을 보이는 회귀 계수가 산출된다.here,

Is an input value of the logistic regression model, and the 18 feature values shown in FIG. Can be an input value. Also,

Is a regression coefficient, and a regression coefficient having the best performance is calculated by the regression model predictor 124.

An example in which the change frequency is predicted by the regression model predictor 124 is illustrated in FIG. 5. Here, the change frequency of the web page indicates how often the web page changes. FIG. 5 (a) shows a distribution map after the change frequency is calculated using the number of times the web page that has changed in a certain period of time for the selected sample data is actually changed. At this time, the value of the X axis "1.0" means that the content of the web page has changed every day for a certain period, the value of "X" of the X axis means that it has not changed at any time. FIG. 5B shows a distribution diagram of the change frequency actually predicted by the regression model predictor 124. Thus, as shown, it can be seen that FIG. 5 (b) which is a distribution of change frequency predicted through the prediction model is very similar to FIG. 5 (a) which is a distribution of actual values.

An example in which the degree of change is predicted by the regression model predictor 124 is illustrated in FIG. 6. Here, the degree of change of the web page indicates how much the web page changes. FIG. 6A illustrates a distribution diagram of the degree of change over a period of time with respect to the selected sample data, while FIG. 6B illustrates a distribution diagram of the degree of change actually predicted by the regression model predictor 124. . Thus, as shown in Figure 6 (b) which is a distribution of the degree of change predicted through the prediction model it can be seen that very similar to Figure 6 (a) of the distribution of the actual value. Meanwhile, looking at (a) and (b) of FIG. 6, although the web pages are actually changed a lot, the changes are mostly small. This means that even if the web page changes, the change is insignificant, so the effect of re-visit is almost insignificant. Thus, if there is a web page with the same change frequency, it is more effective to revisit the web page with the larger change.

Referring back to FIG. 1, the priority determiner 140 determines the re-visiting priority of the web page using at least one of the degree of change and the frequency of change predicted by the predictor 120. It includes a ranking unit 142 and the second priority unit 144.

The first priority determiner 142 normalizes the change frequency predicted by the predictor 120 and then determines the priority. In other words, the higher the frequency of change, the higher priority of the first priority determining unit 142 may be higher. However, if the frequency of change is too high, the first priority determining unit 142 may change the web page even if the frequency of change is too high. Normalization is performed on the predicted frequency of change to take into account the ineffectiveness.

An example of determining the priority by the first priority determining unit 142 is expressed by Equation 2 below.

"에 대해 정규화를 수행하기 위해 "

"를 곱한 것을 알 수 있다. 한편, 수학식 2에서는 변경 빈도 정도 및 정규화한 변경 빈도를 가중합하여 우선순위를 결정하고 있으나, 본 발명은 이에 한정되지 않으며, 변경 정도 및 변경 빈도를 이용한 다양한 방법에 의해 우선순위가 결정될 수 있다.In Equation 2, the predicted change frequency is "

To perform normalization on "

In addition, in Equation 2, the priority is determined by weighting the degree of change and the normalized change, but the present invention is not limited thereto. Priority may be determined by.

The second priority determining unit 144 determines the priority by using the elapsed time since the last visit to the web page. When the priority is determined in the same manner as the first priority determiner 142, sometimes a specific web page may not have a chance of revisiting over time due to a low priority. Therefore, in order to prevent this, the second priority determining unit 144 determines the priority by using the elapsed time from the last visit to the web page along with the frequency of change and the degree of change.

An example of determining the priority by the second priority determining unit 144 may be expressed by Equation 3 below.

"을 이용하고 있는 것을 알 수 있다. 한편, 수학식 3에서는 변경 빈도 정도, 정규화한 변경 빈도 및 경과 시간을 가중합하여 우선순위를 결정하고 있으나, 본 발명은 이에 한정되지 않으며, 변경 정도, 변경 빈도 및 경과 시간을 이용한 다양한 방법에 의해 우선순위가 결정될 수 있다.In Equation 3, the elapsed time since the last visit to the web page "

In the equation 3, the priority is determined by weighting the degree of change frequency, normalized change frequency, and elapsed time, but the present invention is not limited thereto. And priority may be determined by various methods using elapsed time.

The index updater 160 revisits the webpage according to the priority determined by the priority determiner 140 to update the index of the webpage. In other words, the index updater 160 re-visits the corresponding web page for each revisit priority of each web page to perform an index update operation.

Hereinafter, a method of updating a web page index according to an embodiment of the present invention will be described with reference to FIG. 7.

First, at least one of a change degree and a change frequency of a web page is predicted (S710). In one embodiment, the degree of change and the frequency of change are at least one of a dynamic feature including information related to a change history of a web page or a static feature comprising information extracted from the web page. It can be predicted using one.

Here, the static feature may include at least one of information regarding the length of the body of the web page and the number of anchor texts, and the dynamic feature may include at least one of information using a hamming distance with respect to a hash value of the web page. It may include.

Here, the Hamming distance is the number of different characters at the same position in two strings of the same length. Therefore, a large hamming distance between the hash value of the past web page and the present web page may mean that the difference between the two hash values is large, that is, the difference between the past web page and the current web page may be large. have.

In other embodiments, the degree of change and the frequency of change may be predicted using chunks extracted from the web page. First, let's look at chunks. To crawl a web page, you can go through a parser, such as an HTML parser, to get the contents of that web page. In this case, the chunk may be extracted from the body of the web page, where chunk refers to each chunk when the body is divided into at least one chunk.

Meanwhile, in the web page index update method according to an embodiment of the present invention, the degree of change and the frequency of change may be predicted using the shingle method using the above-described chunk. Here, the phrase means a set of consecutive chunks appearing in the document.

For example, the string "a rose is a rose is a rose" may be divided into "a, rose, is, a, rose, is, a, rose". Here, assuming that a shingle consists of four words in chunks, the given string is "(a, roes, is, a), (rose, is, a, rose), (is, a, rose, is), (a , rose, is, a), (rose, is, a, rose) ", and if you remove duplicates," (a, roes, is, a), (rose, is, a The words "rose," (is, a, rose, is) "will be the final result.

In this case, assuming that there are two documents, the similarity of the two documents may be calculated as the ratio of the same number of articles to the total number of articles in which the duplicate articles of the two documents are removed. For example, assuming that each document has 10 articles and the same article has 5 articles, the similarity of the two documents may be calculated as "5/15".

Therefore, the web page index updating method according to an embodiment of the present invention extracts the chunks consecutively, for example, by a predetermined word from the past web page and the current web page, and uses the extracted chunks to apply the same chunks to the total number of words. By calculating the percentage of shingles, you can predict how often and how often a web page will change.

In this case, the degree of change and the frequency of change may be predicted using a logistic regression model using at least one of the above-described information using the dynamic feature, the static feature, and the chunk. Here, the logistic regression model is a model that is usually used to predict the probability of occurrence of an event in statistics and predicts the probability of occurrence of an event by receiving various features. The web page index updating method according to an embodiment of the present invention uses a logistic regression model to input information using the above-described dynamic feature, static feature, and chunk as input values, and a change frequency related to how often a particular web page changes. You can estimate the degree of change associated with and how much changes will occur.

In detail, a method of estimating the degree of change and the frequency of change will be described. The change history of the web page is constructed, and the degree of change and the frequency of change are predicted using the change history and the logistic regression model. Here, the change history means a process of changing from a previous web page to a current web page of the web page.

In one embodiment, the change history may be constructed using a hash value calculated using at least one of information about an image URL extracted from a web page, anchor text, and body text. In one embodiment of the present invention, the hash value may be a deep hash value (SIMHASH) value, which is only one embodiment of the present invention, and other hash values may be used.

Here, the SIMHASH value is a function value of the hash function using the algorithm introduced in the paper "Moses S. Charikar, Similarity estimation techniques from rounding algorithms, 2002", and serves as a fingerprint of the web page. In addition, the deep hash value is a hexadecimal value composed of 8 bytes, that is, 64 bits, and can be regarded as a value obtained by compressing the contents of the web page.

In other words, the web page index update method according to an embodiment of the present invention after crawling a web page using a web robot, and extracts information such as image URL, anchor text, body, etc. through the crawled document through an HTML parser will be. Here, the anchor text is represented by underlined words or phrases on the web, and when the anchor text is clicked, it refers to text that is linked and moved to another screen or page.

In one embodiment, the method of building a change history calculates a hamming distance using a hash value of a web page, and generates at least one of a mean, standard deviation, weighted average, and zero hamming distance of the hamming distance. .

Here, the hamming distance means the number of different bits in two bit strings. This is to determine how much the web page has changed by using the hamming distance by comparing the past and present deep hash values serving as fingerprints of the web page.

In addition, the mean and standard deviation of the Hamming distance can be generated using a commonly used mean and standard deviation calculation method. On the other hand, the weighted average of the hamming distance means the average value normalized by dividing the hamming distance by the elapsed time after obtaining the elapsed time using the time from the time of visiting the web page to the current visit in the past. Therefore, even if the Hamming distance is the same, if the elapsed time is short, the weighted average of the Hamming distance may have a higher value.

The ratio of the hamming distance of zero may be obtained as the ratio of the number of times of the hamming distance of the total number of visits of the web page. In other words, since the hamming distance of 0 means that the web page has not changed, the ratio of the hamming distance of 0 can be obtained by dividing the number of times that the web page has not changed by the total number of visits.

Meanwhile, the web page index updating method according to an embodiment of the present invention maintains and manages the visit history from the current time point to the past N times for the web page, thereby changing the average and standard of the hamming distance as the change history of the web page. Ratios with zero deviation, weighted average, and hamming distance, and the like.

As a result of the prediction, the revisit priority of the web page is determined using at least one of the predicted change degree and the change frequency (S720). In one embodiment, the method of determining priority may be prioritized after performing normalization on the predicted frequency of change. In other words, in the web page index updating method according to an embodiment of the present invention, the higher the frequency of change, the higher the priority of re-visiting the web page, but if the change frequency is too high, the web page continues to change even if the re-visit is too high. In order to consider the effect of revisit, the normalization of the predicted change frequency is performed.

In another embodiment, the method of determining the priority may determine the priority using the elapsed time since the last visit of the web page. In other words, the web page index updating method according to an embodiment of the present invention sometimes changes the frequency of frequent changes in order to prevent a certain web page from failing to have a chance to revisit over time due to a low priority. And elapsed time from the last visit to the web page along with the degree of change.

Finally, the web page is re-visited according to the determined priority to update the index of the web page (S730). In other words, the web page index update method according to an embodiment of the present invention is to re-visit the corresponding web page for each revisit priority of each web page to perform an index update operation.

On the other hand, the above-described web page index updating method is implemented in the form of program instructions that can be executed by various computer means can be recorded in a computer-readable recording medium. In this case, the computer-readable recording medium may include program instructions, data files, data structures, and the like, alone or in combination. On the other hand, the program instructions recorded on the recording medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software.

The computer-readable recording medium includes a magnetic recording medium such as a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical medium such as a CD-ROM and a DVD, a magnetic disk such as a floppy disk, A magneto-optical media, and a hardware device specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. The recording medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, or the like.

In addition, program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

On the other hand, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without changing the technical spirit or essential features.

Therefore, it is to be understood that the embodiments described above are exemplary in all respects and not restrictive. The scope of the present invention is shown by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

1 is a schematic block diagram of a web page index update system according to an embodiment of the present invention.

2 is a diagram illustrating examples of dynamic and static features.

3 is a block diagram illustrating a detailed configuration of the change history construction unit shown in FIG. 1.

4 is a diagram illustrating an example in which a hamming distance is calculated using a deep hash value.

5 is a diagram illustrating an example in which a change frequency is predicted.

6 is a diagram illustrating an example in which a degree of change is predicted.

7 is a flowchart showing an index update method according to an embodiment of the present invention.

<Explanation of symbols for the main parts of the drawings>

100: index update system 120: prediction unit

122: change history construction unit 124: regression model prediction unit

140: priority determination unit 142: first priority determination unit

144: second priority determining unit 160: index updating unit

Claims (25)

The degree of change and the frequency of change of the web page using at least one of a dynamic feature including information related to the change history of the web page or a static feature including information extracted from the web page. Predicting at least one; And Determining a revisit priority of the web page using at least one of the predicted change rate and the change frequency, The dynamic feature And at least one of information using a hamming distance of a hash value of the web page. delete The method of claim 1, wherein the static feature And at least one of information on the length of the body of the web page and the number of anchor texts. delete The method of claim 1, wherein the degree of change and the frequency of change Web page index updating method, characterized in that predicted by using the chunk extracted from the web page. The method of claim 1, And the change degree and the change frequency are predicted using a logistic regression. The method of claim 1, wherein the predicting the degree of change and the frequency of change Building a change history of the web page; And Predicting the degree of change and the frequency of change using the change history and logistic regression model. Building a change history of the web page; Predicting at least one of a change degree and a change frequency of the web page by using the change history and a logistic regression model; And Determining a revisit priority of the web page using at least one of the predicted change rate and the change frequency, The change history is And a hash value calculated using at least one of information about an image URL, anchor text, and body text extracted from the web page. Calculating a hamming distance using the hash value of the web page; Generating a change history of the web page by generating at least one of the average, standard deviation, weighted average of the Hamming distance, and the ratio of the Hamming distance being 0; Predicting at least one of a change degree and a change frequency of the web page by using the change history and a logistic regression model; And Determining a revisit priority of the web page using at least one of the predicted change rate and the change frequency. Constructing a change history of the web page by using a deep hash value of the web page; Predicting at least one of a change degree and a change frequency of the web page by using the change history and a logistic regression model; And Determining a revisit priority of the web page using at least one of the predicted change rate and the change frequency. Predicting at least one of a change degree and a change frequency of the web page; And Determining a revisit priority of the web page using at least one of the predicted change rate and the change frequency. Including, And if the predicted change frequency is used, determining the priority, after normalizing the predicted change frequency, determining the priority. The method of claim 1, wherein determining the priority is And determining the priority by using the elapsed time since the last visit of the web page. The method of claim 1, Re-visiting the web page according to the priority to update the index of the web page. A recording medium having recorded thereon a program for performing the method according to any one of claims 1, 3 and 5-7. The degree of change and the frequency of change of the web page using at least one of a dynamic feature including information related to the change history of the web page or a static feature including information extracted from the web page. A prediction unit for predicting at least one; And And a priority determiner configured to determine revisiting priority of the web page using at least one of the predicted change degree and the change frequency. The dynamic feature, Web page index update system, characterized in that it comprises at least one of the information using the Hamming Distance (Hamming Distance) for the hash value of the web page. delete The method of claim 15, The static feature includes at least one of information about the length of the body of the web page and the number of anchor texts, And wherein the dynamic feature comprises at least one of information using a Hamming Distance for a hash value of the web page. The method of claim 15, wherein the prediction unit And predicting the degree of change and the frequency of change using the chunk extracted from the web page. The method of claim 15, wherein the prediction unit The web page index updating system, characterized in that for predicting the degree of change and the frequency of change using a logistic regression (Logistic Regression). The method of claim 15, wherein the prediction unit A change history building unit for building a change history of the web page; And And a regression model predictor for predicting the degree of change and the frequency of change using the change history and the logistic regression model. Extracts at least one of information about an image URL, anchor text, and body text from a web page, and calculates the deep sea calculated using at least one of the extracted image URL, anchor text, and body text; A change history building unit for building a change history of a web page based on a SIMHASH value; A regression model predictor for predicting at least one of a change degree and a change frequency of the web page by using the change history and a logistic regression model; And And a priority determiner configured to determine revisiting priority of the web page using at least one of the predicted change degree and the change frequency. A change history for calculating a hamming distance using a deep hash value of a web page, and generating at least one of a mean, standard deviation, weighted average, and a ratio of the hamming distance of 0 to build a change history of the web page. Construction unit; And A regression model predictor for predicting at least one of the degree of change and the frequency of change using the change history and the logistic regression model; And And a priority determiner configured to determine revisiting priority of the web page using at least one of the predicted change degree and the change frequency. A prediction unit for predicting at least one of a change degree and a change frequency of the web page; And Priority determining unit for determining the re-visit priority of the web page using at least one of the predicted change degree and the change frequency Including, The priority determining unit And when the predicted change frequency is used, normalizing the predicted change frequency and then determining the priority. The method of claim 15, wherein the priority determiner And a second priority determiner configured to determine the priority by using an elapsed time since the last visit of the web page. The method of claim 15, And an index updater for re-visiting the web page according to the priority to update the index of the web page.

Images