JP2010511246A - Search log misuse prevention method and apparatus - Google Patents

Abstract

  The present invention discloses a method for preventing misuse of search logs. A method for preventing abuse of a search log according to an embodiment of the present invention includes a step of selecting a target to be inspected for abnormal behavior from the search log, and scoring the degree of deviation from normal for the selected target to detect abnormal behavior. Stages. The search log abuse prevention method may further include a step of removing the detected abnormal action from the search log using a predetermined deduction logic and correcting the search log. Therefore, by effectively removing abnormal acts detected from the search log, misuse of the search log can be prevented and the search log can be kept clean.

Description

  The present invention relates to Internet search, and more particularly, to a method and apparatus for effectively preventing abuse of search logs (Abusing).

  In recent years, with the development of the Internet, various services using the Internet have been provided to users, the most representative of which is a search service. The search service means that when a user inputs a search word in a search window of a search site provided by the search service provider, the search service provider provides information corresponding to the input search word as a search result. .

  In this way, the search terms entered by each user to use the search service and the information about each user's search behavior are stored in the form of a search log, and the search service provider analyzes the search log by analyzing the search log. Can provide a variety of search services.

For example, for keyword advertisements, charging is determined based on keyword popularity. Here, popularity is determined based on the aspect of the search term acquired through search log analysis. Based on such popularity, the search service provider provides a neutral and legitimate charging basis for the advertisement requester. In addition, search service providers provide various primary and secondary services using search logs. For example, popular search terms, related search term services, and the like use search logs to present search terms that are currently receiving user interest and related search terms. This service was successful because the vast search log met the premise that it was a product of the pure intention of Internet users.

  However, recently, attempts to add distortion to the search log so as to reflect the unintended intentions of specific individuals and specific groups are increasing. It is estimated that these specific gravity will continue to increase. Such an abusing act of search logs has a problem in that it contaminates the search logs, leading to a decline in the reliability of the profit model dependent on the search logs and a decrease in service quality.

  The present invention is intended to solve the above-described problems, and an object of the present invention is to detect an abnormal action by tracking and analyzing a search log, and to prevent a search log abuse for removing a contaminated part. It is to provide such a device.

  In order to achieve the above object, a search log misuse prevention method according to an aspect of the present invention includes a step of selecting a target to be inspected for abnormal acts from a search log, and scoring the degree of deviation from normal for the selected target. , Including detecting abnormal behavior. In one embodiment, the search log abuse prevention method may further include correcting the search log by removing the detected abnormal behavior from the search log using a predetermined deduction logic.

  The inspection object selection step includes: search word summary information obtained by statistically analyzing the number of times of input of each specific search word included in a predetermined time window from the search log; and the number of times each search word is input in the specific IP. Generating at least one of statistically analyzed IP summary information, and detecting the abnormal behavior from at least one of the search word summary information and the IP summary information in the abnormal behavior detection step. Here, IP means the Internet Protocol.

  Here, in the summary information generation step, at least one of an input frequency vector for each IP of a specific search word and an input frequency vector for each search word in a specific IP included within a predetermined time window from the search log is obtained. Generating the search term summary information, reducing the dimension of the input frequency vector for each IP of the specific search term to generate the search term summary information, or generating the IP summary information for each search term in the specific IP Including reducing the dimension of the input count vector.

  Meanwhile, in the dimension reduction step of the input count vector, a limited number of input count vectors for each IP of the specific search word and input count vectors of each search term in the specific IP using a hashed bucket. Convert to a vector of times for a bucket of.

  At this time, the search word summary information and the IP summary information are modeled into a multi-dimensional distribution using a statistical method.

  On the other hand, the abnormal action detection step includes calculating the degree of abnormality as a score by the degree away from the center with respect to at least one of the search word summary information and the IP summary information modeled in the multidimensional distribution, Determining that an abnormal action is included in at least one of the search word summary information and the IP summary information whose calculated score is equal to or greater than a predetermined reference value, wherein the abnormal action detecting step includes The method may further include compressing data by reducing at least one dimension of the modeled search word summary information and the IP summary information before the calculation step.

  Further, the calculation step calculates a score for the degree of abnormality as a ratio with respect to a predetermined reference value using a statistical value modeled through a sum of samples of a standard normal distribution independent of each other in a reduced dimension.

  In the correction step, the abnormal action is removed from at least one of the search word summary information and the IP summary information in which the abnormal action is detected, using a deduction logic applying an information theory for measuring a distribution difference. Including stages.

  A search log abuse prevention device according to another aspect of the present invention for achieving the above-described object is a pre-processing unit that selects a target to be inspected for abnormal behavior from the search log, and the selected target is not normal. An abnormal action detection unit for scoring the degree and detecting an abnormal action; and removing the detected abnormal action from the search log using a predetermined deduction logic, and correcting the search log; including.

1 is a schematic block diagram of a search log abuse prevention device according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating a search log misuse prevention method according to an embodiment of the present invention. It is a flowchart which shows the detail of the test object selection process by one Example of this invention. 4 is a flowchart illustrating details of an abnormal action detection process according to an exemplary embodiment of the present invention. It is a reference figure for demonstrating the statistical method used in an abnormal action detection process. 5 is a flowchart illustrating details of a search log correction process according to an exemplary embodiment of the present invention. FIG. 6 is a diagram illustrating a deduction logic according to an embodiment of the present invention used in a search log correction process. FIG. 6 is a diagram illustrating a user interface screen according to an exemplary embodiment of the present invention. It is a figure which shows the performance experiment result of the search log abuse prevention apparatus by one Example of this invention. It is a figure which shows the performance experiment result of the search log abuse prevention apparatus by one Example of this invention. It is a figure which shows the performance experiment result of the search log abuse prevention apparatus by one Example of this invention. It is a figure which shows the performance experiment result of the search log abuse prevention apparatus by one Example of this invention. It is a figure which shows the performance experiment result of the search log abuse prevention apparatus by one Example of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing the present invention, if it is determined that a specific description of a related known function or configuration will obscure the gist of the present invention, it will be omitted as appropriate. Further, terms to be described later are defined in consideration of the functions in the present invention, and these can be different depending on the user, the intention of the operator, the custom, or the like. Accordingly, each term should be defined based on the content throughout this specification.

  FIG. 1 is a schematic block diagram of a search log abuse prevention apparatus according to an embodiment of the present invention. In FIG. 1, the search log abuse prevention device includes a preprocessing unit 10, an abnormal action detection unit 20, and an abnormal action correction unit 30.

  The preprocessing unit 10 selects a target to be checked for abnormal behavior from the search log. Here, the reason for selecting the object to be inspected for abnormal action through the pre-processing unit 10 instead of inspecting the abnormal action for the entire search log is that the number of IPs for which the search word is input, the number of search words, Alternatively, considering the combination thereof, the number of search logs becomes too large.

  For this purpose, the preprocessing unit 10 first generates an IP and a search word candidate that are noted at the time of inspection, and generates an input value to be used in the inspection stage.

  The pre-processing unit 10 generates, from the search log, an input frequency vector for each specific search word included in a predetermined time window and / or an input frequency vector for each search word in the specific IP, and each generated input frequency Reduce the dimension of the vector and generate search term summary information and / or IP summary information.

  As described above, the pre-processing unit 10 performs the search word summary information obtained by statistically analyzing the number of times the specific search word is input by IP, the IP summary information obtained by statistically analyzing the number of times each search word is input in the specific IP, or these And the generated search term summary information and / or IP summary information can be modeled into a multi-dimensional distribution using statistical methods.

  On the other hand, as another embodiment of the present invention, in order to reduce the number of subjects to be inspected for abnormal acts, the “real-time rapid rise search word detection method and the actual By applying the concept of “prompt search term detection system”, it is possible to select only search terms and / or IPs that have attracted some attention as inspection targets.

  The abnormal action detection unit 20 scores the degree of deviating from the target selected by the preprocessing unit 10 and detects the abnormal action from the selected target. That is, a score technique based on statistical methodology is introduced, and a score calculation procedure for abnormal actions by IP and / or search terms is performed.

  The abnormal action detection unit 20 calculates the degree of abnormality as a score by the degree away from the center with respect to the search word summary information and / or the IP summary information modeled in a multidimensional distribution using a statistical method. Then, it is determined that the search term summary information and / or IP summary information whose calculated score is equal to or greater than a predetermined reference value includes an abnormal action. At this time, in order to increase data processing efficiency, the data can be compressed and processed by reducing the dimension of the modeled search word summary information and / or IP summary information before calculating the score.

  The abnormal action correcting unit 30 corrects the search log by removing the abnormal action detected by the abnormal action detecting unit 20 from the search log using a predetermined deduction logic. In one embodiment, the abnormal action correcting unit 30 uses a deduction logic to which information theory for measuring a distribution difference is applied, thereby causing a contaminated part from the search word summary information and / or the IP summary information in which the abnormal action is detected. Can be removed. That is, a procedure for deducting the number of abnormal action searches using deduction logic and leaving only normal actions from the search log is performed. Accordingly, it is possible to detect and treat a search word abuse (Abusing) act due to fraudulent intentions, and to maintain a clean search log.

  Hereinafter, a search log abuse prevention method according to an embodiment of the present invention will be described in detail based on the configuration of the search log abuse prevention apparatus according to an embodiment of the present invention described above.

  FIG. 2 is a flowchart illustrating a search log misuse prevention method according to an embodiment of the present invention.

  Referring to FIG. 2, in order to prevent abuse of the search log, first, a target to be checked for abnormal behavior is selected from the search log (S100). That is, in one embodiment, as an object for inspecting abnormal acts from a search log, search word summary information obtained by statistically analyzing the number of times of input of a specific search word by IP and / or the number of times of input of each search word in a specific IP are statistically analyzed. The IP summary information that was analyzed manually. At this time, the search term summary information and the IP summary information can be modeled into a multi-dimensional distribution using a statistical method.

  Next, the degree to which the selected search word summary information and / or IP summary information is not normal is scored, and an abnormal action is detected (S200). In one embodiment, the method may further include correcting the search log by removing the abnormal action detected using a predetermined deduction logic (S300).

  Next, the process of selecting the inspection object will be described more specifically with reference to FIG.

  Referring to FIG. 3, in order to select an object to be inspected for abnormal behavior, first, an input count vector for each specific search word and / or each search word in the specific IP is input from a search log within a predetermined time window. A frequency vector is generated (S110). Thereafter, the generated input frequency vector, that is, the input frequency vector for each IP of the specific search word and / or the dimension of the input frequency vector of each search word in the specific IP is reduced, and the search word summary information and / or IP summary information is reduced. Is generated (S120).

  Hereinafter, the above-described inspection object selection process will be described in more detail using specific examples. However, the following embodiment is merely an example of the inspection object selection method, and it is needless to say that various modifications are possible.

1. First stage-preprocessing stage It is necessary to generate IP summary information and search word summary information from the search log DB for a search word abuse investigation. One IP inputs a large number of search terms at a specific time. It is necessary to generate IP summary information in order to measure the degree to which the search aspect performed by this IP differs from the search aspect performed by other ordinary IPs. One search term is input from various IPs. Therefore, it is necessary to generate summary information about the IP in which the corresponding search term is input.

  However, since the number of IPs, the number of search terms, and a combination thereof are very large, it is necessary to select IPs and search terms to be inspected. Processing all of these results in memory problems.

1) Representation of input vector In order to generate IP and search word summary information, first, the following vector representation can be introduced.

If the entire IP number N _I, the entire search word count N _Q, the information for a specific IP in the predefined time window W to a specific search point, the number of times each search word a specific IP as follows is input It can be expressed by a vector of

here,

Means the number of times the k-th search term is input with the specific IP.

  Similarly, the information related to the specific search word can be expressed by a vector of the number of times the specific search word is input in each IP.

here,

Means the number of times that a specific search word is inputted with the k-th IP.

However, since the entire IP number N _I and the entire search word number N _Q is very defense large, to maintain all of the vector representation can not help hitting the memory problems.

2) Selection of inspection target IP and search terms using hash bucket On the other hand, among the search terms input by the specific IP within the specific time window W, the number of different search terms is the number N _Q of the total search terms. There are only a few in comparison. In addition, of the IP you enter a specific search terms within a specific time window W, only a small minority compared to the different IP number also the entire IP number N _I of each other. By generating summary information about a specific IP and summary information about a specific search word using such characteristics, the memory problem described above can be solved. That is, a hash bucket having a number of buckets that is much smaller than the total number of search words or the total number of IPs is used.

Bucket number D << N _I, if N _Q, summary information of a specific IP can be expressed by the count vector of hash buckets as described below.

here,

Means the number of hits of the k-th bucket for a specific IP. When there is a search word q input by the specific IP, the index k of the bucket associated with the search word q is calculated using a hash function as follows: calculate.

  Next, the count of buckets corresponding to the calculated index k is increased.

  Through this process, the IP summary information can be generated by summarizing and expressing the information about the specific IP with a vector having a length corresponding to the number of buckets D as expressed in the above equation 5. . Similarly, search term summary information can also be generated by summarizing and expressing information about a search term with a vector having a length corresponding to the number of buckets D.

Thus, the IP information and keyword information, by expressing summarized by a vector with a length of only a few D of very few buckets from whole IP number N _I and the entire search word number N _Q, the solution to memory problems be able to.

  Meanwhile, the search term summary information and the IP summary information generated through the above-described process can be modeled into a multidimensional distribution using a statistical method.

  Hereinafter, a method of scoring the degree of abnormal action based on the vector expression using the hash bucket expressed in Equation 5 above will be described with reference to FIG. 14 showing the flow of the abnormal action detection process in detail. Explained.

  Referring to FIG. 4, in order to detect anomalous behavior, data is first compressed by reducing the dimension of search term summary information and / or IP summary information modeled in a multidimensional distribution using statistical methods. (S210). In one embodiment, as a method for compressing data, a principal component analysis (Principal Component Analysis: hereinafter abbreviated as “PCA”) that maps input data to mutually orthogonal coordinate systems is used. it can.

  Subsequently, the degree of abnormality is calculated as a score according to the degree away from the center for the reduced-dimension search word summary information and / or IP summary information (S220). In one embodiment, in order to calculate the degree of anomaly, the score for the degree of anomaly is calculated as a percentage of a predetermined reference value using statistics modeled through the sum of samples of a standard normal distribution independent of each other in a reduced dimension. Can be calculated.

  Finally, it is determined that the search term summary information and / or IP summary information whose calculated score is equal to or greater than a predetermined reference value includes an abnormal action (S230). That is, search term summary information and / or IP summary information whose calculated score is greater than or equal to a reference value is detected as an abnormal action.

  Hereinafter, the abnormal action detection process described above will be described in more detail with reference to specific examples. The following embodiment is merely an example of the abnormal action detection method, and it is needless to say that various modifications are possible.

2. Second Stage-Abnormal Action Detection Stage As expressed in Equation 5 above, the IP summary information and the search word summary information are respectively the number-of-inputs information of each search word and the number-of-inputs information by IP of the specific search word in the specific IP. Can be expressed as a vector with element as.
This vector

This shows a discrete distribution,

Can be expressed as

  Here, p is a probability vector and is calculated as follows.

  Finally, in the present invention, the IP summary information and / or the search word summary information is expressed by a set of probability vectors as follows using the probability vector p.

  Hereinafter, a method for scoring the degree of deviating from the normal action with respect to the search word summary information and / or the IP summary information expressed using the probability vector p as shown in Equation 11 above will be proposed.

1) Data Compression Using Principal Component Analysis According to one embodiment of the present invention, a data compression process is performed for smoother data processing. Specifically, data is compressed by reducing the D dimension, which is the number of buckets, using PCA. That is, this method uses a discrete probability distribution representing IP summary information or search term summary information.

From this, the principal component vector that increases the variance of the mapped values is searched, which means searching for several eigenvectors that best explain the characteristics of the corresponding discrete probability distribution.

  In such a PCA method, as the principal component vector, only d principal component vectors (here, d <D) that sufficiently explain the variance of the corresponding discrete probability distribution among the total variances may be used. It is common. At this time, in the input data mapped to the d principal component vectors, there is no correlation between values mapped with different variances for each component, and each principal component vector is orthogonal. Become. Since the PCA method uses a well-known method that is already widely known, a specific description of PCA is omitted.

  Using such a PCA method, the data is obtained by reducing the dimensionality of a discrete probability distribution representing IP summary information or search word summary information in which the number of buckets is D-dimensional to a much smaller number of d-dimensions. Data processing efficiency can be increased by compressing.

  Hereinafter, a method for scoring the degree of deviation from the normal action using the d-dimensional input data subjected to principal component analysis will be described in detail.

2) Scoring method for measuring the degree of anomaly It can be seen that the input data mapped to the d-dimensional principal component vector through the PCA method described above has a different variance for each component. This means that the scaling is different for each dimension. In this case, it is possible to use a prewhitening method that scales the principal component vector so that the variance becomes 1 for each dimension so as to be useful for visualization and post-processing.
Prewhitened mapping matrix

When there is a mapping value of the input vector x to this, a d-dimensional vector

It is expressed as At this time,

When

Are uncorrelated with each other and distributed

It is.

Now, in order to score abnormal acts according to the present invention, the following assumptions are made.
1) Each

Follows the standard normal distribution N (0,1).
2) When i ≠ j,

When

Are independent of each other.

  Here, in general, the absence of correlation does not mean that they are independent of each other, but the present invention uses strong assumptions to increase the efficiency of data processing.

  Under such assumptions, the following statistical values can be defined.

In general, in statistics, the chi-square distribution with d degrees of freedom

Is modeled through the sum of d independent samples of a standard normal distribution. Therefore, under the assumption of Equation 21 above, the statistical value

Can be considered to follow a chi-square distribution with d degrees of freedom.
In the following, for the critical value α,

The smallest s value that satisfies

It is defined as here

Represents a cumulative probability distribution value up to the s boundary, and α is an error level or a significance level, and is preferably 0.05 or 0.01. After all,

Means the maximum boundary of the normal range that does not exceed the critical value α.

All beyond

Can be considered to be included in the range of anomalies.

  Therefore, in the present invention, in order to score the degree of deviating from the normal action, the abuse score is defined by the following equation.

That is,

The greater the value of, the higher the probability vector

Has a small value and the probability vector p is smaller than the critical value α. After all, this provides a basis for determining that it is extremely rare under the given assumptions. In other words, if the score value defined by Equation 30 above is greater than 1, it is a rare case that falls outside the normal range, and this can be determined as an abnormal action.
The statistical method described above will be specifically described with reference to the example of FIG. FIG. 5 shows an example of a chi-square distribution with one degree of freedom.

Means the maximum boundary 902 of the normal range in the chi-square distribution when the critical value representing the error level or significance level is α.

All beyond

Can be considered to be included in the abnormal range.
That is, cumulative probability distribution from probability vector 1 to normal boundary range

The area 904 obtained by subtracting the ”means an abnormal area, and all the areas included in this area 904 are

Can be considered to be included in the abnormal range.

  Next, a search log correction process according to an embodiment of the present invention will be described in detail with reference to FIG.

  Referring to FIG. 6, in order to correct a search log, a contaminated portion is detected from search word summary information and / or IP summary information detected as anomalous activity using a deduction logic applying information theory for measuring a distribution difference. It is removed (S310).

  At this time, the deduction logic for correcting the search log is the difference in distribution between the probability model of the population and the probability model of the search word summary information and / or IP summary information in which the abnormal action is detected. An abnormal action can be removed using a KL distance (Kullback-Leibler Distance).

  Hereinafter, the search log correction process will be described in detail with reference to a specific embodiment. The following embodiment is merely an example of a search log correction method, and it goes without saying that various modifications are possible.

3. Third Stage—Search Log Correction Stage 1) Means for Measuring Distribution Difference—KL Distance As described above, the deduction logic used for search log correction according to one embodiment of the present invention detects abnormal actions. The KL distance is used as a means for measuring the distribution difference between the probability model of the search word summary information and / or the IP summary information and the probability model of the population.

  Such KL distance is based on Information Theory (Cover and Thomas (1991)). For example, if there are two distributions p and q, the KL distance between these two distributions is It can be obtained as follows.

  Therefore, the KL distance has a value of 0 when both distributions are the same.

2) Deduction logic For convenience, the N pieces of data used to construct the model are used as a population, and this is represented by an NXD matrix M. In this case, I-th row m _i of M is the vector saved the number of hash buckets. The matrix M is normalized on the basis of the rows to obtain a discrete probability model m.

  If the hash bucket vector of the abnormal pattern is h and its discrete probability model is p, the KL distance between the discrete probability distribution m of the population and the discrete probability distribution p to be examined is calculated as follows.

When the value of the specific element is reduced by the hash bucket vector h using the above equation 40, the difference between the modified discrete probability model and the population discrete probability model can be reduced.
Specifically, a hash bucket i

The larger the value, the greater the KL distance between the two distributions, which makes the distribution p abnormal. Therefore, in order to keep the search log clean by removing abnormal behavior, when the critical value is β,

The hash bucket is a proofreading candidate to apply the deduction logic.

On the other hand, FIG. 7 is a diagram for explaining a deduction logic used in the search log correction process according to an embodiment of the present invention.
FIG. 7 shows the overall deduction logic. Here, the “find ()” function is a function that retrieves the index of the element that satisfies the condition in (). The “ceil ()” function is a function that retrieves the smallest integer larger than the factor in (). The operator “. *” Performs multiplication between elements of a vector. “Score” means the number of abuse points defined in Equation 30 above.

Is the number of search term input, p is

, Β is a critical value for selecting a calibration candidate, and f is a KL distance between the discrete probability distribution m of the population and the discrete probability distribution p to be examined.

  The overall deduction logic will be described. First, the probability of obtaining a probability function is obtained by normalizing the number of times of input of a specific search word per IP or the number of times of input of each search word in the specific IP, and the KL distance based on the difference from the population discrete probability model. Is calculated (904). An index i in which the obtained KL distance is larger than the critical value β is obtained. The obtained index means a search term or IP including an abnormal action. The number of searches corresponding to the obtained index is decreased (906), and the critical value β is adjusted.

  The above deduction logic repeats until there is no candidate that belongs to the normal range of score <1 or exceeds the critical value β. In particular, the reason why β is increased at each iteration is that the core abnormal deductions are already made at the beginning of the iteration, so that the deduction criteria are made more strict at the next iteration.

  FIG. 8 is a user interface screen provided by the search log abuse prevention apparatus according to an embodiment of the present invention.

  Referring to FIG. 8, the search word list and IP list selected as search targets are displayed in the left window, and the count number to be deducted is displayed in the intermediate window according to the abuse points calculated for the degree of abnormality.

  On the other hand, FIGS. 9 to 13 are diagrams showing experimental results by the search log abuse prevention method according to one embodiment of the present invention.

  In order to confirm the performance of the search log misuse prevention method according to one embodiment of the present invention, the result at around 12:30 on July 7, 2006 is examined. In this experiment, the time window W was set to 1 hour, and the number D of hash buckets was set to 32. The critical values α = 0.01, β = log (1.8), and scale = log (1.3) were set.

  After constructing a model from the search word summary information and performing the above-described abuse test on the test candidate set, the top 20 search words with the highest abuse score are shown in FIG. This is a discrete probability model of each sample expressed in the form of a histogram. The vertical axis is a probability value, and the scale of the axis is fixed at [0, 1]. The horizontal axis represents the hash bucket index. At the top of the picture, the name of the search term and the score of abuse were recorded. Since the top 20 search terms all have an abuse score of about 3 to 9, and all are 1 or more, it is predicted that abnormal acts were included.

  On the other hand, FIG. 10 shows an example of a deduction processing result for the top 20 abuse search words detected by the present invention. Looking at each row, the original hash bucket before the deduction and the hash bucket reflecting the deduction logic are shown as a pair. Regarding the score, after the deduction, the abuse score is less than 1, and it can be confirmed that the abnormal action has been removed.

  On the other hand, FIG. 11 shows a result of comparing the deduction processing result and the discrete probability distribution value according to the present invention. It can be confirmed that the abuse score of the search term that was abnormal with an abuse score of about 3 to 9 was 1 or less and was corrected to the normal range. For example, in the case of the search word “type”, the misuse score which was 9.673833 was about 0.211166 after the deduction processing procedure for removing the abnormal action, and it can be seen that it was corrected within the normal range. For convenience, the vertical axis is scaled to [0, 0.1] after deduction.

  A population probability model used as a reference for calculating the KL distance by the deduction logic is shown in FIG.

  When comparing the score before and after deduction, it can be said that the abuse score of the search word that was abnormal returned to the normal level by removing the abnormal action through the deduction logic.

  FIG. 13 shows an example of deduction processing results for the top 40 search terms. The number of abuse points is written on the left, and the total number of searches before deduction and the number of deductions calculated by the deduction logic are displayed. That is, by subtracting the number of deductions calculated by the deduction logic from the total number of searches, it is understood that the contaminated portion can be removed from the search term detected as an abnormal action and the search log can be corrected.

  The method for reducing the number of searches for a search term determined to be abnormal using the search term summary information and maintaining only normal action information on the search log has been described above. Most of the search term abuse problems can be sufficiently solved only by the abuse detection and treatment method using the search term summary information as described above. However, although the abuse word score is less than 1 in the search word summary information, it is determined as a normal action. In such a case, misuse can be further corrected using the IP summary information. Since such a method is substantially the same as the abuse detection and treatment method for the search word summary information, a detailed description thereof will be omitted.

The present invention has proposed a search log abuse prevention method and apparatus for maintaining a clean search log through diagnosis and post-processing of search abuse in the search log. That is, a hash bucket-based data structure was constructed to represent IP summary information and / or search term summary information, and this was converted to a discrete probability model to represent input data. A technique that can detect the sample was proposed. The input data is transferred to the space of principal component vectors orthogonal to each other by the PCA method, where a statistically based scoring technique for measuring the degree away from the center is presented.

  Finally, a deduction technique is proposed to convert abnormal samples to normal samples based on information theory.

  On the other hand, the above-described method for preventing the abuse of the search log can be created as a computer program. Codes and code segments constituting the program can be easily inferred by a computer programmer in the field. Further, the program is stored in a computer readable information storage medium (computer readable media), and is executed by being read and executed by a computer, thereby embodying a search log misuse prevention method. The information storage medium includes a magnetic recording medium, an optical recording medium, and a carrier wave medium.

  The present invention has been described with reference to preferred embodiments thereof. Those skilled in the art to which the present invention pertains can understand that the present invention can be embodied in a modified form without departing from the essential characteristics of the present invention. Accordingly, the disclosed embodiments should be considered in an illustrative, not a limiting sense. The scope of the present invention appears not in the above description but in the claims, and any difference within the scope equivalent to the scope of the present invention should be construed as being included in the present invention.

  As described above, according to the present invention, an abusive search word including an abnormal action is efficiently detected from the search log, and only an effective search number among the search times of the search word determined to be an abuse is left, and the abnormal action is detected. By removing it, it is possible to prevent abuse of the search log and keep the search log clean.

Claims (23)

In a method to prevent misuse of search logs,
Selecting an object to be inspected for abnormal behavior from the search log;
Scoring the degree of deviation from normal with respect to the selected object, and detecting abnormal behavior;
A search log misuse prevention method, comprising:   The method according to claim 1, further comprising correcting the search log by removing the detected abnormal action from the search log using a predetermined deduction logic. . The inspection object selection step includes:
Search term summary information obtained by statistically analyzing the number of times of input of each specific search word included in a predetermined time window from the search log, and IP summary obtained by statistically analyzing the number of times of input of each search word in the specific IP. Generating at least one of the information,
The method of claim 2, wherein the abnormal action is detected from at least one of the search word summary information and the IP summary information in the abnormal action detection step. The summary information generation step includes:
Generating at least one of an input frequency vector for each IP of a specific search word and an input frequency vector for each search word in a specific IP included within a predetermined time window from the search log;
In order to generate the search word summary information, the dimension of the input frequency vector for each IP of the specific search word is reduced, or the dimension of the input frequency vector of each search word in the specific IP to generate the IP summary information The search log misuse prevention method according to claim 3, further comprising a step of reducing the search log. The dimension reduction step of the input frequency vector includes:
Converting the IP input frequency vector of each specific search word and the input frequency vector of each search word in the specific IP into a frequency vector for a limited number of buckets using a hashed bucket; The search log misuse prevention method according to claim 4.   The method of claim 3, wherein the search word summary information and the IP summary information are modeled into a multidimensional distribution using a statistical method. The abnormal action detection step includes:
Calculating the degree of abnormality as a score according to the degree away from the center with respect to at least one of the search word summary information and the IP summary information modeled in the multi-dimensional distribution;
Determining that at least one of the search word summary information and the IP summary information whose calculated score is equal to or greater than a predetermined reference value includes an abnormal action;
The search log misuse prevention method according to claim 6, further comprising: The abnormal action detection step includes:
The search log of claim 7, further comprising reducing at least one dimension of the modeled search word summary information and IP summary information and compressing data before the calculation step. How to prevent abuse. The data compression step includes
9. The search log misuse prevention method according to claim 8, wherein the method is performed using a principal component analysis method for mapping input data to mutually orthogonal coordinate systems. The calculation step includes:
The score for the degree of abnormality is calculated as a ratio to a predetermined reference value using a statistical value modeled through a sum of samples of the standard normal distribution independent of each other in the reduced dimension. Search log abuse prevention method. The score for the degree of abnormality is expressed by the equation

Where statistic

Follows a chi-square distribution with degrees of freedom d modeled through the sum of samples of standard normal distributions independent of each other,

11. The search log abuse prevention method according to claim 10, wherein represents a maximum boundary of a normal range not exceeding a critical value α. Said

All beyond

12. The search log abuse prevention method according to claim 11, wherein it is determined that is included in an abnormal range.   The deduction logic uses a KL distance (Kullback-Leibler Distance) representing a distribution difference between at least one probability model of the probability model of the population and the search word summary information and the IP summary information in which the abnormal action is detected. The method according to claim 6, wherein the abnormal action is removed. The search log correction step includes:
The abnormal action is removed from at least one of the search word summary information and the IP summary information in which the abnormal action is detected using a deduction logic to which an information theory for measuring a distribution difference is applied. Item 4. The search log abuse prevention method according to Item 3. In the inspection object selection step, a search word and / or IP is selected as an object for inspecting the abnormal action,
The search log abuse prevention method according to claim 1, wherein an abnormal action is detected from the selected search word and / or IP in the abnormal action detection step.   A computer-readable recording medium on which a program for causing a computer to execute the search log misuse prevention method according to any one of claims 1 to 15 is recorded. A preprocessing unit for selecting an object to be inspected for abnormal behavior from the search log;
An abnormal action detection unit that detects an abnormal action by scoring the degree of deviation from normal with respect to the selected target;
Removing the detected abnormal action from the search log using a predetermined deduction logic, and correcting the search log;
A search log misuse prevention device, comprising: The pre-processing unit is a search word summary information obtained by statistically analyzing the number of input of each specific search word included in a predetermined time window from the search log according to each IP in order to select an object to be examined for the abnormal action. And / or generate IP summary information that statistically analyzes the number of input times of each search term in a specific IP,
The search log abuse prevention device according to claim 17, wherein the abnormal action detection unit detects the abnormal action from at least one of the search word summary information and / or IP summary information.   The search log abuse prevention apparatus according to claim 18, wherein the search word summary information and / or IP summary information is modeled into a multidimensional distribution using a statistical method. The pre-processing unit is
Reducing the dimension of the input frequency vector for each IP of the specific search word to generate the search word summary information, and reducing the dimension of the input frequency vector of each search word in the specific IP to generate the IP summary information. The search log misuse prevention apparatus of Claim 19 characterized by these. The abnormal action detection unit,
The degree of abnormality is calculated as a score by the degree away from the center with respect to the modeled search word summary information and / or IP summary information, and the calculated search word summary information is a predetermined reference value or more, and 20. The search log abuse prevention apparatus according to claim 19, wherein it is determined that an abnormal act is included in the IP summary information. The abnormal action detection unit,
The search log misuse prevention apparatus according to claim 21, wherein before the score is calculated, the dimension of the modeled search word summary information and / or IP summary information is reduced and the data is compressed. The abnormal action correction unit
The abnormal act is removed from the search term summary information and / or the IP summary information in which the abnormal act is detected using a deduction logic to which an information theory for measuring a distribution difference is applied. Search log abuse prevention device.

Images