KR102426056B1 - Method, system, and computer program for dedecting multimodal abusing pattern to select document - Google Patents

Abstract

A method, system, and computer program for detecting multimodal abusing patterns for document screening are disclosed. The abusing pattern detection method includes: detecting a burstiness pattern representing a repeating pattern of a document based on a multimodal feature between documents; and selecting an original document by using the explosive pattern.

Description

METHOD, SYSTEM, AND COMPUTER PROGRAM FOR DEDECTING MULTIMODAL ABUSING PATTERN TO SELECT DOCUMENT

The description below relates to a technique for detecting an abusing pattern document.

Due to the commercialization of the Internet, many users can search for information on various topics using the Internet. However, it takes time and effort to find the optimal information desired by the user in the vast amount of information.

Recently, search engines have introduced various search methods so that users can find the optimal information they want. One of them is a document intent-based ranking algorithm that allows documents with rich experiences or opinions or documents with high information to be exposed at the top of search results.

Meanwhile, an abusing problem of intensively producing similar documents in order to occupy document rankings for advertising or malicious purposes is increasing.

A lot of techniques for solving this abusing problem are being studied, for example, in Korea Patent Publication No. 10-2012-0005797 (published on January 17, 2012), a document determined to be a filtering target is registered or blocked from being exposed. The technique is disclosed.

A method and system for detecting multi-modal abusing patterns for documents produced for a certain period of time and selecting original documents based on the detected patterns are provided.

A method and system for detecting abusing characteristics of document patterns and network information between document authors and selecting intentionally produced abusing documents through these patterns are provided.

A method for detecting an abusing pattern executed in a computer system, wherein the computer system comprises at least one processor configured to execute computer readable instructions contained in a memory, and the method for detecting an abusing pattern is performed by the at least one processor , detecting a burstiness pattern representing a repeating pattern of the document based on a multimodal feature between the documents; And by the at least one processor, by using the explosive pattern provides an abusing pattern detection method comprising the step of screening an original document (original document).

According to an aspect, the detecting may include detecting the explosive pattern appearing in the document set with respect to a document set within a time window.

According to another aspect, in the detecting, the explosive pattern appearing in the document set using at least one content of a word, a link, a title, a text, and an image included in each document for a document set within a time window. It may include a step of sensing.

According to another aspect, the detecting may include detecting the explosive pattern by using a proportion in which a word or expression selected based on the frequency of occurrence of a word in the document set appears in each document.

According to another aspect, the detecting may include: measuring a plurality of features representing the repeating pattern using content included in the document; and calculating an explosive score by summing the plurality of features.

According to another aspect, the detecting may include: selecting a document set using the plurality of features; and learning the plurality of features or adjusting a weight for each feature based on the document set.

According to another aspect, the detecting may include converting the content included in the document into a numeric vector and then assigning a pattern ID corresponding to the numeric vector; and calculating the explosive score by scoring the number of pattern IDs in the time window by applying a time window to the pattern ID according to time.

According to another aspect, the detecting may further include measuring sharpness of the explosive pattern by adjusting the time window when the explosive score is equal to or higher than a predetermined level.

According to another aspect, the method for detecting the abusing pattern may further include, by the at least one processor, generating network information between a document in which the explosive pattern is detected and an author who wrote the document.

According to another aspect, the detecting includes converting the content included in the document into a numeric vector and then assigning a pattern ID corresponding to the numeric vector, wherein the generating includes the pattern ID and generating the network information by linking a link between the author and the author of the document.

According to another aspect, the selecting may include selecting at least one of an abusing document and an abuser by using the network information together with the explosive pattern.

It provides a computer program stored in a non-transitory computer-readable recording medium for executing the abusing pattern detection method in the computer system.

It provides a non-transitory computer-readable recording medium in which a program for executing the abusing pattern detection method in a computer is recorded.

A computer system comprising: at least one processor configured to execute computer readable instructions contained in a memory, the at least one processor configured to generate an explosive pattern representing a repeating pattern of the document based on multimodal features between the documents. a pattern detection unit to detect; and a document sorting unit for sorting original documents by using the explosive pattern.

A computer system comprising: at least one processor configured to execute computer readable instructions contained in a memory, the at least one processor configured to generate an explosive pattern representing a repeating pattern of the document based on multimodal features between the documents. a pattern detection unit to detect; a network generating unit generating network information between the document in which the explosive pattern is detected and the author who created the document; and a document sorting unit for sorting original documents by using the explosive pattern and the network information.

According to embodiments of the present invention, it is possible to detect a multi-modal abusing pattern for a document produced for a certain period of time, and select a document having an abusing pattern and an original document based on the detected pattern.

According to embodiments of the present invention, document screening performance and document retrieval quality can be improved by selecting intentionally produced abusing documents using network information between document authors along with the abusing characteristics of document patterns.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
2 is a block diagram illustrating the internal configuration of an electronic device and a server according to an embodiment of the present invention.
3 is a diagram illustrating an example of components that a processor of a server may include according to an embodiment of the present invention.
4 is a flowchart illustrating an example of a method that a server may perform according to an embodiment of the present invention.
5 is an exemplary diagram for explaining the collective characteristics that may appear in a document according to an embodiment of the present invention.
6 to 7 are exemplary views for explaining a process of expressing a document as a numerical vector according to an embodiment of the present invention.
8 is an exemplary diagram for explaining a process of learning an abusing pattern using detailed features according to an embodiment of the present invention.
9 is an exemplary diagram for explaining network information between a document and an author according to an embodiment of the present invention.
10 to 13 are exemplary views for explaining a process of selecting an abusing document and a normal document according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to a technique for screening an abusing pattern document.

Embodiments including those specifically disclosed herein may search for a repeating abusing pattern without originality and conversely select an original document, thereby improving document screening performance and document search quality.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices 110 , 120 , 130 , 140 , a plurality of servers 150 , 160 , and a network 170 . 1 is an example for explaining the invention, and the number of electronic devices or the number of servers is not limited as in FIG. 1 .

The plurality of electronic devices 110 , 120 , 130 , and 140 may be a fixed terminal implemented as a computer system or a mobile terminal. Examples of the plurality of electronic devices 110 , 120 , 130 , 140 include a smart phone, a mobile phone, a navigation device, a computer, a notebook computer, a digital broadcasting terminal, a personal digital assistant (PDA), and a portable multimedia player (PMP). ), tablet PCs, game consoles, wearable devices, Internet of things (IoT) devices, virtual reality (VR) devices, augmented reality (AR) devices, and the like. As an example, in FIG. 1 , the shape of a smartphone is shown as an example of the electronic device 110 , but in embodiments of the present invention, the electronic device 110 is substantially configured to be different through the network 170 using a wireless or wired communication method. It may refer to one of various physical computer systems capable of communicating with the electronic devices 120 , 130 , 140 and/or the servers 150 and 160 .

The communication method is not limited, and a communication method using a communication network (eg, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network, a satellite network, etc.) that the network 170 may include, as well as a short-distance wireless communication between devices may be included. can For example, the network 170 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , the Internet, and the like. In addition, the network 170 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network, etc. not limited

Each of the servers 150 and 160 communicates with the plurality of electronic devices 110 , 120 , 130 , 140 and the network 170 through a computer device or a plurality of computers providing commands, codes, files, contents, services, etc. It can be implemented in devices. For example, the server 150 may be a system that provides a first service to a plurality of electronic devices 110 , 120 , 130 , 140 connected through the network 170 , and the server 160 is also a network ( It may be a system that provides the second service to the plurality of electronic devices 110 , 120 , 130 , and 140 connected through 170 . As a more specific example, the server 150 provides a service (eg, a search service, etc.) targeted by the application through an application as a computer program installed and driven in the plurality of electronic devices 110 , 120 , 130 , 140 . The first service may be provided to the plurality of electronic devices 110 , 120 , 130 , and 140 . As another example, the server 160 may provide a service for distributing a file for installing and driving the above-described application to the plurality of electronic devices 110 , 120 , 130 , and 140 as the second service.

2 is a block diagram for explaining the internal configuration of an electronic device and a server according to an embodiment of the present invention. In FIG. 2 , the internal configuration of the electronic device 110 and the server 150 will be described as an example of the electronic device. In addition, other electronic devices 120 , 130 , 140 or server 160 may also have the same or similar internal configuration as the aforementioned electronic device 110 or server 150 .

The electronic device 110 and the server 150 may include memories 211 and 221 , processors 212 and 222 , communication modules 213 and 223 , and input/output interfaces 214 and 224 . The memories 211 and 221 are non-transitory computer-readable recording media, and are non-transitory computer-readable recording media, such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), flash memory, and the like. It may include a permanent mass storage device. Here, a non-volatile mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in the electronic device 110 or the server 150 as a separate permanent storage device distinct from the memories 211 and 221 . In addition, in the memories 211 and 221 , an operating system and at least one program code (eg, a browser installed and driven in the electronic device 110 or codes for an application installed in the electronic device 110 to provide a specific service) can be stored. These software components may be loaded from a computer-readable recording medium separate from the memories 211 and 221 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, the software components may be loaded into the memories 211 and 221 through the communication modules 213 and 223 rather than the computer-readable recording medium. For example, the at least one program is a computer program installed by files provided through the network 170 by a file distribution system (eg, the above-described server 160 ) for distributing installation files of developers or applications. It may be loaded into the memories 211 and 221 based on (eg, the above-described application).

The processors 212 and 222 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. The instructions may be provided to the processors 212 and 222 by the memories 211 and 221 or the communication modules 213 and 223 . For example, the processors 212 and 222 may be configured to execute received instructions according to program codes stored in a recording device such as the memories 211 and 221 .

The communication modules 213 and 223 may provide a function for the electronic device 110 and the server 150 to communicate with each other through the network 170 , and the electronic device 110 and/or the server 150 may communicate with each other through the network 170 . A function for communicating with an electronic device (eg, the electronic device 120 ) or another server (eg, the server 160 ) may be provided. For example, a request generated by the processor 212 of the electronic device 110 according to a program code stored in a recording device such as the memory 211 is transmitted to the server 150 through the network 170 under the control of the communication module 213 . ) can be transferred. Conversely, a control signal, command, content, file, etc. provided under the control of the processor 222 of the server 150 passes through the communication module 223 and the network 170 to the communication module 213 of the electronic device 110 . ) through the electronic device 110 may be received. For example, a control signal, command, content, or file of the server 150 received through the communication module 213 may be transmitted to the processor 212 or the memory 211 , and the content or file may be transmitted to the electronic device ( 110) may be stored as a storage medium (the above-described permanent storage device) that may further include.

The input/output interface 214 may be a means for interfacing with the input/output device 215 . For example, the input device may include a device such as a keyboard, mouse, microphone, camera, and the like, and the output device may include a device such as a display, speaker, haptic feedback device, and the like. As another example, the input/output interface 214 may be a means for an interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 215 may be configured as one device with the electronic device 110 . In addition, the input/output interface 224 of the server 150 may be a means for interfacing with a device (not shown) for input or output that is connected to the server 150 or that the server 150 may include. As a more specific example, when the processor 212 of the electronic device 110 processes a command of a computer program loaded in the memory 211 , a service configured using data provided by the server 150 or the electronic device 120 . A screen or content may be displayed on the display through the input/output interface 214 .

In addition, in other embodiments, the electronic device 110 and the server 150 may include more components than those of FIG. 2 . However, there is no need to clearly show most of the prior art components. For example, the electronic device 110 is implemented to include at least a part of the above-described input/output device 215 or other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. may include more. As a more specific example, when the electronic device 110 is a smart phone, an acceleration sensor or a gyro sensor, a camera module, various physical buttons, a button using a touch panel, an input/output port, and a vibrator for vibration generally included in the smart phone Various components such as may be implemented to be further included in the electronic device 110 .

Hereinafter, a specific embodiment of a method and system for detecting a multimodal abusing pattern for document screening will be described.

In the present specification, a document may refer to an information unit to be searched on the Internet.

Documents produced for advertising or malicious purposes have a characteristic in which similar content is repeated, without originality, and mainly written by authors of similar groups with similar content and intensively at the same time.

In the present embodiment, the burstiness pattern is for measuring document quality, and refers to an abusing pattern that appears as a repeating pattern as documents are written with similar content by authors of a similar group at a similar time. The explosive pattern of a document may mean a multimodal feature between documents produced within a specific period rather than a single document feature.

The present embodiments are to find documents of a repeating pattern without originality and, on the contrary, select original documents that search service users may prefer.

3 is a block diagram illustrating an example of components that a processor of a server may include according to an embodiment of the present invention, and FIG. 4 is an example of a method that the server may perform according to an embodiment of the present invention is a flowchart showing

The server 150 according to the present embodiment serves as a platform that provides a search service. In particular, the server 150 may detect an explosive pattern, which is a repeating pattern, with respect to a document to be searched, and select an original document through this pattern.

The processor 222 of the server 150 is a component for performing the abusing pattern detection method according to FIG. 4 , and as shown in FIG. 3 , the pattern detection unit 310 , the network generation unit 320 , and the document selection It may include a unit 330 . Depending on the embodiment, components of the processor 222 may be selectively included in or excluded from the processor 222 . In addition, according to an embodiment, the components of the processor 222 may be separated or combined to express the functions of the processor 222 .

The processor 222 and the components of the processor 222 may control the server 150 to perform the steps S410 to S430 included in the method for detecting the abusing pattern of FIG. 4 . For example, the processor 222 and components of the processor 222 may be implemented to execute instructions according to the code of the operating system included in the memory 221 and the code of at least one program.

Here, the components of the processor 222 may be expressions of different functions performed by the processor 222 according to instructions provided by the program code stored in the server 150 . For example, the pattern detection unit 310 may be used as a functional representation of the processor 222 that controls the server 150 according to the above-described command so that the server 150 detects an explosive pattern indicating a repeating pattern of a document. have.

The processor 222 may read a necessary command from the memory 221 in which the command related to the control of the server 150 is loaded. In this case, the read command may include a command for controlling the processor 222 to execute steps S410 to S430 to be described later. Steps S410 to S430 to be described later may be performed in an order different from that shown in FIG. 4 , and some of the steps S410 to S430 may be omitted or additional processes may be further included.

In step S410 , the pattern detecting unit 310 may detect an explosive pattern indicating a repeating pattern of a document for a document generated during a predetermined period. Since the documents produced by the abusers mainly have some commonalities and tend to appear intensively in a specific period, it is possible to select the abusive documents by finding the explosive patterns corresponding to these repeating patterns. As an example, the pattern detection unit 310 includes various features (hereinafter, referred to as 'detailed features') that indicate a repeating pattern using words, images, links, titles or texts of documents, whether or not they are included in the document. ) to detect explosive patterns. In this case, the pattern detection unit 310 periodically (eg, 4 hours) measures detailed features for a document set produced or modified in a corresponding period, and then sequentially accumulates detailed features for each period for a specific period (eg, 8 hours). Explosive patterns can be found from detailed features accumulated over time, 1 day, etc.). The pattern detecting unit 310 may find an explosive pattern by increasing/decreasing a specific pattern generated within the time window while adjusting the time window.

In step S420 , the network generating unit 330 may generate network information between authors who have created a corresponding document for a document in which an explosive pattern is detected. The network generator 330 may generate a network between the document and the author and between the author and the author based on the explosive pattern of the document. The network generator 330 may generate network information by linking links between authors who have created documents of similar repeating patterns and linking links between documents of similar repeating patterns and authors who created the documents.

In step S430 , the document sorting unit 330 may select a document based on the explosive pattern detected in step S410 and the network information generated in step S420 . The document sorting unit 330 may filter documents including a repeating pattern and authors who write a document in a repeating pattern by using the explosive pattern of the document and network information between the authors, and select normal documents with original patterns.

The process of detecting the explosive pattern of a document will be described in detail as follows.

Referring to FIG. 5 , in the present embodiment, the explosive pattern is a collective feature that may appear between documents produced within a specific period (time window), and includes a document explosive pattern 501, an individual explosive pattern 502, and a group explosive pattern ( 503) may be included. The document explosive pattern 501 refers to a pattern in which documents including similar words, similar titles, and similar texts are concentrated for a certain period of time, and the personal explosive pattern 502 refers to a pattern in which documents classified as similar documents by authors are displayed for a certain period of time. It means a pattern that is intensively written, and the group explosive pattern 503 may mean a pattern in which similar documents are intensively written in a similar period by a specific group of authors.

The explosive pattern of the document may be created by a combination of detailed features related to at least one of the document explosive pattern 501 , the individual explosive pattern 502 , and the group explosive pattern 503 as described above.

Detailed features may be calculated in units of document sets of a certain period, and most may be normalized using rank statistics in order to easily scale between features.

Examples of detailed features used to calculate the explosive score are as follows.

(1) word burstiness score

Word explosiveness is a feature that measures how many times more frequently used words appear in a document over a time window. In this case, the word is a space separation criterion, and only words including nouns are counted. In addition, whether or not it is used more than usual can be measured by a TF-IDF ratio in the time window of each word in comparison with TF-IDF (Term Frequency-Inverse Document Frequency).

For example, word explosiveness (WBScore) can be scored through [Equation 1].

[Equation 1]

WBScore = sum{ log(TF + 1) × IDF / (NORM_TF × IDF) } (if DF >= 5 else 0)

(2) link burstiness score

Link explosiveness is a feature that measures how often a link contained in a document appears within other documents within a time window. In other words, link explosiveness is a measure of how many different authors have written links included in a document.

For example, link explosiveness (LBScore) can be scored through [Equation 2].

[Equation 2]

LBScore = 1 (if WF == 1)

LBScore = (DF + WF) × 0.8 (if WF == 2)

LBScore = DF + WF (if WF > 2)

Here, WF stands for Writer Frequency.

(3) title burstiness score

Title explosiveness is a feature that measures how often the title of a document appears in other documents within a time window. The titles of documents can be vector-quantized to measure the same number of titles. At this time, a 128-dimensional vector is used for the word vector, and the title vector is created by adding all the word vectors appearing in the title. When quantizing a title vector, it can be converted into a numeric ID by changing a positive (+) value to 1 and a negative (-) value to 0 for each dimension of 128. After measuring DF (Document Frequency), WF (Writer Frequency), etc. based on numeric ID, it can be scored in the same way as link explosiveness.

For example, the title explosiveness (TBScore) can be scored through [Equation 3].

[Equation 3]

TBScore = 1 (if WF == 1)

TBScore = (DF + WF) × 0.8 (if WF == 2)

TBScore = DF + WF (if WF > 2)

(4) body burstiness score

The body explosive score can be calculated by creating a numeric ID in a similar way to the heading explosive. For example, a plurality of numeric IDs may be created by cutting the body of a document by a certain number of words (eg, 100 words).

For example, the body explosiveness (BBScore) can be scored through [Equation 4].

[Equation 4]

BBScore = 1 (if WF == 1)

BBScore = (DF + WF) × 0.8 (if WF == 2)

BBScore = DF + WF (if WF > 2)

(5) Reward points

The cost score is a measure of whether the document was written for a fee. It means a feature that is classified as 1 if the author directly specified in the document, and 0 otherwise. For example, by learning a part (eg, 100 bytes) at the end of a document through probabilistic learning (eg, Naive Bayes), it is possible to classify whether or not it is costly. The cost score can be used to build a learning set of other features rather than being used alone.

(6) pay word burstiness score

The weight of the word weight corresponds to a feature that calculates the word explosiveness in the documents classified as high performance, selects the top n words for each topic, and measures how much weight the selected word exists in the document.

For example, PWBScore may be scored through [Equation 5].

[Equation 5]

PWBScore = sum{ log(TF + 1) × IDF / (NORM_TF × IDF) }

(7) expression burstiness score

Explosive high-level document expression weight score can be calculated in a similar way to proverbial word weight score. However, the unit of summation uses a 'two-word expression' containing a verb or an adjective rather than a noun. Explosive parent document representation weight is a feature for detecting expressions that appear frequently in explosive parent documents. In this case, the high-level explosive document means documents having a combined explosive score of a predetermined level or higher. The combined explosive score will be described again below.

For example, the explosive upper document expression weight (EBScore) can be scored through [Equation 6].

[Equation 6]

EBScore = sum{ log(TF + 1) × IDF / (NORM_TF × IDF) }

(8) expression rate burstiness score

The explosive upper document expression ratio score is a value obtained by dividing the explosive upper document expression weight score by the number of words in the entire text, and may be defined as in [Equation 7].

[Equation 7]

ERBScore = EBScore / (word count)

(9) Explosive score Total score by author (by author)

The summed score for each explosive score author is a value obtained by adding the summed explosive score for each author and converting it into a ranking statistical score.

(10) image burstiness score

Image explosiveness is a feature that measures how repeatedly similar images are used in multiple documents over a time window. Image similarity is expressed as a single vector ID, where image explosiveness can be measured in a similar way to text-based explosiveness (word explosiveness, title explosiveness, text explosiveness) using the vector ID of the image.

The processor 222 may convert the text or image included in the document into a numeric vector according to the detailed feature, and then convert it back into a numeric ID for identification.

In the case of text, as shown in FIG. 6 , a numerical vector for the text may be generated using a deep learning technique such as word2vce. Text is a numeric ID (eg, simhash ID) for estimating the similarity between two sets using a vector for one word, a vector for two words, a vector for a title, a vector for content such as a link, etc. can be expressed

Referring to FIG. 7 , in the case of an image 600, ID can be achieved by clustering based on the similarity between images by utilizing the image search deep feature. As an example, an image can be IDed by clustering an image deep learning vector (4096 dimension) with k nearest neighbors (kNN) algorithm, and then listing the cluster IDs in nearest order. For example, if the cluster distances are close in the order of 1, 3, 5, and 29, the image ID can be expressed as [1.3.5.29]. In this case, the image deep learning vector may be clustered using the image signature obtained by the SHA-256 hash algorithm.

The detailed features described above can be converted into rank statistic scores to scale between features. Rank statistics scores are listed in order of score size and normalized to 0.99 for the top 1%, 0.5 for the top 50%, and 0.01 for the bottom 1%. At this time, if the score size is the same, the ranking statistics score is also made to be the same.

Additionally, summary metrics can be applied to simultaneously measure how large the highest-valued features are while all detailed features have high values. For example, the summary indicator may be defined as in [Equation 8].

[Equation 8]

Summary indicator = 0.7 × score of the first-place feature + 0.3 × (average score of the rest of the features)

The above summary indicator may be used for the purpose of detecting when the feature with the highest score shows explosiveness with great certainty even if the scores of other detailed features are low.

The processor 222 sums up the scores of detailed features to select a document set having a higher sum score than a predetermined level, and then re-learns detailed features based on the selected document set, adjusts the combined weight between detailed features, or new detailed features can be further measured. The processor 222 may adjust the detailed features using the correct answer data set evaluated by the tester so that the detailed features are not learned in the wrong direction while repeating the above process.

8 shows an example of a result of learning an advertisement document as an example of an abusing document as a decision tree.

The processor 222 may firstly select a document having a high probability of being an advertisement by applying a predetermined filtering condition in the learning process. For example, referring to FIG. 8 , the processor 222 classifies a document having an explosive upper document expression weight score (expression density) exceeding 0.888 as an advertisement document, and among documents having an explosive upper document expression weight point score of 0.888 or less, the text explosiveness Documents with a score exceeding 0.472 are classified as advertisement documents, and among documents with a body explosive score of 0.472 or less, documents with a link explosive score exceeding 0.487 are classified as advertisement documents, and the rest of the documents can be classified as original and normal documents. have. As another example, a document having a word explosive score exceeding 0.824, a document having a high explosive document expression ratio score exceeding 0.426, etc. may be additionally selected as advertisement documents, and the remaining documents may be classified as normal documents.

In FIG. 8 , a score serving as a document classification criterion corresponds to an exemplary numerical value, but is not limited thereto.

The processor 222 may create a final explosive score as a linear model for a document classified as an advertisement document through the above-described filtering conditions. For example, the final explosive score is a summed explosive score obtained by summing all scores for each detailed feature [Mathematics] It can be defined as Equation 9].

[Equation 9]

dq_score = 0.5639 × total + 0.012 × w + 0.189 × l + 0.210 × t + 0.245 × b + 0.024 × pt + 0.005 × p + 0.127 × max(er, e)

dq_score = - min(dq_score, 0.99)

(where total is the summary indicator, w is the word explosive score, l is the link explosive score, t is the title explosive score, b is the body explosive score, pt is the cost performance score, p is the proxy word weight score, er is the explosive parent document expression ratio score, r is the weight score of the explosive top document expression)

The weight included in Equation 9 corresponds to an exemplary numerical value, and is not limited thereto.

The above-described summed explosive score does not include the image explosive score, but this is only an example, and a different method of summing the explosive score may be applied according to the importance of detailed features. For example, the sum explosive score may be calculated in such a way that the highest weight is applied to the word explosiveness score and the image explosiveness score.

The processor 222 may finally use only the explosive score in a predetermined range (eg, -0.5 to -1.0), and post-process the explosive score to increase the accuracy of the score and expand the application range.

As an example of the post-processing, the processor 222 may include a model for correcting the explosive score when a document with an explosive score is created by the same author multiple times in a short time (eg, a white paper advertisement, etc.). If several documents with similar patterns are written by several people in a short period of time, the probability of an abusing pattern is very high. For example, in order to calibrate the explosive score, three time windows of 1 hour, 10 minutes, and 1 minute may be used. When measuring the explosive score, a basic time window of about 8 hours to 1 day is first applied, and then the time window is reduced to 1 hour by post-processing, so that the sharpness of the explosive pattern can be additionally measured.

The explosive score is based on the method of finding the abusing pattern through the increase or decrease of a specific pattern that occurred within the period while moving the time window due to the nature of the explosive pattern. In this way, to find all the documents of the repeating pattern in the entire document collection, the entire period must be repeatedly calculated. As another example of post-processing, the processor 222 may shorten the calculation time for the explosive score by using network information between document authors. In detail, the processor 222 may vectorize all documents and assign a pattern ID to each detailed feature through quantization so as to have similarities between documents. A pattern ID-author ID-pattern ID network can be created by linking only the pattern ID with a score of a certain level or higher in the pattern ID of each document, and then connecting the left pattern ID and the link between the author of the document. The calculation time for explosive scores can be shortened by finding clusters with a certain distribution or more in the pattern ID-creator ID-pattern ID network and propagating detailed feature scores of the corresponding pattern ID.

Referring to FIG. 9 , the processor 222 connects the link between the document and the author who created the document to create the author-document-author network 900, so that the same pattern ID can be given to the author who created the document of the repeating pattern. have. In other words, clusters gathered in a star-like shape in the author-document-writer network 900 corresponds to the abusing condition because several people wrote the document of the corresponding pattern ID at the same time. When the processor 222 averages and applies the explosive scores of the documents belonging to the cluster corresponding to the abusing condition, the explosive score for all documents in the entire collection can be estimated, and the author (ie, the abuser) who wrote the document of the repeating pattern can be found additionally.

The process of detecting explosive patterns in documents is summarized as follows.

Referring to FIG. 10 , the processor 222 expresses the text included in the document 1010 as a word vector 1011 for a recently created document 1010 , and converts the image included in the document 1010 into an image vector 1012 . ), both text and images included in the document 1010 may be expressed as numeric IDs. In this case, the same pattern ID is assigned to the similar text and the similar image based on the similarity. In addition, the processor 222 may measure and score the explosive pattern of the document 1010 through the pattern ID assigned to the document 1010 , and may select the abusing document 1001 of the repeating pattern based on this.

As shown in FIG. 11 , when the same pattern ID of a document is accumulated over time, the processor 222 may calculate an explosive score within a corresponding period by applying a time window. In this case, the explosive score is a score of pattern number quantiles within the time window. First, the processor 222 applies the basic time window (eg, 7 days) and when the explosive score is above a certain level, the time window is gradually reduced (eg, 1 day, 1 hour, 10 minutes, 1 minute) to the explosive score can be calculated.

Referring to FIG. 12 , the processor 222 may assign a pattern ID to each detailed feature, for example, a document set 1201 to which a pattern ID of a similar word is assigned, a document set to which a pattern ID of a similar sentence is assigned ( 1202), documents belonging to two or more sets at the same time in the document set 1203 to which the pattern ID of the same author is assigned, and the document set 1204 to which the pattern ID of a similar image is assigned, that is, there is a common explosive pattern in several detailed features The appearing document set 1210 may be selected as a candidate document suspected of being an abusing document.

The processor 222 may re-learn detailed features based on the document set 1210 selected as a candidate document, adjust a combined weight between detailed features, or add new detailed features. Referring to FIG. 13 , the processor 222 may classify a document set 1210 selected as a candidate document into an abusing document having a repeating pattern and a normal document with originality while repeating reinforcement machine learning for a document set 1210 selected as a candidate document. As an example, the processor 222 samples a high-ranking document with a high explosive score from the candidate document set 1210 or a document occupying a high ranking in the search result, and selects the abusing pattern according to the tester's abusing tagging or pattern selection for the sampled document. can be relearned. For example, the processor 222 may exclude documents including images that have been pre-tagged or labeled from the candidate document set 1210, and relearn documents incorrectly classified as abusing documents through weight adjustment between detailed features. You may.

The processor 222 may apply the document selection result based on the multi-modal explosive pattern between documents to the document ranking. For example, in the case of a document finally selected as an abusing document of a repeating pattern, it ranks higher than that of a normal document in the search result. Penalties can be applied to avoid doing so.

As such, according to embodiments of the present invention, it is possible to detect a multi-modal abusing pattern for a document produced for a certain period of time, and select a document having an abusing pattern and an original document based on the detected pattern. And, according to embodiments of the present invention, document screening performance and document search quality can be improved by selecting intentionally produced abusing documents using network information between document authors along with the collective characteristics of document patterns.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the devices and components described in the embodiments may include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general purpose or special purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more of these, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be embodied in any type of machine, component, physical device, computer storage medium or device for interpretation by or providing instructions or data to the processing device. have. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be to continuously store the program executable by the computer, or to temporarily store the program for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute other various software, and servers.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (15)

In the method for detecting an abusing pattern executed in a computer system,
the computer system comprising at least one processor configured to execute computer readable instructions contained in a memory;
The abusing pattern detection method,
detecting, by the at least one processor, a burstiness pattern corresponding to the repeating pattern of the document by measuring a plurality of features representing the repeating pattern of the document using content included in the document; and
After selecting, by the at least one processor, the abusing document including the explosive pattern, selecting the remaining documents as original documents
including,
The step of detecting the explosive pattern corresponding to the repeating pattern of the document,
selecting a document set using the plurality of features; and
Learning the plurality of features based on the document set, adjusting a weight for each feature, or adding new features
A method of detecting an abusing pattern comprising a. According to claim 1,
The step of detecting the explosive pattern corresponding to the repeating pattern of the document,
Detecting the explosive pattern appearing in the document set with respect to the document set within a time window
Abusing pattern detection method further comprising. According to claim 1,
The step of detecting the explosive pattern corresponding to the repeating pattern of the document,
Detecting the explosive pattern appearing in the document set using at least one content of a word, a link, a title, a text, and an image included in each document for the document set within a time window
Abusing pattern detection method further comprising. 4. The method of claim 3,
The step of detecting the explosive pattern corresponding to the repeating pattern of the document,
Detecting the explosive pattern by using the proportion in which a word or expression selected based on the frequency of occurrence of words in the document set appears in each document
Abusing pattern detection method further comprising. According to claim 1,
The step of detecting the explosive pattern corresponding to the repeating pattern of the document,
calculating an explosive score by summing the plurality of features
Abusing pattern detection method further comprising. delete In the method for detecting an abusing pattern executed in a computer system,
the computer system comprising at least one processor configured to execute computer readable instructions contained in a memory;
The abusing pattern detection method,
detecting, by the at least one processor, an explosive pattern corresponding to the repeating pattern of the document by measuring a plurality of features representing the repeating pattern of the document using content included in the document; and
After selecting, by the at least one processor, the abusing document including the explosive pattern, selecting the remaining documents as original documents
including,
The step of detecting the explosive pattern corresponding to the repeating pattern of the document,
converting the content included in the document into a numeric vector and assigning a pattern ID corresponding to the numeric vector; and
Calculating an explosive score by scoring the number of pattern IDs in the time window by applying a time window to the pattern ID over time
A method of detecting an abusing pattern comprising a. 8. The method of claim 7,
The step of detecting the explosive pattern corresponding to the repeating pattern of the document,
measuring sharpness of the explosive pattern by adjusting the time window when the explosive score is above a certain level
Abusing pattern detection method further comprising. In the method for detecting an abusing pattern executed in a computer system,
the computer system comprising at least one processor configured to execute computer readable instructions contained in a memory;
The abusing pattern detection method,
detecting, by the at least one processor, an explosive pattern corresponding to the repeating pattern of the document by measuring a plurality of features representing the repeating pattern of the document using content included in the document;
generating, by the at least one processor, network information between the document in which the explosive pattern is detected and the author who created the document; and
Selecting, by the at least one processor, documents corresponding to the abusing condition including the explosive pattern and the network information, and then selecting the remaining documents as original documents
including,
The step of detecting the explosive pattern corresponding to the repeating pattern of the document,
converting the content included in the document into a numeric vector and then assigning a pattern ID corresponding to the numeric vector
including,
The generating step is
Creating the network information by linking the link between the pattern ID and the author who created the document
A method of detecting an abusing pattern comprising a. delete 10. The method of claim 9,
The selecting step is
Selecting at least one of the abusing document and the abuser using the network information together with the explosive pattern
A method of detecting an abusing pattern comprising a. A computer program stored in a non-transitory computer-readable recording medium for executing the method for detecting an abusing pattern of any one of claims 1 to 5, 7 to 9, and 11 in the computer system. A non-transitory computer-readable recording medium in which a program for executing the abusing pattern detection method of any one of claims 1 to 5, 7 to 9, and 11 in a computer is recorded. In a computer system,
at least one processor configured to execute computer readable instructions contained in memory
including,
the at least one processor,
a pattern detecting unit for detecting an explosive pattern corresponding to the repeated pattern of the document by measuring a plurality of features representing the repeating pattern of the document by using the content included in the document; and
A document sorting unit that sorts the abusing documents including the explosive pattern and then selects the remaining documents as original documents
including,
The pattern detection unit,
selecting a document set using the plurality of features;
Learning the plurality of features based on the document set, adjusting weights for each feature, or adding new features
A computer system characterized by a. In a computer system,
at least one processor configured to execute computer readable instructions contained in memory
including,
the at least one processor,
a pattern detecting unit for detecting an explosive pattern corresponding to the repeated pattern of the document by measuring a plurality of features representing the repeating pattern of the document by using the content included in the document;
a network generating unit generating network information between the document in which the explosive pattern is detected and the author who created the document; and
A document sorting unit that selects documents corresponding to the abusing condition including the explosive pattern and the network information and then selects the remaining documents as original documents
including,
The pattern detection unit,
After converting the content included in the document into a numeric vector, a pattern ID corresponding to the numeric vector is given,
The network generating unit,
Creating the network information by linking the link between the pattern ID and the author of the document
A computer system characterized by a.

Images