KR20050111566A - Spam mail filtering system via link structure analysis of e-mail - Google Patents

Abstract

The present invention measures the number of times a web document pointed to by a hyper-link present in an email is cited (linked) by another arbitrary web document, and then the number of hyperlinks present in the email and the hyperlink. Machine learning with the number of other web documents quoting qualities and classify spam mail.

Also, classify spam mail after machine learning using only host URL part of hyperlink of email. In addition, after the machine learning through the content-based analysis in addition to the hyperlink analysis of the email, the email content analysis results are integrated to improve spam filtering efficiency.

Description

Spam Mail Filtering System via Link Structure Analysis of E-mail}

The present invention relates to a system and method for blocking e-mail, and more particularly, to a system and method for classifying random spam mail sent without a user's consent through an internet communication network.

Spam mail is advertising email that is sent randomly to an unspecified number of people, even if they do not want it or do not request it. Spam mail deprives individuals of their working hours and stresses them, and random spam mail is a major dysfunction of the information society that causes excessive load on the network and causes great damage to individuals, companies and the nation.

Existing spam classification algorithm can be classified as follows.

① List-Based Filtering

Create blacklists and whitelists at the email server level to block spam. However, this method has the disadvantage that spammers continue to attack by changing their email address, are expensive to update the list, and react slowly to the latest spam mail.

Rule-Based Filtering

This is a method of classifying an email when it finds a specific word to classify by analyzing the header, subject and body of the email. However, if a spammer modifies a word by mixing special characters or Chinese characters with specific words that classify spam mail, such as "advertisement", "advertisement", "advertisement", etc., it is difficult to classify it by rule-based method. There is this.

Statistical Content-Based Spam Filtering

It uses spam mail and non-spam mail as a sample to remove stop words, preprocessing such as stemming, normalization, and decision tree and naïve. This is a method of classifying e-mail after learning by machine learning methods such as Naive Bayesian, Support Vector Machine (SVM), and Artificial Neural Networks.

However, most recent spam mails have a limitation of statistical spam classification because the contents of the body are mostly sent as images rather than texts.

④ Collaborative Filtering

This is a method of classifying mails which are reported as spam mails by users in one e-mail server as spam mails to other users. This method is mainly used by email portal service providers and has a good spam classification rate, but has a limitation that it can be used only within one email server because it does not share information about spam between different email servers. In addition, spammers insert random code into spam mails for spam mails with the same content, thereby reducing the classification rate. It also lowers the classification rate because it constantly changes the sender address of the spam mail or the address of the sending server.

⑤ Spam classification through social network analysis

This is a recent method proposed by Boykin that represents email senders as nodes on the graph, and links to send emails as links so that people who send and receive emails frequently are grouped on the graphs. Clusters represent friends who know each other. So nodes that are not clusters on the graph can be classified as spammers.

However, this method has the disadvantage of having to keep track of the relationship of sending the email, so it should be used only within one email server.

The present invention is a method of classifying by measuring the number of web documents linking a web document pointed to by a hyperlink (hereinafter referred to as a link) existing in an email, rather than a conventional rule-based or content-based spam mail classification method. Therefore, you should be able to extract the various types of links that exist within an email, and sometimes even out-of-format links.

In order to measure the number of web documents linking the web documents pointed to by the extracted links, the Google web API provided by the search engine Google using the PageRank algorithm is used.

The result is a new method of classifying spam mails, which is a new method of classifying spam mails after learning the decision tree based on the sum of the number of links in the email and the number of times the web documents of the links are cited by other web documents. .

In addition, by measuring the number of web documents linking only the host URL portion in the link of the e-mail can be learned by the decision tree, it can further increase the spam classification rate by merging the content-based classification method.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Email can be viewed as a web document, with most links to other web documents. In the case of spam mails, in particular, the purpose of advertising is mostly to have a link to an advertising site. This is because in order to realize email marketing for advertising purposes, it must be directed to the site to be advertised. On the other hand, normal mail, not spam mail, may have a link to another site such as a newsletter, or it may be divided into a case in which there is only a text and no link, such as an email between individuals.

On the other hand, the search engine Google has a result of indexing by calculating the fitness of all web documents using the page rank algorithm. The goodness of fit here is a quantification of how many links a web document is referenced by other web documents. In other words, the number of in-links coming into a web document is quantified. Thus, the high relevance of a web document means that the web document is linked by many other web documents and is therefore more popular and more useful. On the other hand, a poor fit of a web document means that the web document is not popular and useful because it is rarely linked by other web documents.

If you give 'link:' to Google's search box, you'll see the number and URLs of the web documents that link to the web documents that Google has already indexed. For example, if you search by giving the 'link:' option to the Seoul homepage, such as link: http: //www.seoul.go.kr, in Google's search box, 212 links to the Seoul homepage (as of November 2005). You can view the web document at. Since there are 212 web documents linking the Seoul homepage, it can be said that the Seoul homepage is popular and useful. On the other hand, if you search for 'link:' option on 'Applicant's personal blog' ('link: http: //blog.naver.com/bearrhee'), there will be 0 web documents linking to Applicant's blog. It can be said that blogs are not popular and are not objectively useful.

In the system of the present invention, the suitability of the link was measured using the Google web API provided by Google.

Referring to FIG. 1, most spam mails have links to other web documents. But few web documents link to them.

On the other hand, referring to Figure 2, when the normal mail has a link, there are relatively many web documents that link the web documents.

At this time, through the link structure analysis of e-mail, the following facts can be found.

"If an email has a link and there are very few web documents linking to the web document on that link, it is likely spam. On the other hand, if the email does not have a link (consists of text only) or has a link, the web document If you have a lot of web documents linking to, it's likely a normal mail. "

3 shows a spam mail classification process based on link structure analysis. It can be divided into learning process and classification process. In the learning process, a sufficient number of spam mails and normal mails are used for learning data. In addition to the links in the form of HTML tags from the learning data, it extracts URLs in the form of 'http: //' from the text. The extracted link is measured using the Google Web API to measure the number of web documents that link to the web document corresponding to the link. If there are several links in an email, the number of web documents that link each web document corresponding to each link is used as the appropriateness of the email. At this time, the model is generated by training the decision tree, which is one of the machine learning methods, using the sum of the number of links in the email and the goodness of fit.

Trees of various sizes are created as the reliability of rule pruning of the decision tree changes during the model generation. That is, the number of generated rules varies according to the reliability of rule pruning. An example of a rule generated from real data is as follows.

Suppose the email you want to classify is D, the number of links in the email is L, and the goodness of fit of the web document corresponding to the link is F,

Rule 1. If L> 0 and F> 42 then D is normal mail

Rule 2. If L> 6 and F <= 42 then D is normal mail

Rule 3. If L = 0, D is normal mail

Rule 4. If 0 <L <= 6 and F <= 42 then D is spam

The categorization process categorizes e-mails that correspond to new inputs that don't know whether they are spam or not. As in the learning process, we first preprocess the email to extract the link (hyperlink) and measure the fitness with the Google Web API. In the present invention, the goodness of fit is simply defined as the sum of other web documents linking links existing in a page and is the same as the equation of FIG. Then, the link and goodness-of-fit are applied to the decision tree model created in the learning process to classify spam mail or not.

You can also measure the suitability of the host URL by cutting the host (server) portion from the URL of the link extracted from the email. For example, in addition to extracting a link like “http://www.makdrim.com/md/shaver/shaver.php?pcode=inventad” from an email and measuring the suitability of the web document for this link, For example, you can re-cut the host section, such as 'http://www.makdrim.com/', to measure the goodness of fit of the web document for this host.

You can also view and measure the URL of an image in an email as a link. For example, in an email

When you have a tag for an image like "<img src =" http://www.seoul.go.kr/images/main_eventhead.gif ">", you'll see "http://www.seoul.go.kr/images /main_eventhead.gif 'or' http://www.seoul.go.kr/ 'with only the host part cut out.

In addition, it can cut the link and the host part of the link, measure the fitness of each, and merge the content analysis-based method with each other to further increase the spam classification rate.

The present invention classifies whether the e-mail is spam or not by extracting a link in the form of a URL such as a hyperlink or an image in the e-mail and measuring the fitness of the web document of the link. Conventionally, there was a method of classifying links by pattern matching and pattern matching, but this is a disadvantage that cannot be classified in the case of new spam mail that is not in the spam list. However, the present invention measures and categorizes the suitability of the web document pointed to by the link in the e-mail, so that any new e-mail can be classified. Also, in recent years, the contents of spam mails are often sent in the form of images instead of texts, but there are limitations because the existing contents analysis methods cannot analyze the images. However, the present invention is a completely new form of breakthrough spam classification that overcomes the disadvantages of such content-based methods.

Therefore, the present invention is robust against any type of attack to date, such as spammers frequently processing the contents of spam mails, inserting arbitrary code in the text, or frequently changing the sending server.

This is because if the spammer's web page linked and induced in the spam mail is not linked by many other web documents, it is judged as low suitability by the present invention and classified as spam mail.

In addition, since the method of the present invention is not a method of analyzing the contents of an email, it is advantageous in that it can be effectively classified as a spam mail log of any language as well as Korean and English.

1 is a diagram analyzing the link structure on the web of spam mail.

Figure 2 is a view of analyzing the link structure on the web of non-spam normal mail.

Figure 3 is a flow chart illustrating a spam mail classification process through the proposed link structure analysis.

4 is a goodness of fit of the web page used in the present invention.

Claims (2)

Spam e-mail classification method that extracts URLs such as hyperlinks and images in e-mails, and measures and classifies the extent to which web documents corresponding to the URLs are linked from other web documents. The method of claim 1, wherein not only the entire URL of the link but also the host (server) URL part is cut and the degree of the host URL is measured by the linkage from other web documents.