KR101067650B1 - System for constructing whitelist - Google Patents

Abstract

The present invention relates to a white list construction system, comprising: a crawling unit for searching various sites on the web; A site extracting unit for extracting a related site by determining whether the site is related to a specific field, for each site searched through the crawling unit; And a white list extracting unit extracting a white list using the domain information of the corresponding site from the site extracted from the site extracting unit. It includes.

Crawling, White List, Domain

Description

White List Construction System {SYSTEM FOR CONSTRUCTING WHITELIST}

The present invention relates to a white list construction system, and more particularly, to maintain and manage a high-quality white list continuously and actively to fundamentally solve the rapidly increasing phishing problem, and to prevent phishing, which is a variant phishing technique. It's about a new whitelisting system called Anti-Phishing Crawler.

As the number of Internet users is increasing, crimes using the Internet environment are increasing rapidly. Phishing is one of the most popular new crimes using the characteristics of the cyber environment.

Phishing is a term for a criminal mechanism that attempts to steal your personal and account confidential information. According to Gartner's research, financial damages from phishing have been reported at $ 2.6 billion in 2006 and $ 3.2 billion in 2007, and the damage from phishing continues to increase.

Currently, the most commonly used technique for anti-phishing is a blacklist-based technique. The blacklist technique collects and stores URLs of known phishing sites to block a site from a web browser when a user visits the stored phishing site. Most antiphishing commercial solutions or common web browsers use this technique. However, most phishing sites have a very short lifespan, so even if regular updates are made, they cannot be blocked until they are reported on the list. That is, there is a disadvantage that it does not detect unknown phishing sites. Also, phishing sites are very expensive to maintain lists because their number is constantly increasing.

On the other hand, the whitelist technique is a technique to solve the shortcomings of the blacklist technique. Whitelist-based anti-phishing techniques are a way to tell a user whether or not they are safe when they visit a site by collecting and storing the URLs of those sites.

Whitelisting techniques are generally easier to maintain than blacklisting because they target secure, long-lived sites, and can detect unknown phishing attacks. However, it is not easy to determine how much and how to build the whitelist in the whitelist technique, and despite the above advantages, it is not used much and the proposed method is extremely rare.

Among the conventionally proposed methods of constructing a white list, there is a method of constructing a white list based on a personal site, taking advantage of the fact that the number of sites frequently entered by individuals is very limited. Although this proposal specifically suggested how to build a whitelist, it has a disadvantage in that it cannot determine the safety of other sites because it builds a whitelist of individual-based sites. We also downgraded the need for regular updates because sites on the whitelist could not change frequently, but because sites on the whitelist are safe sites, they are important even if they do not change frequently. You can find the limitations here.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is a crawler method for implementing automatic, extensive and periodic updates which are most important for creating a whitelist in a whitelist-based phishing prevention technique. To build whitelists efficiently.

It is also an object of the present invention to efficiently select financial related sites through topic crawlers using appropriate seed pages and queries to crawl financial related sites most known as phishing target sites.

It is also an object of the present invention to effectively determine the safety of sites to be entered into the whitelist by using page characteristics and four domain characteristics, which are the most prominent characteristics of phishing sites and normal sites, to determine whether the selected sites are safe sites. There is also in the ship.

In addition, the object of the present invention is to prevent farming, a new breed of fraudulent method, which is popular by storing the IP address of the site.

The present invention for achieving the above technical problem relates to a white list building system, the crawling unit for searching various sites on the web; A site extracting unit for extracting a related site by determining whether the site is related to a specific field, for each site searched through the crawling unit; And a white list extracting unit extracting a white list using the domain information of the corresponding site, targeting the site extracted from the site extracting unit. It includes.

On the other hand, the present invention relates to a whitelist building method, the method comprising: (a) searching for various sites on the web by the crawling unit; (b) receiving and storing the site searched by the site extracting unit through the crawling unit; (c) determining whether or not the similarity value is not zero by calculating a similarity related to a specific field for each site stored by the site extracting unit through Equation 1; (d) if the similarity value is not 0, the site extracting unit determines the relevant site as a specific field-related site and extracts it, and records the calculated similarity score and the corresponding link if the similarity value is not 0; (e) determining, by the whitelist extractor, whether a domain of a current site domain and a link site is identical to a site extracted from the site extractor; (f) determining that the whitelist extractor determines that the site is not a phishing site and extracts the current site domain if the domain of the link site matches the domain of the link site; (g) collecting information regarding the lifetime, age, popularity, and DNS ranking of the site domain of the extracted site by determining that the whitelist extractor is not a phishing site; (h) digitizing the information collected by the whitelist extractor using neural networks using characteristic values; And (i) extracting a site having a high input value for a domain characteristic item as a white list based on the numerical value of the white list extracting unit, and storing the extracted site as a URL and IP information of the extracted site; It includes.

According to the present invention as described above, by automatically picking up as many sites as possible through a technique called crawler whitelist that is difficult to build in a whitelist-based phishing prevention technique to allow the user to check the latest list periodically This has the effect of increasing the reliability.

In addition, according to the present invention, it is possible to provide a more secure site by selecting a site to be included in the white list by effectively combining and using various characteristics that are distinguished between a normal site and a phishing site.

In addition, according to the present invention, there is an effect of preventing phishing as well as pharming novel.

Specific features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings. In the meantime, when it is determined that the detailed description of the known functions and configurations related to the present invention may unnecessarily obscure the subject matter of the present invention, it should be noted that the detailed description is omitted.

Hereinafter, with reference to the accompanying drawings will be described in detail the present invention.

The whitelist building system and method thereof according to the present invention will be described with reference to FIGS. 1 to 7.

1 is an overall configuration diagram conceptually showing a white list building system S according to the present invention. As shown, the crawling unit 100, the site extracting unit 200, and the white tries extracting unit 300 are illustrated. It is made, including.

The crawling unit 100 performs a function of searching various sites on the web.

The site extracting unit 200 performs a function of extracting a related site by determining whether each site searched through the crawling unit 100 is a site related to a specific field, as shown in FIG. The site management module 210 and the similarity calculation module 220 as shown. In the present embodiment, a specific field will be set to "finance", but the present invention is not limited thereto.

In detail, the site management module 210 receives and stores a site searched through the crawling unit 100.

The similarity calculation module 220 calculates "finance" related similarity through the following [Equation 1] to determine whether each site stored in the site management module 210 is a site related to "finance".

[Equation 1]

The cosine similarity (sim (q, p)) of page p and query q is

Where Vq and Vp are the term frequencies based on the vector representation of the query q and page p, respectively, and VqVp is the dot product of the two vectors, | v || Is the Euclidean gambling of the vector V.

The similarity calculation module 220 determines whether the similarity value is not 0, and if it is not 0, determines the relevant site as a financial related site and extracts it. On the other hand, if the determination result is 0, it is determined that it is not a financial-related site, and receives another site from the site management module 210 to perform a similarity calculation.

At this time, the site determined to be a finance-related site records the calculated similarity score and the corresponding link and transmits the same to the site management module 210. That is, the similarity calculation module 220 may receive the similarity calculation from the site management module 210 in the order of sites having the high similarity score.

For reference, the site extraction unit 200 according to the present invention uses a technique called topic crawler. Topic crawlers are topic crawlers that attempt to download only web pages that are related to the topic that is defined first. In this case, the crawler can be smoothly implemented through multi-thread crawling using 256 threads simultaneously using the highest priority algorithm using the cosine similarity of Equation 1, which is known to be the most efficient among the topic crawler algorithms. 2 is an exemplary view illustrating a multithreaded crawler model through a site extractor.

At this time, the site extraction unit 200 has two important elements.

The first is a query, which corresponds to the topic here. The more accurate the query, the higher the number of sites that will be relevant to that topic. In the present invention, the query "bank" is used to effectively crawl not only banks but also sites in charge of financial transactions. Although the query is simple, it is very popular because it is the most used term on financial sites.

Second is the seed page. The performance of a finance selector can depend on how much and collectively the seed page contains links related to the topic.

In addition, the white list extracting unit 300 performs a function of extracting a white list using the domain information of the corresponding site from the site extracted from the site extracting unit 200, as shown in FIG. The page information determining module 310, the domain information determining module 320, and the list extracting module 330 are included.

In this embodiment, the stability is determined based on the domain matching of the current site domain and the link site. Few links on the phishing site currently use the domain of the phishing site. Because phishing sites are designed to steal your personal information and account number, you don't create any other sites to save money, except for sites you've created for phishing. However, most normal sites will have their own domain and link domain matching. Therefore, if you compare the domain of the current domain with the link, it is very likely that you are a phishing site if none of the domains match.

Therefore, the page information determination module 310 determines whether the current site domain and the domain of the link site match the target site extracted from the similarity calculation module 220 of the site extraction unit 200.

As a result, if the current site domain and the link site domain match, it is determined that the site is not a phishing site, and if it does not match, it is determined to be a phishing site.

3 is an exemplary view showing a phishing site according to the present invention, it can be seen that the domain (a) of the current site and the domain (b) of the link site does not match. As above, the site where the domain of the current site domain and the link site match is determined again as a phishing site based on the detailed characteristics of the domain.

The domain information determination module 320 collects information on the lifetime, age, popularity, and DNS ranking of the site domain for the extracted site by determining that it is not a phishing site through the page characteristic determination module 310, and the neural network. Using Neural Networks, the collected information is digitized into characteristic values.

Domain characteristics can be divided into four types.

First, the domain lifetime, which is the time from the creation date of the domain to its expiration date.

Phishing sites have a very short domain life span because they deceive and deceive users in the short term than regular sites.

Second, the domain age, which is the time from the creation date of the domain so far.

The shorter the phishing site, the younger the domain, the more likely it is. 4 is an exemplary view showing an information search result for calculating the lifetime and age of a specific domain using a domain lifetime and age search site.

Third, domain popularity, which indicates how many sites using the same domain are registered with a regular search engine site.

In general, a well-known normal site provides a number of sites around the domain, so the number of sites that use the domain is large, but phishing sites register sites that use the domain to deceive users. Do not use. Search engine sites also remove these phishing sites from the search engine database for their safety, so it is not easy to find phishing sites in search engines. 5 shows a search result of a domain popularity of a specific bank using a search engine site.

Fourth, DNS ranking indicating how many times a user requested the domain.

A normal site uses a lot of domain lookups for users to access the site, whereas phishing sites rarely look up a domain because the domain is not well known. 6 is an exemplary view illustrating a ranking ranking of a specific domain using a ranking site.

The properties of the domain consist of a combination of the four properties above. Overall, the longer the domain lifetime, the older, the more popular, and the higher the ranking, the more likely it is to be a normal site.

If these four characteristics are judged independently, there is a high possibility that a reliable whitelist cannot be built. Therefore, in the present invention, a site is determined by combining four characteristics using a neural network technique, which is one of data mining techniques.

Since the input value in the neural network technique must have a value between 0 and 1, it is necessary to convert the input values of domain characteristics in which each value has a categorical type.

According to the present embodiment, information about the lifespan, age, popularity, and DNS ranking of domains is quantified as shown in Table 1 below.

Characteristics 0 0 to 1 One Domain lifetime 2 years or less (Domain lifetime-2) / 6 More than 8 years Domain age Less than 1 year (Domain age-1) / 3 More than 4 years Domain popularity 0 Number of domains / 100,000 More than 100,000 DNS ranking If not in the ranking (10000001-DNS ranking) / 1 million First place

Table 1 Digitization of Domain Attribute Input Values

Table 1 shows the process of changing domain property values from categorical to numeric. First, the domain life is set to 0, which is the lowest input value when the life span is 2 years or less, and 1, which is the highest value when it is 8 years or more. If the domain lifetime is between 2 and 8 years, the value is given in proportion. The domain age is similar to the domain life, and the value is proportional to 0 when the age is less than 1 year, 1 when the age is over 4 years, and between. In the case of domain popularity, the lowest input value is 0 when there are no domain lookups for the site, and the highest value is 1 when there are more than 100,000 sites. When the popularity is in between, the value is given in proportion. Lastly, in DNS ranking, if it is not in the ranking, the smallest value is 0. Otherwise, the value is inversely proportional to the rank. This is because the ranking should have a larger value as the value of the rank is smaller. Overall, the longer the domain lifetime, the older, the more popular, and the higher the ranking, the greater the input value.

The list storage module 330 extracts and stores the white list based on the numerical value through the domain information determination module 320. That is, a site having a high input value for a domain characteristic item is extracted, and it is determined that the extracted site is not a phishing site at last, extracted as a white list, and the URL and IP information of the extracted site are stored.

Hereinafter, a construction method using the white list construction system S as described above will be described with reference to FIG. 7.

7 is a flowchart illustrating a whitelist construction method according to the present invention. As shown, the crawling unit 100 searches for various sites on the web (S10), and the site management module of the site extraction unit 200 ( 210 receives and stores a site searched through the crawling unit 100 (S20).

Thereafter, the similarity calculation module 220 calculates the "financial" related similarity with respect to each site stored in the site management module 210 through [Equation 1] above, and determines whether or not the similarity value is not 0 ( S30).

As a result of the determination in step S30, if the similarity value is not 0, the similarity calculation module 220 determines the relevant site as a finance-related site, extracts it, and records the calculated similarity score and the corresponding link to the site management module 210. In operation S40, when the similarity value is 0, the procedure is performed in step S10 to receive another site from the site management module 210 to perform similarity calculation.

Subsequently, the page information determining module 310 of the white list extracting unit 300 determines whether the current site domain and the domain of the linking site correspond to the site extracted from the similarity calculating module 220 (S50). .

As a result of the determination in step S50, when the current site domain and the domain of the link site match, the page information determination module 310 determines that the site is not a phishing site and extracts it (S60). Judging by the site, the procedure proceeds to step S10.

Next, the domain information determination module 320 collects information on the lifetime, age, popularity, and DNS ranking of the site domain for the extracted site by determining that it is not a phishing site through the page characteristic determination module 310. In operation S70, the collected information is digitized into characteristic values using neural networks (S80).

Subsequently, the list storage module 330 extracts, as a white list, a site having a high input value for the domain characteristic item based on the numerical value through the domain information determination module 320, and extracts the URL and IP of the extracted site. Store the information (S90). Thereafter, by performing the procedure to step S10, it may be repeated.

As described above and described with reference to a preferred embodiment for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described as described above, it is a deviation from the scope of the technical idea It will be understood by those skilled in the art that many modifications and variations can be made to the invention without departing from the scope of the invention. Accordingly, all such suitable changes and modifications and equivalents should be considered to be within the scope of the present invention.

1 is an overall configuration diagram conceptually showing a white list building system according to the present invention;

2 is an exemplary view showing a multithreaded crawler model through a site extraction unit.

3 is an exemplary view showing a phishing site according to the present invention.

Figure 4 is an exemplary view showing the results of information search to calculate the age and age of a particular domain.

5 is an exemplary view showing domain popularity search results of a specific bank using a search engine site.

6 is an exemplary view showing a specific domain ranking ranking using the ranking site.

7 is an overall flowchart of a whitelist building method according to the present invention;

** Description of symbols for the main parts of the drawing **

100: crawling unit 200: site extraction unit

300: white list extraction unit 210: site management module

220: similarity calculation module 310: page information determination module

320: domain information determination module 330: list extraction module

Claims (10)

In the white list building system, Crawling unit 100 for searching various sites on the web; A site extracting unit (200) for extracting related sites by determining whether each site is searched through the crawling unit (100), whether the site is related to a specific field; And A whitelist for a site extracted from the site extracting unit 200, determining whether or not it is a phishing site using domain information of the corresponding site, and extracting a whitelist by digitizing the collected information when the site is not a phishing site. Extracting unit 300; Including, The site extraction unit 200, A site management module 210 for receiving and storing a site searched through the crawling unit 100; And A similarity calculation module 220 for calculating a similarity related to a specific field through [Equation 1] to determine whether each site stored in the site management module 210 is a site related to a specific field; Including, The similarity calculation module 220, It is determined whether the similarity value is not 0, and if it is not 0, it is determined that the site is related to a specific field and extracted. If it is 0, it is determined that the site is not related to a specific field. And receiving a site and performing a similarity calculation, and recording a similarity score and a corresponding link of a site determined to be a specific field related site and transmitting the same to the site management module 210. [Equation 1] The cosine similarity (sim (q, p)) of page p and query q is

Where Vq and Vp are the term frequencies based on the vector representation of the query q and page p, respectively, and VqVp is the dot product of the two vectors, | v || Is the Euclidean gambling of the vector V. delete delete delete The method of claim 1, The white list extraction unit 300, A page information determination module 310 for determining whether a current site domain matches a domain of a link site, targeting a site extracted from the site extraction unit 200; Collecting information on the lifetime, age, popularity and DNS ranking of the site domain for the extracted site determined to be not a phishing site through the page characteristic determination module 310, using Neural Networks (Neural Networks) A domain information determination module 320 for digitizing the collected information into characteristic values; And A list storage module 330 for extracting and storing a white list based on the numerical value through the domain information determination module 320; White list building system comprising a. Claim 6 was abandoned when the registration fee was paid. The method of claim 5, The page information determination module 310, And if the domain of the current site domain and the link site match, determine that the site is not a phishing site, and if not, determine that the site is a phishing site. The method of claim 5, The list storage module 330, And extracting a site having an input value of a predetermined value or more with respect to the domain characteristic item as a white list, and storing the extracted site URL and IP information. delete delete delete

Images