TWI634769B - Method for detecting domain name transformation botnet through proxy server log - Google Patents

Abstract

The invention relates to a method for detecting a domain flux botnet through a proxy server log, and the website analysis module performs the following steps, including: accessing the proxy server log data; and then filtering the algorithm according to a node. Filtering out the regular user agent connection in the log data; and grouping the remaining records according to a website grouping algorithm to obtain the associated collection between the external websites in the log content; obtaining the connection topology of the external website And use a link feature algorithm to determine whether the set conforms to the link characteristics of the domain flux; and filter out the remaining records in the domain connected to the normal application or Content Delivery Network (CDN), and the rest of the network The domain is judged as the domain flux botnet.

Description

Method for detecting domain name transformation botnet through proxy server log

The present invention relates to a method for detecting a malicious network through a proxy log, and more particularly to a method for detecting a domain flux botnet through a proxy server log.

The organization and analysis of the proxy server log data is the basis of today's information security control. With the advent of the huge data era, how to extract valuable information from a huge amount of log data is the technical staff of the field. The main subject, if it can develop a cluster that can screen out malicious domains, and further lock the victim host within the enterprise or company, it will have obvious benefits for enterprises or companies that need to carry out strict security control.

For example, the Republic of China Patent Publication No. I455546, which is a method and system for detecting a malicious domain of rapidly changing domain technology, mainly uses router host name and network address autonomous system number and the like included in the router information. With the same name of the specific part of the router host, or the network packet transmission time is greater than the preset one to determine whether it is a malicious domain, however, its accuracy is inaccurate and has a normal application. The program error reported the possibility of a malicious domain.

In addition to the previous technology, the domain has also developed a network search engine and the WHOIS website to obtain the domain name related domain. Collecting results to find suspicious websites with only a small number of search results, and finally determining whether it is a technical method of the relay station domain name of the suspicious botnet based on the collection of related network domains and the number of search results, but it still cannot accurately filter out the domain name. Domain flux botnet, domain flux technology, if applied to the botnet, is a way to achieve multiple domain names associated with the same IP address through existing DNS services or domain generation algorithms, thereby escaping The technology of Uniform Resource Locator (URL) detection.

In view of the various malicious website protection methods extended by the proxy server log data, there are still many possibilities in response to the ever-changing technology, which can be improved for each problem, and a screening method for the domain flux botnet. It is the need of people in the current field.

The present invention proposes a method for detecting a domain flux botnet through a proxy server log, because the attacker often uses domain flux technology to avoid being easily used in order to improve the survival rate of the botnet (Botnet). The seizure is further blocked, but since the behavior of the malicious program connecting to a specific external website is recorded in the log data of the proxy server in detail, the present invention is made by analyzing the log data of the proxy server and communicating through the domain. The result of the set is to obtain a method that is consistent with the domain flux connection behavior and that the behavior may be a domain flux botnet.

The invention discloses a method for detecting a domain flux botnet through a proxy server log, which mainly performs a plurality of steps through a website analysis module, firstly, the website analysis module accesses the proxy server log data. And filtering the connection status of the user-agent information representative in each log data according to a node filtering algorithm, so as to belong to the common The log content of the website connection is filtered out; wherein the node filtering algorithm uses the node dimension value of the user agent to filter the famous website. In detail, this step is summarized by mapping (MapReduce) The architecture uses the terminal URL of the external website as the key (Key), and the user agent acts as a value (Value), which can efficiently obtain a list of the times each external website is linked by different user agents, by filtering the dimension values in the list. A large website can initially filter out relatively well-known websites. The present invention retains an external website whose dimension value is less than a predetermined threshold (for example, a threshold of 10) in long-term traffic information, and is intended to be filtered out. In addition to the more famous websites, only the external websites that are only connected by a very small number of user-agents (User-agents) usually indicate that they have special uses. These external websites have a high probability of being a malicious relay station.

The next step is that the website analysis module matches the external client's internet protocol address (Client IP) in the connection record of the remaining log content in the previous step according to a website grouping algorithm. And the user agent connection record, to find out the associated collections between the external websites in the log content by grouping; the purpose of this step is to link the external websites through the information provided by the traffic information in the log. Sexuality, which is intended to treat external websites linked to by the same program as related websites; in detail, the present invention divides external websites having the same Client IP and user agent connection records into the same collection. In the meantime, it means that these websites are related to each other, and because of the characteristics of the set of external websites (Client IP and user agent), it can be judged by whether the contents of the log records are Exact Match. Therefore, this kind of website grouping algorithm can also be implemented through the key-value architecture in MapReduce.

The website grouping algorithm can be grouped by the method of association search, that is, the connection module of the website analyzes the remaining log content. After establishing a set of records, the following steps are performed: the individual connection records in each set are used as elements, and if the external sites connected to the connection record have the same Client IP and user agent, the intersection is judged and the elements are The set of intersections is merged; the set of elements in the set is greater than the preset threshold; the set of disjoint sets with the remaining sets is determined as a separate set; and the above three steps are repeated until all the sets are judged as independent sets.

Following the steps, the website analysis module obtains the connection record of the Client IP in the log content and the connection topology of the client IP and the external website of the connection; wherein the present invention uses the above steps to set the URLs of the disjoint sets After the collection records are classified, the connection records of the same collection will only be linked to the same set of domains. The meaning of the connection records of the same collection is generated by programs with the same purpose. Out of the network behavior, this step will get the connection topology of the Client IP and the external URL, and obtain the connection information of the Client IP in the proxy server log, for example, the hash of the URL string of the disjoint collection (hash) The method obtains the group code, and cooperates with the connection information of each website recorded in the connection record, and uses the multiple input mechanism of MapReduce to use the URL as the key to compare the connection record set corresponding to the group code and the group. The connection topology of the group.

Then, the website analysis module presents the relationship between the external website and the associated collection through the connection topology, and uses a connection feature algorithm to determine whether the collection conforms to the connection characteristics of the domain flux, wherein the so-called domain The connection feature of flux means that the connection is connected to many different external URL domains through a small number of Client IPs and the same User-agent; in practice, the connection feature algorithm is It is used to calculate the ratio of the number of terminal URLs in a set to the number of Client IPs. If the calculated ratio exceeds a preset threshold, it means that the website of the set meets the domain change line of the domain flux connection feature. for.

Finally, the website analysis module extracts the collections that conform to the domain flux connection characteristics, and filters out the representative and normal network applications (for example, anti-virus software) or content delivery network (CDN). Connected to the domain, the website analysis module can identify the remaining domains as domain bots and alert them.

The above is the method for detecting the domain flux botnet through the proxy server log, which can mine the suspected domain in the log data of the huge long-term period through the characteristics of the domain flux botnet. The external website of the flux botnet for further precautions.

A‧‧‧ computer

B‧‧‧ computer

C‧‧‧ computer

E‧‧‧ computer

F‧‧‧ computer

G‧‧‧ computer

H‧‧‧ computer

1‧‧‧ Attacker

2‧‧‧ Attackers

10‧‧‧ cluster

11‧‧‧ Relay Station

12‧‧‧ Relay Station

13‧‧‧Relay station

20‧‧‧ cluster

21‧‧‧ Relay Station

22‧‧‧Relay station

23‧‧‧ Relay Station

30‧‧‧Relay station

S201~S205‧‧‧ method steps

FIG. 1 is a schematic diagram of a scenario for detecting a domain flux botnet through a proxy server log according to the present invention.

2 is a flow chart of a method for grouping external websites by proxy logs according to the present invention.

FIG. 3 is a schematic diagram showing an example of a statistical list of the number of times a user agent is connected.

FIG. 4 is a schematic diagram showing an example of converting a simple log data into an external website collection according to the present invention.

FIG. 5 is a schematic diagram of an example of a set back push network connection group according to the present invention.

6 is a schematic diagram showing an example of a network behavior topology diagram of a domain flux according to the present invention.

FIG. 7 is a schematic diagram showing an example of an antivirus software performing a connection behavior through a domain flux method.

Figure 8 is a schematic diagram showing an example of proxy server log data.

Figure 9 is a schematic diagram showing an embodiment of screening a domain flux botnet in accordance with the method of the present invention.

Figure 10 is a schematic diagram showing an embodiment of screening a domain flux botnet in accordance with the method of the present invention.

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

FIG. 1 is a schematic diagram of a scenario for detecting a domain flux botnet through a proxy server log. The attacker can embed a malicious program into a victim host to make it a zombie computer (Bots), and the zombie computer can steal. Malicious behavior such as confidential or sensitive data, and today's attackers use domain flux technology to avoid checking for the survival rate of the botnet, but the behavior of malicious programs connected to specific websites will be recorded in the agent. In the server log data, the present invention can still be parsed out. In Figure 1, the attacker 1 uses a malicious program to turn the computer A and the computer B and the computer C in the target enterprise internal network into zombie computers, computer A and The malicious programs in computer B and computer C are connected to the relay station 11, the relay station 12, and the relay station 13, and the computer C connects to the famous website when performing normal connection; in addition, the attacker 2 uses the malicious program to target the internal network of the enterprise. Computer D and Computer E and Computer H become zombie computers, computer A and computer B and computer C in the evil The program connects to the relay station 21, the relay station 22, and the relay station 23, and the computer D connects to a famous website when performing normal connection; wherein the computer C and the computer D are connected to the relay station and connected to the website; in addition, the famous website 30 It is connected to the computer C, computer D, computer E, computer F and computer G that execute the normal application. The computer and the relay station have a staggered connection relationship, and the botnet avoids the general inspection through the domain flux. The method of the present invention can efficiently trace the suspected botnet network domain through the records of the computer A, the computer B, the computer E and the computer H which become zombies in the log data of the Proxy Server.

2 is a flow chart of a method for grouping external websites by proxy logs. First, step S201 is to filter a famous website by using a user agent dimension value. Through the MapReduce architecture, the website analysis module obtains each external part of the log. A list of the number of times the website is linked by different user agents. If the website with too large a dimension value is filtered, the famous website can be filtered out first. As shown in Figure 3, it is an example of a statistical list of the number of times the user agent is connected. It can be observed that most of the websites have only records that have been connected by a group of user agents (92.77%), while the famous websites such as www.google.com are connected by 22,825 different user agents. Through the above observations, it can be known that the present invention can be initially screened for a famous website by checking the external website for different user agent connection records. As discussed in the example of Figure 1, the well-known website 30 will be considered first in the step of S201.

Then, please continue to refer to FIG. 2, step S202 is to obtain an association set of the external website through the association search, mainly to convert the proxy server log data into a form of an external website collection; the present invention is based on the website analysis mode. The group divides the external websites with the same Client IP and user agent connection records into the same collection, which means that the websites are related, and they can be transparent. MapReduce's Key-value architecture is implemented; please refer to Figure 4, which is a schematic diagram of a simple log data converted into an external website collection. It can be observed in Figure 4, which is based on Client IP and user agent. The key, and the URL of the terminal (Dest Url) is used as a value for the joint search. Finally, the results of the sample log data through the association search are "CnC1, CnC2, CnC3" and "CnC4, CnC5, CnC6". A disjoint set, and a detailed method of how the union lookup is implemented, will be explained in the following paragraphs.

Next, step S203 is to push back the network connection group, wherein the website analysis module groups the connection records through the mutually disjointed address sets, and the connection records of the same group are only connected. To the same set of domains, the connection records of the same group represent the meaning of the network behavior generated by the program of the same purpose; in this step, the website analysis module will obtain the Client IP. The connection with the external URL is topological, and the connection information of the Client IP in the proxy server log is obtained; as shown in the example diagram of the set back push network connection group in FIG. 5, the website analysis mode of the present invention After the two disjoint sets of "CnC1, CnC2, CnC3" and CnC4, CnC5, CnC6" are obtained in a hash manner, the CnC1~CnC6 of the terminal URL is given according to the set of the group to which it belongs. Group-ID, together with the connection information recorded in the connection record CnC1~CnC6, through the MapReduce Multiple Input as the URL, you can compare the connection record group and group corresponding to the Group-ID. The connection topology is shown at the bottom of the figure.

Then, step S204 is to determine that the behavior conforms to the domain flux feature, and the website analysis module respectively presents each connected group with its network connection topology, and then determines whether the group conforms to the domain flux connection feature. Among them, the domain flux connection feature is a small number of Client IP connections to a number of external URLs, as shown in Figure 6, which is a domain flux network behavior extension In the simple example, only a few Client IPs use the same User-agent to connect multiple different domains. As shown in the figure, the same user agent is used by the Client IP of 10.107.56.20. Connect to the domain starting with sp-install, c-sp-storage, sp-download, sp-alive, sp-setting, sp-storage, Orbtr-install, spms-download, c-api.sec, etc. It conforms to a typical connection feature of a domain flux that uses a single Client IP to connect multiple different domains with the same User-agent.

Then, step S205 is to filter the legitimate behavior domain, wherein the website analysis module takes out the collection that conforms to the domain flux connection feature, and then filters out the representative and normal network application (for example, anti-virus software) or The domain to which the CDN and so on are connected; as shown in Figure 7, it is an example of a connection behavior of an antivirus software through the domain flux method, but this known normal web application behavior will not be listed. The botnet behavior to be detected by the present invention will be excluded in this step, and the remaining domains will be judged as domain bots of the domain flux; for example, in the example of FIG. 7, the "iavs9x" can be accessed. The regular expression of .u.avast.com" is added to the filter list to filter out this normal web application.

Next, as shown in FIG. 8, it is an example of a proxy server log data, wherein each row of data represents a log record of the established timestamp (Timestamp), client IP (Client IP) Information such as the terminal URL (Dest Url), the terminal port (Dest Port), and the user agent (User-agent) may additionally include a Sent Byte, a Receive Byte, a Method, and a Path ( Path) and the like, and the present invention mainly uses the client IP, terminal URL and user agent information to detect the domain flux botnet, and judges whether the website is a famous website by the size of the node dimension value. Low representation is only external to a small number of user agents The website has a higher chance of being a malicious relay station.

In step S202, the website grouping algorithm is used to merge the collections with the intersections in the external website collection, thereby generating disjoint sets. The steps of the association search method are as follows: (1) searching for the elements in the collection as a unit The intersections of the collections are merged, and the collections with intersections are merged into one collection (the collection size exceeds the preset threshold and filtered in this step); (2) Determine which collections are disjoint collections (no intersection with other collections) Separate them and the rest of the collections go back to step (1); (3) repeat steps (1) and (2) until all the collections are independent. Taking Figure 4 as an example, in the MapReduce architecture, in Item 1~Item 8, with Client IP and user agent as the key, and the terminal URL as the value, you can first cluster four sets, which are Item 1 and Item 2 respectively. Clustered out CnC1, CnC2 value, Item 3 and Item 4 will cluster CnC2, CnC3 value, Item 5 and Item 6 will cluster CnC4, CnC5 value, Item 7 and Item 8 will cluster CnC4, CnC6 value, These four sets can be used as input sets for re-association, and the final result of the union is "CnC1, CnC2, CnC3" and two sets of disjoint URLs (CnC4, CnC5, CnC6); then, According to the example of FIG. 5, after the present invention proceeds to step S203, a connection topology of the two sets is obtained.

The present invention is based on the same Client IP connection to a number of different terminal URLs, and the one-to-many relationship presented conforms to the connection topology of the domain flux connection feature to determine the botnet. For example, it can be based on The value of the terminal URL in a set, divided by the result value of the client IP value, exceeds a certain preset threshold, and is used as a basis for judging the change behavior of the domain conforming to the domain flux connection feature.

Finally, as shown in FIG. 9 and FIG. 10, it is a schematic diagram of an embodiment of screening a domain flux botnet according to the method of the present invention. After the process shown in FIG. 2 of the present invention, the obtained group 739663648 conforms to the characteristics of the domain flux botnet collection, and further, the domain on the group is queried on the Virus Total website, as shown in the figure. As shown in FIG. 10, it can be observed that the entire group of domains obtained by the present invention are registered as malicious websites, so that the method of detecting the domain flux botnet through the proxy server log of the present invention is effective.

The detailed description of the preferred embodiments of the present invention is intended to be construed as the invention The patent scope of this case.

In summary, the present invention is innovative in terms of technical ideas, and also has various functions that are not in the prior art, and has fully complied with the statutory invention patent requirements of novelty and progressiveness, and has filed a patent application according to law, and invites you to approve the invention. The patent application was inspired to invent, and it was a matter of feeling.

Claims (4)

A method for detecting a domain flux botnet through a proxy server log, wherein the website analysis module performs the following steps, including: accessing at least one log data stored by the external proxy server; The filtering algorithm filters the connection status of the user-agent information representative in each log data, and filters the log content belonging to the connection with the regular website; according to a website grouping algorithm, the remaining log content In the connection record, match the external client with the same client Internet Protocol address (Client IP) and user agent connection record, and perform grouping to obtain the associated collection between the external websites in the log content; The connection record of the Client IP in the content and the connection topology of the Client IP and the connected external website; the relationship between the external website and the associated collection is presented through the connection topology, and judged by a connection feature algorithm Whether the set conforms to the connection characteristics of the domain flux, wherein the connection feature of the domain flux refers to the use of the same user agent for the Client IP. Domain connection; and take out the collection that conforms to the domain flux connection feature and filter out the domain that is connected to the normal web application or content delivery network (CDN). The remaining domains are Determined as a domain flux botnet. The method for detecting a domain flux botnet through a proxy server log according to claim 1, wherein the node filtering algorithm is performed by the website analysis module according to each of the long-term periods. The terminal uniform resource locator (URL) in the log data is used as a key, and the user agent information is used as a value. The mapReduce architecture is used to calculate and obtain each external website in the log data. A list of the number of connections of different user agents. The number of times that each external website is connected by a different user agent is equal to the dimension value (Degree) on the node. By the size of the dimension value, it can be determined whether the external website represented by the node is a famous website. Only external sites with dimension values less than the preset threshold will be retained. The method for detecting a domain flux botnet through a proxy server log according to the first aspect of the patent application, wherein the website grouping algorithm is grouped by a cluster search method, that is, the website analyzes After the module establishes several sets of connection records of the remaining log contents, the following steps are performed: the individual connection record data in each set is taken as an element, and if the external website of the connection record connection has the same Client IP and user agent , judging as an intersection, and merging the set of elements with intersection; deleting the set of elements in the set greater than the preset threshold; determining the set of disjoint sets with the remaining sets as independent sets; Repeat the above three steps until all the sets are judged as independent sets. The method for detecting a domain flux botnet through a proxy server log according to claim 1, wherein the connection feature algorithm is used to calculate the number of terminal URLs in a set and the client. The ratio of the number of IPs, if the ratio exceeds the preset threshold, represents the behavior of the domain that matches the domain flux connection characteristics of the aggregated website.