CN108337255B - Phishing website detection method based on web automatic test and width learning - Google Patents

Abstract

The invention discloses a phishing website detection method based on web automatic test and breadth learning, and belongs to the technical field of computer network safety. According to the invention, traditional feature extraction is carried out based on url and html pages, interactive feature extraction is carried out by using a web automatic testing technology, and finally width learning training is carried out by using a preprocessed training sample after feature extraction, so that phishing websites are accurately and quickly identified and detected, and the network information and property safety of people are protected.

Description

A phishing website detection method based on web automated testing and breadth learning

technical field

The invention belongs to the technical field of computer network security, and more particularly relates to a phishing website detection method based on web automatic testing and breadth learning.

Background technique

Phishing is an attack method that steals sensitive information such as users' personally identifiable data and financial account numbers by sending a large number of deceptive spam emails claiming to be from banks or well-known institutions, false advertisements on web pages, etc. The most typical phishing attack is to lure users to a well-designed phishing website that is very similar to the target organization's website, obtain personal sensitive information entered by users on the website or defraud users to send money. Because the victims of this type of attack are not easily alerted, phishing websites have become one of the most serious Internet crimes, and the detection of phishing websites has also become one of the most popular research directions in the field of network security.

In 2016, the National Engineering Laboratory of Internet Domain Name Management Technology led by CNNIC, the International Anti-Phishing Working Group (APWG), and the China Anti-Phishing Alliance (APAC) jointly released the "Statistical Analysis Report on the Status Quo of Chinese Phishing Websites in the World (2016)" ” (hereinafter referred to as the “Report”). The data shows that in 2016, the number of phishing websites in my country increased by 150.96% year-on-year. The main counterfeit objects are Taobao, China Mobile, and major banks. The domain names used are mainly .COM, .CC, .PW, and .NET.

In the third quarter of 2017, 360 Mobile Guard blocked 790 million phishing websites for mobile phone users nationwide, an increase of 102.6% over the third quarter of 2016. The intercepted mobile phone phishing websites were classified, among which gambling and gambling phishing websites accounted for 80.2% of the total, and the proportions of false shopping, false recruitment, financial securities, counterfeit medicines and phishing advertisements decreased in order.

Although there are many blocked websites, most of the blocked websites have existed for a long time, and it is difficult to catch and block the latest phishing websites. The average life cycle of phishing websites is only 4.684 days, while the average reporting cycle is 13.327 days. For phishing websites, they must be identified and blocked in a very short period of time, otherwise it will pose a threat to people's property safety.

At present, the identification and blocking technologies for phishing websites are performed by antivirus software and browsers themselves. The technologies are divided into the following categories:

①Blacklist filtering technology: Add the phishing websites that are manually detected and reported by the public to the blacklist. When the visited URL (Uniform Resource Locator, Uniform Resource Locator) exists in the blacklist, it will be intercepted and warned. This method cannot identify the latest phishing websites and requires manual verification.

②The feature extraction of url: The corresponding features, such as domain names, are extracted through the accessed url, but this method of determination is unreliable, because the url does not have the decisive characteristics of phishing websites, and the misjudgment rate and omission rate of this method are higher.

③Combining various website page elements as features to detect phishing websites: Because it takes a certain amount of time to obtain the characteristics of web pages, this method is more accurate than the second type of method, but the speed and efficiency of execution are both not tall.

SUMMARY OF THE INVENTION

Aiming at the above defects or improvement requirements of the prior art, the present invention provides a phishing website detection method based on web automated testing and width learning, the purpose of which is to perform traditional feature extraction based on url and html pages, and use web automated testing technology to perform Interactive feature extraction, using the preprocessing training samples after feature extraction for breadth learning training, so as to accurately and quickly identify and detect phishing websites, and protect people's network information and property security.

To achieve the above object, according to one aspect of the present invention, a method for detecting a phishing website based on web automated testing and breadth learning is provided, comprising the following steps:

(1) Perform static feature extraction, dynamic feature extraction and interactive feature extraction on a large number of phishing websites and normal websites in the data set on the PC (Personal Computer, personal computer) end to form a feature vector set;

The data set comes from phishing websites and normal websites collected on the Internet, or obtained directly from network security companies;

(2) The feature vector set in step (1) is divided into a training set and a verification set by using the k-fold cross-validation method;

(3) use the training set to carry out the training of width learning, use the verification set to carry out test comparison, build a basic model and optimize the performance of the classifier;

The classifier is a model trained by the width learning algorithm. When using, input a website address in the classifier and output whether it is a phishing website; the performance of the classifier refers to the correct rate of the classifier identifying the phishing website; Performance optimization refers to improving the recognition accuracy;

(4) Collect misjudged websites and newly included websites as a new set of feature vectors, and perform incremental learning to add input to the model to optimize the model.

Preferably, step (1) is specifically:

(1.1) Perform static feature extraction on url;

(1.2) Use web automated testing technology to simulate a non-interface browser to access the url of the dataset;

(1.3) Dynamic feature extraction for pages accessed by url;

(1.4) Simulate the browser to interactively click and browse the page, and return to the interactive feature.

Preferably, the static features in step (1.1) include:

①Whether there is an IP address in the url;

②Whether the domain name of the url is a pure number from the beginning to the first point;

③Whether the url contains sensitive characters, such as @;

④Whether the url port is port 80;

⑤ Whether the length of the url is less than 23 characters;

⑥Whether the url contains keywords related to shopping or property accounts, such as account, banking, taobao;

The above six static features are recorded as <F1, F2, F3, F4, F5, F6>.

Preferably, the dynamic features in step (1.3) include:

①Whether the title (title) of html contains sensitive characters, such as 'lottery', 'overseas gambling', 'winner';

②Whether there is a form form;

③ Whether the resource (resource) of the picture is the same domain name as the original url;

④ Whether the href of the link has the same domain name as the url; the href is the abbreviation of Hypertext Reference, which is the url of the designated hyperlink target;

The above four dynamic features are marked as <F7, F8, F9, F10>.

Preferably, the interactive features in step (1.4) include:

①Whether the form is rigorous;

②Click the link, whether it is empty;

③ Click on the link, whether URL redirection occurs;

The above three interactive features are marked as <F11, F12, F13>.

Preferably, step (2) is specifically:

(2.1) Set the k value;

(2.2) Use the k-fold cross-validation method to divide the data set in step 1 into the training set and the validation set.

Preferably, step (3) is specifically:

(3.1) using the feature vector set of the webpage samples in the training set of step (2) to train the width learning model and test the performance of the classifier; the webpage samples refer to the phishing website URL and the normal website URL in the training set;

(3.2) Continuously adjust the network architecture by adding feature nodes and enhanced nodes for training and testing until the classifier achieves the expected performance, obtain the weight information of each layer and save the model.

Preferably, step (3.1) is specifically:

(3.11) Initialize the number of feature windows N2, the number of feature nodes in the window N1, and the value of the number of enhanced nodes N3; randomly initialize the weight matrix of the feature nodes of the classifier model, and use the sparse auto-encoding to process the weight of the feature nodes;

(3.12) Perform matrix multiplication between the feature vector set of the webpage sample and the weight matrix obtained in step (3.11) to obtain a feature node matrix;

(3.13) Randomly initialize the enhanced node weight matrix;

(3.14) Multiply the feature node matrix obtained in step (3.12) and the weight matrix obtained in step (3.13) to obtain an enhanced node matrix;

(3.15) The feature node matrix obtained in step (3.12) and the enhanced node matrix obtained in step (3.14) are horizontally spliced by column to obtain an input matrix;

(3.16) Obtain the plus generalized inverse of the input matrix obtained in step (3.15) and perform matrix multiplication with <Y> to obtain a weight matrix; the <Y> is a matrix composed of labels of web page samples; the labels of the web page samples Indicates whether it is a phishing website or not, for example, label 1 means it is a phishing website, and label 0 means it is not a phishing website;

(3.17) Since step (2) is k-fold cross-validation, step (3.1) is repeated k times to average the accuracy of k times;

(3.18) Gradually increase the values of N1, N2, and N3, observe whether the accuracy of the width model is improved, and find the optimal parameters.

Preferably, step (3.2) is specifically:

(3.21) Use the incremental learning method of increasing the number of feature nodes and enhancing the number of nodes to adjust and test the model obtained in step (3.1);

(3.22) Set the number of loops in step (3.21) and record the obtained test accuracy, compare and determine the optimal number of feature nodes and the number of enhancement nodes, and save the optimal model.

Preferably, step (4) is specifically as follows: collecting misjudged websites and newly included websites as a new set of feature vectors, performing incremental learning of increasing input to the model, and obtaining an adjusted weight matrix, thereby realizing the optimization model;

Preferably, in step (1), a regular expression is used to extract the static features of the url, and Phantomjs is used to simulate a non-interface browser to perform an automated UI test without an interface; the PhantomJS is a webkit-based JavaScript API, which is a non-interface browser. .

The invention uses the url static feature extraction, html dynamic feature extraction and the interactive feature extraction technology of web automatic testing technology, firstly extracts the static feature of the url; Code extraction, extracting dynamic features from html; simulating link clicks, simulating account input and login in form forms, eliminating the process of page rendering, and quickly extracting interactive features; for new phishing websites that are not in the blacklist, by simulating Click the link to test whether the link is empty; simulate the login form to test whether the form is normal; simulate clicking the link to test whether there is url redirection. Quickly and accurately detect phishing sites with these new interactive features.

The invention uses the width learning model to improve the ability to identify new fishing websites. Breadth learning is a new machine learning method and idea. Different from deep learning, breadth learning has a shallow architecture and requires less computing resources. In addition, deep learning needs to re-improve the entire model when accepting new samples, which takes a lot of time, but the breadth learning algorithm does not need to re-train the original model, only the newly added fishing Feature extraction is performed on website samples, and the existing models are adjusted and supplemented, and the detection accuracy is continuously improved in self-update.

In general, compared with the prior art, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

1. According to the method provided by the present invention, first perform static feature extraction based on url, then perform dynamic feature extraction based on html page, and then use web automated testing technology to simulate browser to perform interactive feature extraction on the page, and the features are from static to dynamic. , and then to interactive feedback, feature mining proceeds from shallow to deep, ensuring the quantity and quality of features; finally, using the breadth learning model requires only a small amount of resources, fast training and incremental learning features, to achieve accurate, fast and adaptive Identification technology for phishing websites;

2. Interactive features combined with url static features and html dynamic feature extraction can greatly improve the identification accuracy of phishing websites, and are suitable for the latest phishing websites, which can accurately and quickly detect and identify phishing websites;

3. The feature extraction of newly added phishing website samples through breadth learning, and the adjustment and supplement of the existing model, the time required for updating the model is greatly reduced, and the requirement for computing resources is low; at the same time, the detection accuracy of breadth learning is improved. Able to achieve continuous self-improvement in self-renewal.

Description of drawings

Fig. 1 is the general flow chart of a kind of phishing website detection method based on web automated testing and width learning in the preferred embodiment of the present invention;

2 is a schematic diagram of feature extraction of a phishing website in a preferred embodiment of the present invention;

3 is a schematic diagram of preparing a k-fold cross-validation data set for width learning in a preferred embodiment of the present invention;

FIG. 4 is a schematic diagram of training and optimization of width learning in a preferred embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

The present invention provides an intelligent detection method for phishing websites based on web automated testing and breadth learning. As shown in FIG. 1 , it is the main flowchart of the present invention, which clearly shows the relationship between the process and steps of the entire invention. The following specifically describes the implementation of the steps:

(1) Step 1: Perform static feature extraction, dynamic feature extraction and interactive click access on a large number of phishing websites and normal websites in the dataset on the PC side.

As shown in Figure 2, step 1 is as follows:

Step 1.1, perform static feature extraction on the url itself, including the following six features:

①Whether there is an IP address in the url: IP addresses can be used to evade domain name registration and user inspection;

②Whether the domain name of the url is pure numbers from the beginning to the first point: Regular URLs rarely use pure numbers to insert domain names, such as regular URL Baidu https://www.baidu.com/, phishing website http://www.030033. com/;

③Whether there are sensitive characters such as @ in the url: the front of the @ character is the account, and the back is the real address, which is often used by phishing websites;

④Whether the url port is port 80: regular urls are accessed through port 80, and non-port 80 is suspected of being a phishing website;

⑤Whether the length of the url is less than 23 characters: According to statistics, the url of a regular website generally does not exceed 23 characters;

⑥Whether the url contains keywords such as account, banking, taobao, etc.: When it comes to shopping, the bank should be alerted;

The above six features use six regular expression pairs of url to match and extract corresponding features, for example, re.match(r'.*?//(.*)/.*',url) combined with split('.') can Extract the domain name from the url. Other features can be extracted in the same way. If the above features appear, return 1, otherwise return 0, forming a total of six feature sets <F1, F2, F3, F4, F5, F6>.

In step 1.2, the access to the url of the browser without an interface is realized through the code, the rendering process of the browser page is omitted, and time is saved for the subsequent feature extraction. The specific code is as follows:

self.dirver=webdriver.PhantomJS()

dirver.get('http://www.douyu.com/directory/all')

Step 1.3, through HTML (HyperText Markup Language, hypertext markup language) source code for dynamic feature extraction for the page accessed by the url, including the following four features:

①Whether the title (title) of html contains sensitive characters such as 'lottery', 'overseas gambling', 'winner', some illegal phishing websites like to take advantage of netizens' desire for profit;

②Whether there is a form form: The form is also a sensitive feature, because the ultimate purpose of the phishing website is to steal the account password;

③ Whether the resource (resource) of the picture is the same domain name as the original url: because phishing websites often steal pictures from other genuine websites;

④ Whether the href of the link has the same domain name as the url: because the phishing website does not write the content of the article itself, most of it is a chain; href is the abbreviation of Hypertext Reference, which is the URL that specifies the hyperlink target.

The above four features use the driver object in step 1.1, and use the driver to parse these four features from the source code, such as driver.find_element_by_tag_name("title").text, extract the content of the title. Other features can be extracted in the same way. If the above features appear, return 1, otherwise return 0, which constitutes a total of four feature sets <F7, F8, F9, F10>.

Step 1.4, interactive click access to the page itself, with the following three features:

①Whether the form is rigorous: whether the random account password can be logged in: because the phishing website usually does not have the original database, generally any account password can be logged in;

②Click on the link, whether it is empty: If most of the links are empty links, the possibility of this website being a phishing website is higher;

③ Click on the link, whether URL redirection occurs: the life cycle of phishing websites is short, and URL redirection is often used to jump to the unsealed IP address;

The above three features can use the driver to simulate the browser to click the page in real time, so as to extract the features according to the return value, such as elem. If it is successful, it can be predicted that it is a phishing website with a high probability. Other features can be extracted in the same way. If the above features appear, return 1, otherwise, return 0, which constitutes a total of three feature sets <F11, F12, F13>;

(2) Step 2: k-fold cross-allocation training set and validation set, as shown in Figure 3, step 2 is as follows:

Step 2.1, set the k value: the parameter k represents the number of repeated training and testing, which is proportional to the resources consumed;

Step 2.2, use k-fold cross-validation to separate the training set and the validation set to train the model, divide the original data set into k parts, k-1 as the training set and 1 as the validation set, and repeat k times to improve The generalization ability of the model.

Figure 4 is a schematic diagram of training and optimization of width learning. For details, see steps 3 and 4 below:

(3) Step 3: Train the width model and optimize itself. Step 3 is as follows:

Step 3.1, use the phishing website sample feature vector set obtained in step 1 to train the width learning model and test the performance of the classifier; specifically, the following steps are included:

Step 3.1.1, initialize the number of feature windows N2, the number of feature nodes in the window N1, and the value of the number of enhanced nodes N3, according to the experience of many experiments, initialize N1*N2=samples/600, N3=samples/10, samples represents the number of samples ; Randomly initialize the classifier model feature node weight matrix We, and use sparse auto-encoding to process the feature node weights;

Step 3.1.2, perform matrix multiplication with the weight matrix We obtained in step 3.1.1. to obtain the feature node matrix Z;

Step 3.1.3, randomly initialize the enhanced node weight matrix Wh;

Step 3.1.4, multiply the feature node matrix Z obtained in step 3.1.2 with the weight matrix Wh obtained in step 3.1.3 to obtain an enhanced node matrix;

Step 3.1.5: The input matrix A is obtained by horizontally splicing the feature node matrix Z obtained in step 3.1.2 and the enhanced node matrix H obtained in step 3.1.4 by column;

Step 3.1.6, obtain the plus generalized inverse of the input matrix A obtained in step 3.1.5 and perform matrix multiplication with the applied training sample label set to obtain the weight matrix W. The specific code is as follows:

W=np.linalg.inv((A.T).dot(A)+lamda*

np.eye((A.T).shape[0])).dot((A.T).dot(train_y));

Step 3.1.7, since step 2 is k-fold cross-validation, repeat step 3.1 k times to average the accuracy of k times to enhance the generalization ability of the classifier;

Step 3.1.8, according to the experimental experience, adjust the N1, N2, N3 parameters, obtain the highest accuracy at the peak value, and save the values of these three parameters.

Step 3.2, adjust the network architecture by increasing the number of nodes in the matrix for training and testing until the classifier achieves the expected performance or the adjustment reaches a certain number of times, and obtains the weight information of each layer in the optimal case and saves it;

Step 3.2 specifically includes the following processing:

Step 3.2.1, use the incremental learning method of increasing the number of feature nodes and enhancing the number of nodes to adjust and test the model obtained in step 3.1;

Step 3.2.2, set the number of loops to perform step 3.2.1 and record the obtained test accuracy, compare and determine the optimal number of feature nodes and the number of enhancement nodes, and save the optimal model.

(4) Step 4, collect misjudged websites and newly included websites as a new feature vector set, and carry out incremental learning of increasing input to the model in a timely manner;

Step 4 specifically includes the following processing:

4.1, perform feature extraction and save for the examples that fail to be classified in step 3;

4.2, perform feature extraction for the new website and save it in a local file;

4.3, when the number reaches the preset value, a unified input-based incremental learning is performed, and the W weight matrix is adjusted to update the model;

To sum up, using the technical solution provided by the present invention, the intelligent detection method for phishing websites based on web automated testing and breadth learning combines traditional features with interactive features, uses breadth learning to train the model, consumes less resources, and realizes rapid self-development. It adapts to the update while ensuring the accuracy of the model, and can achieve precise interception and strike in the extremely short life cycle of phishing websites.

The k-fold cross-validation method described in the present invention is to scramble the samples, and then evenly divide them into k parts, select k-1 parts in turn for training, the remaining part for verification, calculate the sum of the squares of the prediction errors, and finally put the k-folds for training. The sum of squared prediction errors is averaged as the basis for selecting the optimal model structure. Suppose there are N samples, and the special k is N, which is the leave oneout method.

Assuming that the matrix A is an m*n matrix, the plus-sign generalized inverse of the matrix A in the present invention refers to (A'A)A', where A' represents the transposed matrix of A.

The weights mentioned in the present invention refer to the parameters of the width learning model.

The sparse autoencoder refers to a technology that automatically learns features from unlabeled data and provides a better feature description than the original data.

Those skilled in the art can easily understand that the above are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, etc., All should be included within the protection scope of the present invention.

Claims (9)

1. a phishing website detection method based on width learning, is characterized in that, comprises the steps: (1) Perform static feature extraction, dynamic feature extraction and interactive feature extraction on a large number of phishing websites and normal websites in the website dataset on the PC side to form a feature vector set; (2) The feature vector set in step (1) is divided into a training set and a verification set by using the k-fold cross-validation method; (3) Use the training set to carry out the training of breadth learning, use the verification set to test and compare, build a basic model and optimize the performance of the classifier; the performance of the classifier refers to the correctness of the classifier to identify the phishing website. Rate; (4) Collect misjudged websites and newly included websites as a new set of feature vectors, and perform incremental learning to increase the input of the model to optimize the model; Step (1) is specifically: (1.1) static feature extraction is carried out for url; the static feature includes: whether the domain name of url is a pure number from the beginning to the first point, and whether the length of the url is less than 23 characters; (1.2) Use web automated testing technology to simulate a non-interface browser to access the url of the dataset; (1.3) Dynamic feature extraction is performed for the page accessed by the url; the dynamic features include: whether the resource of the picture has the same domain name as the original url; (1.4) Simulate the browser to interactively click and browse the page, and return to the interactive feature. 2. a kind of phishing website detection method based on width learning as claimed in claim 1, it is characterised in that the static feature described in step (1.1) also comprises: ①Whether there is an IP address in the url; ③Whether there are sensitive characters in the url, and the sensitive characters include @; ④Whether the url port is port 80; ⑥ Whether the url contains keywords related to shopping or property accounts, the keywords include account, banking, taobao. 3. a kind of phishing website detection method based on width learning as claimed in claim 1, is characterized in that, described in step (1.3), dynamic feature also comprises: ①Whether the title of html contains sensitive characters, the sensitive characters include 'lottery', 'overseas gambling', 'winner'; ②Whether there is a form form; ④ Whether the href of the link is the same domain name as the url; the href is the abbreviation of Hypertext Reference, which is the url of the designated hyperlink target. 4. a kind of phishing website detection method based on width learning as claimed in claim 1 is characterized in that, the interactive feature described in step (1.4) comprises: ①Whether the form is rigorous; ②Click the link, whether it is empty; ③ Click on the link to see if URL redirection occurs. 5. a kind of phishing website detection method based on width learning as claimed in claim 1, is characterized in that, step (2) is specially: (2.1) Set the k value; (2.2) Use the k-fold cross-validation method to divide the data set in step (1) into the training set and the validation set. 6. a kind of phishing website detection method based on width learning as claimed in claim 1, is characterized in that, step (3) is specially: (3.1) Use the feature vector set of the web page samples in the training set of step (2) to train the width learning model and test the performance of the classifier; (3.2) Continuously adjust the network architecture by adding feature nodes and enhanced nodes for training and testing until the classifier achieves the expected performance, obtain the weight information of each layer and save the model. 7. a kind of phishing website detection method based on width learning as claimed in claim 6 is characterized in that, step (3.1) is specially: (3.11) Initialize the number of feature windows N2, the number of feature nodes in the window N1, and the value of the number of enhanced nodes N3; randomly initialize the weight matrix of the feature nodes of the classifier model, and use the sparse auto-encoding to process the weight of the feature nodes; (3.12) Perform matrix multiplication between the feature vector set of the webpage sample and the weight matrix obtained in step (3.11) to obtain a feature node matrix; (3.13) Randomly initialize the enhanced node weight matrix; (3.14) Multiply the feature node matrix obtained in step (3.12) and the weight matrix obtained in step (3.13) to obtain an enhanced node matrix; (3.15) The feature node matrix obtained in step (3.12) and the enhanced node matrix obtained in step (3.14) are horizontally spliced by column to obtain an input matrix; (3.16) Obtain the plus generalized inverse of the input matrix obtained in step (3.15) and perform matrix multiplication with <Y> to obtain a weight matrix; the <Y> is a matrix composed of labels of web page samples; the labels of the web page samples represent a phishing website or not; (3.17) Since step (2) is k-fold cross-validation, step (3.1) is repeated k times to average the accuracy of k times; (3.18) Gradually increase the values of N1, N2, and N3, observe whether the accuracy of the width model is improved, and find the optimal parameters. 8. a kind of phishing website detection method based on width learning as claimed in claim 6 is characterized in that, step (3.2) is specially: (3.21) Use the incremental learning method of increasing the number of feature nodes and enhancing the number of nodes to adjust and test the model obtained in step (3.1); (3.22) Set the number of loops in step (3.21) and record the obtained test accuracy, compare and determine the optimal number of feature nodes and the number of enhancement nodes, and save the optimal model. 9. a kind of phishing website detection method based on width learning as claimed in claim 1, is characterized in that, step (4) is specially: collect misjudged website and newly included website as new feature vector set, carry out model to model. Incremental learning of the input is added to obtain an adjusted weight matrix to optimize the model.

Images