RU2811375C1 - System and method for generating classifier for detecting phishing sites using dom object hashes - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: method for generating a classifier, including a machine learning model, for detecting phishing sites using hashes of page DOM objects contains stages at which pages corresponding to a set of links are obtained; a tree structure is generated in the form of a DOM tree for each page; at least one line is formed consisting of DOM tree elements for each page; at least two hashes are created for each page; two sets of data are formed from the generated hashes; a check for data diversity is performed in each data set; training of the classifier is carried out based on the generated data sets.

EFFECT: reduced proportion of false positives and increased proportion of true positives when detecting phishing sites.

11 cl, 9 dwg

Description

Technical field

The present invention relates to the field of information technology, and more specifically to systems and methods for detecting phishing sites and optimizing the detection of phishing sites.

State of the art

Over the past decade, computer attacks involving imitations of popular company websites, such as phishing attacks, have become a major problem facing many Internet users. Phishing (English phishing from fishing “fishing, fishing”) is a type of Internet fraud, the purpose of which is to gain access to confidential user data, in particular logins and passwords. This is achieved by sending mass emails on behalf of popular brands, as well as personal messages within various services, for example, on behalf of banks or within social networks. The letter often contains a direct link to a site that is externally indistinguishable from the real one, or to a site with redirection. After the user lands on a fake page, attackers try various techniques to induce the user to enter his login and password on the fake page, which he uses to access a certain site, which allows attackers to gain access to the user's accounts and bank accounts.

There are other ways to steal users' confidential data. Some phishers use JavaScript to change the address bar. This is achieved, for example, by performing a so-called “browser-in-browser” attack, in which the attacker creates a fake browser window using HTML directly on the site’s web page.

Attackers can also exploit vulnerabilities in trusted site scripts. This type of fraud, known as Cross-Site Scripting, is the most dangerous because the user is interacting with an official (trusted) site. A cross-site scripting attack is the injection of malicious code into the content of a trusted website and its subsequent execution in dynamic content displayed in the user's (victim's) browser. This kind of phishing is very difficult to detect without special skills.

Solutions are known from the prior art aimed at detecting cyber attacks using fraudulent sites.

For example, from publication US 20200252428 A1, a solution for detecting phishing cyber attacks is known. This solution involves analyzing the code downloaded from a web page at a specific URL (uniform resource locator) to identify links. The links collect URLs from site pages and analyze the code corresponding to the collected URLs. If the code matches code related to known phishing cyberattacks (malicious URLs), then the original URL is determined to be a malicious URL. One of the disadvantages of the presented solution is the increase in analysis time with a large volume of data and the impossibility of making a decision in the case of incomplete compliance of the code when comparing it with other code. One of the solutions that eliminates this drawback is the use of machine learning models to speed up the processing of large volumes of information.

In addition, the huge amount of information available on the Internet also poses a challenge for quickly and effectively monitoring and analyzing websites. So, for example, in the process of processing a stream of phishing URLs, namely when HTML pages that correspond to them are downloaded, instead of the real phishing page, they potentially receive a lot of typical information, for example: messages that these pages are already blocked; various server errors if the page is blocked or deleted by the administrator; typical cloaking (from the English Cloak), i.e. a method of deceiving search engines and automated analysis systems. The amount of such data can be quite large. To solve this problem, content processors and technologies derived from them are used, working on the basis of machine learning, and the quality of their work is extremely dependent on the quality of the training data. Practice has shown that it is precisely this type of typical information that prevents high-quality detection of a real phishing attack. First, the machine learning model is trained to detect things that are easy to detect but no longer need to be detected, for example, because these pages are already blocked, and not really complex phishing pages. Secondly, the likelihood of false positives increases due to safe (legitimate) HTML pages being included in phishing samples.

Accordingly, solutions are needed that can detect changes in sites or similarities to phishing sites, while reducing the number of false positives and increasing the rate of true positives. Embodiments of the present invention achieve at least these objectives individually and collectively.

Disclosure of the Invention

The present invention relates to solutions for detecting phishing sites and is aimed at reducing the number of false positives and increasing the proportion of true positives when detecting phishing sites using a database of DOM tree hashes for pages.

The technical result of the present invention is to reduce the proportion of false positives and increase the proportion of true positives by generating a classifier, including a machine learning model, for detecting phishing sites using hashes of DOM objects for pages.

In one of the embodiments of the invention, a method is proposed for generating a classifier, including a machine learning model, for detecting phishing sites using hashes of DOM objects of pages, and this method contains the steps of: obtaining pages corresponding to a set of links; form a tree structure in the form of a DOM tree for each page by parsing the page, during which data and information are extracted from the page; generating at least one line consisting of DOM tree elements for each page based on the template; creating at least two hashes for each page, with the first hash being generated from the entire page, and the second and subsequent hashes from the generated rows; form two sets of data from the generated hashes, where the first set of data contains information about safe pages, and the second set of data contains information about phishing pages; check for data diversity in each data set; carry out training of a classifier, including a machine learning model, based on the generated data sets.

In another embodiment of the method, pages are obtained using at least one of the following approaches: website pages are downloaded according to links in the form of text files or pages are obtained from a database.

In another embodiment of the method, the template determines the method for forming a string from elements of the page's DOM tree.

In another embodiment of the method, the at least one template is one of the following templates:

a template according to which a line is formed based only on the names of the tags in the DOM tree of the page (the first template);

a template according to which a string is formed based on tag names and tag attribute names (second template).

In yet another embodiment of the method, at least two lines are formed, the first line being formed based on the first template, and the second line being formed based on the second template.

In another embodiment of the method, the check is carried out by determining whether there are groups of similar hashes in the data sets, and these groups are sufficiently numerous.

In another embodiment of the method, data is additionally cleaned from the data sets, where the cleaning consists of searching and removing from one of the data sets hashes that are included in both data sets.

In another embodiment of the method, a check is additionally carried out for the presence of data diversity in each of the specified data sets.

In another embodiment of the method, after training the classifier, its operation is checked on a test subset of data.

In another embodiment of the method, the classifier is retrained when a false positive is detected during testing on a test subset of data.

In another embodiment of the invention, a system for generating a classifier, including a machine learning model, is proposed for detecting phishing sites based on DOM page objects, wherein said system includes: a data collection tool, an analysis tool, including a DOM object generation tool and a hash generation tool, a storage means data, a learning tool for training a classifier including a machine learning model for finding phishing sites, and in which a computing device comprising at least one processor and at least one memory storing machine-readable instructions executable by the at least one processor performs the mentioned method.

Brief description of drawings

Additional objects, features and advantages of the present invention will become apparent from a reading of the following description of the invention with reference to the accompanying drawings, in which:

Fig. 1 shows an example of a system for creating a classifier incorporating a machine learning model for detecting phishing sites.

Fig. Figure 2 shows an example of a system for detecting phishing sites with the possibility of subsequent retraining of the classifier.

Fig. Figure 3 shows an example of a method for training (retraining) a machine learning model to detect phishing sites.

Fig. 4 shows an example of a method for detecting phishing sites.

Fig. 5a-5b shows an example of a DOM object for a site page and generating strings from the DOM object.

Fig. Figure 6 shows an example of a database containing classified hash functions of website pages for training a machine learning model.

Fig. 7 shows examples of suspicious pages and the application of the present invention.

Fig. 8 illustrates an example of a computer system with which the present invention is implemented.

Although the invention may take various modifications and alternative forms, the characteristic features exemplified in the drawings will be described in detail. It should be understood that the purpose of the description is not to limit the invention to a specific embodiment. On the contrary, it is the purpose of the description to cover all changes and modifications falling within the scope of the present invention as defined in the appended claims.

Description of Embodiments of the Invention

Objects and features of the present invention, methods for achieving these objects and features will become apparent by reference to exemplary embodiments. However, the present invention is not limited to the exemplary embodiments disclosed below, but may be embodied in various forms. The following description is intended to assist one skilled in the art to fully understand the invention, which is defined only by the scope of the appended claims.

The present invention is a technical solution that allows you to automatically detect phishing sites and inform about detected changes or similarities to phishing sites using a component containing a machine learning model. Within the framework of the present invention, based on information about sites, including their pages, a training sample is formed for training or retraining a machine learning model, while being able to “clean” the data obtained in the process of detecting phishing sites from “junk” data. The performance of a machine learning model depends on the quality of the data on which the model is trained. Therefore, “garbage” data is considered to be data that influences the appearance of errors of the first and second types. In other words, “junk” data reduces the likelihood of detecting phishing sites or increases the likelihood of identifying legitimate sites as phishing, which increases the likelihood of a false positive for the specified model. Thus, the present invention makes it possible to analyze a large number of sites more efficiently and quickly to identify phishing sites than existing technologies.

It is worth noting that the approaches presented in the description of the present invention, depending on the implementation, can determine similarities between a target site and a large set of sites more accurately and quickly than prior art solutions designed for comparing sites.

The present invention can be implemented in conjunction with such systems that are designed to collect information about sites, including site content, from various servers connected to the Internet, analyze the collected information to determine whether the site information corresponds to previously posted information on the site, classification sites based on determining the similarity in the content of sites with previously classified sites, including determining whether the site is phishing. The present invention can mediate between sites and other network participants who access the sites, for example to determine whether a site is phishing or malicious.

Embodiments of the present invention use in their work the analysis of website pages through the “Document Object Model” (from the English Document Object Model) or DOM object. A DOM object is generated for each page, converted to a specific form, and based on it, several types of hashes (English hash) are formed, which allow more accurate comparison of site pages and/or sites themselves. After this, a machine learning model is trained, which is subsequently used to identify phishing sites. This approach makes it possible to more accurately determine similarities between pages or sites, which leads to a reduction in the number of false positives when identifying phishing pages and/or sites and, accordingly, increases the proportion of true positive positives, i.e. detection of phishing sites.

A site page is an independent part of a web resource, which is a text file in one of the document markup languages (for example, in HTML or XHTML format), and the page has its own unique address (URL). Each page is created using markup language and contains at least the main tags: head, containing the title (Title) and metadata (for example, Keywords, Description), and the body (Body), containing the content. Tags have values and attributes, which also have values.

The DOM object is a platform- and language-independent interface that allows you to dynamically access and update the content, structure, and style of HTML, XHTML, and XML text files (hereinafter referred to as text files). Text files can contain any data. For example, a text file might include the HTML code for a page received from a web server. The DOM object defines the logical structure of a text file, or how a text file is accessed, and allows it to be manipulated through a tree structure called a DOM tree. The basis of every marked up text file is tags, in particular HTML tags. According to the DOM object, each text file tag is an object. Nested tags are "children" of the parent element. The text that is inside the tag is also an object. The DOM object allows client applications to dynamically access, interact with, and update site information received from one or more Web servers.

The "site information" attribute may include any relevant information associated with the site or the host (eg, server) on which the site is hosted. For example, site information may include: site URL; HTML code received after contacting the web server; headers of the response from the web server; metadata associated with HTML code; and any other information that may be received from the Web server to display the page associated with the Site.

In FIG. 1 shows an example implementation of a system for creating a classifier, including a machine learning model, for detecting phishing sites 100 (hereinafter referred to as the system for creating a classifier 100 ). The classifier creation system 100 is designed to train a machine learning model, which in turn is designed to detect phishing sites.

In one embodiment, the system for creating the classifier 100 is implemented using a computer system, such as the one shown in FIG. 8 .

In a preferred embodiment, the classifier creation system 100 includes a data collection engine 110 , an analysis engine 115 , a data storage engine 140 , and a classifier training engine for searching phishing sites 150 . The parser 115 in turn includes a DOM object generator 120 and a hash generator 130 . Depending on the implementation, all tools can be interconnected both at the hardware and software levels, and it is possible to implement the joint use of hardware and software levels.

Data Collection Tool110 designed to interact with websites160, which are usually located on external devices, and with devices such as cloud storage165, to collect the necessary information about sites. It is worth noting that the data collection tool110 can be implemented, inter alia, by any suitable and known method designed to search, receive and store information about sites.

Interaction with sites 160 and devices 165 occurs via both wired and wireless communications, such as through network 170 . In general, an example of network 170 is the Internet. The external devices on which the sites 160 are located are, for example, computer systems such as hosting that host and store data. This data includes not only the sites themselves, but also information associated with them, such as the site URL or site pages. Also, the data collection tool 110 can interact with various DNS servers (from the English Domain name server) and search engines to collect information about sites. The external device 165 is a cloud storage, which in turn interacts with search engines and is designed to collect and store information about sites. In yet another embodiment, device 165 is a cloud infrastructure, such as Kaspersky Security Network (KSN). KSN is a cloud service infrastructure that provides access to an operational knowledge base about the reputation of files, Internet resources (sites) and software. Device 165 can contain both information about legitimate files, Internet resources (sites) and software, and about malicious ones, such as phishing sites and pages.

The data collection tool 110 can collect information about sites either automatically or by interacting with various search engines, or at the request of other means of the system for creating a classifier 100 or a request received from a user of the system 100 through one of the input means (not shown in Fig. 1 ). An example of the operation of the data collection tool 110 in automatic mode is the approach used by search robots (English Web crawler) to regularly and/or periodically crawl sites 160 hosted on hosting sites and accessible via the network 170 .

In one embodiment, tool 110 receives a set of URLs 190 from external device 165 . In this case, tool 110 collects site information based on URL links from the set of URLs 190 . Information is collected by following the received URL links and downloading the necessary information, for example, the site itself or its page in the form of a text file and other site resources.

After collecting information about the sites, the data collection tool 110 transmits it to the analysis tool 115 .

The analysis tool 115 is designed to create a set of hashes for each site and/or for each page thereof. The hash set for each page or site contains at least two hash types. In a preferred embodiment, the hash set consists of three hash types. Three types of hashes allow you to most effectively search for similar sites and/or detect phishing sites. At the same time, depending on the implementation of the invention, it is possible to use at least one type of hash, for example, for tasks of determining the similarity of pages or sites. It is worth noting that in an implementation where only one type of hash is used, the hash must be created using a DOM object.

The parser 115 , using the DOM object generation engine 120 , parses each page received from the data collector 110 . Page parsing refers to the automated process of extracting data or information from a page that is in the form of a text file. During the parsing process, the parser 115 , using the DOM object generation tool 120, creates a tree structure of the page, namely a DOM tree. An example of a DOM tree is shown in Fig. 5a . In one implementation, the DOM 120 renderer uses the Python lxml library to parse the page. In another embodiment, using the lxml library, the mentioned tool 120 also performs page restoration, for example, adjusting (correcting) the DOM tree and generating code (for example, HTML code) of the page based on the tree structure. For example, code generation consists of restoring missing closing tags. After generating the DOM tree, said tool 120 passes it to the hash generation tool 130 . The hash generator 130 converts the resulting DOM tree into a certain form, namely, it forms at least one line from the elements of the page DOM tree according to a certain pattern. Each template defines how a string is formed from elements of the page's DOM tree.

Thus, to form the first line, the first template is used, according to which only tag names are used to form the line. To do this, tool 130 removes all text and all values and generates a chain of tags, for example: html, head, title, body. We can say that the resulting string is unique. To form the second line, a second template is used, according to which the principle remains the same as when forming the first line, only the names of the tags and additionally the names of the tag attributes are used. In FIG. Figure 5b shows examples of both strings, which are generated based on the DOM tree presented in FIG. 5a .

Next, tool 130 generates a hash for each generated line, and also additionally generates a hash from the page (text file) itself. In one embodiment, tool 130 generates hashes based on the MD5 algorithm. Thus, the tool 130 at least generates three types of hashes for each page, where the first hash is generated from the entire page (content hash or content_md5), the second hash (hash_light) is generated from the first line, i.e. line generated according to the first template, and the third hash (hash_full) generated from the second line, i.e. line generated according to the second pattern. After generating all the hashes for each page, the means 130 passes them to the data storage means 140 . It is worth noting that when generating additional strings based on additional patterns, tool 130 will also generate additional hashes based on them.

In one implementation, the tool 115 checks each page to see if it is new at the start of its operation. In other words, whether the tool 115 previously received the corresponding page for analysis. In this case, first, the tool 115 , using the tool 130 , will generate a hash from the page itself (the first hash) and check for the presence of this hash in the data storage tool 140 . If the tool 115 detects a hash in the data storage tool 140 , the tool 115 will exclude this page from further analysis. Otherwise, if the tool 115 does not detect a corresponding hash in the data storage means 140 , then the tool 115 will conduct further analysis, taking into account that the hash from the page itself has already been generated.

Data storage means 140 is a computer-readable medium that is designed to store information about sites and their pages, where the stored information includes both generated hashes and additional information about the sites, such as URLs of HTML pages. Additional information may be provided by either the analysis tool 115 , the data collection tool 110, or the learning tool 150 . An example of data storage by the data storage means 140 is shown in FIG. 6 . In one embodiment, data storage means 140 is a database expressed as a table in which each row contains information for one hash. The hash information at least includes:

• Hash - hash value,

• Hash_type - hash type (for example, hash from the page “content_md5” (first hash), hash “hash_light” (second hash), hash “hash_full” (third hash)).

Also, the data storage means 140 further contains information for the hash, which includes the purpose of the hash and the decision that is made on the corresponding hash. For example, the purpose of the hash can indicate that the hash is used during training (train_action) and in what way - for training or for testing. An example of a hash judgment is to indicate that the hash refers to hashes pointing to a phishing site (mark_as_phish) or hashes pointing to a “clean” page or site (mark_as_clean).

It is worth noting that tool 140 contains information about both benign pages and phishing pages. Safe pages are pages that do not pose a threat to the loss of third party data and on which detection mechanisms for phishing sites do not work. Also, such pages are pages that are formed and modified only with the permission of the site owner, if the site owner has a confirmed identity.

In one embodiment, the parser 115 determines the popularity of each hash type in the data store 140 and ranks them according to the determined popularity. In one embodiment, popularity is determined based on analysis of pages over a six (6) month period. Thus, the analysis tool 115 determines how many identical hashes from pages have been generated over the past six months. Accordingly, the greater the number of pages that have the same hash, the more popular this hash is.

The training tool 150 is designed to train or retrain a classifier including a machine learning model to find phishing sites. The training tool 150 interacts with the data storage tool 140 when training the classifier. The learning engine 150 generates two sets of data. The first dataset contains information about benign pages, and the second dataset contains information about phishing pages. Page information refers to previously collected information about sites, in particular information about hashes. Each data set is generated based on information about hashes, in particular information about the purpose of the hashes. In a particular implementation, data sets are formed taking into account the frequency of occurrence of a hash among all created hashes and the length of the DOM tree on the basis of which the hash was created.

In one implementation, the information contained in each data set includes page and site information, which can be divided into three groups. The first group includes information about the content contained on the pages, in particular information about organizations, the need to enter a password, or other user data. The second group includes information about URLs, in particular information about the form of writing the line and the presence of spaces and special characters. The third group includes information about pages obtained from external sources, such as the WHOIS service. Also, sets can additionally include information about the popularity of a particular domain.

It is worth noting that the formation of data sets using hashes generated on the basis of DOM trees makes it possible to obtain the most balanced data sets, which allows further training of the classifier with the greatest efficiency. In other words, this approach makes it possible to maintain a balance between homogeneous and diverse information in data sets, in particular by excluding “junk” data.

Depending on the implementation option, when generating data sets, all hashes and associated information stored in the data storage means 140 can be used, or only part of them. The limitations when generating data sets are the number of hashes stored in the data storage medium 140 and the date the hash was generated, i.e. hash relevance.

Next, the training engine 150 divides each data set into two parts, where the first part is a training subset (training set) and the second part is a testing subset (test set). On the training subset, the training tool 150 trains the classifier, namely the machine learning model, and using the test subset, it carries out verification (checking the quality of work) of the trained classifier. For example, dividing into two parts is based on the proportion of the total data: 70% training set and 30% test set. Also, the proportion may change depending on the amount of data in each data set.

In a particular implementation, the learning tool 150 divides the data set into three parts. In this case, a third is added to the specified two subsets - a validation subset (validation sample). The purpose of the validation set is to select a decision threshold for a machine learning model trained on the training set. After choosing a decision threshold, a final check of the quality of the specified model is carried out.

In one embodiment, the training tool 150, when generating two sets of data, additionally performs so-called data cleaning in order to improve the sample of data for training. Data cleaning is based on the fact that the generated hashes, especially hashes corresponding to the hash_full hash type (third hash), are fairly unique page identifiers up to the variability associated with reflecting the characteristics of a particular user or session on the site. For example, a home page on a social network is the same page regardless of what first name, last name, photo, and similar data fills the corresponding slots (for example, input fields) of the page. Repeated unique identifiers (hashes) of pages with different content are also typical for service pages, such as error messages. Thus, the more often a particular hash appears in both data sets, the more likely it is that it is a service page. Data cleaning uses an approach in which the learner 150 eliminates both duplicate hashes and those related to service pages in at least one of the data sets. This data cleaning allows you to prepare the most optimal data sets for training a classifier containing a machine learning model. This approach eliminates one of the shortcomings of machine learning, namely training on data sets in which very similar objects were present. For example, the same hashes were present in both data sets. When training on similar data, the performance of the trained machine learning model decreases and false positives occur when such a trained machine learning model makes decisions.

In yet another embodiment, the learning engine 150 cleans both data sets as follows. First, the learning engine 150 generates a set of the most popular hashes for clean hashes and a set of the most popular hashes for phishing hashes by accessing the data storage engine 140 and then searching, provided that the engine 140 does not already contain hash popularity information. Next, the learner 150 identifies hashes that correspond to hashes related to both data sets. After identifying at least one such hash, the learner 150 determines which page the hash belongs to and, based on this, removes the hash from one data set and leaves the hash in the other data set. For example, if the hash refers to a service page, then this page is safe, and, accordingly, the hash is removed from the data set related to phishing sites and remains in the data set related to safe sites. In addition, the learning engine 150 enters or changes information about the specified hash in the data storage engine 140 as follows: the “Learning Action” column indicates “to delete (none)” and the “Decision” column indicates “mark_as_clean” " In yet another example, if the hash refers to a phishing page, then the learner 150 leaves such a page only in the dataset related to phishing sites.

Next, the learner 150 checks for data diversity in each specified data set. Since in each data set there is a group of similar objects (hashes), and such a group is quite large, then such a data sample can also negatively affect the quality of classifier training. For example, a data set contains 100 (one hundred) thousand groups, and the objects of one of the groups occupy more than 1% of the entire set. In another example, the data set contains 90 (ninety) thousand groups, and the objects of one of the groups occupy more than 5% of the entire set. In yet another example, multiplicity is determined empirically. Therefore, in each data set, the learner 150 selects at least one group of hashes related to a page that appears more than N times in the corresponding data set, where N is a configurable parameter and is determined by practical means. In one implementation, the parameter N is determined based on the ratio of the total number of hashes in the data set and hashes related to one page. If the training tool 150 did not select any groups, then each set of hashes is optimal for further training of the classifier, including the machine learning model. Otherwise, the learning tool 150 in each selected group randomly selects K pages, where K is a configurable parameter and is determined in a practical way. The learner 150 retains the pages whose hashes belong to the selected K pages in the data set and removes the remaining pages whose hashes belong to the selected group from the data set. It is worth noting that if the data set relates to phishing sites, then the learner 150 adds the relevant information about the page, namely the hash, to the data store 140 . For example, an entry will be added to the specified hash line that the decision being made is “phishing.” Also, the learner 150 may further check whether the page to which the hash relates is phishing. In one embodiment, the additional check consists of querying the security components (not shown in FIG. 1 ) related to phishing site analysis and detection solutions, and receiving a response from them. If the data set is a "clean" site, then the learner 150 adds to the hash entry in the data store 140 that the decision being made is a "clean page".

Next, the training tool 150 trains a classifier, including a machine learning model, based on the generated and optimized data sets. Depending on the implementation, training a machine learning model is based on at least one of the following principles: supervised learning, such as logistic regression, linear regression, or k-nearest neighbors algorithm (k-NN); or neural networks. After training the classifier, the training tool 150 carries out verification (checking the quality of work) of the trained classifier on the test subset. If the trained classifier does not exceed a given threshold when making incorrect decisions, i.e. the number of false positives is below the maximum permissible level, then the learning tool 150 sends the classifier to work. Otherwise, if the trained classifier made the number of incorrect decisions exceeding the specified threshold, i.e. the number of false positives has exceeded the maximum permissible level, then the learning tool 150 retrains the classifier. During retraining, the training engine 150 generates a new or updates at least one training data set. To update the data set, the learning engine 150 queries the data collection engine 110 to collect new or additional information about the sites.

In a particular retraining implementation, the learner 150 may further implement changes to hyperparameters of the machine learning model, such as learning rate and data set size. Also, the learner 150 may change the dimensions of the machine learning model, such as the number of nodes in the neural network.

In one embodiment, the learner 150 passes a classifier for detecting phishing sites to security components 185 responsible for detecting phishing sites on the client side 180 . Security component 185 is introduced in the description of FIG. 2 . Depending on the implementation options, protection components 185 are technical solutions that are aimed both at protecting against cyber attacks and malware, and at ensuring security against data theft, while these solutions can be implemented both on clients 180 and on various servers corporate networks.

In yet another embodiment, in addition to transmitting the specified classifier, at least one type of generated hashes is additionally transmitted to the security components, for example, a third type of hashes. These hashes can also be used in conjunction with the classifier when comparing sites for similarity or detecting phishing sites as a preliminary step.

In another implementation, if the learning engine 150 , when generating a data set related to clean sites, during the data cleaning process determines that the page does not have phishing content, then its corresponding hashes are marked as clean and the decision made is determined as a clean page. For example, the page refers to a server error message.

Fig. Figure 2 shows an example of a phishing site detection system. The phishing site detection system (hereinafter referred to as the detection system) 200 is implemented using at least one computer system, such as the one shown in FIG. 8 .

Depending on the implementation option, the detection system 200 can be implemented either as part of the security component 185 or separately from the security component 185 (not shown in FIG. 2 ) on the client device 180 , or in conjunction with the classifier creation system 100 on a single device, wherein the system detection 200 interacts with the classifier creation system 100, regardless of the implementation option. In addition, the detection system 200 and the classifier generation system 100 may be implemented by a single system.

In one embodiment, detection system 200 includes tools such as a pre-checker 220 , a processor 230 , a classifier 240 including a machine learning model, and a database 260 . Depending on the implementation options, all tools can be interconnected both at the hardware and software levels, and it is possible to implement and jointly use the hardware and software levels.

The security component 185, upon receiving a task to scan a site or its page, transmits the corresponding data to the detection system 200 . The task can be received in the form of a URL (link), in which case the security component 185 will first download the content (text of the HTML code of the page) of the corresponding resource from the received link. In addition, the security component 185 optionally converts each HTML page into a text file.

The pre-validator 220 is designed to validate the page received by the detection system 200 using hashes from the database 260 . The specified tool 220 generates a hash from the page itself (text file). In one embodiment, the hash is generated based on the MD5 algorithm. In another implementation, the hash can be generated based on any other known hashing algorithm, for example CRC32, SHA256, SHA512. The tool 220 then compares the generated hash with hashes from the database 260 . The database 260 contains at least two sets of hashes, where one set includes hashes from benign pages and a second set includes hashes from phishing pages. If the tool 220 finds the generated hash in the database 260 , then the detection system 200 will determine the file being checked as phishing or benign, depending on the set of hashes in which the hash being checked was found. Otherwise, if the hash is not found in the database 260 , the engine 220 passes the page to the processing engine 230 .

In one embodiment, detection system 200 does not include pre-checker 220 . In this case, the received page immediately goes to the processing tool 230 , which performs a preliminary analysis of the page.

Processor 230 is configured to convert the page into at least one string of page DOM tree elements according to a specific pattern, generate at least one hash for each generated string, and check the generated hashes against database 260 . It is worth noting that processing a page to generate a specific type of hash is similar to the work of a parser 115 . The functionality of the processing means 230 can be said to correspond to the functionality of the analysis means 115 . Thus, first the processing engine 230 parses the page, as a result of which it creates a tree structure of the page, i.e. DOM tree. Next, processing engine 230 converts the DOM tree into at least one line according to a specific pattern of page DOM tree elements. Processing means 230 then generates a hash from each generated string. In one embodiment, the engine 230 generates at least two types of hashes for each page, where one hash (hash_light) is generated from a string generated according to the first pattern, and another hash (hash_full) is generated from a string generated according to the second pattern. After all hashes for a page have been generated, tool 230 checks the resulting page by comparing the generated hashes with similar hashes from database 260 . In the event that the hashes have been found in the database 260 , the detection system 200 makes a decision as to whether the site is a phishing site. Otherwise, if no hashes were found, the engine 230 passes the resulting page to the classifier 240 . In particular, the transfer of the received page also means the transfer of information about the content and metadata of the page.

In yet another embodiment, the pre-checker 220 and the processing engine 230 are implemented together. In this case, preliminary verification of the resulting page is carried out both using a hash generated from the entire page, and using at least one hash generated from a string of elements of the page's DOM tree. For example, a check is first carried out using a hash from the entire page, and if the hash was not found in the database 260 , then a check is carried out using hashes generated from strings from the elements of the DOM tree of the page, i.e. hashes are applied sequentially. Also, the use of the mentioned hashes can be carried out in parallel.

The classifier 240 includes a machine learning model and is designed to determine whether a page is phishing based on information received from the processor 230 . The classifier 240 provides the learned information as input to the trained machine learning model. The output of the machine learning model provides the classifier 240 with a decision about whether the page matches known phishing pages or sites. The classifier 240 provides the security component 185 with a decision whether the site is a phishing site.

The database 260 is a computer-readable medium that is designed to store and make stored data available to the detection system 200 on demand. As mentioned earlier, the stored data is at least two sets of hashes, where one corresponds to benign pages and the other corresponds to phishing pages. It is worth noting that these sets include pages that can clearly be identified as safe pages or phishing pages.

In one embodiment, detection system 200 has the ability to retrain classifier 240 . In this case, the detection system 200 forwards each decision made to the classifier generation system 100 . In turn, the classifier creation system 100 evaluates each decision made. If the classifier creation system 100 determines the solution to be false, it will retrain the classifier, namely the machine learning model. Retraining is carried out taking into account at least one false decision. The classifier generation system 100 will then provide the retrained classifier to the detection system 200 .

It is worth noting that the 240 classifier is designed to determine whether a site is phishing or not in cases where other protection mechanisms have failed to make a decision. Such situations are typical when attackers disguise their websites by changing them. As a rule, changes relate to the content of the site.

Let us consider the features of using the hashes presented in the present invention, generated on the basis of DOM trees of HTML pages presented in Fig. 7 .

Let's assume the first page contains some HTML and content and the page is a suspicious page. For example, the first page is assumed to be a phishing page designed to steal banking information.

The second page is also a suspicious page and a different version of the first page. The difference is that the second page has the same functionality, but at the same time counteracts detection by substituting some characters and adding false values for class attributes. It is worth noting that in Fig. 7 to demonstrate the substitution is obvious, in the present samples homography attacks are performed more skillfully.

For each page, hashes are generated according to the previously stated principle, namely:

• Generate an md5 hash from each page;

• Obtain a DOM tree by parsing an HTML page using the lxml library;

• Convert the generated DOM tree for each page into two lines:

• For the first line, use the first template, according to which only the tag names are saved,

• For the second line, a second template is used, according to which the tag names and tag attribute names are stored.

• Form hashes for the received strings.

Thus, MD5-based hashes for each page of the entire page will look like:

For the first page – 397ac9d66c21ec19ae55a3c758d6443d,

For the second page – 39c7706b85321872a6e4bae79eff3064.

It can be seen that the changes made to the content of the second page also changed the hash of the second page. Therefore, it will not be possible to group pages based on the hashes of such pages (files), and you will need to generate new samples (hashes) each time to block phishing sites (for example, when blocking by signature). In other words, if a hash is generated from the entire page, then a small change in the content of the page will change that hash.

At the same time, the structure of the pages remained the same:

First line (tag structure):

For the first page:

doc | html | head | title | /title | /head | body | h1 | /h1 | /body | /html | /doc

For the second page (with masking):

doc | html | head | title | /title | /head | body | h1 | /h1 | /body | /html | /doc

Second line (tags and attributes structure):

For the first page:

doc | html | head |title | /title | /head | body | h1:class | /h1 | /body | /html | /doc

For the second page (with masking):

doc | html | head |title | /title | /head | body | h1:class | /h1 | /body | /html | /doc

It can be seen that the lines for both pages are identical. Therefore, having generated hashes based on the received strings, the hashes for both pages will be the same. For example, strings of the second type in the form of hashes (the structure of tags and attributes) will look like:

for the first page – fbec7f8965c8f1c9c5986c076b7de5cd,

for the second page – fbec7f8965c8f1c9c5986c076b7de5cd.

Thus, when using the claimed solution, it is additionally possible, based on the conversion of page DOM objects into strings, to form hashes and group similar pages into clusters.

In one embodiment, the presented principle is intended not only to search for phishing sites, but also to detect similar pages.

As an example, consider two pages:

https://www.google.com/error

https://www.google.com/another_error.

Generated hashes based on entire pages will be different, while generated hashes based on the presented solution (using DOM objects and strings) will not:

Hashes based on entire pages:

1) cfa900f6311e2e675c95de0788684d2b,

2) 696531ac087387b995e3376a36b333d1.

Hashes based on the first type string for the specified pages:

1) 4841ea35c8deb90b749f2f1b2aed8a11,

2) 4841ea35c8deb90b749f2f1b2aed8a11.

Hashes based on the second type of string for the specified pages:

1) d8085a405ad249ece59f79744c94158f,

2) d8085a405ad249ece59f79744c94158f.

This way, you can easily find large groups of pages that are similar to within the structure and remove them from the training set (or balance the sample, leaving only a few examples from each group).

Fig. Figure 3 shows an example of a method for training (retraining) a machine learning model to detect phishing sites.

At step 310, a set of URLs is obtained using the data collection tool 110 from various sources.

At step 320 , using the data collection tool 110, pages and/or sites are downloaded in the form of text files in HTML format according to the URL from the resulting set.

In one embodiment, steps 310 and 320 may not be performed. In this case, the pages for training are obtained from the data storage means 140 or from a similar database that stores information about the sites.

In a preferred embodiment, steps 330 , 335 and 340 are performed using analysis tool 115 .

At step 330 , each received page is parsed, during which a DOM tree of the corresponding page is formed. During the parsing process, a tree structure of the page is created, namely a DOM tree.

At step 335 , at least one row of DOM tree elements is generated for each resulting page according to a specific template. Each template defines how a string is formed from elements of the page's DOM tree. It is worth noting that to form the first line, the first template is used, according to which only tag names are used to form the line. To form the second line, a second template is used, according to which the names of tags and additionally the names of tag attributes are used to form the line.

At step 340 , a hash is created for each generated line and a hash for the entire resulting page. In a particular implementation case, a hash of the entire received page is not created. In the preferred embodiment, three types of hashes are created for each page, where the first hash is generated from the entire page (content hash), the second hash (hash_light) is generated from the first line, and the third hash (hash_full) is generated from the second line, while at step 335 two lines for each page. After all hashes have been generated for each page, proceed to step 350 .

At step 350 , the popularity for each type of hashes generated for the pages corresponding to the resulting set of URLs is determined. Popularity is determined based on page analysis over a period of 6 (six) months. Thus, they determine how many identical hashes from pages have been generated over the past six months. Accordingly, the greater the number of pages that have the same hash, the more popular the hash becomes. In one embodiment, this step may be omitted, in which case, after step 340 , proceed to step 360 .

At step 360 , two data sets are generated, where the first data set contains information about safe pages, and the second data set contains information about phishing pages. Page information at least refers to previously collected information about sites. Each data set is generated based on hash information. In particular, data sets are generated taking into account the frequency of occurrence of a hash among all created hashes and the length of the DOM tree on the basis of which the hash was created.

In one particular embodiment, at step 360 , the information contained in each data set further includes page and site information, which can be divided into three groups. The first group includes information about the content contained on the pages. The second group includes information about URLs. The third group includes information about pages obtained from external sources, such as the WHOIS service.

In yet another particular embodiment, at step 360 , each data set is further divided into two parts, where the first part is a training subset (training set) and the second part is a test subset (test set).

In another particular embodiment, at step 360 , data is further cleaned to improve the training data sample. During data cleaning, hashes are identified in the data sets that correspond to hashes related to both data sets, and the hashes are generated based on DOM trees. After identifying at least one similar hash, it is determined which page the hash belongs to, and based on this, the page corresponding to the hash is removed from one data set and the page corresponding to the hash is left in the other data set. An example implementation is presented in the description of FIG. 1 .

At step 370 , an analysis is performed to determine whether there is data diversity in each data set. During the analysis, it is determined whether there are groups of similar objects (hashes) in the data sets, and these groups are quite numerous. If so, then in the corresponding data set at least one group of hashes related to pages that occur more than N times in the corresponding data set is selected. In one implementation, the parameter N is determined based on the ratio of the total number of hashes in the data set and hashes related to one page. Next, K pages are randomly selected from each selected group. After this, the pages whose hashes belong to the selected K pages are left in the data set and the remaining pages whose hashes belong to the selected group are removed from the data set. If no groups are identified, then each data set is already optimal.

At step 380 , a training set is formed by adding the obtained data sets. Each dataset contains information about the pages corresponding to the remaining hashes.

Additionally, at step 385 , it is checked whether the training set has been formed. If formed, the training sample is passed on to the machine learning model for training. If not formed, then return to step 310 where a new set of URLs is obtained.

At step 390 , the machine learning model, which is part of the classifier, is trained based on the generated sample.

Additionally, at step 390 , the trained machine learning model is verified. If the trained machine learning model did not exceed a given threshold when making incorrect decisions on the test subset, i.e. the number of false positives is below the maximum permissible level, then the method stops working. Otherwise, if the trained machine learning model made the number of incorrect decisions exceeding a given threshold, i.e. the number of false positives has exceeded the maximum permissible level, then the machine learning model is retrained using the presented method.

Fig. 4 shows an example of a method for detecting phishing sites.

At step 410, at least one site page is obtained for review. The site page is provided as a text document in HTML format.

At step 420, a hash is created from the entire page and at step 430, the hash is searched in database 260 . If the hash is found in the database 260 , then proceed to step 435 . Otherwise, if the hash is not found in the database 260 , then proceed to step 440 .

At step 435 , it is determined which set of hashes the hash found at step 430 in the database 260 belongs to. If the hash belongs to the set of “clean” hashes, then proceed to step 410 to check the next page. Otherwise, if the hash belongs to the set of phishing hashes, then proceed to step 480 , where a decision is made about the detected phishing site.

At step 440 , each received page is parsed, during which a DOM tree of the corresponding page is formed. During the parsing process, a tree-like representation of the page structure is created, namely a DOM tree.

At step 450, at least one row is formed from the elements of the page's DOM tree according to a specific pattern. It is worth noting that to form the first line, the first template is used, according to which only tag names are used to form the line. To form the second line, a second template is used, according to which the names of tags and additionally the names of tag attributes are used to form the line.

At step 460 , hashes are created for the page from the at least one generated string. In the preferred embodiment, three types of hashes are generated for each page, where the first hash is generated from the entire page (content hash), the second hash (hash_light) is generated from the first line, and the third hash (hash_full) is generated from the second line.

On the move465 searches for at least one hash in a database260. If at least one hash is found in the database260, then go to step490. Otherwise, if no hash is found in the database260, then go to step470.

At step 470 , the page is analyzed using a classifier based on the at least one hash generated. To do this, information about the page corresponding to the specified hash is passed to the input of the machine learning model. In particular, the transfer of information about a page also includes the transfer of information about the content and metadata of the page.

At step 475 , a decision is made whether the page is phishing based on the output of the analysis by the machine learning model. If the machine learning model decides that the page is not phishing, then proceed to step 410 to check the next page, or finish the job if all pages of the site are checked. Otherwise, if the machine learning model has decided that the page is phishing, then proceed to step 480 .

At step 480 , a decision is made about the detected phishing site based on the information obtained after step 475 or step 435 .

In a particular implementation case, steps 420 and 430 may be omitted. In this case, after receiving the page for review at step 410 , proceed to step 440 .

In yet another particular implementation, the creation of a hash for the page provided in step 420 is performed in conjunction with the creation of hashes in step 460 . In this case, step 430 is performed at step 465 .

In FIG. 8 illustrates a computer system on which various embodiments of the systems and methods disclosed herein may be implemented. Computer system 20 may be a system configured to implement the present invention and may be in the form of a single computing device or multiple computing devices, such as a desktop computer, a laptop computer, a notebook computer, a mobile computing device, a smartphone, a tablet computer, a server, a mainframe computer. , embedded device, and other forms of computing devices.

As shown in FIG. 8 , the computer system 20 includes: a central processing unit 21 , a system memory 22 , and a system bus 23 that communicates various system components, including memory, coupled to the central processing unit 21 . System bus 23 is implemented like any bus structure known in the art, which in turn includes a bus memory or bus memory controller, a peripheral bus, and a local bus capable of interfacing with any other bus architecture. Examples of buses include: PCI, ISA, PCI-Express, HyperTransport™, InfiniBand™, Serial ATA, I2C and other suitable connections between computer system components 20 . The central processing unit 21 contains one or more processors having one or more cores. The central processing unit 21 executes one or more sets of computer-readable instructions implementing the methods presented herein. System memory 22 may be any memory for storing data and/or computer programs executable by the central processing unit 21 . System memory can contain both read-only memory (ROM) 24 and random access memory (RAM) 25 . The basic input/output system (BIOS) 26 contains the basic procedures that ensure the transfer of information between elements of the computer system 20 , for example, when the operating system is loaded using ROM 24 .

Computer system 20 includes one or more data storage devices, such as one or more removable storage devices 27 , one or more non-removable storage devices 28 , or combinations of removable and non-removable devices. One or more removable storage devices 27 and/or non-removable storage devices 28 are connected to the system bus 23 via an interface 32 . In one embodiment, removable storage devices 27 and associated computer-readable storage media are nonvolatile modules for storing computer instructions, data structures, program modules, and other computer system 20 data. System memory 22 , removable storage devices 27 , and non-removable storage devices 28 may use various computer readable media. Examples of computer-readable storage media include computer memory such as cache memory, SRAM, DRAM, capacitorless RAM (Z-RAM), thyristor memory (T-RAM), eDRAM, EDO RAM, DDR RAM, EEPROM, NRAM, RRAM, SONOS, PRAM; flash memory or other memory technologies such as solid-state drives (SSDs) or flash drives; magnetic cassettes, magnetic tapes and magnetic disks such as hard disks or floppy disks; optical media such as compact discs (CD-ROMs) or digital versatile discs (DVDs); and any other media that can be used to store the desired data and that can be accessed by the computer system 20 .

System memory 22 , removable storage devices 27, and non-removable storage devices 28 contained in the computer system 20 are used to store the operating system 35 , applications 37 , other program modules 38 , and program data 39 . Computer system 20 includes a peripheral interface 46 for transmitting data from input devices 40 such as a keyboard, mouse, stylus, game controller, voice input device, touch input device, or other peripheral devices such as a printer or scanner through one or more input/output ports, such as a serial port, parallel port, universal serial bus (USB), or other peripheral interface. A display device 47 , such as one or more monitors, projectors, or embedded displays, is also connected to the system bus 23 through an output interface 48 , such as a video adapter. In addition to the display devices 47 , the computer system 20 is equipped with other peripheral output devices (not shown in FIG. 8 ), such as speakers and other audiovisual devices.

The computer system 20 may operate in a networked environment using a network connection to one or more remote computers 49 . The remote computer(s) 49 is a working personal computer or server that contains most or all of the components noted previously in describing the nature of the computer system 20 shown in FIG. 8 . There may also be other devices in the network environment, such as routers, network stations or other network nodes. The computer system 20 may include one or more network interfaces 51 or network adapters for communicating with remote computers 49 over one or more networks, such as a local area network (LAN) 50 , a wide area network (WAN), an intranet, and the Internet. Examples of network interface 51 are an Ethernet interface, a Frame Relay interface, a SONET interface, and wireless interfaces.

Embodiments of the present invention may be a system, a method, or a computer-readable medium (or storage medium).

A computer-readable storage medium is a tangible device that stores and stores program code in the form of machine-readable instructions or data structures that are accessible by the central processing unit 21 of a computer system 20 . A computer-readable medium may be an electronic, magnetic, optical, electromagnetic, semiconductor storage device, or any suitable combination thereof. By way of example, such a computer-readable storage medium may include random access memory (RAM), read-only memory (ROM), EEPROM, compact disc read-only memory (CD-ROM), digital versatile disk (DVD) ), flash memory, hard drive, portable computer floppy disk, memory card, floppy disk, or even a mechanically encoded device such as punch cards or embossed structures with instructions written on them.

The system and method of the present invention can be thought of in terms of means. The term "device" as used herein refers to an actual device, component, or group of components implemented in hardware, such as an application-specific integrated circuit (ASIC) or FPGA, or a combination of hardware and software. providing, for example, using a microprocessor system and a set of machine-readable instructions to implement the functionality of a tool that (in the process of execution) turns the microprocessor system into a special-purpose device. The tool may also be implemented as a combination of these two components, with some functions being implemented by hardware alone and other functions being implemented by a combination of hardware and software. In some embodiments, at least a portion, and in some cases all, of the means may be executed on the central processing unit 21 of the computer system 20 . Accordingly, each means may be implemented in various suitable configurations and should not be limited to any particular embodiment set forth herein.

In conclusion, it should be noted that the information given in the description are examples that do not limit the scope of the present invention as defined by the formula. One skilled in the art will appreciate that in developing any actual embodiment of the present invention, many decisions specific to the particular implementation must be made to achieve specific objectives, and these specific objectives will differ from one embodiment to another. It is understood that such development efforts can be complex and time consuming, but will nevertheless be a routine engineering task for those of ordinary skill in the art using the present disclosure.

Claims (25)

1. A method for generating a classifier, including a machine learning model, for detecting phishing sites using hashes of DOM page objects, wherein this method contains the steps of: a) get pages corresponding to a set of links (URL); b) form a tree structure in the form of a DOM tree for each page by parsing the page, during which data and information are extracted from the page; c) form at least one line consisting of DOM tree elements for each page based on the template; d) create at least two hashes for each page, with the first hash being formed from the entire page, and the second and subsequent hashes from the generated lines; e) form two sets of data from the generated hashes, where the first set of data contains information about safe pages, and the second set of data contains information about phishing pages; f) check for data diversity in each data set; g) train a classifier, including a machine learning model, based on the generated data sets. 2. The method according to claim 1, in which pages are obtained by at least one of the following approaches: downloading site pages according to links in the form of text files or obtaining pages from a database. 3. The method according to claim 1, in which the template determines the method of forming a string from the elements of the DOM tree of the page. 4. The method according to claim 3, wherein at least one template is one of the following templates: a) a template according to which a line is formed based only on the names of the tags in the DOM tree of the page (the first template); b) a template according to which a string is formed based on the tag names and tag attribute names (second template). 5. The method according to claim 1, in which at least two lines are formed, the first line being formed based on the first template, and the second line being formed based on the second template. 6. The method according to claim 1, in which at step e) the check is carried out by determining whether there are groups of similar hashes in the data sets, and these groups are sufficiently numerous. 7. The method according to claim 1, in which data is additionally cleaned from the data sets, where the cleaning consists of searching and removing hashes from one of the data sets that are included in both data sets. 8. The method according to claim 1, in which they additionally check for the presence of data diversity in each of the specified data sets. 9. The method according to claim 1, in which additionally, after training the classifier, its operation is checked on a test subset of data. 10. The method according to claim 9, in which the classifier is retrained when a false positive is detected during testing on a test subset of data. 11. A system for generating a classifier, including a machine learning model, for detecting phishing sites based on page DOM objects, wherein the specified system includes: a) data collection means, b) an analysis tool, including a DOM object generation tool and a hash generation tool, c) data storage device, d) a learning tool designed to train a classifier that includes a machine learning model for finding phishing sites, e) and wherein a computing device comprising at least one processor and at least one memory storing machine-readable instructions executable by the at least one processor, implements the method according to any one of claims. 1-10.