RU2427890C2 - System and method to compare files based on functionality templates - Google Patents

Abstract

FIELD: information technologies.

SUBSTANCE: method to determine belonging of files to collections of available files on the basis of files comparison with the help of functionality templates includes stages, at which functionality templates are generated on the basis of information on the executed file. Then extracted noise information is deleted from functionality templates of the executed file. Then units of functionality templates of the executed file are reduced to normalised view. Then these units are compared to units of functionality templates of available files, and using comparison results, decision is made on belonging of the unit to one of functionality templates of available files. Creating functionality templates by available malicious software, newly arrived files may be compared with them, and automatic records may be added with condition of similarity; characteristic logical units are extracted from collections of malicious programs, and heuristic rules are created by these units; automatic descriptions are generated. Also the possibility appears to carry out clusterisation of objects, which helps to accelerate their further processing.

EFFECT: increased reliability and accuracy of malicious software detection, achieved by comparison of executed files by means of functionality templates.

14 cl, 16 dwg

Description

Technical field

The invention relates to systems and methods for creating descriptions and comparing the functionality of executable files, and more specifically to determining their belonging to known collections.

State of the art

Currently, there is a significant increase in the number of malware. Every day, several tens of thousands of new variants of malware appear, which makes it more and more difficult for antivirus companies to provide the user with a decent level of protection. One reason for this is the emergence of polymorphic and metamorphic malware. Polymorphism consists in the formation of a unique program code directly at runtime, while the procedure that generates the code itself is also unique with each new infection. Metamorphism is the encryption or distortion of some parts of the program, making it “unrecognizable”. “New option” does not mean a completely different malicious program, but for the most part, they are new only for the existing one of the main detection technologies - signature.

The growing threat from polymorphic and metamorphic viruses has forced antivirus developers to complement signature analysis with other methods, which include reduced mask technology, cryptographic and statistical analysis, and code emulation.

The technology of the reduced mask consists in applying a certain transformation to the encrypted body of the virus, which at the output produces a unique combination that identifies the virus, regardless of the choice of encryption key.

Code emulation was also used for more efficient heuristic detection, as it enabled dynamic code analysis. However, the basic method of signature detection was practically not improved.

An alternative solution to the problem of a sharp increase in the number of threats was behavioral analysis. While traditional anti-virus scanners stored malware signatures in the databases and scanned the code against the signatures in the databases, the behavioral blocker determined whether the program was malicious based on its behavior in the system. If a program performed actions not permitted by rules defined in advance, then the execution of this program was blocked.

The main advantage of a behavioral blocker is its ability to distinguish between “good” and “bad” programs without the help of a professional virus analyst. Since there is no need to analyze each new threat, regular updating of the signature databases becomes unnecessary, and meanwhile, users are reliably protected from new threats.

The problem is that there is some kind of intermediate zone between clearly malicious and acceptable actions. In addition, the same actions can be malicious in a program designed to cause harm, and useful in legitimate software. For example, the low-level data recording used by a malicious program to erase information from a hard disk is completely legitimately used by the operating system. In addition, it is difficult for a behavioral blocker installed on a file server to determine whether a user legally modifies or deletes a document, or is the result of a malicious program.

In this regard, the bulk of antiviruses are still focused on the detection of known malicious programs using the signature method. Nevertheless, behavioral analysis was further developed: in some modern solutions to protect against information threats, it is used in combination with other methods of searching, neutralizing and removing malicious code.

A large percentage of false positives and a low level of detection of malicious software through new technologies does not allow abandoning the signature method of detection and raises the question of its improvement.

There is another method for describing the functionality of executable files, described, for example, in patent US 5752039, which discloses the idea of comparing executable files, updating one executable file to the functionality of another executable file, and also describes a system for structuring executable files by creating logical blocks similar to that described in the invention.

However, the patent does not describe the logical associations of the elements of the executable files present in this invention.

There are other methods for extracting information that is useless for comparing information in the code of an executable file, as described, for example, in application US 20050028149 A1, which discloses the idea of creating a circuit that recognizes a cyclic code, or US 7349931, which discloses the idea of scanning a computer system for searches Files with added code that work with potentially malicious processes.

However, the implementation of these methods does not say anything about getting rid of noisy code from the compiler.

There are also methods for comparing unknown executable files with a collection of known files, as well as compiling collections, described, for example, in application US 20080195587 A1, which describes a method and system that determine the similarity of set models, or in US 7103913, which discloses the idea of a method comparing verified records with already stored records in the system, or in US 7447703, which discloses the idea of creating collections of information that improve employee productivity.

However, the present invention uses various types of malware collections - the list is wider than in the above patents. A file comparison method similar to this invention is used in the presented patent, however, there are differences in its implementation.

An analysis of the prior art and the possibilities that arise when combining them in one system, allows you to get a new result. Accordingly, the technical result of the claimed invention is the provision of a system, method and computer-readable medium that increase the reliability and accuracy of malware detection by comparing executable files through functionality templates.

Disclosure of invention

The present invention is intended to describe and compare the functionality of executable files in order to determine their belonging to collections of known files.

The technical result of the present invention is to increase the reliability and accuracy of detection of malicious software and is achieved by comparing executable files through templates of functionality.

According to a method for determining whether a file belongs to collections of known files based on file comparison using functionality templates, it comprises the steps of: (a) generating functionality templates based on information about an executable file; (b) remove the selected noise information from the functionality templates of the executable file; (c) bring the blocks of the templates of the functionality of the executable file to a normalized form for subsequent comparison; (d) comparing the normalized blocks of the templates of the functionality of the executable file with the blocks of the templates of the functionality of known files and based on the results of the comparison decide on whether the normalized block of templates of the functionality belongs to one of the templates of the known files functionality.

In a private embodiment, information about the executable file can be collected through information collection tools such as a module that emulates program execution; software virtualization module; application control module; a module that disassembles the program and allows you to facilitate further alignment between certain communication functions.

In another private embodiment, the formation of the functionality templates of the executable file is carried out in two stages: dynamic and static. The dynamic stage involves collecting information about the executable file using information collection tools; and the result of the dynamic stage during the formation of the functionality templates for the executable file is a memory dump and an event log. The static stage involves direct tracing of a memory dump and the formation of functionality templates for the executable file.

In another particular embodiment, the executable file functionality templates are of two types: a functionality template that is based on a sequence of blocks; and a functionality template that is based on a sequence of edges. Blocks are collections of elements, such as API functions, links to lines of code, and links to other logical blocks. Ribs are the areas between calls from one block to another.

In another particular embodiment, the extracted noise information includes basic compiler libraries and statically linked functions of standard libraries.

In another particular embodiment, the comparison of the functionality templates of the executable file with the templates from the collections of known files is carried out according to different mathematical algorithms for each type of functionality template, such as the Hamming distance; Levenshtein distance; algorithm for calculating weights; algorithm for extracting characteristic information.

A system for determining file ownership of collections of known files based on file comparisons using functionality templates, comprising: an emulation tool that collects information about an executable file; means for creating functionality templates for the executable file, which, on the basis of the information collected by the emulation tool, forms blocks and makes up functionality templates for the executable file from them; means for processing the functionality templates of executable files, which removes noise information extracted from the functionality templates and brings the blocks of functionality templates to a normalized form; means for storing collections of functionality templates of known files; means for comparing functionality templates of executable files, which compares normalized blocks of functionality templates received from means for processing functionality templates with functionality templates of known files from said storage medium of a collection in accordance with the classification of functionality templates for executable files.

In a particular embodiment, the emulation means includes information collection tools such as a module emulating program execution; software virtualization module; application control module; a module that disassembles the program and allows you to facilitate further alignment between certain communication functions.

A machine-readable medium for determining whether a file belongs to collections of known files based on file comparisons using the functionality templates on which the computer program is stored, when the computer executes the following steps: (a) form the functionality templates based on information about the executable file; (b) remove the selected noise information from the functionality templates of the executable file; (c) bring the blocks of the templates of the functionality of the executable file to a normalized form for subsequent comparison; (d) comparing the normalized blocks of the templates of the functionality of the executable file with the blocks of the templates of the functionality of known files and based on the results of the comparison decide on whether the normalized block of templates of the functionality belongs to one of the templates of the known files functionality.

Additional features and advantages of the invention will be established from the description that is attached, and will in part be apparent from the description, or may be obtained by using the invention. The advantages of the invention will be realized and achieved by means of the structure specifically emphasized in the written description and formulas, as well as the accompanying drawings.

It is understood that the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the claimed invention.

Brief Description of the Drawings

Additional objectives, features and advantages of the present invention will be apparent from reading the following description of an embodiment of the invention with reference to the accompanying drawings, in which:

Figa shows a diagram of the interaction of the means of this invention.

Figb - method of operation of the present invention.

Figure 2 shows an example of a logical block generated as a result of the operation of the present invention.

3A and 3B show examples of signatures of various compilers.

Figure 4 shows a schematic representation of the structure of a functional template.

5 shows a data structure of a functional template.

6 shows a schematic representation of sets of files for the algorithm for allocating standard logical blocks according to the "value of information content".

7 shows a comparison diagram of logical blocks of two functional patterns.

Fig. 8 shows a one-to-one correspondence with a similarity value between logical units of two functional patterns.

Fig.9 shows a table comparing the logical paths of two information templates.

Figure 10 shows a binary representation of the logical paths of a functional template in the form of a sequence of hash functions taken from the names of events / block elements.

11 shows an example of a comparison of blocks according to the Levenshtein distance measurement algorithm.

12A and 12B show the extraction of information from functional patterns for a detection algorithm based on the extraction of characteristic information.

13 shows an example of a functional pattern for a detection algorithm based on the extraction of characteristic information.

Description of Embodiments

The objects and features of the present invention, methods for achieving these objects and features will become apparent by reference to exemplary embodiments. However, the present invention is not limited to the exemplary embodiments disclosed below, it can be embodied in various forms. The essence described in the description is nothing more than the specific details provided to assist the specialist in the field of technology in a comprehensive understanding of the invention, and the present invention is defined only in the scope of the attached claims.

This invention allows to describe and compare the functionality of executable files, collecting and structuring it into functionality templates. In this case, information is collected by both static and dynamic systems, which allows you to collect maximum information about the actions of the program.

Unlike heuristic detection, the invention allows you to build complete structured descriptions of executable files, rather than highlighting specific “malicious” functionality. The method is based on the collection and comparison of exactly the functionality of the executable file; therefore, changing the binary representation of the executable file does not affect the description of the program, and consequently, the comparison results.

Powerful tools for collecting information about the executable file, such as Emulator - a tool for modeling the state of a simulated system, to execute the original machine code; “Sandbox” - a virtualization tool that controls the interaction between the operating system and the program; HIPS, an application control tool, and Tracer, a module that disassembles the program and makes it easy to further build logical functions between certain functions, make it possible to collect a large amount of information about the executable file received for verification. Information in itself collected from various sources in its pure form is not amenable to analysis and comparison. To conduct a qualitative comparison, it needs to be normalized and structured. One option for presenting the collected information is a functionality template.

A functionality template is a collection of information collected and integrated into logical blocks, as well as the relationships between them. A logical block (Logic Block) is a collection of elements such as API functions, links to lines of code, links to other logical blocks, etc. An example of a logical block is shown in Figure 2. Combining the collected information into functionality templates provides several advantages:

the creation of functionality templates follows the clearly defined logical rules described below, therefore, within one family of malicious programs, modifying the functionality templates that describe them will be very limited;

locality: the elements responsible for performing a certain action, whether obtaining privileges or executing requests to the server, are usually located either in one logical block or in a set of blocks interconnected;

since the functionality template has a clear structure, it can be compared with various mathematical algorithms described below;

the structure of the functionality template does not depend on the tools for collecting information, therefore, for different tasks and limitations, the collection tools can be combined;

the number of unique functionality templates is orders of magnitude smaller than the number of unique malicious files, because most files can be similar in functionality and at the same time differ binary from each other. A unique functionality template can be compiled for each program (executable file) and for each family (collection) - it characterizes the functionality of the program or the family of programs as a whole.

By creating functionality templates for already known malware, you can:

compare newly arrived files with them and add automatic records to the databases, subject to similarity;

select characteristic logical blocks from malware collections and create heuristic rules for these blocks;

generate automatic descriptions.

Also, it becomes possible to cluster unknown objects, as well as objects in general, which helps to accelerate their further processing.

The interaction scheme of the means of this invention is shown in Fig.1A. The process of creating functionality templates for executable files consists of two stages. At the first stage, emulation tools 110 dynamically collect information about the executable file during its execution. The emulator 110 may include various information collection tools, such as a module emulating program execution; program virtualization module - “sandbox”; application control module; or a module that disassembles the program and performs some actions that then allow you to build a logical connection between certain functions. At the second stage, using the means of creating functionality templates for executable files 120, functionality templates are generated based on the accumulated information according to logical rules. Further, the tool for processing the functionality templates of executable files 130 separates the templates of the functionality of the executable files by types, based on the logical relationships between sequences of logical elements, and also extracts noise information from the templates of functionality and brings the internal structural elements of the templates of functionality to a normalized form. The functionality template prepared for further analysis is transmitted to the executable file functionality template comparator 140, which compares the internal structural elements of the function templates with the known file functionality templates from the storage tool collections of known file templates 150 using mathematical algorithms in accordance with the classification of the executable file functionality templates . As a result of the comparison, the executable file related to the functionality template refers to the corresponding collection of files.

On figb shows the method of operation of the present invention. According to this method, at step 170, information on the executable file 160 is accumulated using the information collection tools. This data mainly consists of a memory dump 171 and an event log 172. Then, based on the received data, at step 180, executable file functionality templates are generated that are of two types : functionality templates that are based on a sequence of blocks 181; and functionality templates that are based on a sequence of edges 182. The generated functionality templates of the executable file are subjected to additional processing at step 190, during which noise information is extracted from them. After that, at step 191, the blocks of templates of the functionality of the executable file are brought to a normalized form for subsequent comparison. At step 192, the normalized blocks of the templates of the functionality of the executable file are compared with the blocks of the templates of the known files and the results of the comparison at step 193, the decision is made whether the normalized block of the templates of functionality belongs to one of the templates of the known files.

Next, all stages of the operation of this invention will be described in detail.

Creating an executable file functionality template using the functionality template creation tool 120 is carried out in two main steps.

Dynamic stage:

preliminary processing of the executable file by a dynamic information collection system operating within the framework of emulation tools 110. For example, an emulator can be used, although, as mentioned earlier, the ideology of functionality templates does not limit the use of other tools, such as SandBox, HIPS, etc., as well as associations these tools. At the exit from this stage there should be two main components of further analysis:

memory fingerprint (dump);

the event log.

The main information from the event log that is required is a set of the following pairs: {VA ₁ - HASH ₁ , VA ₂ - HASH ₂ , ..., VA _n - HASH _n }, where VA _i is the virtual address of this event element, HASH; 210 in FIG. 2 is a hash function (for example, CRC32) taken on behalf of the event, 220. The event may be a call to an API function, reading the memory of system dlls, etc.

Static stage:

At this stage, the trace of the memory fingerprint is directly generated and the functionality template of the executable file is formed on the basis of all the information collected. To search for primary entry points in the fingerprint, compiler signatures are used.

FIG. 3A shows an example of a signature of a Microsoft Visual C ++ 6 compiler. × l (MSVCRT):

558BEC6AFF68 :::::::: 68 :::::::: 64A100000000506489250000000083 :::: 5356 578965E833DB895DFC6A :: FF15 :::::::: 59830D :::::::: FF830D :::: :::: FFFF15 :::::::: 8 BOD, where :: (two colons) denotes any byte. The hash of the signature is highlighted in the column Hashes 310. Columns 320 and 330 show the corresponding commands and values.

An example of the signature of the Borland C ++ compiler is shown in FIG. 3B:

EB1066623A432B2B484F4F4B90E9. The hash of the signature is highlighted in the Hashes 340 column. Columns 350 and 360 show the corresponding commands and values.

To form logical blocks, we have the following rules:

the logical block begins with the “entry point” of the trace;

The call command, which does not call the API function and leads to the allocated memory area, is a call to the next or existing logical block;

tracing occurs before the ret (ret N) command, int N, jmp N, or before any of the invalid opcode exceptions appear. After that, the presence of a transition for this command is checked. If there is no transition, the logic block ends. If there is, then an attempt is made to trace for this command;

any reference to the code encountered is put in the list of "entry points", i.e. places in the code where the program can start execution;

when a call to an API function, a string reference, or a traceable address matches a dynamic system log, a new corresponding element is added to the logical block.

Schematically, the structure of the template functionality shown in Fig.4. HASH _i , HASH _{i + 1} in blocks 440 and 450 are hashes from functions (including their arguments), for example, from the WriteFile function, the hash of which is equal to ССЕ95612 in block 430. In this case, 2 types of sequences of elements connected among themselves are distinguished:

directly logical blocks and

areas between calls of one logical block to another, called edges. For example, Edge # 1.1 and Edge # 1.2 of block 410.

As mentioned earlier, a logical block is a collection of elements such as API functions, links to strings, links to other logical blocks, etc. In the drawing, the logical blocks are Logic Block # 1 410, Logic Block # 2 420, Logic Block # 3 430, Logic Block # 4 440 and Logic Block # 5 450.

A logical block consisting only of calls to API functions that does not contain transitions is the simplest logical block and is entirely an edge. When a call occurs in a logical block to another logical block, the first logical block will be divided into two edges: from its beginning to the call of the second logical block and from the beginning of the call of the second logical block to the end of the first logical block. Accordingly, if a logical block refers to n other logical blocks, then it will consist of n + 1 edges. For example, in the logic block Logic Block # 1 410 there are edges Edge # 1.1, Edge # 1.2, etc., where CALL Logic Block # 2, CALL Logic Block # 3 - calls to logic blocks Logic Block # 2 220 and Logic Block # 3,430, respectively.

Based on the logical relationships between sequences of elements, two types of functionality templates for executable files were created:

an ordered set of logical blocks, hereinafter we will use the term simple template - a functionality template in which the fundamental factor is the dependencies of the elements inside the logical blocks;

a set of interconnected edges, then we will use the term boundary template - a functionality template in which the dependencies between the edges are a fundamental factor.

For the separation of patterns by type, the means of processing patterns of functionality 130 of the present invention, shown in FIG. 1, are responsible.

In the presented example, FIG. 4, the data structure of the functionality templates will look as follows, as shown in FIG. 5: an ordered set of logical blocks 510 — a list of logical blocks (sorting by size to optimize the process is possible).

To compile a set of interconnected ribs 520, the interconnection of the ribs is built in the order in which they are called in the program code. Each edge occurs in set 520 only once to avoid duplication and cyclization of information.

A logical path (Logic Path), for example, a logical path 530 or 540 - is the path from the "entry point" into the program, i.e. from the place in the code where the program can start execution. Accordingly, there are as many paths as there are "entry points" in the program. In the list of “entry points”, as already mentioned, at the static stage, any encountered links to the code are placed.

Among all the information describing the file and the functionality gathered in the template, there may be “noise” information, such as the compiler’s base libraries, statically linked functions of standard libraries, etc. If you do not remove the noise information, then it is quite possible that a clean program and a malicious program will be more than 90% similar, from which we can conclude that they are similar. It is clear that without isolating, evaluating, and pre-processing such information, comparing functionality templates is dangerous, because under certain conditions, the probability of false positives is high. Logical blocks containing such “noise information” will be called standard logical blocks in the future. In the present invention, the allocation of logic blocks containing noise information from the functionality templates is performed by the functionality template processing means 130 depicted in FIG. 1.

One of the algorithms for the allocation of standard logical blocks is the selection according to the "value of information content", i.e. in magnitude, which characterizes the level of non-informative blocks in terms of our method (explained below). Let's take two sets of files: X - malicious programs, Y - clean programs shown in Fig.6. Moreover, the set will be characterized by signs:

the same compiler for all files in sets;

similar used system libraries (± several libraries);

similar total number of logical blocks (± 15-20%) in the pattern;

similar total number of hashes in each pattern (± 15-20%).

These characteristics are selected, because most of the noise is the base code of the compiler libraries or standard compiler libraries. To automatically find noise blocks for the Visual C ++ compiler, you need to take the set of files compiled by this compiler.

The total number of logical blocks and the number of hashes in the functionality template are the most common similarity parameters. Those. if one functionality template consists of one hash in one logical block, and another of hundreds of hashes in hundreds of logical blocks. The likelihood of similar (noisy) sections appearing in such files is small. As already mentioned, a hash is a hash function taken on behalf of an event.

Select all the unique logical blocks on such a set, i.e. a unique set of hashes. For each unique block, we determine the number of files on each set in which it is present. For each block we get 4 values:

X - total number of malicious patterns 610;

Y is the total number of pure patterns 620;

x is the number of malicious patterns 630 containing a logical block;

y is the number of pure patterns 640 containing a logical block.

Thus, for each block, you can calculate the value of informativeness iGain (X, Y, x, y), which will show how significant this logical block is for dividing the set {X + Y} into two subsets of X and Y, that is, how logical is this the block determines whether the functionality template belongs to a particular collection. It should be noted that a truly malicious block will not occur on many clean templates of 620 functionality, but if it is a compiler block (the same MD5 sum), then it will be found in both clean and malicious files, that is, it there will be very low information content. With low information content, it is possible that the unit belongs to noise information. Subsequently, this uninformative block is deleted from the database. When deleting an element, all logical connections must be saved to save the paths. Such blocks are interesting that are present in the set {X + Y} in more than half of the files and at the same time will have a fairly low information content:

{x + y} ≥ {X + Y} / 2;

iGain≤ε, where ε is a given sufficiently small threshold for information content.

After selecting the standard logical blocks, we move on to algorithms for comparing information patterns.

For example, there is a logical unit consisting of the following functions (runtime of the Visual C ++ compiler):

_except_handler3

_set_app_type

_f_fmode

_p_commode

_adjust_fdiv_XREFF

_setusermatherr

_initterm

_getmainargs

_initterm

_acmdln_XREFF

GetStartupInfoA

GetModuleHandleA

exit

_XcptFilter

In the collection there are:

malicious patterns X = 67;

pure patterns Y = 84.

Moreover, in pure patterns, such a block occurred y = 71 times, and in malicious x = 62 times. It turns out that iGain (P, N, p, n) = 1134 · 10 ^-6 .

Consider another logical block consisting of such functions (obtaining API functions for raising privileges):

"ADVAPIXX.DLL"

GetModuleHandleA

"ADJUSTTOKENPRIVILEGES"

GetProcAddress

"LOOKUPPRIVILEGEVALUEA"

GetProcAddress

"OPENPROCESSTOKEN"

GetProcAddress

GetCurrentProcess

Closehandle

For the same subsets of pure and malicious patterns, such a block among pure patterns occurred y = 2 times, and among malicious ones x = 65 times. It turns out that iGain (P, N, p, n) = 81461 · 10 ^-6 .

A quantitative difference is visible between the iGain values for these logical units.

The first logical block occurs quite a large number of times, both in clean files and in malicious ones, therefore it has low information content for dividing the set M = X + Y into 2 subsets of X and Y. That is, it is not informative and will not be used when creating a functionality template.

The second logical block is often found in malicious files, but rarely in clean ones, therefore it has a high information content value for dividing the set into subsets, therefore it can be used in comparison.

The comparison of the functionality templates between themselves is carried out by means of comparing the functionality templates 140 shown in FIG. 1, and consists in comparing logical blocks (or edges, depending on the type of functionality template). Since there is no direct correspondence between blocks in the functionality templates, the task is to heuristically select only those blocks that can potentially be similar. Moreover, for different types of patterns, the selection heuristic is different. As a result of comparing the functionality template with the functionality templates from the collections of known files 150 shown in FIG. 1, the executable file related to the functionality template will be placed in the corresponding file collection.

Comparison of simple templates:

The comparison scheme is presented in Fig.7. Logical blocks in simple templates are sorted by the number of elements in the block and only those blocks in which the number of elements are approximately equal are compared (the deviation is set by the parameter). A limiting factor is the number of calls inside the logical block of other blocks. We get the matrix 710 M × N, where M and N are the number of logical blocks in a simple template, and in each cell lies the value of similarity between the corresponding logical blocks.

After comparing the logical blocks with each other, a search is made for the best option for combining similar blocks. The best option is that in which, by comparing the "similar" logical blocks of two simple patterns with each other, the best overall percentage of similarity of patterns is obtained. At the same time, to speed up the comparison, the search for the best option occurs only for those logical blocks that may be potentially similar. If you look for the best option without acceleration in all cells of the table, then the complexity of the algorithm would be K !, where K = Max (M, N), because each of the logical blocks is compared with all blocks of another simple template.

On Fig depicts a similar example of a comparison of simple patterns, where 810 are the logical blocks of the first functionality template, 820 are the logical blocks of the second functionality template. Having put them in a one-to-one correspondence with the similarity value between the blocks of both files, it is necessary to find the most optimal combination of similarities. For example, for Fig. 8, the best match would be 1.1-2.3, 1.3-2.4, 1.4-2.2.

Comparison of edge patterns:

Figure 9 shows a comparison table of 910 logical paths (LP _x ). The size of this task increases as Fig. 8 was a comparison of logical blocks, and Fig. 9 shows a comparison of logical paths. Since in the data structure of edge templates there is already not one type of elements, but two (edges and logical paths), respectively, and the comparison is a little more complicated. The comparison takes place in two stages. First, the logical paths are sorted by the number of edges and elements in them. After that, only potentially similar paths are chosen and only they are involved in the comparison and in the search for the best option. The second stage is a comparison of directly two paths with each other, which happens in the same way as comparing ordered sets of logical blocks, only edges are compared instead of logical blocks. The logical path in this situation becomes equivalent to an ordered set of logical blocks.

The “lower” element of comparison (the most elementary element of comparison) is a logical block or edge, depending on the type of functionality template, respectively. Logical blocks / edges are a sequence of elements (events) and can be compared by various algorithms. But in order to compare the blocks with each other, you need to bring them to a certain normalized form, ready for comparison. Bringing to a normalized look is performed by the means of processing templates functionality 130, shown in figure 1.

One of the normalized options is to represent logical blocks in the form of binary strings. The alphabet for the strings will be the hash functions of the names of the events / elements. In this form, working with elements seems to be the most convenient and logically understandable.

Calls of logical blocks, for clarity, will be designated as 00000000 + logical block number, which will be shown later on the example of ordered sets of logical blocks in the form of 00000002 or 00000005.

Having performed the transformations on the above example of logical blocks / edges, we have:

an ordered set of logical blocks, depicted in Figure 5:

Logic Block # 1:

C24FA5F4 447D086B 6CC098F5 00000002 5C856C47 00000003 00000005, where 00000002, 00000003, 00000005 are transitions between hashes. One way to use these values is to restore logical paths. It is possible to implement the storage of these values inside an ordered set of logical blocks.

Logic Block # 2:

FFF372BE 919 V6 BCB 553 V5C78

Logic Block # 3:

C79DC4E3 1DDA9F5D 553 V5C78 CCE95612 00000004 B09315F4

Logic Block # 4:

HASH ₁ , HASH ₂ , HASH ₃ , HASH _i , ..., HASH _N

sets of interconnected ribs:

Logic Path # 1 is represented as the sequence depicted in FIG. 10.

In this form, functionality templates can easily be subjected to various mathematical transformations and comparisons.

Next, we will consider algorithms for comparing logical blocks / edges that were already used for comparison and when working with which positive results were obtained.

Hamming distance is a measure of the difference between code combinations (binary vectors) in the vector space of code sequences. In this case, the Hamming distance between two binary sequences (vectors) X and Y of length n is the number of positions in which they are different. In our case, the vectors X and Y are determined by unique elements in these logical blocks;

Levenshtein distance is a measure of the difference between two sequences of characters (lines) relative to the minimum number of insert, delete and replace operations necessary to translate one line to another. In our case, the rows are logical blocks / edges. Each element (HASH) inside a logical block / edge is a single letter. Elements have no weight, i.e. equivalent. Figure 11 shows the process of calculating the Levenshtein distance when comparing two blocks 1110 and 1120. Based on the definition of this algorithm, it is clear that in this case the Levenshtein distance is 5.

Algorithm for calculating weights (iGain):

each element is assigned a generated weight. Weights are generated by the set {X + Y}, where X is the total number of malicious functionality templates, Y is the total number of pure functionality templates;

X and Y are selected either by a third-party algorithm, or by indirect signs.

As a result, each ID receives the weight w _id , which shows how much it characterizes the functionality template on the X subset. Next, the logical blocks / edges are compared with each other and the best option is found. The functionality template / path has a total weight W = W ₁ + W ₂ + ... + W _n , where W _i is the weight of the ith logical block / edge.

W _i = W _id1 + W _id2 + ... + W _idm - the sum of the weights of all elements of the logical block / edge.

When two blocks are compared, the elements are divided into 2 groups:

there are elements in both logical blocks (and taking into account the quantity). Their weight is W ₊ ;

there are no elements in both logical blocks. Their weight is W _- .

W _cpi is the weight of the comparison. W _cp = W ₊ -W _- . If W _cpi ≤0, then W _cpi = 0 and logical blocks not considered absolutely similar. If W _cpi > 0, then W _cpi / W _i · 100 is the percentage of similarity of logical blocks. W _cpi / W · 100 - percentage of similarity inside the functionality / logical path template.

Total value of interest:

U _max = (W _cp1 + W _cp2 + ... + W _cpn ) · 100 / W.

For example, for the Trojan Downloader family there are functions that it often uses (CreateProcess, UrIDownload, etc.). Also for this family there is a set of functions that are never used or accidentally used. A set of similar Trojan Downloader programs of various versions (X) is taken. For these programs, we allocate logical blocks, identifiers, used libraries. As a Y-set, pure programs or programs that are similar in their primary characteristics and which do not belong to this family are taken. Then, for each hash, the values of x and y are calculated (how many times it was found among malicious functionality templates and among pure functionality templates, respectively). Then, for each hash, the weight of iGain is calculated. Because Since the malware family was initially accurate, iGain will be great for the functions that define this family. For each logical block, the weight is calculated as the sum of the weights of the hashes inside the logical block. Then, for the hashes to be verified, the sum of its occurrences in the logical blocks of the programs from the sets X and Y is compiled. If the weight is positive, then this value is a percentage of the total weight;

algorithm for extracting characteristic information:

the detection algorithm is based on isolating from the functionality template only families of behavioral-determining information in the form of a Logical block or Logical path. Those. A family of similar functionality templates is highlighted. Next, it is analyzed which information is determining (characterizing), which logical blocks (logical paths) are found constantly and are characteristic of the family, and which are mutated or uninformative (for example, performing any auxiliary - not malicious actions, which are also found in clean programs). ) The analysis can take place either automatically (frequency statistics) or carried out by the analyst (or jointly, the statistician is provided with statistical information, and he generates a record from it).

Schematically, the selection of information is presented in Fig.12A and 12B. Where A 1210 - all the information from the functionality template, B 1220 - characteristic information that defines the family, or a set of malicious actions, information.

For two templates of functionality from the family, we can ignore uninteresting (not falling into the pool) information and search only for characteristic information 1250 in FIG. 12. Based on this characteristic information 1320 in FIG. 13, collections of known files 150 depicted in FIG. 1 can be compiled.

In conclusion, it should be noted that the information provided in the description are only examples that do not limit the scope of the present invention defined by the claims. One skilled in the art will recognize that there may be other embodiments of the present invention consistent with the spirit and scope of the present invention.

Claims (13)

1. A method for determining the ownership of files in collections of known files based on file comparisons using functionality templates, comprising the steps of
(a) form patterns of functionality based on information about the executable file;
(b) remove the selected noise information from the functionality templates of the executable file;
(c) bring the blocks of the templates of the functionality of the executable file to a normalized form for subsequent comparison;
(d) comparing the normalized blocks of the templates of the functionality of the executable file with the blocks of the templates of the functionality of known files and, based on the results of the comparison, decide on whether the normalized block of templates of the functionality belongs to one of the templates of the known files functionality. 2. The method according to claim 1, in which information about the executable file can be collected by means of information collection tools such as a module emulating program execution; software virtualization module; application control module; a module that disassembles the program and allows you to facilitate further alignment between certain communication functions. 3. The method according to claim 1, in which the formation of patterns of functionality of the executable file is carried out in two stages: dynamic and static. 4. The method according to claim 3, in which the dynamic stage involves collecting information about the executable file using the information collection tools, and the result of the dynamic stage when forming the functionality templates for the executable file is a memory dump and an event log. 5. The method according to claim 3, in which the static step involves the direct tracing of a memory dump and the formation of functionality templates for the executable file. 6. The method according to claim 1, in which the functionality templates of the executable file are of two types:
functionality template that is based on a sequence of blocks and
A functionality template that is based on a sequence of edges. 7. The method according to claim 6, in which blocks are called sets of elements, such as API functions, links to lines of code and links to other logical blocks. 8. The method according to claim 6, in which edges are called sections between calls from one block to another. 9. The method according to claim 1, in which the selected noise information includes basic compiler libraries and statically linked functions of standard libraries. 10. The method according to claim 1, in which the comparison of patterns of functionality of the executable file with patterns from collections of known files is carried out according to different mathematical algorithms for each type of patterns of functionality, such as the Hamming distance; Levenshtein distance; algorithm for calculating weights; algorithm for extracting characteristic information. 11. A system for determining file ownership of collections of known files based on file comparisons using functionality templates, comprising
emulation tool that collects information about the executable file;
means for creating functionality templates for the executable file, which, on the basis of the information collected by the emulation tool, forms blocks and makes up functionality templates for the executable file from them;
means for processing the functionality templates of executable files, which removes noise information extracted from the functionality templates and brings the blocks of functionality templates to normalized form;
means for storing collections of functionality templates of known files;
means for comparing functionality templates of executable files, which compares normalized blocks of functionality templates received from means for processing functionality templates with functionality templates of known files from said storage medium of a collection in accordance with the classification of functionality templates for executable files. 12. The system according to claim 11, in which the emulation tool includes such information collection tools as a module emulating program execution; software virtualization module; application control module; a module that disassembles the program and allows you to facilitate further alignment between certain communication functions. 13. A machine-readable medium for determining whether files belong to collections of known files based on file comparisons using functionality templates on which a computer program is stored, the execution of which on a computer carries out the following steps:
(a) form patterns of functionality based on information about the executable file;
(b) remove the selected noise information from the functionality templates of the executable file;
(c) bring the blocks of the templates of the functionality of the executable file to a normalized form for subsequent comparison;
(d) comparing the normalized blocks of the templates of the functionality of the executable file with the blocks of the templates of the functionality of known files and, based on the results of the comparison, decide on whether the normalized block of templates of the functionality belongs to one of the templates of the known files functionality.

Images