RU2778979C1 - Method and system for clustering executable files - Google Patents

Abstract

FIELD: computing technology.

SUBSTANCE: method for clustering executable files implemented on a computer apparatus and containing the stages of: obtaining a set of executable files; determining the format of each executable file separately for each file format: finding repeating sequences of a set length in the files; determining the most frequent sequences; attributing files containing at least one most frequent sequence to one family; clearing all files attributed to this family from further processing; repeating the search for the most frequent sequences; attributing files containing at least one most frequent sequence to the next family and clearing said files from further processing until all files are attributed to some family or until the remaining files do not contain repeating sequences; in response to the remaining files not containing repeating sequences, attributing each of said files to a separate family.

EFFECT: ensured automatic clustering of executable files.

17 cl, 7 dwg

Description

FIELD OF TECHNOLOGY

This technical solution relates to the field of computer technology, namely to methods and systems for clustering executable files, which may be necessary when determining whether software belongs to a well-known software family, as well as whether software belongs to one or another group of software authors.

BACKGROUND OF THE INVENTION

This technical solution relates to the field of computer technology, namely to methods and systems for clustering executable files, which may be necessary when determining software ownership. Under the definition of belonging in this case, one can understand both the belonging of the software to a well-known software family, and the belonging of the software to one or another group of software authors.

It is known that professional cybercriminals carefully develop their attack strategy and rarely change it, moreover, they usually use the same malware for a long time, only slightly modifying it.

On the other hand, malware developers who create tools for cybercriminals can use the same software solution for a long time, for example, a function that implements an encryption algorithm, in various malware samples, moreover, created for different cybercriminal groups and related to to different malware families.

Therefore, in the field of cybersecurity, the knowledge that a given malware sample belongs to a certain malware family, as well as the knowledge that a given malware sample was created by a certain author or group of authors, can be very important.

A well-known malware detection method is signature analysis. This method is based on searching files for a unique byte sequence, i.e. a signature that is specific to this particular malware. For each new sample of malware, anti-virus laboratory specialists perform code analysis, on the basis of which they determine the signature of this new malware. The resulting signature is placed in the virus signature database that the antivirus program works with, and this subsequently helps the antivirus detect this malware.

This method is well known to cybercriminals as well. Therefore, almost all modern malicious programs are constantly modified in one way or another, and the purpose of such modification is not to change the main functionality of malware. As a result of the modification, the files of the next version of malware should acquire new properties from the point of view of anti-virus signature analyzers. Then they will no longer be "identified" as malicious using existing signature databases, which will increase the secrecy of cybercriminals' actions.

In addition to modification, a technique called obfuscation is also widely used. It is a reduction of the source text or executable code of the program to a form that preserves its functionality, but makes it difficult to analyze and understand the operation algorithms. The listed changes in malware can be carried out both by a human and automatically, for example, by the so-called polymorphic generator, which is part of a malicious program.

At the same time, it is important that the main functions of HPE, as a rule, do not undergo significant changes. After modification, the malicious program will "look" differently for signature analyzers, its code may be obfuscated and unsuitable for human analysis, but the set of functions that this program performed before modification will most likely remain the same after modification.

Patent RU2728497A1 is known from the prior art (Slipenchuk P.V., Pomerantsev I.S., METHOD AND SYSTEM FOR DETERMINING SOFTWARE OWNERSHIP BY ITS MACHINE CODE, publ. 29.07.2020. class G06F 8/74). It describes a method for determining whether software belongs to a particular family of programs by its machine code, in which a file containing the machine code of the software is obtained; determine the format of the received file; extract and save the code of the functions present in the received file; remove from the saved code functions that are library; allocate commands in each function; allocate a pair of "action, argument" in each command; convert each "action, argument" pair to a number; save, separately for each selected function, the resulting ordered sequences of numbers; accumulating a predetermined number of machine code analysis results and detecting repeating sequences of numbers (patterns) in them; for each detected pattern, a parameter characterizing its frequency is calculated; based on the calculated set of parameters, the classifier is trained to determine the ownership of the software by the sequence of pairs "action, argument"; a trained classifier is used to subsequently determine whether the software belongs to a particular family of programs.

As it is easy to see, for the operation of the described method, a significant array of files containing machine code is required, and this array must contain representatives of all software families, the definition of belonging to which is supposed to train the classifier. Accordingly, the classifier will not be able to identify software samples that belong to families that are not represented in the array of files by the time it is used. And in order to teach him this, you will need to add several representatives of the new family to the array and repeat the described procedures again, from scratch. At the same time, the manual formation of such an array of files containing an assortment of software sufficient for training the classifier is an extremely time-consuming task.

In essence, it is the problem of fast, automatic formation of such arrays of files, clustered by belonging to certain families, that currently hinders the active development of solutions similar to that described in RU2728497A1.

At the same time, the prior art solution US20150178306A1 (YI YANG et al., Method and apparatus for clustering portable executable files, publ. (Portable Executable, PE) files. The method involves extracting the characteristics of the PE file from the PE file; generating a PE file identifier for the PE file based on the characteristics of the PE file; and clustering the base of PE files by PE file IDs.

First of all, it should be noted that this method is initially focused on the analysis of PE format executable files used mainly in the Microsoft Windows operating system. Accordingly, executable files of other formats that do not have the characteristics of a PE file are not subject to analysis by the described method. Whereas malware is quite diverse; it exists for many operating systems such as *nix systems, Android, Apple operating systems, etc.

Also known from the prior art is the solution EP 2743854 B1 (TAO YU, Clustering processing method and device for virus files, publ. files, including: static analysis of binary data of virus files to be classified in order to obtain portable data of the executable structure of virus files; comparing the portable executable structure data of the virus files to be classified and classifying the virus files that have the PE structure data satisfying the predetermined similarity condition into the same category; and performing secondary clustering of virus files in each of the categories classified in the previous step.

It is easy to see that this solution has the same drawbacks as the previous one, that is, it is not designed to work with executable files that do not have the PE format.

Also known from the prior art is the solution EP 2946331 B1 (SOURABH SATISH et al, Classifying samples using clustering, publ. 17.08.2016. class G06F 21/56). This solution discloses a method for classifying a file, including creating a set of samples associated with a file containing marked and unmarked samples; collection of feature values from labeled and unlabeled samples; selection of a subset of functions; clustering together tagged and untagged samples having at least a threshold measure of similarity among the collected values of the selected subset of features to create a set of clusters, each cluster having a subset of samples from the set of samples; recursively repeating the selection and clustering steps on a subset of samples in each cluster in the cluster set until at least one stopping condition is reached, given that there are many iterations of the selection and clustering steps, the first clustering iteration uses the first similarity threshold measure and subsequent clustering iterations use increasingly stringent similarity thresholds, iterations create a cluster that has at least one labeled sample and at least one unlabeled sample, and in the selection step, different subsets of features are selected for different iterations of the set of iterations to determine the amount of deviation in the values of the features of the samples in the cluster from the set clusters created by clustering iteration; and selecting a feature subset for a subsequent clustering iteration in response to a determined amount of variance in feature values of the samples in the cluster from the set of clusters generated by the clustering iteration; and upon reaching at least one stop condition, propagating the label from the at least one tagged instance in the cluster to the at least one untagged instance in the cluster to classify the file associated with the at least one untagged instance.

In relation to this solution, first of all, it should be noted that it requires the presence of marked files, that is, those that are previously marked as malicious, respectively, it is not suitable for working with a sample of arbitrary composition. Further, the solution involves examining data extracted from the internal structure of the file, and is therefore unsuitable for clustering strongly encrypted or obfuscated executables.

The present solution is designed to solve the problems identified in the analysis of the prior art and the creation of an improved complex method for clustering executable files.

SUMMARY OF THE INVENTION

Within the scope of this description, unless otherwise specified at the place of application, the following specific terms are used with the following meanings:

Library functions - functions that implement the most common, typical actions. Library functions are widely used by a wide variety of programs, so their presence in code or in program workflows is not specific to any one program or to any particular developer.

The sequence length is the number of instructions that make up the given sequence. It is important that this value is not related to the way the commands are written (in machine code and in the source code of the program, different ways of writing are used), and does not depend on how many bytes are needed to write the commands that make up the sequence. In other words, the expression "sequence of length 40" means that this sequence consists of 40 commands. About how many bits or bytes are needed to write it, this expression does not say anything.

File clustering is the division of an unordered set of files into clusters (subsets), where each cluster or subset corresponds to a specific software family. The task of identifying the family, that is, determining the functional purpose of the programs included in the family, is not set in this case. The main task of clustering is to assign each file included in the initial unordered set to one of the families, and the number of families required for clustering this set is unknown before clustering.

Obfuscation or obfuscation of the code is bringing the program code to a form that preserves the functionality of the program, but makes it difficult to analyze it statically and understand the operation algorithms.

A software family is a set of programs united by one functional purpose (for example, cryptographers, loaders, etc.) and/or a common team of developers, as well as a basic execution algorithm. Programs that make up the same family differ from each other by various modifications, as a result of which their well-known characteristics, such as checksum, file size, file name, etc., differ. However, all programs included in the same family are designed for the same goal, and all of them perform the same basic steps of the algorithm to achieve this goal. For the purposes of this description, it is assumed that programs having identical functionality but different formats belong to different families, for example, "PE ciphers" and "Mach-O ciphers" are not one, but two different families.

A sequence is a sequence, a "chain" of commands following one after another in the code of the file under study. Moreover, each command itself can have a rather complex structure, consisting, say, of an action and one or more arguments. Commands can be of various lengths, usually in terms of the number of bytes required to write them. As already mentioned, the length of a sequence does not depend on the length of its constituent commands, and is expressed not in bits or bytes, but in units. Speaking, for example, of a sequence of length 50, it is implied that this sequence is a sequence of 50 consecutive commands.

A sample (from the English sample, “sample, instance”) is a non-functional part of a file used only for research, for example, a file containing program code. From the source file of the program, ready for launch and execution, the sample is obtained by deleting various code fragments from this file that are not of interest in the framework of the study. For example, a sample can be obtained from a source file by removing library functions and stable constructs from it.

Durable constructs are pieces of code that can be found in a wide variety of programs. Such constructions are found not only in software of a certain purpose or a certain author, but almost everywhere. As an example of stable constructions, for example, prologues of functions can be named.

A function is a piece of code that can be accessed from somewhere else in the program. In most cases, an identifier (name) is associated with a function. The function name is inextricably linked with the address of the first instruction (operator) included in the function, to which control is transferred when the function is called. After the function has been executed, control returns to the return address, that is, the point in the program where the function was called.

The technical problem to be solved by the claimed technical solution is the creation of a computer-implemented method and system for clustering executable files, which are described in independent claims. Additional embodiments of the present solution are presented in dependent claims.

The technical result consists in providing automatic clustering of executable files.

In a preferred implementation, a computer-implemented method for clustering executable files is claimed, which consists in performing steps in which, using a computing device:

get a lot of executable files,

define the format of each executable file,

separately for each file format:

find repeating sequences of a given length in files,

determine the most frequent sequences,

files that contain at least one most frequent sequence belong to the same family,

exclude all files belonging to this family from further processing,

repeat the search for the most frequent sequences, assigning files containing at least one most frequent sequence to the next family and excluding these files from further processing until all files are assigned to any family, or until there are files left, not containing repeating sequences,

in response to the fact that there are files that do not contain repeated sequences, each of these files is assigned to a separate family.

In a particular embodiment of the method, the sequence length is selected depending on the number of files in which the given sequence was found.

In another particular embodiment of the method, the sequence length is a fixed, predetermined value.

In another particular embodiment of the method, only sequences are considered whose entropy exceeds a predetermined threshold.

In a particular embodiment of the method, the most frequent sequences are found by means of a hash table search.

In another particular embodiment of the method, the decision on whether each file belongs to a given one family is made based on the results of calculating for this file the value of the weighting function, which is the sum of the coefficients of at least two of the most frequent sequences found in the given file.

In another particular embodiment of the method, each coefficient of the weighting function is calculated as the ratio of the number of files in which this sequence is present at least once to the initial number of files.

In another particular implementation of the method, in response to the fact that there are files that do not contain repeated sequences, these files do not belong to any family.

In another preferred implementation, a computer-implemented method for clustering executable files is claimed, which consists in performing the steps in which, using a computing device:

get a lot of executable files,

define the format of each executable file,

run each executable file in a virtual environment corresponding to its format,

a sample is created from each executable file, in which library functions and stable constructions are excluded from the code, and the resulting sample is saved,

after receiving samples corresponding to all received files:

find repeating sequences of a given length in samples,

determine the most frequent sequences,

files corresponding to samples that contain at least one most frequent sequence belong to the same family,

exclude files and samples corresponding to this family from further processing.

repeat the search for the most frequent sequences, assigning files corresponding to samples containing at least one most frequent sequence to the next family and excluding these files and samples from further processing until all files are assigned to any family, or until there are samples left that do not contain repeating sequences,

in response to the fact that there are samples that do not contain repeated sequences, each of the files corresponding to these samples is assigned to a separate family.

In a particular embodiment of the method, the length of the sequence is selected depending on the number of samples in which this sequence was found.

In another particular embodiment of the method, the sequence length is a fixed, predetermined value.

In another possible particular embodiment of the method, only sequences are considered whose entropy exceeds a predetermined threshold.

In another possible private embodiment of the method, the most frequent sequences are found by searching the hash table.

It is also possible to implement a particular variant of the method, in which the decision on whether each file belongs to a given family is made based on the results of calculating for this file the value of the weighting function, which is the sum of the coefficients of at least two of the most frequent sequences found in the sample corresponding to the given file.

In another particular embodiment of the method, each coefficient of the weighting function is calculated as the ratio of the number of samples in which this sequence is present at least once to the initial number of samples.

In another particular embodiment of the method, in response to the fact that there are samples that do not contain repeating sequences, the files corresponding to these samples do not belong to any family.

The claimed solution is also implemented through a system of clustering executable files, containing:

- long-term memory configured to store used files and data;

- a computing device configured to perform the described method.

DESCRIPTION OF THE DRAWINGS

The implementation of the invention will be described hereinafter in accordance with the accompanying drawings, which are presented to explain the essence of the invention and in no way limit the scope of the invention. The following drawings are attached to the application:

Fig. 1A-B illustrate a preferred implementation of a computer-implemented method for clustering executable files.

Fig. 2A-B illustrate another preferred implementation of a computer-implemented method for clustering executable files.

Fig. 3 illustrates the manufacturing steps of the helper program used in one of the preferred embodiments of the method.

Fig. 4 illustrates an example general layout of a computing device.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the implementation of the invention, numerous implementation details are provided to provide a clear understanding of the present invention. However, one skilled in the art will appreciate how the present invention can be used, both with and without these implementation details. In other cases, well-known methods, procedures and components have not been described in detail so as not to obscure the features of the present invention.

Furthermore, it will be clear from the foregoing that the invention is not limited to the present implementation. Numerous possible modifications, changes, variations and substitutions that retain the spirit and form of the present invention will be apparent to those skilled in the subject area.

The present invention is directed to providing an implementation of a computer-implemented method and system for clustering executable files.

As shown in FIG. 1A, the claimed computer-implemented method for clustering executable files (100) can be implemented as follows:

The method starts at a step (SW) in which a plurality of executable files to be analyzed are obtained. These may be pre-compiled files such as EXE or DLL containing a program or dynamic link library, electronic document files with an embedded scripting language (such as VBA), such as DOC or XLS files, as well as files containing program source code in any interpretable language. programming languages such as JavaScript, Python, etc. In all these files or in some part of them, fragments of code or data can also be additionally encrypted, all these files or part of them can also be obfuscated, including using algorithms that are resistant to disclosure.

Executable files can be in any known format, not limited to the PE format; for example, it can also be, without limitation, executable files of the Mach-O, ELF, COM, COFF and other formats.

Next, at step (120), the format of each executable file is determined and a code analysis tool corresponding to the format is selected. Determination of the file format is performed in any known way, for example, by means of a script that compares in turn the signature of the file being checked with the known signatures of various file formats and, if the signatures match, displays a message about the actual file format (which in general may differ from the format associated in the operating system used with the given file extension). For files containing source code in one of the interpreted programming languages, at the same step, the programming language in which the code is written is determined, and the code analysis tool corresponding to the language is selected. Programming language detection can also be done by any known method, such as the Linguist program (https://github.corn/github/linguist) or the Ohcount program (https://github.com/blackducksoftware/ohcount).

The method then proceeds to step (130) where repeated sequences of a given length are found in the files. This step will be described in detail with reference to FIG. 1B.

The step begins with the selection step (131), where, depending on whether the given file contains source code or not, a sequence of further actions is selected.

In response to the fact that the file contains source code, the method then proceeds to step (133).

In response to the fact that the file does not contain source code (ie, the file contains machine or operational code), the method proceeds to step (132) where the code contained in the file is disassembled. Disassembly is performed by a specialized disassembler program that converts machine or operational code into a set of instructions in assembly language or IL language. The disassembler program can be any known one, for example, IDA Pro, Sourcer or another. As a result of the operation of this program, the machine code turns out to be marked up: the boundaries of functions are explicitly distinguished in it.

The method then proceeds to step (133) where the functions are retrieved and stored as a list. This is done by the code analyzer selected earlier in step (120) selected for the given file format. This tool is a parser program, the algorithm of which, in the case of a source code, is based on the syntax of the programming language in which this code is written. In the case of machine code, a parser program is used, the algorithm of which is based on the specification of the computing architecture used.

All code located within the boundaries of functions selected by the disassembler is saved, for example, as a separate file having a text format. The rest (remaining outside these boundaries) code in the described method is considered program data and is not used.

The same is done when analyzing the source code. The parser finds boundaries in the analyzed source code, code fragments inside of which are functions. For this, one can use, for example, the well-known fact that in most high-level programming languages, the body of a function is enclosed in a block statement. Accordingly, one of the possible algorithms for the parser can be to identify pairs of words or symbols in the code, which in the syntax of this language denotes a block operator, additional verification of the fact that the found block operator contains exactly the body of the function, as well as moving the code that makes up the body of the found function , to a file where the processing results are saved. Checking the fact that the block statement contains exactly the body of the function can be performed, for example, by searching in the line preceding the opening of the block statement, a symbolic expression corresponding to the function name in the syntax of this programming language.

Alternatively, if the source code is written in a C-like programming language, such as JavaScript, then each function will begin with a function header in a known format. For example, in the JavaScript code for the following example, the header of each of the two functions can be identified by the presence of the function keyword. The header in such code will be followed by the body of the function, also enclosed in a block statement, in this case denoted by a pair of curly braces, { and }:

Actions on source code left outside block statements (that is, function boundaries) are performed depending on whether the syntax of a given programming language allows execution of code outside functions. For example, for source code written in C or C#, such code is ignored. In this example, for code written in JavaScript, such code is considered to be related to one more function, separate from all those already selected. The code of this function is also transferred to the list of functions and further processed in the same way as all other code placed in this list. Thus, at step (133), the code of all functions available in the analyzed file is obtained and stored as a separate file.

The method then proceeds to step (134) where the file containing the found functions is parsed and commands are extracted within the functions. To extract commands, the results of the previously performed disassembly are used. For example, when disassembling the following piece of machine code

IDA Pro will highlight the following command:

having a machine code representation

Said analysis can be performed in any known way that allows to obtain this result, for example, by a script specially written for this purpose. The algorithm of such a script must parse the analyzed fragment of machine code in accordance with the specification of the architecture used; in this example, this is the x86 architecture.

The fragments identified in the machine code corresponding to individual commands are stored, for example, each on a separate line of a text file. In one possible implementation of the method described, arguments such as the 10000h argument are considered part of the command and stored. In another possible implementation, arguments are ignored and only commands are stored, such as the mov dword_423F3C command.

What exactly is considered a separate command when analyzing the source code depends on the syntax of the programming language in which the code under study is written. So, in one of the possible implementation options, the beginning of the line can be considered the beginning of the command, and the end is the character closest to the beginning of the line, which means the end of the command in this programming language, such as the semicolon character (;) in the C language.

In an alternative implementation of the method, a command can be considered a sequence of characters located on a separate line. For example, within the framework of this approach, as part of the disabieSecurity function from the above source code fragment (1), written in JavaScript, six commands will be selected, according to the number of lines:

As in the analysis of machine code, the fragments corresponding to individual commands are stored, for example, each command on a separate line of a text file. After that, the method proceeds to step (135), in which sequences of a given length are formed from the selected commands with a given step.

Within the framework of this description, we will call a sequence a sequence, a “chain” of commands following one after another in the code of the file under study. It is clear that depending on whether the source code is analyzed or machine code, commands can have a rather complex structure and different lengths, contain various arguments, modifiers, and so on. All these circumstances are not taken into account in the formation of sevections.

So, in one of the possible implementations of the described method, the found commands can be preliminarily converted into numbers for convenience. This can be done, for example, by taking a hash function of the full text of each command, or by any other well-known way of converting a string to a number. It is clear that the same commands will be converted to the same numbers. In addition, for this purpose, such a conversion algorithm can be chosen, in which commands, the text of which differs slightly, will be converted into numbers close in absolute value, for example, the fuzzy hashing algorithm SSDeep.

For the sake of simplicity of the example explaining the formation of sequences, we will conditionally denote individual commands by letters of the alphabet, moreover, we will assume that each command is denoted by one letter, and the same commands are denoted by the same letters. Of course, in reality, the number of commands that can be extracted from a single executable file greatly exceeds the number of letters in any alphabet. So, in this example, as a result of the execution of the above stage (134), for some executable file, we received the following set of commands:

Suppose, in accordance with the predefined processing conditions, sequences of length 5 should be formed from this set of commands with a step of 1.

It is important to emphasize that the length of a sequence does not depend on the length or other parameters of its constituent commands; sequence length is expressed not in bits or bytes, but in units. Speaking, for example, of a sequence of length 5, it is implied that this sequence is a sequence of five consecutive commands.

The parameter, called the step, in this case determines how many commands the beginning of each next sequence will be from the beginning of the previous sequence. In the case of example (b), with step equal to 1, the first sequence will start with command A, and the second with command B. With step equal to 2, the second sequence will start with command C, with step equal to 3, the second sequence will start with command D, and so on.

Under the given conditions (length 5, step 1), the following sequences will be formed and stored, for example, in a database, from the instruction set (b):

Conditions for the formation of sequences, length and step, can be fixed values and be set in advance, at the stage of developing a system that implements the method. In an alternative embodiment of the described method, the length of the sequence and the step of its formation can be selected automatically depending on the fulfillment of predetermined conditions. For example, the length of the sequence was initially set to 100 commands, and during the analysis of the resulting sample of files, it was found that there was not a single repeating sequence in the resulting sample of files, that is, there is not a single sequence that would be present in at least two different files. Then the length of the sequence is reduced to 90 commands, the sequences are re-formed and it is checked whether at least one sequence repeated in two or more files is found.

Alternatively, if the length of the sequence was initially set to 100 commands, and during the analysis of the resulting sample of executable files it was found that one of the sequences is present in all files, the sequence length can be automatically increased to 110 commands, after which the sequences are formed anew and it is checked that none of the sequences is present in all files that make up the resulting sample.

In another possible implementation, if the described requirements for the frequency of the sequence are not met, the step can be automatically increased or decreased. A variant is also possible in which the step and length of the sequence are increased simultaneously, as well as a variant in which the step is increased and the length of the sequence is reduced.

In one of the possible embodiments of the described method, at step (135), the entropy of each generated sequence is additionally calculated, and the sequence is stored in the database only if its entropy exceeds a predetermined threshold. The meaning of such a check is to exclude sequences from processing where there are too many repetitive commands. In this case, the entropy can be calculated in any well-known way, for example, as the average message entropy, according to the Shannon formula:

where n is the number of commands (length) of the sequence, and pi is the probability of the presence of a certain command in the i-th position within this sequence.

On the example of sequences (7), it is easy to see that the entropy of sequences in which there are no repeated commands, for example, sequences a in c d e, will be higher than the entropy of sequences in which repeated commands are present, as, for example, in the sequence D E F D E.

In another possible embodiment of the described method, at step (135), only those sequences are stored whose entropy is greater than the first predetermined threshold, but less than the second predetermined threshold. Thus, sequences are selected and stored that have a certain number of repeated commands, but those where this number is not more than a given value.

It should be noted that at step (135), for each sequence, an indication of the file from which it was obtained is also stored in the database, for example, by specifying the file name or the full path to the file, also including its name and extension.

As a result of step (135), a database is obtained, in which, separately for each file of the initial set obtained at step (110), all sequences found in this file and having a given entropy value are stored. It is also possible to implement the described method, in which the entropy of sequences is not calculated, but stored in the database, separately for each file of the original set, all the sequences found in it.

Said database can be implemented in any suitable way, for example, it can be a hash table or any other well-known data structure that implements the associative array interface and allows storing sequence-value pairs, where the value field implies the storage of a numerical value.

At this step (130), described in relation to FIG. 1B ends and the method proceeds with reference to FIG. 1 A, to step (140).

At step (140), the most frequent sequences are determined, that is, those sequences that are most often found in the set of executable files obtained at the step (PO). This may be done in any conventional manner. For example, first, for each next sequence from the total set of sequences, it is calculated how many times it occurs in the studied set of executable files; the counted number is stored in the "value" field corresponding to this sequence. Then, when the counting for all sequences is completed, the actual frequency is calculated, that is, the ratio of the number of occurrences of this sequence L to the total number of sequences K. This parameter can be either less than one if the sequence is not found in every executable file, or more if the sequence occurs on average several times throughout each file:

The frequency value calculated for each sequence is stored in the database in the "value" field corresponding to this sequence. This completes step (140) and the method proceeds to step (150).

In step (150), all files that contain at least one most frequent sequence are assigned to the same program family. This can be done in any suitable manner, such as hash table lookups, machine learning techniques, and so on.

At this step, one or more sequences are selected that have the highest numerical frequency value. Since the database also stores for each sequence an indication of the file from which it was obtained, it is not difficult to determine in which particular files of the set of executable files obtained at the step (software) this one or these several sequences occur. These files are considered to constitute one family of programs.

In an alternative implementation, the decision on whether each file belongs to a given family is made based on the results of calculating for this file the value of the weighting function of the form F(A, B, C, D, D, k1, k2, k3, k4, k5), where A ... D - the five most frequent sequences, and k1…k5 - coefficients, meaning the frequency of each of these sequences. If the value of F is higher than the threshold value selected in advance, then the given file is considered to belong to the given family. It is also possible to implement the described method, which takes into account a different number of the most frequent sequences, for example, without limitation, two, three, and so on.

In this case, the actual weighting function F can be calculated, for example, as

where bin(A) is a binary value equal to 1 if sequence A is found in the given file, and equal to 0 if not found, bin(B) is a similar value for sequence B, etc.

In one embodiment of the method, the coefficients k1...k5 can be calculated as the frequency of this sequence (9). In another possible implementation, they can be calculated as the ratio of the number of files in which the given sequence is present at least once, to the total number of files. Any other options for calculating the function F are also possible, taking into account the contribution of each of the coefficients and the presence of specific sequences in this file.

The actual assignment of files to one family may mean, for example, creating a separate folder and copying the corresponding files to this folder. In an alternative implementation, in a text file that stores a general list of file names obtained in step (110), the names of files assigned to the same family are labeled, for example, "Family 1". Any alternative implementation of this step is also possible, allowing you to indicate the fact that specific files included in the original set of executable files belong to the same family.

This completes step (150) and the method proceeds to step (160) where all files belonging to the given family are excluded from further processing. At step (150), the analysis is carried out in a database that stores the sequences created at step (140) and their corresponding numerical parameters that have the meaning of frequency. Therefore, exclusion from further processing of files related to the same family can technically mean, for example, exclusion from the database of those sequences for which the database stores references to these files.

In an alternative implementation for those sequences that relate to the files to be excluded, the database can be set to zero numerical values that have the meaning of frequency. As a result, due to the peculiarities of the algorithm used in step (150), the sequence data and their corresponding files will be excluded from further analysis.

This completes step (160) and the method proceeds to step (170). At this step, the search for the most frequent sequences is repeated, the assignment of files containing at least one most frequent sequence to the next family and the exclusion of these files from further processing until all files are assigned to any family, or until there will be files that do not contain repeated sequences.

In other words, step (170) is a set of sequentially performed steps (150) and (160) described above, and they are performed cyclically until all files from the set obtained in step (110) are assigned to one or family.

In the event that at the end of step (170) in the original set of executable files there is not a single file that does not contain repeating sequences, the described method (100) is completed.

It is possible that at the end of step (170) in the original set of executable files, a certain number of files remain, and they do not contain a single repeating sequence. In this case, the method proceeds to step (180).

At step (180), each of the remaining files, in which there is no repeating sequence, is assigned to a separate family. In other words, at step (180), as many new families of programs are created as there are files left that do not contain repeated sequences, and one of the remaining files is assigned to each of these new families. This completes the described method (100).

As an alternative to method (100) described above with reference to FIG. 1A-B, it is also possible to implement the claimed computer-implemented method for clustering executable files, as described below with reference to FIG. 2A-B.

As shown in FIG. 2A, the claimed computer-implemented method for clustering executable files (200) can be implemented as follows:

The method starts at step (202) where a plurality of executable files to be analyzed are obtained. Completely similar to the above step (BUT), these can be already compiled files like EXE or DLL containing a program or dynamic link library, electronic document files with an embedded scripting language (like VBA language) like DOC or XLS files, as well as files containing the source code of programs in any interpreted programming languages, such as JavaScript, Python, etc. In all these files or in some part of them, fragments of code or data can also be additionally encrypted, all these files or part of them can also be obfuscated, including using algorithms that are resistant to disclosure. Executable files can be in any known format, not limited to the PE format; for example, it can also be, without limitation, executable files of the Mach-O, ELF, COM, COFF and other formats.

The method (200) then proceeds to step (204) where the format of each executable file is determined and a code analyzer corresponding to the format is selected. This is done in exactly the same way as step (120) described above.

After step (204) is completed, the method (200) proceeds to step (206), where each executable file is launched in an isolated environment corresponding to its format. The sandbox itself can be any suitable environment for running the file, and can be selected automatically based on the file format information obtained in the previous step (204).

Thereafter, the method proceeds to step (208) where a sample is created from the executable file. A sample is a non-functional, research-only part of an executable file. Step (208) will be described in detail below with reference to FIG. 2B.

As shown in FIG. 2B, step (208) begins at step (210) where dumps are taken and machine code is extracted from the dumps. Indeed, any executable running in an isolated environment spawns a number of processes in the working memory of the environment. At this stage, by accessing the working memory of the environment, dumps are obtained, that is, "fingerprints" of the working memory of the processes generated by the analyzed file. If necessary, such dumps can be done very often, literally for every clock cycle of the main processor of a computer system, which allows you to study in detail the processes taking place in memory. Dumping can be done by any known method, for example, using the ProcDump utility.

Since each memory dump is a text of the machine code in memory at the time the dump was taken, the machine code of the program in the file under investigation is obtained in the above manner at step (210). At the same time, the fact that the data source is the memory of working processes makes it possible to obtain "clean", not encrypted and not obfuscated machine code, even if the file under study was strongly encrypted, obfuscated, etc. Whatever modification the previously examined file was subjected to, at the moment when the processes generated by it are in memory, the machine code of these processes characterizes exactly those actions for which the file was created.

The machine code extracted from the dumps is saved, for example, as a text format file. At this stage (210) is completed and the method proceeds to stage (220), where the code of all functions present in it is extracted and stored from the received machine code. This is done by a disassembler program, as well as a parser program, the algorithm of which is based on the specification of the computing architecture corresponding to this file, such as the x86 architecture.

At step (220), functions are extracted from the machine code and stored as a list of functions. This is performed in a completely analogous manner to step (133) described above. At the end of step (220), the code of all functions available in the analyzed file is received and stored as a separate file, after which the method proceeds to step (230).

At step (230), those functions that are library functions are found and removed from the received function code. To do this, analyze the file obtained during step (220) and containing the found functions. With the help of signature analysis, those functions that are related to library ones are excluded from the list. Library functions are standard tools. They are widely used by a wide variety of programs, so their presence in native code or in workflows is not specific to any one particular program. Exclusion of library functions makes it possible to significantly simplify the analysis and at the same time obtain a better result due to the fact that only those instructions that best characterize a particular program will remain in the code.

Signature analysis and removal of library functions are performed by an auxiliary script program. The algorithm of this script can be, for example, comparing each of the functions in turn with a set of signatures (regular expressions) prepared in advance. Each of these signatures corresponds to a specific, pre-described in the form of a library function signature; when a function is found that matches any signature, all code that makes up the body and header of such a function is excluded from the file.

Upon completion of processing by the script, the file containing the remaining functions is saved and thus the text of the code of the functions that are present in the analyzed file, and, moreover, are not library functions, is obtained. At this point, step (230) ends and the method proceeds to step (240), where code sections that correspond to stable constructs are removed from the remaining code, that is, code fragments that are equally common in any programs, for example, function prologues. This is done through a pre-made program.

The production of the named program (300) can be performed as shown in FIG. Z, as follows.

At step (310) a sufficient number, for example, 1000 copies of programs belonging to different families, are selected. This auxiliary collection can include both obviously malicious programs, for example, datastealers, downloaders, worms, and obviously legitimate ones, for example, text editors, image viewers, archivers, and so on. The collection may also include electronic document files with an embedded scripting language (for example, VBA), such as DOC or XLS files, as well as files containing the source code of programs in any interpreted programming languages, such as JavaScript, Python, and etc.

At the next step (320), the specified auxiliary selection is processed by an algorithm that detects repeated character sequences and counts the number of occurrences of this character sequence in the total code array. In this case, the minimum length of the sequence is set, for example, equal to 20 characters, and the maximum length of the sequence is not limited. In an alternative embodiment, both the minimum and maximum sequence lengths are specified, eg, 15 and 250 characters, respectively.

The described algorithm returns a list of found repeating sequences of characters, where each sequence is assigned the number of its occurrences in the auxiliary collection of files.

Then, in step (330), the most frequent sequences are selected from the list of stable sequences returned by the algorithm. For example, those sequences are selected that occur on average at least once in each program, i.e. in this example, those that occur 1000 times or more.

At the next step (340), a program or script is created that will remove exactly those sequences that were selected in the previous step from any code.

Thus, a program is obtained that is capable of processing arbitrary code fragments, removing from them those sequences of characters that are included in the "removal list" as frequently occurring in different programs.

Returning to Fig. 2B, after processing by the program (300) at step (240), a file is obtained in which stable constructions have been removed from all functions identified in the analyzed file, that is, such code fragments that are quite often found in any arbitrarily chosen programs.

At this point, step (240) ends and the method proceeds to step (245). At this stage, the file obtained in the previous stage is analyzed and the commands are isolated within each function. Step (245) is performed in exactly the same way as step (134) described above, and as a result, for each source set file, a sample is obtained in which both library functions and stable constructs are removed from the code.

After step (245) is completed, the method proceeds, with reference to FIG. 2A to step (248) where the received samples are stored. It should be noted that at step (248), for each sample, an indication of the file from which it is obtained is also stored in the database, for example, by specifying the file name or the full path to the file, also including its name and extension, or otherwise , by any known method. That is, in one way or another, but for each saved sample, information is also stored about which particular data file the sample was obtained from.

At this stage (248) ends and the method proceeds to stage (250), where the samples are analyzed and the files corresponding to them are assigned to certain families. The execution of step (250) is described in detail below with reference to FIG. 2B.

As shown in FIG. 2B, the execution of step (250) begins at step (255), where repeated sequences of a given length are found in the samples obtained in the previous step. This is done in the same way as described previously with reference to FIG. 1B to stage (135), with the only difference that the sequences in this case are formed from the commands following one after another in the code of the sample under study, and not in the code of the file under study. In other words, in step (255), the same actions are performed as previously described for step (135), but with respect to samples, and not to files.

Completely similarly to the above, as a result of step (255) a database is obtained in which, separately for each sample, all sequences found in this sample and having a given entropy value are stored. It is also possible to implement the described method, in which the sequence entropy is not calculated, but stored in the database, separately for each sample, all the sequences found in it. It is clear that on the basis of this information, it is not difficult to build an inverse relationship in any known way, that is, to find in which samples this sequence is present. This completes step (255) and the method proceeds to step (260).

In step (260), the most frequent sequences are determined, that is, those sequences that are most often found in the resulting set of samples. This can be done in a similar manner to step (140) above. The frequency value calculated for each sequence is stored in the database in the "value" field corresponding to this sequence. This completes step (260) and the method proceeds to step (265).

At step (265), files corresponding to those samples that contain at least one most frequent sequence are assigned to the same family. This is performed similarly to step (150) described above, with the only difference being that at least one of the most frequent sequences is found in the samples, and files corresponding to these samples are assigned to one program family.

As described above, at this step, one or more sequences are selected that have the highest numerical frequency value. Since the database also stores for each sequence an indication of the sample or samples where it occurs, and for each sample at step (248) an indication of the corresponding file was stored, it is not difficult to determine which files of the set obtained at step (110) executable files correspond to this one or these several sequences. These files are considered to constitute one family of programs.

This completes step (265) and the method proceeds to step (270), which excludes files and samples corresponding to this family from further processing. Since the analysis is carried out in a database that stores the created sequences and their corresponding numerical parameters that have the meaning of frequency, the exclusion from further processing of samples assigned to the same family can technically mean, for example, the exclusion from the database of those sequences for which the database contains pointers to these samples.

In an alternative implementation for those sequences that relate to samples to be excluded, the database can be reset to zero numerical values that have the meaning of frequency. As a result, due to the peculiarities of the algorithm used, the sequence data and their corresponding samples will be excluded from further analysis. It is also clear that after performing this operation, the most frequent sequences remaining in the database will be sequences with frequency scores following those of the sequences just excluded.

This completes step (270) and the method proceeds to step (275). At this step, the search for the most frequent sequences is repeated, the assignment of files corresponding to samples containing at least one most frequent sequence to the next family and the exclusion of these files and samples from further processing until all files are assigned to any family, or until there are no samples that do not contain repeating sequences.

In other words, step (275) is a set of sequentially performed steps (260) ... (270) described above, and they are performed cyclically until all files from the set obtained at step (110) are assigned to some or family.

In the event that at the end of step (275) in the analyzed set of samples there is not a single sample left that does not contain repeating sequences, the described method (200) ends here.

A situation is possible in which, at the end of step (275), a certain number of samples remain in the analyzed set of samples, and they do not contain a single repeating sequence. In one of the possible implementations of the method, the files corresponding to these sequences do not belong to any of the families, the method (200) also ends. In another possible implementation in such a case, the method proceeds to step (280).

At step (280), each of the files corresponding to the samples, in which there is no repeating sequence, is assigned to a separate family. In other words, at step (280), as many new families of programs are created as there are samples left that do not contain repeated sequences, and one of the remaining files is assigned to each of these new families.

This completes the described method (200).

The following are additional embodiments of the present invention.

In a particular embodiment of the method, the sequence length is selected depending on the number of files in which the given sequence was found.

In another particular embodiment of the method, the sequence length is a fixed, predetermined value.

In another particular embodiment of the method, only sequences are considered whose entropy exceeds a predetermined threshold.

In a particular embodiment of the method, the most frequent sequences are found by means of a hash table search.

In another particular embodiment of the method, the decision on whether each file belongs to a given one family is made based on the results of calculating for this file the value of the weighting function, which is the sum of the coefficients of at least two of the most frequent sequences found in the given file.

In another particular embodiment of the method, each coefficient of the weighting function is calculated as the ratio of the number of files in which this sequence is present at least once to the initial number of files.

In another particular implementation of the method, in response to the fact that there are files that do not contain repeated sequences, these files do not belong to any family.

In another particular embodiment of the method, the length of the sequence is selected depending on the number of samples in which this sequence was found.

It is also possible to implement a particular variant of the method, in which the decision on whether each file belongs to a given family is made based on the results of calculating for this file the value of the weighting function, which is the sum of the coefficients of at least two of the most frequent sequences found in the sample corresponding to the given file.

In another particular embodiment of the method, each coefficient of the weighting function is calculated as the ratio of the number of samples in which this sequence is present at least once to the initial number of samples.

In another particular embodiment of the method, in response to the fact that there are samples that do not contain repeating sequences, the files corresponding to these samples do not belong to any family.

The claimed solution is also implemented through a system of clustering executable files, containing:

- long-term memory configured to store said files, as well as a database;

- a computing device configured to perform the described method.

On FIG. 4, the general scheme of the computing device (400) will be presented below, providing the data processing necessary to implement the claimed solution.

In general, the device (400) contains such components as: one or more processors (401), at least one memory (402), data storage medium (403), input/output interfaces (404), I/O means ( 405), networking tools (406).

The processor (401) of the device performs the basic computing operations necessary for the operation of the device (400) or the functionality of one or more of its components. The processor (401) executes the necessary machine-readable instructions contained in the main memory (402).

The memory (402) is typically in the form of RAM and contains the necessary software logic to provide the desired functionality.

The data storage facility (403) can be implemented in the form of HDD, SSD disks, raid array, network storage, flash memory, optical information storage devices (CD, DVD, MD, Blue-Ray disks), etc. The facility (403) allows perform long-term storage of various types of information, such as the above-mentioned files, intermediate data, lists, databases, etc.

Interfaces (404) are standard means for connection and operation, such as USB, RS232, RJ45, LPT, COM, HDMI, PS/2, Lightning, Fire Wire, etc.

The choice of interfaces (404) depends on the specific implementation of the device (400), which can be a personal computer, mainframe, server, server cluster, thin client, smartphone, laptop, and the like.

The keyboard may be used as data I/O (405). In addition to the keyboard, the following I/O devices can also be used: joystick, display (touchscreen), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

Means of network interaction (406) are selected from devices that provide network reception and transmission of data, for example, an Ethernet card, WLAN/Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. With the help of means (406) the organization of data exchange over a wired or wireless data transmission channel, for example, WAN, PAN, LAN (LAN), Intranet, Internet, WLAN, WMAN or GSM, is provided.

The components of the device (400) are coupled via a common data bus (410).

In these application materials, two options for the preferred disclosure of the implementation of the claimed technical solution were presented, which should not be used as limiting other, private embodiments of its implementation, which do not go beyond the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims (39)

1. A method for clustering executable files performed on a computer device, a method comprising steps in which: get a lot of executable files, define the format of each executable file, separately for each file format: find repeating sequences of a given length in files by converting the file into source code, analyzing the source code in order to determine the boundaries in it, code fragments inside which are functions, then selecting commands inside the functions from which sequences of a given length are formed, determine the most frequent sequences, files that contain at least one most frequent sequence belong to the same family, exclude all files belonging to this family from further processing, repeat the search for the most frequent sequences, assigning files containing at least one most frequent sequence to the next family, and excluding these files from further processing until all files are assigned to any family or until there are files left that do not containing repeating sequences, in response to the fact that there are files that do not contain repeated sequences, each of these files is assigned to a separate family. 2. The method according to claim 1, characterized in that the length of the sequence is selected depending on the number of files in which this sequence was found. 3. The method according to claim. 1, characterized in that the length of the sequence is a fixed, predetermined value. 4. The method according to claim 1, characterized in that only sequences are considered whose entropy exceeds a predetermined threshold. 5. The method according to claim 1, characterized in that the most frequent sequences are found by searching the hash table. 6. The method according to claim 1, characterized in that the decision on whether each file belongs to this one family is made based on the results of calculating for this file the value of the weighting function, which is the sum of the coefficients of at least two of the most frequent sequences found in this file. 7. The method according to claim 6, characterized in that each coefficient of the weighting function is calculated as the ratio of the number of files in which the given sequence is present at least once to the initial number of files. 8. The method according to claim 1, characterized in that, in response to the fact that there are files that do not contain repeating sequences, these files do not belong to any family. 9. A method for clustering executable files, performed on a computer device, a method comprising steps in which: get a lot of executable files, define the format of each executable file, run each executable file in a virtual environment corresponding to its format, a sample is created from each executable file, in which library functions and stable constructions are excluded from the code, and the resulting sample is saved, after receiving samples corresponding to all received files: find repeating sequences of a given length in samples by analyzing the machine code of the sample in order to determine the boundaries in it, code fragments inside which are functions, then selecting commands inside the functions, from which sequences of a given length are formed, determine the most frequent sequences, files corresponding to samples that contain at least one most frequent sequence belong to the same family, exclude files and samples corresponding to this family from further processing, repeat the search for the most frequent sequences, assigning files corresponding to samples containing at least one most frequent sequence to the next family and excluding these files and samples from further processing until all files are assigned to any family or until there will be no samples that do not contain repeated sequences, in response to the fact that there are samples that do not contain repeated sequences, each of the files corresponding to these samples is assigned to a separate family. 10. The method according to claim 9, characterized in that the length of the sequence is selected depending on the number of samples in which this sequence was found. 11. The method according to claim 9, characterized in that the length of the sequence is a fixed, predetermined value. 12. The method according to claim 9, characterized in that only sequences are considered whose entropy exceeds a predetermined threshold. 13. The method according to claim 9, characterized in that the most frequent sequences are found by searching the hash table. 14. The method according to claim 9, characterized in that the decision on whether each file belongs to a given family is made based on the results of calculating for this file the value of the weighting function, which is the sum of the coefficients of at least two of the most frequent sequences found in the sample corresponding to this file . 15. The method according to claim 14, characterized in that each coefficient of the weighting function is calculated as the ratio of the number of samples in which the given sequence is present at least once to the initial number of samples. 16. The method according to claim 9, characterized in that, in response to the fact that there are samples that do not contain repeating sequences, the files corresponding to these samples do not belong to any family. 17. System for clustering executable files, comprising: - long-term memory configured to store used files and data; - a computing device configured to perform the method according to any one of paragraphs. 1-16.

Images