RU101223U1 - SYSTEM FOR QUICK DETECTION OF SIMILAR OBJECTS USING CONVOLUTIONS - Google Patents

Abstract

 1. A system for detecting similar objects when comparing unknown objects with already known objects, which includes! a) a means of storing a collection of known objects designed to store known objects and associated with a means of counting convolutions; ! b) a convolution counting tool for counting convolutions of known objects, also associated with the convolution update tool; ! c) a convolution update tool associated with a convolution collection tool of known objects storage means for updating convolution of known objects in a convolution collection tool of known objects; ! d) said means for storing a collection of convolutions of known objects, intended for storing convolutions of known objects and associated with a local means of updating the convolution; ! e) said local convolution updating tool for updating convolutions of known objects in a local convolution collection of known objects associated with a local convolution collection tool for known objects; ! e) said local means for storing convolution collections of known objects, intended for storing convolutions of known objects, associated with a local means for comparing convolutions; ! g) said local convolution comparison tool for detecting similar objects by comparing convolutions of known objects from a local means of storing a collection of convolutions of known objects and convolutions of unknown objects from a local convolution counter associated with the local convolution counter; ! h) the mentioned local remedy �

Description

Technical field

The utility model relates to systems for detecting similar objects when comparing unknown objects with already known objects.

State of the art

Currently, one of the main problems of the antivirus industry is the ever-growing number of malware, such as viruses, trojans, worms, rootkits, and others. To detect all new types of malware, antivirus companies constantly update their antivirus databases that contain malware signatures, as well as improve other methods for analyzing unknown programs for malicious functionality. However, it is becoming increasingly difficult to detect malware that sometimes differs very slightly, for which different signatures are obtained. At the same time, the number of analyzed files is constantly growing, which also creates a large computational load and leads to the fact that anti-virus databases are updated with a delay or they do not have new malware.

Thus, antivirus companies have an important task - to determine that a new unknown file contains malicious functionality for a limited time. One of the problems that arise when solving this problem is the fact that malicious functionality can be implemented in different ways, each of which will give the same result. Consider an example of a small piece of code written in C:

int i = 0, j = 0;

int main ()

{

if (i == 1000)

return i;

else

return j;

}

and compare it with the second code fragment, which turned out after compiling the above fragment into a low-level assembly language:

_main proc near

mov ecx, dword_414184

mov ecx, 3E8h

cmp ecx, eax

jz short locret_401124

mov eax, dword_414180

locret_401124:

retn

_main endp

Small changes in a fragment of code written in C that will not affect its functionality will be reflected differently in the second fragment. For example, when a conditional jump changes from “if (i == 1000)” to “if (i = 999),” as a result, the fragment will be compiled into another fragment in assembly language. Adding instructions such as NOP (no operation), changing the names of variables and other actions that do not affect the final result can hide or obfuscate (English obfuscate) real functionality, which can also be malicious. In addition, various processor registers can be used in the object code, different from those used by conventional compilers (for example, you can use a different register instead of the AX register). Ultimately, by modifying object code in different ways, it is possible to achieve that there will be practically no fragment left that could be compared with already known fragments of malicious code.

Thus, the signature verification method, despite its simplicity, is far from the most reliable way to detect malware, since virus writers can use numerous methods of modifying and obfuscating code. As a result, a large amount of time, effort and practical skills are required in order to determine the hidden functionality and methods that were used to hide it, even though it was the same malware.

File comparison systems for determining their similarity can be found in the descriptions of patents such as US 6990600, US 6738932, US 5995982, US 6021491, US 5878050 and others. However, the systems described in these patents rely on an accurate comparison of objects and are not effective for small changes.

An analysis of the prior art and the possibilities that arise when combining them in one system allows you to get a new result, namely a system for accelerating the detection of similar objects when comparing unknown data objects with already known objects. This result is achieved by presenting objects in the form of convolution representations and their subsequent comparison.

Utility Model Essence

This useful model is designed to detect similar objects when comparing unknown objects with already known objects.

The technical result of this utility model is to use a system that contains: means for storing a collection of known objects, designed to store known objects and associated with a means of counting convolutions; a convolution counting means for counting convolutions of known objects also associated with the convolution updating means; convolution updater associated with the convolution collection tool of known objects, and intended to update convolution of known objects in the convolution collection tool of known objects; said means for storing a convolution collection of known objects for storing convolutions of known objects and associated with a local means of updating the convolution; said local convolution update tool for updating convolutions of known objects in a local convolution collection of known objects associated with a local convolution collection tool for known objects; said local means for storing a convolution collection of known objects intended for storing convolutions of known objects associated with a local means for comparing convolutions; said local convolution comparison tool for detecting similar objects by comparing convolutions of known objects from a local convolution collection tool of known objects and convolutions of unknown objects from a local convolution counter associated with the local convolution counter; said local convolution counting means for counting convolutions of unknown objects that were received from a local means of analyzing unknown objects and associated with a local means of analyzing unknown objects; said local means for analyzing unknown objects, designed to detect unknown objects necessary for comparison with known objects.

In one embodiment, the object is a file.

In yet another embodiment, the system further comprises means for processing unknown objects, designed to obtain information from unknown objects that have been identified as similar to known objects from the local means for comparing convolutions and the local means for analyzing unknown objects.

In one particular embodiment, the means of storing a collection of known objects is a database of known malicious programs.

Brief Description of the Drawings

Additional objectives, features and advantages of the present utility model will be apparent from reading the following description of the implementation of the utility model with reference to the accompanying drawings, in which:

Figures 1-5 show a graphical display of executable files.

Fig.6 illustrates a histogram of convolution, which is presented in graphical form in Fig.2.

Fig.7 illustrates a comparison in the form of a histogram for convolutions, which are graphically presented in figa and 4B.

Fig. 8 illustrates a histogram comparison for convolutions that are graphically represented in Figs. 4A and 5.

Fig.9 shows an example of a file whose convolution is not suitable for the proposed comparison.

Figure 10 discloses the use of convolutions to find new variants of already known objects.

11 illustrates an exemplary system of the present utility model.

12 illustrates a convolution counting method.

Description of Embodiments of a Utility Model

The objects and features of the present utility model, methods for achieving these objects and features will become apparent by reference to exemplary embodiments. However, the present utility model is not limited to the exemplary embodiments disclosed below, it may be embodied in various forms. The above description is intended to help a specialist in the field of technology for a comprehensive understanding of a utility model, which is determined only in the scope of the attached formula.

In the future, the description will focus on comparing primarily objects related to antivirus analysis, but it is worth noting that the scope of this useful model is not limited to this and you can use the proposed system to compare objects and files of other types, for example, to compare graphic images.

In order to further better understand the approach used in the work of the proposed system, we can consider a graphical representation of several sample files. Figure 1 presents a similar representation (bitmap) of the executable file1. In this image, each byte is represented as a pixel, while the intensity of each pixel corresponds to the byte value, i.e. black corresponds to 0, and white corresponds to 255. Thus, a file of size 10,000 bytes will be represented as a square of 100 × 100, a file of size 40,000 bytes will be represented as a square of 200 × 200, etc.

In the anti-virus industry, it has been noticed that most of the changes in malicious files that are used to hide malicious functionality will be displayed as very small changes in images like Figure 1. This is especially true for various families of malicious programs that have similar functionality and their differences in graphical form are hardly noticeable.

Figure 2 shows another example of a similar representation, in this case for file2, which is used by the scheduler. It is seen that the structure of this file is different from file1, which is displayed in figure 1, and the differences are visible to the naked eye. File3, which is shown in figure 3 also illustrates a completely different representation of the file, which allows us to conclude that its functionality differs from file1 and file2.

Note that different compilation methods for the same file can lead to different representations of this file, which can be seen in FIGS. 4A and 4B for file4. In this case, FIG. 4B illustrates file4 in which the order of operation of some functions has been changed. Also note FIG. 5, which also shows the representation of file4, but which was compiled in debug mode. You can also see the difference between figa and fig.5. This shows that small and non-functional changes to the file do not bring major changes to its final structure.

You can see that the comparison of such representations in the form of images, such as in FIGS. 1-5, is quite difficult for a person to perceive, although it is possible to distinguish the difference. This approach was presented solely for illustration, since the automatic way to detect the difference between the files does not need to form a graphical display of the files, but works with the source data. The approach of representing files (and generally objects) as a two-dimensional raster image is well suited to describe the differences between files. The ultimate goal is to calculate the degree of difference between one file and another, and a similar approach may include creating a graphical display in the form of a histogram, although it should be understood that in this case, the histograms are only an illustration.

Consider the files shown in figa and 4B. Each such image can be divided into a certain number N (for example, in this case N = 100) of horizontal lines. Each line corresponds to a specific part of the file. If, for example, an image has a size of 100 × 100 (that is, it is a file mapping of 100 × 100 = 10,000 bytes in size), then each horizontal line will be formed from 100 pixels. Further, for each line, the average value for all bytes in this line can be calculated, alternatively, the sum of all bytes in the line can be calculated, and as a result, for each file you can get a set of N values. Note that if the file is not completely divided by N, then you can add additional bytes filled with zeros. In the case of large files (for example, 40,000 bytes), the first horizontal line will be formed by 2 rows of pixels. The rest of the calculations remain the same.

Thus, a histogram can be generated for the image, each vertical column of which corresponds to each line. An example of a histogram for the image in figure 3 is shown in figure 6. Accordingly, it is possible to form histograms for various images (for example, for the images in FIGS. 4A and 4B) and then superimpose one histogram onto another for illustrative comparison.

FIG. 7 shows an overlap of two image signatures in FIGS. 4A and 4B, which clearly shows that the difference is very slight and lies in a small area A in FIG. 7, which is displayed in a little more detail. Thus, it can be concluded that if the file displayed in FIG. 4A is malicious, then it can be argued that the file displayed in FIG. 4B is also malicious due to the high level of similarity between them. On the other hand, when comparing the files displayed in FIGS. 4A and 5, the comparison of which is shown in FIG. 8, it can be seen that the files are very different, since there is a big difference between the values for these files.

The approach described above differs from conventional file comparison methods, which are based on the use of hashes and checksums. When calculating such values, even a very small change in the file can lead to the result that a completely different value will be obtained and even files very similar in functionality will have completely different values of hashes or checksums. At the same time, the present system allows us to solve the problem of finding similar files, for example, various variations of malicious programs.

This utility model also has its own function that calculates the characteristic value for a file, in our case it is a “convolution” (in the future we will use this word without quotes). A convolution is an N-dimensional vector or a sequence of N bytes (or words or double words), the graphic representation of which was given above in the example in FIG. 6. The value of each byte in the convolution is the height of the corresponding vertical column in the histogram. Accordingly, the question remains of calculating the convolution in such a way that it accurately reflects the changes in the file, but at the same time it is stable with respect to very minor changes (i.e., it would be possible to avoid the minuses characteristic when calculating hashes or checksums).

We define N as the length of the convolution; for further discussion, we suggest the value N = 100. It was also experimentally established that the values of N, which differ from 100 by an order of magnitude or more, are characterized by a large number of errors of the first and second kind. It is also worth noting that when calculating the convolution, it is necessary to solve the problem of so-called collisions, i.e. situations when the function gives the same value for different input data. Another drawback of the aforementioned methods for computing hashes and checksums is their resource consumption, which can lead to a heavy load, for example, when determining the similarity between hundreds of thousands or millions of malware samples that anti-virus companies receive daily.

In this utility model, to solve all of the above drawbacks, it is proposed to use the following convolution calculation algorithm:

a) the degree of partitioning is selected, for example, bytes, words, double words, etc.

b) a set of bytes A consisting of zeros is initialized. The size of the kit is N.

c) the file is divided into N parts, after which the first part (the beginning of the file) is highlighted

d) the average value of bytes (or words, or double words - depending on the degree of partitioning) is calculated for the first part

e) the previous step is repeated for all other parts of the file

f) the resulting set And now is a convolution of the file and it is saved in the database

Using pseudo-code, you can describe this algorithm as follows:

int blocksize = filesize / N

for i = 0 to N-l

furl [i] = average (file [i * blocksize], ..., file [i * blocksize + blocksize - 1])

12 illustrates a convolution counting method. After the optimal parameters for the number N were selected at step 1202, at step 1204, the file size is determined for which at step 1206 the file will be split into N parts. Then, at step 1208, enumeration of all parts of the file begins, for each of which, at step 1214, a part of the convolution will be calculated. In that case, if the parts of the file have ended, then at step 1210 the convolution is considered received and will be written to the database at step 1212.

Now we give an equation that allows us to calculate the difference between convolutions:

where n _i is the average byte value in the i-th part of the analyzed object (for example, a file), N is the convolution length, Δn _i is the difference between the i-elements of convolution, K _A represents the total difference of the objects (more precisely, the weight of the difference), in while K _n already represents a coefficient for normalizing this difference, and M is the maximum convolution value (for example, when choosing a byte as the degree of partitioning, M will be equal to 255). Thus, K _n shows how different the objects are. And the smaller K _n , the more objects are similar.

Let us consider in more detail the number of differences between convolutions. It is worth noting that the difference between two convolution elements with Δ = 1 and the difference in one convolution element with Δ = 2 mean two different types of similarity between files. In general, the maximum possible difference between files can be defined as M * N and can be used in the equation as follows:

where D is the number of convolution differences (when choosing a byte as the degree of partitioning, D will also equal the number of nonzero | Δn _i |). K _n is a number that ranges from 0 to 1. This alternative method of counting takes into account the different types of differences that were given earlier.

Therefore, if the files have, for example, completely different headers, but the rest of the data is identical, then in this case they will be considered similar. However, if the files have differences in all parts, then they will not be similar, since most of the values in the convolution are different, although they are very close to each other. It can be concluded that the convolution-based method described above will not work well with encrypted or archive files that have a random or pseudo-random structure, such as, for example, in FIG. 9. The same problem is represented by files filled with the same values (for example, zeros) by 95-100%.

Thus, choosing two convolutions for comparison, you can use different metrics, for example, sum up all the differences for convolutions, and then use different methods of normalizing (correlating) the resulting value in order to calculate the total difference between convolutions. You can add some value for all the differences, in order to consider the files similar with a slight difference of 2-3%. Another way is to calculate the mean square size of the differences for the convolution.

It is also worth noting that to optimize the process, it is better to compare files of approximately equal size (accurate to 50% for uncompressed or unencrypted files) to simplify the comparison of convolutions of these files.

Figure 10 discloses the use of convolution to find new variants of already known objects, in particular, to detect new variants of malicious programs. At step 1002, the process of detecting similar files is initialized, after which a known collection of files is downloaded at step 1004, for example, new versions of malicious programs. At step 1006, a convolution is calculated for each of these files, which is then stored in the database at step 1008. At step 1010, the convolution database will be updated, after which you can begin the process of checking (for example, the file system) for undetected malware at step 1012 At step 1014, it is checked whether all files are checked and if there are none, then the process ends at step 1016. At step 1018, the convolution for the file is calculated, and then at step 1020, the nearest convolution is searched in the convolution database and is satisfied count their difference. If at step 1022 it is concluded that the difference lies below the risk threshold (i.e., the files are not identical), then the process continues with the following files. If the threshold has been exceeded, then at step 1024 the file will be quarantined and at step 1026 sent for further analysis, as a potentially malicious program. Note that for anti-virus scanning, the threshold (i.e., the value of K _n ) lies in the range 0.01–0.04.

Figure 11 shows an exemplary system of the present utility model for one embodiment. Using convolution can significantly accelerate the process of detecting similar objects. A collection of 1102 well-known objects can be located on the server side of any major software service provider — it can be an anti-virus company or a company that provides various services for the detection of similar objects, for example, similar images or text files, which can be very important in the field of education when detection of plagiarism or similar abstracts. Then, the convolution counter 1104 counts the convolution for objects from this collection (for example, this can be done periodically for new objects) and sends the data to the convolution update tool 1106, which updates the convolution collection 1108 of known objects. Note that the convolutions themselves already occupy much less space than the objects themselves, so it is much easier to operate when they are transferred to users' computers on which the local convolution update tool 1110 is installed, which uses the obtained data to update the local collection of 1112 convolutions of known objects.

At the same time, the local tool 1114 for analyzing unknown objects (this can be either an anti-virus scan tool or any other file filter with the specified parameters) detects unknown files that need to be compared. For them, convolutions are then counted using the local convolution counting means 1116, which are transmitted to the local convolution comparing means 1118. Based on comparisons, the local convolution comparison tool 1118 concludes that the unknown objects are similar to those already known. Additionally, unknown objects that are similar to those already known can be sent to the tool 1120 for processing unknown objects (for example, for the purpose of additional antivirus analysis in the case of an antivirus company). It is worth noting that the above local funds are preferably located on users' computers, while the remaining funds are on the side of the service provider company using collections of known objects.

A collection of 1102 known objects, a collection of 1108 convolutions of known objects, and a local collection of 1112 convolutions of known objects can be made in the form of such database management systems (DBMS) as MySQL, PostgreSQL, Firebird, DB2. By using a data model, such systems can be hierarchical, network, relational, and object-oriented. Also, such systems can have different access to the database itself in the form of a file-server, client-server or embedded access type.

The convolution counter 1104, as well as the local convolution counter 1116, works similarly to hash functions, such as SHA1 or MD5, which convert an input data array of arbitrary length to an output bit string. The use of systems that calculate hash sums is described in such patents as JP 4148372, JP 11306194. The main difference between the convolution counting is that the convolution itself is ultimately represented as an N-dimensional vector or a sequence of N bytes. As already mentioned in the calculation of hash sums, even a very small change to the file can lead to the result that a completely different value will be obtained and even files very similar in functionality will have completely different values of hashes or checksums. At the same time, the use of convolutions allows us to solve the problem of finding similar files, for example, various variations of malicious programs. The convolution update tool 1106, as well as the local convolution update tool 1110, may be implemented as software update tools in one embodiment. Software update tools are described in patents and applications such as US 2003070087, JP 2003015926, JP 2002318722, and US 7069452. One of the update options is the patch utility (http://ru.wikipedia.org/wiki/Patch_(UNIX)), which is designed to transfer edits (changes) between different versions of text files. As a rule, updating is carried out using files (the so-called "patches" or "patches"), which are used to transfer differences between file versions. Convolutions can be saved as files of a certain format, which allows the use of standard updating tools.

The local convolution comparison tool 1118 may be implemented as a comparator based on logic elements in one embodiment. For a more accurate calculation of the similarity of convolutions, you can also use the comparison tools that are described in such patents as EP 1560446 and JP 11039194.

Local tool 1114 for analysis of unknown objects, as well as tool 1120 for processing unknown objects can be represented as one of the means of anti-virus scanning and analysis. The list of such tools includes heuristic analysis (http://en.wikipedia.org/wiki/Heuristic_analysis), the use of an emulator (http://en.wikipedia.org/wiki/Emulator), running programs in a secure environment (the so-called " sandbox ", http://en.wikipedia.org/wiki/Sandbox_(computer_security)), the use of program controls (http://en.wikipedia.org/wiki/Host-based_intrusion-prevention_system#Host-based), etc. .d. All the above analysis systems allow you to analyze an unknown object (for example, a file) in order to determine its possible malicious essence. The result of the work may be a conclusion about the harmfulness of an unknown object. In another case, an unknown object can be defined as, for example, a legitimately used program.

In conclusion, it should be noted that the information provided in the description are only examples that do not limit the scope of this utility model defined by the formula. The person skilled in the art will understand that there may be other options for implementing this utility model, consistent with the nature and scope of this utility model.

Claims (4)

1. A system for detecting similar objects when comparing unknown objects with already known objects, which includes a) a means of storing a collection of known objects designed to store known objects and associated with a means of counting convolutions; b) a means of counting convolutions, designed to count convolutions of known objects, also associated with a means of updating the convolution; c) a convolution update tool associated with a convolution collection tool of known objects storage means for updating convolution of known objects in a convolution collection tool of known objects; d) said means for storing a collection of convolutions of known objects, intended for storing convolutions of known objects and associated with a local means of updating the convolution; e) said local convolution updating tool for updating convolutions of known objects in a local convolution collection of known objects associated with a local convolution collection tool for known objects; e) said local means for storing convolution collections of known objects, intended for storing convolutions of known objects, associated with a local means for comparing convolutions; g) said local convolution comparison tool for detecting similar objects by comparing convolutions of known objects from a local means of storing a collection of convolutions of known objects and convolutions of unknown objects from a local convolution counter associated with the local convolution counter; h) the said local means of counting convolutions, designed to count convolutions of unknown objects that were received from the local means of analysis of unknown objects, and associated with the local means of analysis of unknown objects; i) said local means for analyzing unknown objects, designed to detect unknown objects necessary for comparison with known objects. 2. The system of claim 1, wherein the object is a file. 3. The system according to claim 1, which further comprises means for processing unknown objects, designed to receive information from unknown objects that were identified as similar to known objects from the local means of comparing convolutions and the local means of analyzing unknown objects. 4. The system of claim 1, wherein the means for storing a collection of known objects is a database of known malicious programs.