TWI514188B - A system for detecting packed program and method thereof - Google Patents

Description

Packing program detection system and method thereof

The present invention relates to a program detection system and method thereof, and more particularly to a package program detection system and method thereof.

As information systems develop rapidly, the number of relatively malicious attacks increases. Therefore, anti-virus software companies need to constantly update the signature of the virus engine in order for the anti-virus software to recognize malicious programs.

The common trend in recent years is that hackers use a method of confusing anti-virus software detection when writing malicious programs. In software engineering, "packing" is a software compression encryption technology and has been widely used. Software engineers usually use this method to protect the software and compressed file size they develop, and to avoid the software being developed being cracked or modified by reverse engineering.

However, the software protection mechanism of this "packing" method has been used by hackers to protect malicious programs developed by them and to disguise their malicious behavior. Therefore, by encrypting the malicious program, the malicious program can successfully avoid the detection of the anti-virus software, thereby reducing the ability of the malicious program to recognize. Because the malware shelling method threatens to change, static analysis has its own needs. The only drawback of static analysis is that it cannot correctly understand the behavior of malicious programs.

In order to solve the above problems of the prior art, the present invention provides a packer detection system and method thereof. The invention is a method for recognizing whether a malicious program file is packed by reverse engineering and dynamic analysis technology, and complementing static analysis to analyze possible errors.

Firstly, the present invention provides a method for detecting a packaged program for detecting a file, which comprises the steps of: translating a file and extracting a combined language code of a file that has been reversed; extracting a combined language One of the predetermined number of lines in the code from the program entry point to analyze the combined language code; the conversion analysis combined language code to generate a support vector machine feature code; and the support vector machine feature code to determine one of the archived attributes of the file, wherein The shell attribute file contains a packed file and an unpacked file.

The predetermined number of lines may be selected from the first line to the fifteenth line from the program entry point in the combined language code.

The method for detecting the packaged program of the present invention optionally further includes the following steps: if the file is judged to be a packed file, then one of the packer categories of the packed file is analyzed. The packer class may optionally include at least one of ASPack, AsProtect, EXE32Pack, FSG, PEBundle, PECompact, PEX, UPX, yoda, and WWPack32.

The method for detecting a package program of the present invention optionally further includes the steps of: storing the file having the attribute of the package to a training set database.

Furthermore, the present invention further provides a packer detection system for detecting a file, comprising a reverse translation module, a capture module, a conversion module, and a determination module. The anti-translation module is used to reverse the translation of the file and retrieve the combined language code of one of the files that have been reversed. The capture module is connected to the reverse translation module for capturing a predetermined entry point in the combined language code One of the number of lines analyzes the combined language code. The conversion module is coupled to the capture module for converting the analysis combined language code to generate a support vector machine signature. The judging module is connected to the conversion module for judging one of the archived attributes of the file according to the feature code of the support vector machine, wherein the file with the packed attribute includes the packed file and the unpacked file.

The predetermined number of rows may be selected from the first row to the fifteenth row from the program entry point in the combined language code.

The packer detection system of the present invention can optionally further comprise a sorting module. The classification module is connected to the judgment module for analyzing the type of the packer of the packed file when the file is a packed file. The packer class may optionally include at least one of ASPack, AsProtect, EXE32Pack, FSG, PEBundle, PECompact, PEX, UPX, yoda, and WWPack32.

The packer detection system of the present invention optionally further includes a training set database. The training set database is connected to the judging module for storing files with packed attributes.

Compared with the prior art, the present invention proposes a packer detection system and method thereof, which have the uniqueness of capturing the order of combined language instructions. The invention uses the dynamic analysis method, and after combining the file reverse translation, the combined language code of the first line to the fifteenth line starting from the program execution point of the program is used as the training set feature, and then the same The way to retrieve the file. Then, through the analysis and classification of the support vector machine, the pack file is identified, and the packer category of the packed file is classified. When it is known that the packer type of the packaged file is available, the packaged file can be unpacked by its corresponding tool, and then detected by the anti-virus software to determine whether it is a malicious program or a non-malicious program. Solve the problem that the malware in the prior art uses the shelling method to avoid the anti-virus software detection. At the same time, the present invention recognizes programs by techniques of reverse engineering and dynamic analysis. Whether the file is packed or not can complement the static analysis method and it may analyze the error.

The advantages and spirit of the present invention will be further understood from the following detailed description of the invention.

1‧‧‧Packer Detection System

10‧‧‧Reverse translation module

20‧‧‧Capture module

13‧‧‧Transformation module

40‧‧‧Judgement module

50‧‧‧Classification module

60‧‧‧ Training Set Database

FIG. 1 is a flow chart showing a method for detecting a packaged program of the present invention in a specific embodiment.

FIG. 2 is a functional block diagram showing a packer detection system of the present invention in a specific embodiment.

In order to make the objects, features and advantages of the present invention more comprehensible, the embodiments of the present invention are described in detail with reference to the accompanying drawings.

Referring to FIG. 1, FIG. 1 is a flow chart showing a method for detecting a packaged program of the present invention in a specific embodiment. In one embodiment, the method for detecting a packaged program of the present invention is for detecting a file, which includes the following steps:

Step 1: Disassemble the file and retrieve the combination code of one of the files that have been reversed.

More specifically, in one embodiment, an executable file is dynamically reverse-translated through Ollydbg, and the combined language code of the reverse-compiled file is retrieved. In practical applications, the file can be a Portable Executable File (PE file), such as an EXE file or a DLL file, but not limited thereto.

Step 2: Extract one of the predetermined number of lines from the entry point (Entry Point) in the combined language code to analyze the combined language code.

Since the packed file is unpacked and initialized first, in an embodiment, the capture module captures the instruction at the initialization stage as an identification feature. More specifically, since each shell has a different length of the unpacking instruction section during the initialization phase, for example, the file of the shelling instruction by the UPX-packed file is relatively short in the initialization phase; When the module takes too long a predetermined number of rows, it will capture the noise of the unpacking command section that is not in the initialization phase, thereby affecting the recognition of the packed file. Therefore, in this embodiment, the capture module captures the content of the combined language code of the fifteen lines starting from the entry point (Entry Point) as the feature code of the subsequent analysis (ie, analyzing the combined language code). In a preferred embodiment, the predetermined number of lines is fifteen lines, meaning that the predetermined number of lines is the first line to the fifteenth line from the program entry point in the combined language code. However, the predetermined number of rows is not limited to fifteen behaviors. In practical applications, the predetermined number of rows may also be any row value between fifteen and twenty-five rows.

Step 3: Transform and analyze the combined language code to generate a support vector machine signature.

Step 4: Determine a property of the file according to the feature code of the support vector machine, wherein the file with the packed property includes a packed file and an unpacked file.

More specifically, in one embodiment, the support vector machine feature codes of the plurality of files are substituted into a Support Vector Machine (SVM) for training as a training set feature. Then, the support vector machine feature code of the file is substituted into the support after training. In the measuring machine, to determine whether the file has been processed. The trained support vector machine is to substitute the file support vector machine feature code into the support vector machine to train a hyperplane. By training the hyperplane to distinguish which category the files belong to. In order to reduce errors during testing, the hyperplane of the present invention has the largest margin (Margin) separating the two categories except that the data can be correctly classified. In addition, because the data is mostly in a Separable Case, it is impossible to classify the data in any way regardless of the hyperplane. Therefore, the present invention further adds a formula of a separable case to a slack variable and processes it via a Lagrangian function to enable subsequent classification. Get more accurate results.

Step 5: If it is determined that the file is a packed file, analyze one of the packer categories of the packed file.

In one embodiment, the Packer category includes at least one of ASPack, AsProtect, EXE32Pack, FSG, PEBundle, PECompact, PEX, UPX, yoda, and WWPack32. However, the packer category is not limited to the above, and the packer can also be a tool that can be used to compress executable files. In addition, in practical applications, when the classifier of the packed file is known, the packaged file can be unpacked by its corresponding tool, and then detected by the anti-virus software and judged to be malicious. Program or non-malicious program.

Step 6: Save the file with the packed property to a training set database.

In an embodiment, if the file is a new type of packed file, it is processed through steps 1 through 4 to obtain a packed attribute, and then has a packed attribute. The new type of packed file is stored in the training set database to update the judgment criteria of the support vector machine. In an embodiment, if the file is an unknown source file, the process proceeds from step 1 to step 3 to obtain a support vector machine feature code, and then the support vector machine feature code of the file of unknown source is substituted into the training. The results in the support vector machine are compared with the training set database to determine the classification results.

Referring to FIG. 2, FIG. 2 is a functional block diagram of a packer detection system of the present invention in a specific embodiment. In one embodiment, the package detection system 1 of the present invention is configured to detect a file, and includes a reverse translation module 10, a capture module 20, a conversion module 30, and a determination module. 40. A classification module 50 and a training set database 60, wherein the reverse translation module 10, the capture module 20, the conversion module 30, the determination module 40, the classification module 50, and the training set database 60 can be Stored in a storage device and executed by a central processor coupled to the storage device.

In this embodiment, the reverse group translation module 10 reverses the translation of the file and retrieves the combined language code (Assembly Code) of the file that has been reversed.

In this embodiment, the capture module 20 is coupled to the reverse translation module 10 for extracting a combined language code from a predetermined number of rows from the entry point of the combined language code. In a preferred embodiment, the predetermined number of lines is fifteen lines, meaning that the predetermined number of lines is the first line to the fifteenth line from the program entry point in the combined language code. However, the predetermined number of rows is not limited to fifteen behaviors. In practical applications, the predetermined number of rows may also be any row value between fifteen and twenty-five rows.

In this embodiment, the conversion module 30 is coupled to the capture module 20 for converting and analyzing the combined language code to generate a Support Vector Machine (SVM). Signature.

In this embodiment, the determining module 40 is connected to the conversion module 20 for determining the attributes of the file according to the support vector machine feature code, wherein the attributes of the file include a packed case and an uncased.

In this embodiment, the classification module 50 is coupled to the determination module 40 for analyzing the type of the packer of the packaged file when it is determined that the attribute of the file is already packed. The Packer category includes at least one of ASPack, AsProtect, EXE32Pack, FSG, PEBundle, PECompact, PEX, UPX, yoda, and WWPack32. However, the type of the sheller is not limited to the above. In practical applications, the packer can also be a tool for compressing executable files. In practical applications, when the type of the packer of the packaged file is known, the packaged file can be unpacked by its corresponding tool, and then detected by the anti-virus software and judged to be a malicious program or Non-malicious programs.

In this embodiment, the training set database 60 is coupled to the determination module 50 for storing files having the attribute of the package. In an embodiment, if the file is a new type of packed file, the same process is performed by the reverse translation module, the capture module, the conversion module, and the judgment module to obtain a packed attribute, and then training. The set database stores a new type of packed file with a packed attribute to update the judgment criteria of the support vector machine. In an embodiment, if the file is an unknown source file, it is also processed by the reverse group module, the capture module, and the conversion module to obtain a support vector machine feature code. The judging module compares the result of the support vector machine feature code of the file of the unknown source into the trained support vector machine (SVM) and compares it with the training set database 60 to determine the classification result.

Compared with the prior art, the present invention proposes a shelling program detection system. And its method, which has the uniqueness of capturing the order of combined language instructions. The invention uses the dynamic analysis method, and after combining the file reverse translation, the combined language code of the first line to the fifteenth line starting from the program execution point of the program is used as the training set feature, and then the same The way to retrieve the file. Then, through the analysis and classification of the support vector machine, the pack file is identified, and the packer category of the packed file is classified. When it is known that the packer type of the packaged file is available, the packaged file can be unpacked by its corresponding tool, and then detected by the anti-virus software to determine whether it is a malicious program or a non-malicious program. Solve the problem that the malware in the prior art uses the shelling method to avoid the anti-virus software detection. At the same time, the invention recognizes whether the program file is packed by the technology of reverse engineering and dynamic analysis, and can complement the static analysis method, which may analyze the error.

The features and spirit of the present invention will be more apparent from the detailed description of the preferred embodiments. On the contrary, the intention is to cover various modifications and equivalents within the scope of the invention as claimed. Therefore, the scope of the patented scope of the invention should be construed in the broadest

Claims (10)

A method for detecting a package program for detecting a file, comprising the steps of: translating the file and extracting a combined language code of the file that has been reversely translated; and extracting the combined language code Converting the combined language code from one of the predetermined number of lines from the program entry point; converting the analysis combined language code to generate a support vector machine feature code; and determining a one of the archived attributes of the file according to the support vector machine feature code, wherein The file of the packed property contains a packed file and an unpacked file. The method for detecting a packer according to claim 1 of the patent application further includes the following steps: if it is determined that the file is the packed file, analyzing one of the packer categories of the packed file. The method of detecting a packer according to claim 2, wherein the packer category comprises at least one of an ASPack, an AsProtect, an EXE32Pack, an FSG, a PEBundle, a PECompact, a PEX, a UPX, a yoda, and a WWPack32. The method for detecting a packaged program as described in claim 2, further comprising the steps of: storing the file having the attribute of the package to a training set database. The method for detecting a packaged program according to claim 1, wherein the predetermined number of lines is the first line to the fifteenth line from the program entry point in the combined language code. A packer detection system for detecting a file, comprising: a reverse group translation module, translating the file, and extracting a combined language code of the file that has been reversely translated; Taking a module, connecting to the reverse group translation module, extracting one of the predetermined number of lines from the program entry point in the combined language code to analyze the combined language code; a conversion module connected to the capture module, converting The analyzing combines the language codes to generate a support vector machine signature; A judging module is connected to the conversion module, and determines a stuffing attribute of the file according to the support vector machine feature code, wherein the file having the packed attribute includes a packed file and an unpacked file. The method for detecting a packer according to claim 6 of the patent application, further comprising: a sorting module connected to the judging module, and analyzing the packed shell when the file is the packed file One of the files is the packer category. The packer detection system of claim 7, wherein the packer category comprises at least one of ASPack, AsProtect, EXE32Pack, FSG, PEBundle, PECompact, PEX, UPX, yoda, and WWPack32. The packer detection system of claim 7, further comprising: a training set database, connected to the judgment module, storing the file having the attribute of the pack. The packer detection system of claim 6, wherein the predetermined number of lines is the first line to the fifteenth line from the program entry point in the combined language code.