WO2010069904A1

WO2010069904A1 - Method and apparatus for protecting vulnerable functions

Info

Publication number: WO2010069904A1
Application number: PCT/EP2009/067036
Authority: WO
Inventors: Vinay Karecha; Hirosh Joseph
Original assignee: F-Secure Oyj
Priority date: 2008-12-16
Filing date: 2009-12-14
Publication date: 2010-06-24

Abstract

A method of detecting malware present on a computer system, the method comprising, installing a scanning function onto the computer system, and installing and maintaining on the computer system a database identifying vulnerable functions and containing respective malicious values and/or properties, redirecting function calls from a given application to said scanning function, using the scanning function and said database to identify function calls that relate to functions identified as being vulnerable, to identify such function calls that include malicious values and/or properties, and to inhibit such function calls that include malicious values and/or properties.

Description

Method and Apparatus for Protecting Vulnerable Functions

Technical Field

The present invention relates to a method and apparatus for protecting vulnerable functions against malicious calls on a computer system, for example a personal computer.

Background

Software applications, and in particular those applications that access the Internet, are vulnerable to attack. For example, a web browser application may be attacked when that application is used to download and view a web page from a remote server. An attacker may include so-called "shellcode" inside the web page and which is written into the browser memory space. A remote shellcode is useful if an attacker wants to "browse around" the target system, but more often an attacker will just want to install some form of malware on the target system. In such cases, download and execute shellcode is often used: this type of shellcode does not spawn a shell, but rather instructs the machine to download a certain executable file off the network, save it to disk and execute it. Nowadays, this approach is commonly used in drive-by download attacks, where a victim visits a malicious webpage that in turn attempts to run such a download and execute shellcode in order to install software on the victim's machine.

An attacker will commonly inject a shellcode into the target process before or at the same time as it exploits a vulnerability to gain control over the program counter. The program counter is adjusted to point to the shellcode, after which the shellcode gets executed and performs its task. Injecting the shellcode is often done by storing the shellcode in data sent over the network to the vulnerable process, by supplying it in a file that is read by the vulnerable process, or through the command line or environment in the case of local exploits.

In particular, input to the application which is not filtered/handled properly in the application can result in the corruption of memory resulting in crash or exploitation. An attacker can take advantage of vulnerable functions within or accessed by an application to achieve this. Functions are often implemented within dynamic-link library (DLL) files such that the can be accessed as required by multiple applications. Whilst some network scanning (IPS) systems can be used to inspect the network traffic (coming from the Internet) to find malicious input known to exploit an application and/or function, since these systems tend to scan all traffic, they consumes more resource than is necessary for the task and are therefore inefficient.

An alternative approach to protecting against attacks on vulnerable functions makes use of so-called "hooking". Hooking is a technique used to detour a function and take full control of its parameters and of the function itself. To achieve hooking a five byte JMP (jump instruction) is placed at the function entry point to be hooked, this JMP directing the control to a "trampoline" function giving control over the pushed parameters in the stack and its manipulation. At this point one can modify parameters, pre-process or post-process the function call: one can also decide not to give control to the hooked function itself. This gives the ability to filter the malicious input passed to certain vulnerable functions. Hooking is more efficient than the IPS approach as it scans only vulnerable data.

Current Host Intrusion Prevention Systems (HIPs) employ one trampoline function for each vulnerable function. In Windows™, the trampoline functions need to be in separate DLL files. Each time a new vulnerability is detected, the HIPS needs to create a new DLL file containing the appropriate trampoline function to protect against the vulnerability, as well as introducing a jump instruction into the vulnerable function itself. This a complex task and has many drawbacks.

These and related issues are considered in the following Internet publications: http://phreedom.org/research/heap-feng-shui/heap-feng-shui.html http://insecure.org/stf/smashstack.html http://milw0rm.com/papers/205 http://www.iss. net/documents/literature/X-ForceNewsjDctOθ.pdf

Summary

According to a first aspect of the present invention there is provided a method of detecting an attack on a computer system. The method comprises installing and maintaining on the computer system a database identifying vulnerable functions and containing respective malicious values and/or properties. Function calls made by applications to the vulnerable functions are then redirected from the vulnerable functions to a common scanning function, which uses the database to identify and inhibit or otherwise restrict function calls that are malicious.

Embodiments of the invention avoid the need for a different scanning (or "trampoline") function for each and every vulnerable function. This makes the scanning software more compact and easier to maintain.

The scanning function may be installed as a Dynamic Link Library file. Similarly, one or more of said vulnerable functions may be implemented within Dynamic Link Library files. Alternatively, the vulnerable functions may installed within an application or applications.

The method may comprise modifying said vulnerable functions to cause said step of redirecting function calls from the vulnerable functions to said scanning function. This may comprise modifying vulnerable instructions comprising inserting into those functions a jump or CALL instruction directing the process flow to the scanning function.

The method may comprise receiving database updates at said computer system, and installing these into said database in order to dynamically update security levels on the computer system.

According to a second aspect of the present invention there is provided a computer programmed to implement the method of the above first aspect of the invention.

According to a third aspect of the present invention there is provided a computer configured to scan for attacks on the computer. The computer comprises a database memory having stored thereon identities of vulnerable functions and respective malicious values and/or properties. There is provided a first processor for dynamically updating said database memory to include identities of newly identified vulnerable functions and respective malicious values and/or properties, and/or to modify malicious values and/or properties of already stored identities. There is further provided a second processor for redirecting function calls made by applications to the vulnerable functions, from the vulnerable functions to a third processor. The third processor is configured to access said database memory to retrieve data therefrom, and to use that data to identify and inhibit or otherwise restrict function calls that are malicious. According to a fourth aspect of the present invention there is provided a method of detecting an attack on a computer system. The method comprises installing a scanning function onto the computer system, and installing and maintaining on the computer system a database identifying vulnerable functions and containing respective malicious values and/or properties. Function calls from a given application are redirected to said scanning function which uses said database to identify function calls that relate to functions identified as being vulnerable, to identify such function calls that include malicious values and/or properties, and to inhibit such function calls that include malicious values and/or properties.

Brief Description of the Drawings

Figure 1 illustrates schematically an approach to handling function calls made by an application;

Figure 2 is a flow diagram illustrating a procedure for handling calls to vulnerable functions;

Figure 3 is a flow diagram illustrating in more detail a process for scanning for malware using a scanning function; and Figure 4 illustrates schematically a computer system implementing the procedures of

Figures 2 and 3.

Detailed Description

The approach described below makes use of a "Generic Hook Technology". This makes use of a single function that takes responsibility for all vulnerable functions. Generic Hook Technology has following advantages:

• No DLL file update required: only a database containing the vulnerability information needs to be updated. • Performance is not affected as the size of code remains same.

• No danger of introduction to vulnerabilities in application since code is not updated, but only the database.

• Size of code remains same, hence application doesn't grow in size, i.e. remains light weight even as the more vulnerabilities are covered. • Since no new code is introduced, vendor is saved from testing new code, hence fast update and release and less response time.

Consider a software application running on a computer platform such as a personal computer (pc) or mobile phone and which is used to access the Internet. An example of such an application is a web browser. Other applications include, for example,

Windows Media Player and Java Messenger. It is common for such applications to make use of functions, either embedded into the application code or embodied in external files. In particular, it is common for functions to be implemented as DLL files.

Data and other parameters are made available to the called function by the application, and in order to prevent attacks on the system it is desirable to scan that data prior to giving control to the function.

It is proposed here to perform said scanning by implementing a generic "trampoline" function (DLL), and hooking vulnerable functions, whether internal or external to an application, to that generic function. The generic function is able to access a database containing dynamically updated vulnerability information such that, whilst the generic function itself does not need to be updated (at least on a regular basis), security is always maintained at a high level.

All function calls made by an application are directed to the generic function (DLL), which takes the following action:

• Detects which vulnerable call was made.

• Loads the vulnerability information from a database according to the call.

• The call is processed by the Generic Function according to the information loaded from database and any vulnerability is detected/blocked.

This approach seeks to eliminate any vulnerability that has been identified and distributed by the protection product (HIPS) vendor. Distribution of updates does not require the distribution of any code, merely requiring downloading, or pushing, of database updates.

The following represent possible example implementation details for this approach:

• The generic function is a " declspec(naked)" call , hence it can hook functions of different calling convention. • The generic function needs to decode the called function. In the case that a CALL instructions is used as the hook (rather than a JMP instruction), the generic function decodes the CALL instruction, which contains the information of called function.

• It should be capable of supporting different comparisons and mathematical operations on parameters of the call. This is implemented via vulnerability database information structure.

• The generic function should be able to load any parameter from the stack to for comparison and analysis purposes. This is implemented by directly reading from a base pointer register, a.k.a. EBP register.

• It is configured to pass control back to the originally called function assuming that the process is not under attack.

Consider by way of example the Windows function "GetFileSize" which is provided within a DLL file. This is configured to use the Windows API "kernel32.GetFileSizeEx" to determine the size of a file identified to the function by a file "handle". Prior to inclusion of a jump instruction, (a portion of) the function code is as follows:

memory address (hex) machine code assembly equivalent

7C810B16 90 NOP

7C810B17 8BFF MOV EDI₁EDI

7C810B19 55 PUSH EBP

7C810B1A 8BEC MOV EBP₁ESP

7C810B1 C 51 PUSH ECX

7C810B1 D 51 PUSH ECX

7C810B1 E 8D45 F8 LEA EAX₁DWORD PTR SS:[EBP-8]

7C810B21 50 PUSH EAX

7C810B22 FF75 08 PUSH DWORD PTR SS:[EBP+8]

7C810B25 E8 7FFFFFFF CALL kernel32.GetFileSizeEx

7C810B2A 85C0 TEST EAX₁EAX

7C810B2C 0F84 2996FFFF JE kernel32.7C80A15B

In order to cause control to be passed to the generic trampoline function, the code is altered to insert a jump (JMP) instruction at the start of the function, pointing to the memory location at which the function is stored (in this case "00095454"). [It is noted that the instructions originally stored at locations 7C810B17 to 7C810B1A are moved to a new location in order to accommodate the jump instruction, and are executed prior to returning to the PUSH ECX instruction when control is returned to the GetFileSize function.] The modified code looks as follows:

7C810B16 90 NOP 7C810B17 E9 38498883 JMP 00095454 7C810B1 C 51 PUSH ECX

7C810B1 D 51 PUSH ECX

7C810B1 E 8D45 F8 LEA EAX₁DWORD PTR SS:[EBP-8]

7C810B21 50 PUSH EAX

7C810B22 FF75 08 PUSH DWORD PTR SS:[EBP+8]

7C810B25 E8 7FFFFFFF CALL kernel32.GetFileSizeEx

7C810B2A 85C0 TEST EAX₁EAX

7C810B2C 0F84 2996FFF

In this example, the generic trampoline function will have access to the handle identifying the file whose size is to be determined, and also to any other data that is passed to the hooked function by the application. Of course, the generic function will also have available to it the identity of the hooked function. Using that identity, the generic function will extract associated data from the dynamically updated vulnerability database, and will use this to analyse the data made available to the hooked function. Any matches and/or patterns that are indicative of an attack on the hooked function and application will result in the generic trampoline function failing to return control to the hooked function. Actions that the generic function might take instead include terminating the application, alerting the user/system administrator, and attempting a disinfection of the active processes.

Figure 1 illustrates schematically a functional architecture adopted by this approach. Three hooked (DLL) functions are illustrated, consisting of a DLL1 , DLL2, and DLL3. A single generic trampoline (DLL) function is provided, with hooks in the hooked functions causing control to be "jumped" to the generic function. The generic function performs appropriate detection and prevention functions using the data stored in the vulnerability database.

Figure 2 is a flow diagram illustrating a high level process according to the approach described above. The process begins at step 100, and proceeds with a series of set-up and ongoing, dynamic maintenance steps. This comprise a step of installing the generic function DLL and populating the associated database, step 200. The database is maintained in a dynamic fashion at step 300. At step 400, hooks linking to the generic function are installed into vulnerable functions. It will be understood that hooks may be installed into further function as and when these are identified as vulnerable. Typically, steps 200 and 400 are carried out when a security application is installed into the computer system. The process then moves to an operation phase. At step 500, when a call is made to a vulnerable function, the hook directs the process to the generic function. At step 600, the generic function accesses the database and scans data associated with the call for malicious properties and/or behaviour.

Figure 3 is a flow diagram illustrating in more detail to behaviour of the generic function. This includes, at step s100, decoding the call type. At step s200, the called address is loaded, whilst at step s300 the database of vulnerable calls is loaded. At step s400, the parameters associated with the call are analysed using the data retrieved from the database. If "bad" parameters are detected at step s500, the call is blocked at step s700 and the process resumed at step s800. If on the other hand no bad parameters are detected at step s700, the process is returned to the called function at step s600 and execution of the process continues. The process is resumed at step s800.

Figure 4 illustrates hardware and software components of a computer system 1 , such as a PC, laptop, or mobile phone, involved in the security mechanism described above. The computer system comprises a main processor 2 and a main memory 3. The main memory contains code for implementing an application 4, such as a web browser, and a number of functions (1 to n) 5. Also provided are first, second, and third processors 6, 7, and 8. It will be appreciated that in practice these processors are likely to be implemented by way of the main processor 2, although it is conceivable that they could be implemented by way of separate, discrete processors. A database memory 9 is provided. Again, this is likely to be implemented within the main memory of the system. The first processor 6 is arranged to maintain within the database memory 9 the vulnerability database as described above. It does this at installation of the security application, and in response to updates pushed to the system 1 via the Internet. The first processor is also responsible for modifying the code, implementing vulnerable functions, to include an instruction or instructions (e.g. a jump instruction) causing control to be passed to the second processor 7 in the event that a vulnerable function is called. This modification process is illustrated in the Figure only for function 1 , by way of a dashed line.

The second processor 5 is responsible for interpreting the redirection instruction in the modified functions so as to cause control to be passed to the third processor 8, illustrated by a further dashed line in Figure 4. This third processor 8 accesses the database memory 9 in order to extract data relevant to the scanning of the called function, and proceeds to scan the function and its data to determine whether or not the function is being hijacked or otherwise misused. If that is the case, the third processor 8 takes appropriate action as described. If it is not the case, the third processor 8 returns control to the function.

It will be appreciated that other mechanisms for causing control to be passed to the generic trampoline function may be used. For example, rather than a JMP instruction, a CALL instruction may be used.

It will be appreciated by the person of skill in the art that various modifications may be made to the above described embodiments without departing from the scope of the present invention.

Claims

1. A method of detecting an attack on a computer system, the method comprising: installing and maintaining on the computer system a database identifying vulnerable functions and containing respective malicious values and/or properties; redirecting function calls made by applications to the vulnerable functions, from the vulnerable functions to a common scanning function; using the scanning function and said database to identify and inhibit or otherwise restrict function calls that are malicious.

2. A method according to claim 1 , wherein said scanning function is installed as a Dynamic Link Library file.

3. A method according to claim 1 or 2, wherein one or more of said vulnerable functions are installed as Dynamic Link Library files.

4. A method according to claim 1 or 2, wherein one or more of said vulnerable functions are installed within an application or applications.

5. A method according to any one of the preceding claims and comprising modifying said vulnerable functions to cause said step of redirecting function calls from the vulnerable functions to said scanning function.

6. A method according to claim 5, said step of modifying vulnerable instructions comprising inserting into those functions a jump or CALL instruction directing the process flow to the scanning function.

7. A method according to any one of the preceding claims and comprising receiving database updates at said computer system, and installing these into said database in order to dynamically update security levels on the computer system.

8. A computer programmed to implement the method of any one of the preceding claims.

9. A computer configured to scan for attacks on the computer and comprising: a database memory having stored thereon identities of vulnerable functions and respective malicious values and/or properties; a first processor for dynamically updating said database memory to include identities of newly identified vulnerable functions and respective malicious values and/or properties, and/or to modify malicious values and/or properties of already stored identities; a second processor for redirecting function calls made by applications to the vulnerable functions, from the vulnerable functions to a third processor; and the third processor being configured to access said database memory to retrieve data therefrom, and to use that data to identify and inhibit or otherwise restrict function calls that are malicious.

10. A method of detecting an attack on a computer system, the method comprising: installing a scanning function onto the computer system, and installing and maintaining on the computer system a database identifying vulnerable functions and containing respective malicious values and/or properties; redirecting function calls from a given application to said scanning function; using the scanning function and said database to identify function calls that relate to functions identified as being vulnerable, to identify such function calls that include malicious values and/or properties, and to inhibit such function calls that include malicious values and/or properties.