AU2007202892A1 - Restricting malicious libraries - Google Patents

Description

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION NAME OF APPLICANT(S):: PC Tools Technology Pty Limited ADDRESS FOR SERVICE: DAVIES COLLISON CAVE Patent Attorneys 255 Elizabeth Street, Sydney, New South Wales, Australia, 2000 INVENTION TITLE: Restricting malicious libraries The following statement is a full description of this invention, including the best method of performing it known to me/us:- 5102 SRESTRICTING MALICIOUS LIBRARIES Technical Field S[001] The present invention generally relates to a method, system, computer readable 00 5 medium of instructions and/or computer program product for restricting a request to load and/or and register a malicious library in a processing system.

Background Art [002] As used herein a "threat" includes malicious software, also known as "malware" or "pestware", which includes software that is included or inserted in a part of a processing system or processing systems for a harmful purpose. The term threat should be read to include possible, potential and actual threats. Types of malware can include, but are not limited to, malicious libraries, viruses, worms, Trojans, adware, malicious active content and denial of service attacks. In the case of invasion of privacy for the purposes of fraud or theft of identity, malicious software that passively observes the use of a computer is known as "spyware".

[003] A hook (also known as a hook procedure or hook function), as used herein, generally refers to a callback function provided by a software application that receives certain data before the normal or intended recipient of the data. A hook function can thus examine or modify certain data before passing on the data. Therefore, a hook function allows a software application to examine data before the data is passed to the intended recipient.

[004] An API ("Application Programming Interface") hook (also known as an API interception), as used herein as a type of hook, refers to a callback function provided by an application that replaces functionality provided by an operating system's API. An API generally refers to an interface that is defined in terms of a set of functions and procedures, and enables a program to gain access to facilities within an application. An API hook can be inserted between an API call and an API procedure to examine or modify function -2nparameters before passing parameters on to an actual or intended function. An API hook may also choose not to pass on certain types of requests to an actual or intended function.

N [005] A process, as used herein, is at least one of a running software program or other S 5 computing operation, or a part of a running software program or other computing operation, that performs a task.

i[006] A hook chain as used herein, is a list of pointers to special, application-defined callback functions called hook procedures. When a message occurs that is associated with a particular type of hook, the operating system passes the message to each hook procedure referenced in the hook chain, one after the other. The action of a hook procedure can depend on the type of hook involved. For example, the hook procedures for some types of hooks can only monitor messages, others can modify messages or stop their progress through the chain, restricting them from reaching the next hook procedure or a destination window.

[007] A kernel, as used herein, refers to the core part of an operating system, responsible for resource allocation, low-level hardware interfaces, security, etc.

[008] An interrupt, as used herein, is at least one of a signal to a processing system that stops the execution of a running program so that another action can be performed, or a circuit that conveys a signal stopping the execution of a running program.

[009] A library is a file containing executable code and data which can be loaded by a process at load time or run time, rather than during linking. There are several forms of a library including, but not limited to, Dynamic Linked Libraries (DLL), Layered Service Provider (LSP), drivers, Active X technologies, and other related services.

[010] In a networked information or data communications system, a user has access to one or more terminals which are capable of requesting and/or receiving information or data from local or remote information sources. In such a communications system, a terminal -3may be a type of processing system, computer or computerised device, personal computer mobile, cellular or satellite telephone, mobile data terminal, portable computer, Personal Digital Assistant (PDA), pager, thin client, or any other similar type of digital electronic device. The capability of such a terminal to request and/or receive information 00 5 or data can be provided by software, hardware and/or firmware. A terminal may include or O be associated with other devices, for example a local data storage device such as a hard disk drive or solid state drive.

('i [011] An information source can include a server, or any type of terminal, that may be associated with one or more storage devices that are able to store information or data, for example in one or more databases residing on a storage device. The exchange of information (ie. the request and/or receipt of information or data) between a terminal and an information source, or other terminal(s), is facilitated by a communication means. The communication means can be realised by physical cables, for example a metallic cable such as a telephone line, semi-conducting cables, electromagnetic signals, for example radio-frequency signals or infra-red signals, optical fibre cables, satellite links or any other such medium or combination thereof connected to a network infrastructure.

[012] A system registry is a database used by all modern operating systems, for example Windows "IM platforms. The system registry includes information needed to configure the operating system. The operating system refers to the registry for information ranging from user profiles, to which applications are installed on the machine, to what hardware is installed and which ports are registered.

[013] Currently software is able to load and register malicious libraries on a user's processing system. Such malicious libraries may interfere with normal operation of the processing system, such as: Conceal certain objects within an operating system of the processing system; Modify memory of running processes; 0 Terminate or restrict processes from starting; Create system instability; Perform spyware-like or virus-like actions; and/or N Modify the behaviour of particular or all running applications.

N, [014] Current methods of restricting such malicious libraries include scanning software 00 5 products or real time software monitoring products.

N

[015] Scanning products scan process memory for known malicious system libraries.

N, Scanning products may additionally scan the system registry for references to known malicious system libraries. However, disadvantages to such scanning products include: 0 A scan needs to be invoked by the user or the user must schedule the scan to be performed. Until a scan is performed, malicious activities can be performed by the malicious libraries.

If a scan detects a malicious library loaded by one or more processes, then either: 0 Those processes need to be terminated, which can be inconvenient to the user; An attempt can be made to unload the malicious system library or libraries, however, this can often leave the process in an unstable state, causing unpredictable results or crashes; If any of the processes are critical system processes, then the system needs to be rebooted for successful deactivation of the malicious system library or libraries which can be inconvenient.

[016] Real time software monitoring products can monitor or protect areas of the system registry which store a list of installed system libraries. This can be achieved by either API hooking or polling for modifications to the list at regular intervals. However, the disadvantages to such real time software monitoring products include: SIf a modification is detected to sections of the system registry which store the list of installed system libraries using API hooking, then either: The process which attempted to make the modification has to be paused until the modified section of the registry has been fully scanned, which can reduce the performance of the processing system; N All required information to perform a scan may not be available as the 00 5 process attempting to perform further changes to the registry has been

(N

O paused or suspended; Ifa modification is detected using the polling technique then: (i A snapshot of installed system libraries has to be maintained which may cause unwanted bursts of CPU activity during polling; and 0 There is difficulty in determining which process performed the modification to the registry.

[017] Real time software monitoring product cannot restrict or prevent malicious system libraries being activated by one or more processes performed in the processing system.

[018] There exists a need for a method, system, computer readable medium of instructions, and/or a computer program product to at least restrict or prevent loading of malicious libraries in a processing system which address or at least ameliorate one or more problems inherent in the prior art.

[019] There also exists a need for a method, system, computer readable medium of instructions, and/or a computer program product to at least restrict or prevent registration of malicious libraries in a processing system which address or at least ameliorate one or more problems inherent in the prior art.

[020] The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as an acknowledgment or admission or any form of suggestion that that prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.

-6- Disclosure Of Invention [021] In a first broad form there is provided a method of restricting a request to load or register a malicious library in a processing system, the method including: (N intercepting, in the processing system, a request to load or register a library; 00 5 determining if the library is malicious; and O in response to determining that the library is malicious, restricting the request to S load or register the malicious library.

[022] In one form, intercepting the request includes intercepting, using an API hook function, an API call to request the library to load or register the library.

[023] In another form, in the event that the library is determined to be malicious, the method includes restricting the API call propagating through an API hook chain associated with the API call.

[024] In one embodiment, the method includes: determining identification data identifying the requested library; and using a detection module and the identification data to determine if the library is malicious.

[025] In another embodiment, the method includes analysing, using the detection module, at least one of the intercepted request and a data storage element of the processing system to determine whether the library is malicious.

[026] In an optional form, the detection module includes one or more submodules including at least one of a cryptographic hash module, a checksum module, a disassembly module, a black-list/white-list module, a relationship analysis module, and a pattern matching module, wherein the method includes analysing, using the one or more submodules, at least one of the intercepted request and a data storage element of the processing system to determine whether the library is malicious.

[027] In another optional form, the method includes: generating, using the cryptographic hash module, a cryptographic hash value of an entity associated with the request, wherein the entity is stored on or associated with the Ndata storage element of the processing system; and comparing the cryptographic hash value to a database to determine whether the

(N

library is malicious, wherein the database includes a plurality of cryptographic hash values identifying malicious entities.

[028] In an optional embodiment, the method includes: generating, using the checksum module, a checksum value of an entity associated with the request, wherein the entity is stored on or associated with the data storage element of the processing system; and comparing, using the black-list/white-list module, the checksum value to a list to determine whether the library is malicious, wherein the list includes records indicative of malicious entities and non-malicious entities.

[029] In another optional embodiment, the method includes: disassembling, using the disassembly module, an entity associated with the request, wherein the entity is stored on or associated with the data storage element of the processing system; and performing a comparison, using the pattern matching module, between the disassembled entity and a list of patterns associated with malicious activity.

[030] In one aspect, in the event that the library is determined to be malicious, the method includes: setting the malicious library as a base entity; determining an entity property of the base entity; determining, using the relationship analysis module, one or more related entities to the base entity which are related by the entity property; and performing, using the detection module, an analysis of the related entities to determine if one or more of the related entities are malicious.

-8c, [031] In another aspect, the method includes: setting the one or more related entities as the base entity; and (Ni repeating steps and followed by step until an end condition is satisfied.

00 [032] In one form, the end condition is at least one of: when no related entities are determined in a particular repetition; when no new related entities are determ ined in a particular repetition; ,q when no new related entities are determined in a particular repetition; when no related entities are determined in a period of time; when the base entity has an entity property which is indicative of the end condition; and when a selected number of repetitions have been perfolbrmed.

[033] In another form, the method includes: in response to determining that the library is malicious: scanning a system registry of the processing system for references to the malicious library; and in the event of detecting a record in the system registry including a reference to the malicious library, removing the reference from the record.

[034] In a second broad form there is provided a system to restrict a request to load or register a malicious library in a processing system, the system being configured to: intercept, in the processing system, a request to load or register a library; determine if the library is malicious; and in response to determining that the library is malicious, restrict the request to load or register the malicious library.

[035] In particular forms, the processing system is configured to perform any one of the above described methods.

[036] In a third broad form there is provided a computer program product for a Sprocessing system, the computer program product including a computer readable medium having a computer program recorded therein or thereon, the computer program product (N being configured to enable restriction of a request to load or register a malicious library in 00 5 the processing system, wherein the computer program product configures the processing O system to: S intercept, in the processing system, a request to load or register a library; Sdetermine if the library is malicious; and in response to determining that the library is malicious, restrict the request to load or register the malicious library.

[037] In particular forms, the computer program product is configured to enable any one of the above-described methods to be performed by the processing system.

Brief Description Of Figures [038] An example embodiment of the present invention should become apparent from the following description, which is given by way of example only, of a preferred but nonlimiting embodiment, described in connection with the accompanying figures.

[039] Figure 1 illustrates a functional block diagram of an example of a processing system that can be utilised to embody or give effect to a particular embodiment; [040] Figure 2A illustrates a known method of loading a library in a processing system; [041] Figure 2B illustrates a known method of registering a library in a processing system; [042] Figure 3 illustrates flow diagram of an example method of intercepting an API call to an API procedure.

[043] Figure 4 illustrates a flow diagram of an example method to restrict a request to load a malicious library in a processing system; (Ni [044] Figures 5A and 5B illustrate a flow diagram of a more detailed example of the 00 5 method illustrated in Figure 4; S[045] Figure 6 illustrates a flow diagram of an example method to restrict a request to Sregister a malicious library in a processing system; (,i [046] Figure 7A and 7B illustrate a flow diagram of more detailed example of the method illustrated in Figure 6; [047] Figure 8A illustrates a functional block diagram representing an example system to restrict a request to load a malicious library in a processing system; [048] Figure 8B illustrates a functional block diagram representing an example system to restrict a request to register a malicious library in a processing system.

[049] Figure 9 illustrates a block diagram representing an example of a detection module and sub-modules; and [050] Figure 10 illustrates a functional block diagram representing an example of the operation of a relationship analysis module.

Modes for Carrying Out The Invention [051] The following modes, given by way of example only, are described in order to provide a more precise understanding of the subject matter of a preferred embodiment or embodiments.

[052] In the figures, incorporated to illustrate features of an example embodiment, like reference numerals are used to identify like parts throughout the figures.

-11- [053] A particular embodiment of the present invention can be realised using a processing system, an example of which is shown in Fig. 1. In particular, the processing system 100 generally includes at least one processor 102, or processing unit or plurality of processors, memory 104, at least one input device 106 and at least one output device 108, coupled together via a bus or group of buses 110. In certain embodiments, input device 106 and output device 108 could be the same device. An interface 112 can also be provided for coupling the processing system 100 to one or more peripheral devices, for example interface 112 could be a PCI card or PC card. At least one storage device 114 which houses at least one database 116 can also be provided. The memory 104 can be any form of memory device, for example, volatile or non-volatile memory, solid state storage devices, magnetic devices, etc. The processor 102 could include more than one distinct processing device, for example to handle different functions within the processing system 100. Input device 106 receives input data 118 and can include, for example, a keyboard, a pointer device such as a pen-like device or a mouse, audio receiving device for voice controlled activation such as a microphone, data receiver or antenna such as a modem or wireless data adaptor, data acquisition card, etc. Input data 118 could come from different sources, for example keyboard instructions in conjunction with data received via a network. Output device 108 produces or generates output data 120 and can include, for example, a display device or monitor in which case output data 120 is visual, a printer in which case output data 120 is printed, a port for example a USB port, a peripheral component adaptor, a data transmitter or antenna such as a modem or wireless network adaptor, etc. Output data 120 could be distinct and derived from different output devices, for example a visual display on a monitor in conjunction with data transmitted to a network. A user could view data output, or an interpretation of the data output, on, for example, a monitor or using a printer. The storage device 114 can be any form of data or information storage means, for example, volatile or non-volatile memory, solid state storage devices, magnetic devices, etc.

[054] In use, the processing system 100 is adapted to allow data or information to be stored in and/or retrieved from, via wired or wireless communication means, the at least -12- Sone database 116. The interface 112 may allow wired and/or wireless communication q between the processing unit 102 and peripheral components that may serve a specialised purpose. The processor 102 receives instructions as input data 118 via input device 106 N, and can display processed results or other output to a user by utilising output device 108.

00 5 More than one input device 106 and/or output device 108 can be provided. It should be 0 appreciated that the processing system 100 may be any form of terminal, server, specialised hardware, or the like.

[055] The processing system 100 may be a part of a networked communications system.

Processing system 100 could connect to network, for example the Internet or a WAN.

Input data 118 and output data 120 could be communicated to other devices via the network. The transfer of information and/or data over the network can be achieved using wired communications means or wireless communications means. A server can facilitate the transfer of data between the network and one or more databases. A server and one or more databases provide an example of an information source.

[056] Referring now to Figure 2A, there is shown a block diagram illustrating an example of a known process of loading a library in a processing system.

[057] In particular, a process 210 transfers, as indicated by arrow 205, an API call to an operating system 220 of the processing system 100, where the API call is indicative of a request to load of a library. The request generally includes an identity associated with the library to be loaded. The operating system queries, as indicated by arrow 215, a system library registry 240, to determine the filename which corresponds to the identity, and the system library registry 240 transfers, as indicated by arrow 225, the filename back to the operating system 220. The operating system 220 then loads an instance of the library from the system library store 230 to the process' address space, as indicated by arrow 235, and the system library store 230 returns an identifier, cookie or handle for the loaded instance of the library to the process 210.

-13- [058] Referring now to Figure 2B, there is shown a block diagram illustrating an example of a known process of registering a library in a processing system.

(Ni [059] In particular, the process 210 transfers, as indicated by arrow 255, an API call to 00 5 the operating system 220 of the processing system 100, where the API call is indicative of (Ni O a request to register a library. The request generally includes a request to obtain an address (,i for a registration function of the library. The operating system 220 queries, as indicated by arrow 265, the system library store 230 for the address of the registration function for the library. The system library store 230 transfers, as indicated by arrow 275, the address of the library to operating system 220, such that the operating system loads the library into the address space of the process 210. The system library store 230 also transfers, as indicated by arrow 285, the address of the registration function to the process 210. The process 210 then invokes the registration function of the library such as to register the library in the system registry 240, as indicated by arrow 295.

[060] Referring now to Figure 3, there is illustrated an example flow chart showing the process 300 of intercepting an API call.

[061] In particular, at step 310, an event occurs in the processing system. At step 320, the operating system running in the processing system 100 registers the occurrence of the event. At step 330, the operating system passes the registered event to the API hook chain.

At step 340, the event is passed to each API hook in the API hook chain such that different applications, processes, and devices may be notified of the registered event. Once the event has propagated throughout the API hook chain, the method includes at step 350 an application receiving notification of the event being registered by the processing system.

At step 360, the application initiates an API call to an API procedure so as to carry out a response to the registered event. If an API hook has been established between the API call and the API procedure, the API call is intercepted at step 370 before it reaches the API procedure. The API call may be allowed to continue calling the API procedure at step 390 or the API call may not be passed on to the API procedure, as shown at step 380.

-14- [062] Referring now to Figure 4, there is shown a flow diagram illustrating an example method to restrict a request to load a malicious library in a processing system 100.

[063] In particular, the method 400 includes at step 410 intercepting an API call to load a 00 5 library. The process of intercepting an API call has previously been discussed in relation to SFigure 3. At step 420, the method includes determining whether the library is malicious.

This can be performed using a detection module 820, as will be described in more detail below.

[064] If the library is determined to be non-malicious, the library is loaded at step 420 in the normal manner indicated in Figure 2A. The method 400 continues to monitor any further API calls to load other libraries in the processing system. In the event that the library is determined to be malicious, the method continues to step 430 where the method includes restricting the request to load the malicious library.

[065] Referring now to Figures 5A and 5B, there is shown a flow diagram illustrating a more detailed example of the method illustrated in Figure 4.

[066] In particular at step 505, the method 500 includes an API interception module 810 intercepting an API call from a process requesting to load a library. At step 510 the method 500 includes determining identification data of the requested library. At step 515, the method 500 includes transferring the identification data to the detection module 820. At step 520, the method 500 includes the detection module 820 determining, using the identification data, if the library is malicious. A number of sub-modules, such as a cryptographic hash module 910, a checksum module 920, a disassembly module 930, a blacklist/whitelist module 940, a relationship analysis module 950, and/or a pattern matching module 960 can be used by the detection module 820 to determine whether the library is malicious. Each of these sub-modules will be discussed in more detail below.

[067] If the detection module 820 determines at step 525 that the library is malicious, the method proceeds to step 530. In particular, at step 530 the method 500 includes the detection module 820 transferring alert data to the API interception module 810. The alert N data is indicative of the identification of the library considered malicious. At step 535 the method 500 includes the API interception module 810 transferring an error to the N requesting process, thus restricting the library to load. The error transferred to the 00 5 requesting process is indicative of the library failing to load. At step 540, the method 500 Soptionally includes presenting the alert data to the user of the processing system 100. In one form, the alert data may be a pop-up window alerting the user of the identification of the request to load the malicious library.

[068] At step 545, the method 500 optionally includes the processing system 100 performing a scan of the system registry 240 for references to the malicious library. In particular, the detection module 820 may be invoked by the processing system 100 such as to determine whether there are any references to the malicious library in the system registry 240, as indicated at step 550. In the event that there are references to the malicious library in the system registry 240, the method 500 includes removing the references to the malicious library from the system registry 240 at step 555.

[069] Returning to step 525 of method 500, in the event that the library is not considered malicious, the scanning engine 820 passes control, via the API interception module 810 to the operating system 220 at step 560. At step 565, the method 500 includes the operating system 220 loading an instance of the library in the requesting process' address space. At step 570, the method 500 includes the operating system 220 returning a pointer, cookie, or handle to the requesting process.

[070] Referring now to Figure 6, there is shown a flow diagram illustrating an example of a method to restrict a request to register a malicious library in a processing system.

[071] In particular, at step 610, the method 600 includes intercepting an API call to register a library. The process of intercepting an API call has previously been discussed in relation to Figure 3. At step 620, the method 600 includes determining whether the library is malicious. In the event that the library is determined to be non-malicious, the method -16- 600 proceeds to step 640 where the library is registered. The method 600 continues to Smonitor API calls to register other libraries. In the event that the library is determined to be malicious, the method 600 continues to step 630 where the library is restricted from N loading.

00 O [072] Referring now to Figures 7A and 7B, there is shown a flow diagram illustrating a more detailed example of the method illustrated in Figure 6.

[073] In particular at step 705 the method 700 includes the API interception module 810 intercepting the API call from the process 210 to the operating system 220, where the API call is requesting the registration function of a library. At step 710 the method 700 includes determining identification data of the library. At step 715, the method includes transferring the identification data to the detection module 820. At step 720 the detection module 820 performs a scan, using the identification data to determine if the library is malicious. At step 725 if the detection module 820 determines that the library is malicious, the method 700 proceeds to step 730.

[074] At step 730, the method 700 includes the detection module 820 transferring alert data to the API interception module 810. The alert data is indicative of the identification of the library being malicious. At step 735 the method 700 includes the API interception module 810 returning to the requesting process 210 an address to a function of a nonmalicious library, thus restricting the registration of the malicious library. In one form, the non-malicious library may be a dummy library which performs no functionality. At step 740, the method 700 optionally includes presenting to the user alert data indicating the identification of the malicious library.

[075] Returning to step 725, in the event that the library is determined to be nonmalicious, the method proceeds to step 745. At step 745, the method 700 includes the detection engine 810 passing control via the API interception module 810, to the operating system 220. At step 750, the operating system 220 returns an address to the registration function of the library to the requesting process. At step 755, the method 700 includes the 17requesting process 210 invoking the registration function of the library to register the library in the system registry 240.

[076] Referring now to Figure 8A, there is shown a functional block diagram of an S 5 example system to restrict a request to load a malicious library in a processing system 100.

O

[077] In particular, the process 210 transfers a request, indicated by arrow 205, to the operating system 220. The request to load the library is intercepted, indicated by arrow 805, by API interception module 810. The API interception module 810 transfers the request to the detection module 820 as indicated by arrow 815. The detection module 820 determines whether the library requested is malicious. The detection module 820 then proceeds to pass control, as indicated by arrow 825, to the API interception module 810. In the event that the detection module 820 determines that the library was malicious, the API interception module 810 passes error data, as indicated by arrow 845, to the requesting process 210, thus restricting the loading of the malicious library. In the event that the scanning engine 820 determined that the requested library was not malicious, the API interception module 810 passes control, as indicated by arrow 835, back to the operating system 220, where the normal process of loading the library as previously discussed in relation to figure 2A is performed such that the library is loaded in the processing system 100.

[078] Referring now to Figure 8B, there is shown a functional block diagram illustrating an example system to restrict a request to register a malicious library in a processing system 100.

[079] In particular, the requesting process 210 passes a request, indicated by arrow 255, to the operating system 220 where the process requests the registration of a library. The API interception module 810 intercepts, as indicated by 855, the request to register the library. The API interception module 810 transfers the request, as indicated by arrow 865, to the detection module 820, wherein the detection module 820 performs an analysis of the library requested for registration. The detection module 820 transfers an indication of -18whether the library was determined to be malicious back to the API interception module F, 810, as indicated by arrow 875. In the event that the indication was indicative of the determination that the library was malicious, the API interception module 810 transfers an address, as indicated by arrow 895, to the requesting process 210 wherein the address 00 5 points to a non-malicious registration function. In the event that the indication was

(N

O indicative of the determination that the library was non-malicious, the API interception (,i module transfers control, as indicated by arrow 885, to the operating system 220, such that the normal process for registering a library is performed, as outlined in Figure 2B.

[080] As shown in Figure 9, the detection module 820 can include a number of further sub-modules to detect if the processing system 100 is being requested to load and/or register malicious libraries, or to determine references to a malicious library.

[081] In particular, the detection module 820 can include the sub-modules of a cryptographic hash module 910, a checksum module 920, a disassembly module 930, a black-list/white-list module 940, a relationship analysis module 950, and a pattern matching module 960. The detection module 820 can be configured to use one or more of these sub-modules exclusively or in combination to determine if the processing system 100 is being requested to load and/or register a malicious library, or to determine references to a malicious library.

[082] Referring now to the sub-modules of the detection module 820, the cryptographic hash module 910 of the detection module 820 is configured to generate a cryptographic hash value of an entity stored on a data storage component of the processing system 100.

An entity can be a data object such as a file stored in the processing system 100. As the cryptographic hash value can be used an identity, the cryptographic hash value can be used in comparisons with the blacklist/whitelist module 940 to determine whether the entity is malicious.

-19- [083] The checksum module 920 of the detection module 820 is configured to determine a checksum of an entity of the processing system 100. The checksum can be compared to a database (blacklist/whitelist module 940) to determine whether the entity is malicious.

00 5 [084] The disassembly module 930 is configured to disassemble the binary code stored O for an entity such that the disassembly module 930 determines processing system t instructions for the entity. The processing system instructions of the entity can then be used Sby the pattern matching module 960 to determine whether entity is malicious. Although strings of instructions can be compared by the pattern matching module 960, the pattern matching module 960 may be configured to perform functional comparisons of groups of instructions to determine whether the functionality of the entity is indicative of malware.

[085] The blacklist/whitelist module 940 of the detection module 820 includes a list of malicious and/or non-malicious entities. The blacklist/whitelist module 940 may be provided in the form of a table or database which includes data indicative of malicious and non-malicious entities. The table may include checksums and cryptographic hash values for malicious and non-malicious entities. The data stored in the blacklist/whitelist module 940 can be used to determine whether an entity in the processing system 100 is malicious or non-malicious. The blacklist/whitelist module 940 can obtain the list of malicious/and or non-malicious entities and related data using the processing system's data storage component, such as the hard drive of the processing system 100, read-only media, read/write media, and/or a network connection.

[086] The relationship analysis module 950 can be used to detect related malicious entities based on a detected base malicious entity 1000. As shown by example in Figure once a malicious entity 1000 has been detected, for example using one or more of the other sub-modules of the detection module 820, a web of related malicious entities 1030 can be determined using the relationship analysis module. Generally, malware includes a bundle of malicious entities. Thus, by only removing a single malicious entity, the malware may not necessarily be disabled from performing some malicious activity.

20 Therefore, detecting a group of malicious entities can be beneficial for disabling the malware.

[087] The relationship analysis module 950 can be configured to determine one or more entity properties of the base malicious entity 1000. For example, the one or more entity properties could include a time which the base malicious entity was created or modified, and/or a directory which the entity is stored in a file system of the processing system 100.

The relationship analyser 950 can then perform a search of the data storage components for related entities 1010 which also share similar entity properties to the base malicious entity 1000.

[088] The one or more related entities 1010 can then analysed using one or more of the other sub-modules 910, 920, 930, 940, 960 of the detection module 820 to determine whether the one or more related entities are malicious. The related entities 1010 which are determined to be malicious 1030 can then be treated as base malicious entities, thereby iteratively determining a group of malicious related entities. In this instance, related entities 1020 can be determined based on malicious base entities 1010. The iterative process can terminate after an end condition is satisfied when no related entities are determined in a particular repetition; when no new related entities are determined in a particular repetition; when no related entities are determined in a period of time; when the base entity has an entity property which is indicative of the end condition; and when a selected number of repetitions have been performed. Other properties such as entity key words, functional relationships, and a network address which the entity was downloaded from which can also be used to determine related entities and potential malicious entities related to the base malicious entity. A more detailed explanation of applying a set of suspicious assessment rules in determining a suspicious/malicious entity is described in the Applicant's co-pending application AU2007200605 entitled "Determination of related entities", the content of which is herein incorporated by cross-reference [089] The pattern matching module 960 of the detection module 820 is configured to search an entity for particular patterns of strings or instructions which are indicative of -21 malware such as malicious libraries. The pattern matching module 960 may operate in Scombination with the disassembly module 930 of the detection module 820.

(Ni [090] Example pseudocode for implementing the method of restricting a request to load a 00 5 malicious library is shown below.

SProcedure initialization() )Begin SCall installApiInterception on sytemlibraryApi; C1 10 End; Replacement systemApi; Procedure systemApi:loadSystemLibraryByld(id) Begin Response SendMessageToScanEngine(ISIDBAD, id); If Response ID_IS_BAD Then Return ERROR CODE; Exit procedure; If (Response IDISOK) or (Response IPCERROR) Then Return call REAL_loadSystemLibraryById(id); Exit procedure; End; Procedure systemApi:loadSystemLibraryByFileName(filename) Begin Response SendMessageToScanEngine(IS_FILEBAD, filename); If Response FILE_IS_BAD Then Return ERROR_CODE; Exit procedure; If (Response FILE IS OK) or (Response IPC_ERROR) Then Return call REAL_loadSystemLibraryByFileName(filename); Exit procedure; End; [091] Example pseudocode for implementing the method of restricting a request to register a malicious library is shown below.

Procedure initialization() Begin Call installApiInterception on sytemlibraryApi; End; Replacement systemApi; Function systemApi:getFunctionAddress(libreference, functionname) Begin If functionname "DllRegisterServer" then begin Response SendMessageToScanEngine(IS_LIB_BAD, libreference); If Response LIBISBAD Then Return AddressOfNonMaliciousFunction; -22- SEnd; Return call REALgetfunctionAddress(libreference, functionname); End; Function NonMaliciousFunction() \Begin 00 SendMessageToUser(PREVENTED_REGISTRATION) C1 Return FAILED; S 10 End; S[092] It will be appreciated that implementation of the above pseudocode in the form of a computer program can configure a processing system to restrict the request to load and/or register a malicious library.

[093] The embodiments discussed in relation to restricting a process loading or registering a malicious library may be implemented separately or in any combination as a software package or component. Such software can then be used to pro-actively notify, restrict, and/or prevent malicious activity being performed using malicious libraries.

Various embodiments can be implemented for use with the Microsoft Windows operating system or any other modern operating system.

[094] In one optional form, such software may be invoked during the start-up process of the processing system 100. Alternatively, the user may invoke the software via the operating system of the processing system 100.

[095] In another optional form, the user may be prompted regarding the identification of the request to load and/or register a malicious library. The user may then provide input, using the input device of the processing system 100, to indicate whether the user wishes for the loading or registration process to continue.

[096] The above-described modules and submodules may be provided in the form of software, hardware, or a combination thereof.

[097] Optional embodiments of the present invention may also be said to broadly consist in the parts, elements and features referred to or indicated herein, individually or -23- Scollectively, in any or all combinations of two or more of the parts, elements or features, and wherein specific integers are mentioned herein which have known equivalents in the art to which the invention relates, such known equivalents are deemed to be incorporated N' herein as if individually set forth.

00 O [098] Although a preferred embodiment has been described in detail, it should be t understood that various changes, substitutions, and alterations can be made by one of Sordinary skill in the art without departing from the scope of the present invention.

t(,i

Claims (14)

1. A method of restricting a request to load or register a malicious library in a (N processing system, the method including: 00 5 intercepting, in the processing system, a request to load or register a library; O determining if the library is malicious; and in response to determining that the library is malicious, restricting the request to Sload or register the malicious library.

2. The method according to claim 1, wherein intercepting the request includes intercepting, using an API hook function, an API call to request the library to load or register the library.

3. The method according to claim 2, wherein in the event that the library is determined to be malicious, the method includes restricting the API call propagating through an API hook chain associated with the API call.

4. The method according to any one of claims 1 to 3, wherein the method includes: determining identification data identifying the requested library; and using a detection module and the identification data to determine if the library is malicious. The method according to any one of claims I to 4, wherein the method includes analysing, using the detection module, at least one of the intercepted request and a data storage element of the processing system to determine whether the library is malicious.

6. The method according to claim 5, wherein the detection module includes one or more submodules including at least one of a cryptographic hash module, a checksum module, a disassembly module, a black-list/white-list module, a relationship analysis module, and a pattern matching module, wherein the method includes analysing, using the Sone or more submodules, at least one of the intercepted request and a data storage element of the processing system to determine whether the library is malicious. S7. The method according to claim 6, wherein the method includes: 00 5 generating, using the cryptographic hash module, a cryptographic hash value of an O entity associated with the request, wherein the entity is stored on or associated with the data storage element of the processing system; and comparing the cryptographic hash value to a database to determine whether the library is malicious, wherein the database includes a plurality of cryptographic hash values identifying malicious entities.

8. The method according to claim 6 or 7, wherein the method includes: generating, using the checksum module, a checksum value of an entity associated with the request, wherein the entity is stored on or associated with the data storage element of the processing system; and comparing, using the black-list/white-list module, the checksum value to a list to determine whether the library is malicious, wherein the list includes records indicative of malicious entities and non-malicious entities.

9. The method according to any one of claims 6 to 8, wherein the method includes: disassembling, using the disassembly module, an entity associated with the request, wherein the entity is stored on or associated with the data storage element of the processing system; and performing a comparison, using the pattern matching module, between the disassembled entity and a list of patterns associated with malicious activity. The method according to any one of claims 6 to 9, wherein in the event that the library is determined to be malicious, the method includes: setting the malicious library as a base entity; determining an entity property of the base entity; -26- determining, using the relationship analysis module, one or more related entities to the base entity which are related by the entity property; and performing, using the detection module, an analysis of the related entities to determine if one or more of the related entities are malicious. 00 O 11. The method according to claim 10, wherein the method includes: (,i setting the one or more related entities as the base entity; and Srepeating steps and followed by step until an end condition is satisfied.

12. The method according to claim 11, wherein the end condition is at least one of: when no related entities are determined in a particular repetition; when no new related entities are determined in a particular repetition; when no related entities are determined in a period of time; when the base entity has an entity property which is indicative of the end condition; and when a selected number of repetitions have been performed.

13. The method according to any one of claims 1 to 12, wherein the method includes: in response to determining that the library is malicious: scanning a system registry of the processing system for references to the malicious library; and in the event of detecting a record in the system registry including a reference to the malicious library, removing the reference from the record.

14. A system to restrict a request to load or register a malicious library in a processing system, the system being configured to: intercept, in the processing system, a request to load or register a library; determine if the library is malicious; and in response to determining that the library is malicious, restrict the request to load or register the malicious library. 27 The system according to claim 1, wherein the processing system is configured to perform the method of any one of claims 1 to 13. (N 16. A computer program product for a processing system, the computer program 05 product including a computer readable medium having a computer program recorded (,i therein or thereon, the computer program product being configured to enable restriction of a request to load or register a malicious library in the processing system, wherein the computer program product configures the processing system to: intercept, in the processing system, a request to load or register a library; determine if the library is malicious; and in response to determining that the library is malicious, restrict the request to load or register the malicious library.

17. The computer program product according to claim 16, wherein the computer program product is configured to enable the method of any one of claims 1 to 13 to be performed by the processing system.

18. A system for determining if one or more entities in a data storage medium of a processing system are malicious, the system being substantially herein before described with reference to the accompanying drawings.

19. A method of determining if one or more entities in a data storage medium of a processing system are malicious, the method being substantially herein before described.

20. A computer program product for enabling restriction of a request to load or register a malicious library in the processing system, the computer program product being substantially herein before described with reference to the accompanying drawings.