RU2817547C1 - System and method of monitoring operability of processes in operating system - Google Patents

Abstract

FIELD: information security.

SUBSTANCE: result is achieved due to steps, according to which: converting the source code of the program into an equivalent machine code; constructing control flow graphs, which reflect the structure and syntax of the source code of the program; converting control flow graphs into call flow graphs using target instructions received from the operating system; generating at least one control flow signature based on the generated call flow graphs; starting a control flow monitor, after which a target process corresponding to said program is launched in the operating system; intercepting, by means of a control flow monitor, instructions of said program executed by the target process; checking the intercepted instructions for compliance with at least one control flow signature of said target process; if the instructions of the target process do not match the signature of the control flow, an error in the operation of the target process is determined.

EFFECT: high efficiency of detecting malfunction of a process in an operating system.

11 cl, 7 dwg

Description

Technical field

The invention relates to the field of information security, and more specifically to methods for monitoring the performance of processes in the operating system of a computer system.

State of the art

Malware is constantly evolving, challenging information security service providers to keep up with the ever-changing threat landscape. Among other things, attackers are developing ways to distribute malicious software (software), using new vulnerabilities to penetrate a computer system.

Currently, various types of attacks are used on the end nodes of a computer network (computer systems). For example, a malicious code injection attack or a stack buffer overflow attack can lead to data corruption and/or disruption of processes in the operating system or the operating system itself as a whole. To protect the end nodes of a computer network, various solutions are used, for example, antiviruses, as well as a class of solutions for detecting and studying malicious activity on end nodes (English: Endpoint Detection and Response or EDR). Such solutions allow you to analyze files using static and dynamic methods to identify behavioral anomalies, exploit vulnerabilities, etc. These violations in one form or another affect the performance of processes, and therefore can be detected during execution.

One of the signs of a process malfunction is a violation of the control flow of the process, namely a violation of the sequence of executable instructions embedded in the computer system program corresponding to the process. Thus, controlling the control flow of a process can be used to monitor its health.

A solution for monitoring control flow is known from the prior art and is presented in patent application WO2009154498A1. This publication describes a method for controlling the operation of a process in an operating system based on a control flow graph (CFG) received from the compiler during program compilation. Based on a sequence of instructions that are executed one by one and cannot be interrupted or skipped during program execution, signatures are created to monitor the health of the process. However, to monitor the operation of the process using the method described in WO2009154498A1, it is necessary to instrument the code. Code instrumentation refers to changing the bytecode of a program in order to create methods for collecting information or performing additional actions before execution, during execution, or after execution of the main program.

Approaches that use code instrumentation have a number of disadvantages. First, it requires the introduction of third-party code into the application, which requires control. Implementing third-party code requires both time and resources, and the application becomes more difficult to maintain. Secondly, instrumenting code in some cases affects the stability of the application and compatibility with future updates. Third, using code instrumentation comes with application security risks. Making changes to an application's source code can create vulnerabilities that can be exploited by attackers.

Another solution is presented in patent US10372902B2. This publication talks about a way to control the operation of a process based on a control flow graph, and unlike the publication presented above, information about the actual control flow is obtained from the processor, and not from the operating system. However, this approach does not allow monitoring the operation of a process on untrusted hardware, and is also limited to monitoring only based on the analysis of machine commands.

The presented solutions monitor the performance of the process, however, they also have a number of disadvantages, in particular the use of code instrumentation. Therefore, there is a need for solutions that, on the one hand, would allow monitoring the performance of the process, and on the other hand, would eliminate existing shortcomings.

Disclosure of the invention

The present invention has been made in view of the problems described above, and the purpose of the present invention is to detect an error in the performance of processes in an operating system, in particular to detect the failure of at least one of the instructions, or the execution of an invalid instruction. The present invention makes it possible to ensure reliability (non-failure operation) in the operation of processes by monitoring the performance of processes by monitoring the control flow using generated models of control flow processes. A process model is understood as a certain pattern or signature. The signature of a controlled process includes possible sequences of execution of instructions, and the set of instructions contained in the signature is determined by the capabilities of monitoring the control flow of the controlled process. Depending on the implementation option, the signature can take the form of either a function control flow graph or a set of graphs, or a formal language description or a finite automaton.

The technical result of the present invention is to increase the efficiency of detecting process failure in the operating system by monitoring the performance of processes in the operating system based on control flow monitoring using generated process models.

As one embodiment of the present invention, there is proposed a method for monitoring the performance of processes in an operating system to identify errors in the operation of processes, the method being implemented in a hardware processor, the method comprising the steps of: converting the source code of the program into equivalent machine code; build control flow graphs that reflect the structure and syntax of the program source code; converting control flow graphs into call flow graphs using target instructions received from the operating system; generating at least one control flow signature based on the generated call flow graphs; launching a control flow monitor, after which a target process corresponding to the specified program is launched in the operating system; intercept instructions of the specified program executed by the target process using a control flow monitor; checking the intercepted instructions against at least one control flow signature of the specified target process; if the instructions of the target process do not match the control flow signature, an error in the operation of the target process is determined.

In another embodiment of the method, the source code of the program is converted into equivalent machine code using a compiler or interpreter.

In another embodiment of the method, when an error is detected in the operation of the target process, the target process is restarted

In another embodiment of the method, an error in the target process associated with one of: failure to execute at least one of the expected instructions is determined; executing an illegal instruction.

In another embodiment of the method, the expected instructions are instructions received from the operating system, within which the performance control of the target process is implemented.

In another embodiment of the method, an invalid instruction is any instruction that is not contained in the source code.

In yet another embodiment of the method, the control flow graph is converted into call flow graphs by excluding instructions from the program code blocks of the control flow graphs that are not target instructions or function calls.

In another embodiment of the method, a control flow signature is generated in at least one of the following ways: by combining call flow graphs and call flow graph connections; by transforming the at least one call flow graph to a state machine view of the calls of the target instructions.

In another embodiment of the method, a control flow signature is generated by generating a set of call flow graphs, wherein the set of call flow graphs is formed by sequentially replacing basic blocks with target instructions with their corresponding call flow graphs.

In another embodiment of the method, when an error is detected in the operation of the target process, the operation of the process is stopped.

As another embodiment of the present invention, there is proposed a system for monitoring the health of operating system processes to detect errors in the operation of processes, consisting of a computer device with a hardware processor configured to perform the method according to any of the above embodiments.

Brief description of drawings

The accompanying drawings illustrate only exemplary embodiments of the invention and therefore should not be considered limiting its scope. Additional objects, features and advantages of the present invention will become apparent from a reading of the following description of the invention with reference to the accompanying drawings, in which:

Fig. 1 illustrates a system that implements a method for monitoring the health of processes in an operating system.

Fig. Figure 2 illustrates the transformation of a control flow graph into a call flow graph for the main() function.

Fig. Figure 3 illustrates the transformation of a control flow graph into a call flow graph for the sleep() function.

Fig. Figure 4 illustrates the transformation of a control flow graph into a call flow graph for the nanosleep() function.

Fig. Figure 5 illustrates the transformation of a control flow graph into a call flow graph for the errno() function.

Fig. 6 illustrates a method for monitoring the health of processes in an operating system.

Fig. 7 shows an example of a computer system with which the present invention can be implemented to monitor the performance of processes.

Carrying out the invention

Objects and features of the present invention, methods for achieving these objects and features will become apparent by reference to exemplary embodiments. However, the present invention is not limited to the exemplary embodiments disclosed below, but may be embodied in various forms. The substance set forth in the specification is nothing more than the specific details necessary to assist one skilled in the art in fully understanding the invention, and the present invention is defined within the scope of the appended claims.

Glossary

A control flow graph or CFG is a representation of the source code algorithm of a function of a compiled program in the form of a graph that displays the control flow in the source code of the said function. A control flow graph consists of nodes representing blocks of program code and arcs that connect these blocks, representing the transfer of control between them.

A block of program code is a sequence of instructions that are executed one at a time and cannot be interrupted or skipped during program execution.

A call flow graph is a transformed control flow graph, which consists of empty blocks, in place of which there may be blocks of program code, basic blocks, as well as arcs that connect blocks, displaying the transfer of control between them.

Basic block - a block in the control flow graph represented by the target instruction of the target process.

A control flow signature (CFS) is a representation of a collection of call flow graphs, consisting of call flow graphs and call flow graph arcs.

The target process is a process in the operating system that is monitored by the control flow monitor.

In FIG. Figure 1 shows an example of a system that implements a method for monitoring the performance of processes in an operating system based on monitoring the sequence of executed instructions.

A process is considered to be faulty if the process:

• failed to execute at least one instruction expected from it, wherein the expected instruction is an instruction received from the operating system, within which the target process is monitored;

• executes instructions that are valid for it, in a different order from the order of instruction execution specified in the source code 110 ;

• executed at least one invalid instruction, where an invalid instruction is any instruction that is not contained in the source code 110 .

The operation of the system can be divided into two stages - the compilation stage and the execution stage.

The execution step is implemented on an operating system 170 installed on the first computer system. The compilation step is implemented on an operating system (not shown in Fig. 1) installed on the second computer system.

In a particular embodiment, the execution stage and the compilation stage are implemented on the same operating system 170 .

Compilation stage .

The compiler 120 receives as input the source code 110 of the program to compile it. The program source code 110 may be represented as code in either a high-level language (eg, C++, Java, etc.) or a low-level language (eg, assembler, C, etc.).

During compilation, compiler 120 converts the source code 110 of a program written in a programming language into equivalent machine code that can be executed on a computer system, such as the one illustrated in the discussion of FIG. 7 . The compilation process involves several steps, namely lexical analysis, parsing and machine code generation. The lexical analyzer breaks the program source code 110 into tokens, such as identifiers, operators, numbers, etc. The parser uses these tokens to construct control flow graphs that reflect the structure and syntax of the program source code 110 . Compiler 120 then generates machine code that corresponds to the control flow graphs. Upon completion of compilation, the compiler 120 passes the constructed control flow graphs to the control flow signature generator 130 and also passes the compiled code (machine code) to the operating system 170 .

In a particular implementation, the parser uses these tokens to construct a single control flow graph containing all control flow graphs of all functions of the program source code 110 . This single control flow graph reflects the structure and syntax of the program source code 110 .

In another embodiment, an interpreter (not shown in FIG. 1 ) is used instead of the compiler 120 . For example, if the source code 110 is in an interpreted programming language (such as Python). It is worth noting that in this embodiment, the interpreter, unlike the compiler 120, performs all the steps described above for compilation every time the program is executed.

Control flow signature generator 130 receives control flow graphs from compiler 120 and target instructions 140 from operating system 170 .

It is worth noting that the target instruction 140 is at least one instruction transmitted from the operating system 170 for the purpose of monitoring the performance of the target process 160 .

The control flow signature generator 130 excludes from the code blocks of the resulting control flow graphs all instructions that are not target instructions 140 or calls to functions containing target instructions 140 . Thus, control flow graphs are converted into call flow graphs. An example of the mapping of control flow graphs to call flow graphs, as well as examples of target instructions 140, are presented in the description of FIG. 2 - Fig. 5 .

Next, the control flow signature generator 130 generates at least one control flow signature based on the generated call flow graphs.

In a preferred embodiment, control flow signature generator 130 generates a control flow signature consisting of call flow graphs and call flow graph relationships.

In another embodiment, control flow signature generator 130 generates a control flow signature represented by a collection of call flow graphs. The control flow signature generator 130 sequentially replaces the basic blocks with target instructions 140 with their corresponding call flow graphs.

In yet another particular embodiment, the control flow signature generator 130 generates a control flow signature by transforming at least one call flow graph to a state machine view of the calls of target instructions 140 .

Call state machine is a graph representation of all possible outcomes of control transfer during execution of the target process 160 . The start of a state machine is the entry point, represented by the target instruction, from which execution of the target process 160 begins.

The control flow signature generator 130 transmits the generated control flow signatures to the control flow monitor 150 , which is located in the operating system 170 .

Execution stage .

Operating system 170 receives compiled code (machine code) from compiler 120 and runs it within target process 160.

The control flow monitor 150 monitors the operation of a process (hereinafter referred to as the target process 160 ) in the operating system 170 during execution of the running compiled source code 110 by monitoring the instructions executed in the target process 160 using at least one acquired control flow signature. Monitoring involves intercepting executable instructions in the target process 160 by the operating system 170 and checking them against at least one received control flow signature. If the executable instructions match at least one signature, then the control flow monitor 150 determines whether the target process 160 is operating correctly when executing the compiled code and continues monitoring. If at least one executable instruction does not match the control flow signature, then an error in the operation of the target process 160 is determined.

In a particular embodiment, to restore the functionality of the target process 160, the control flow monitor 150 restarts the target process 160 through the operating system 170 .

In another particular implementation, when an error is detected in the operation of the target process 160 , the control flow monitor 150 stops the operation of the target process 160 through the operating system 170 .

In yet another particular embodiment, the control flow monitor 150 notifies a security system (eg, SIEM or EDR) or a user of a violation in the target process 160 if the executable instructions of the target process 160 do not match the control flow signature.

In a particular embodiment, control flow monitor 150 is a process started by the operating system 170 before running compiled code in the target process 160 .

Thus, at runtime, the operating system 170 runs a control flow monitor 150 and then runs the compiled code in the target process 160 . The control flow monitor 150 intercepts all instructions of the target process 160 and checks them against the control flow signature. If the instructions of the target process 160 do not match the control flow signature, the control flow monitor 150 restarts the target process 160 or stops its operation.

It is worth noting that the approach to monitoring the health of the target process 160 , in which the control flow monitor 150 intercepts all instructions of the target process 160 , allows you to monitor the health of the process 160 without resorting to instrumentation of the code of the target process 160 . Also, the control flow monitor 150 does not interact with hardware (not shown in FIG. 1 ), allowing the operation of the target process 160 to be monitored on both trusted and untrusted hardware.

In FIG. 2 - Fig. Figure 5 shows an example of generating call flow graphs from control flow graphs for the following program:

#include <unistd.h>

int main(){

sleep(1);

return 0;

}

In this example, the target instructions provided by the operating system 170 to the control flow signature generator 130 are SleepAPI and ErrnoAPI.

To generate a call flow graph from the control flow graph, the control flow signature generator 130 excludes from the control flow graph all code that is not related to the target instructions.

In FIG. Figure 2 shows the transformation of the control flow graph of the main() function into the call flow graph of the main() function. After the conversion, only the target sleep instruction remains, which is specific to the SleepAPI.

In FIG. Figure 3 shows the transformation of the control flow graph of the sleep() function into the call flow graph of the sleep() function. After the conversion, only the target instruction nanosleep, related to the SleepAPI, remains.

In FIG. Figure 4 shows the transformation of the control flow graph of the nanosleep() function into the call flow graph of the nanosleep() function. After the conversion, only the target errno instruction related to ErrnoAPI and SleepAPI remain.

In FIG. Figure 5 shows the transformation of the control flow graph of the errno() function into the call flow graph of the errno() function. After the conversion, only the target ErrnoAPI instruction remains.

Fig. 6 illustrates a method for monitoring the health of processes in an operating system based on monitoring the sequence of executed instructions.

In a preliminary step 605, the program source code 110 is converted into equivalent machine code by the compiler 120 .

In a particular embodiment of the invention, instead of the compiler 120, an interpreter is used (not shown in Fig. 1 ).

At step 610, control flow graphs are constructed that reflect the structure and syntax of the program source code 110 .

At step 620, the control flow signature generator 130 converts the control flow graphs of each function containing at least one target instruction 140 into call flow graphs by eliminating from the program code blocks of the specified control flow graph all instructions that are not the target instructions 140 or calls to other functions containing target instructions 140 .

At step 630 , the control flow signature generator 130 generates at least one control flow signature based on the generated call flow graphs and transmits it to the control flow monitor 150 .

In a preferred embodiment, a control flow signature is generated consisting of call flow graphs and relationships between call flow graphs.

In another embodiment, a control flow signature is generated by converting multiple call flow graphs into one by sequentially replacing basic blocks with target instructions 140 or function calls with their corresponding call flow graphs.

In another particular implementation, a control flow signature is generated by converting call flow graphs to the form of a state machine of calls of target instructions.

At step 640, control flow monitor 150 is launched, followed by target process 160 on operating system 170 .

At step 645, executable instructions of the target process 160 are intercepted by the control flow monitor 150 via the operating system 170 .

At step 650, the intercepted instructions are checked against the control flow signature of the specified target process 160 .

At step 660 , if the instructions of the target process 160 do not match the control flow signature, an error in the operation of the target process 160 is determined. to restore the functionality of the target process 160, the target process 160 is restarted via the operating system 170 .

In a particular embodiment, when an error is detected in the operation of the target process 160, the target process 160 is restarted.

In yet another particular embodiment, if an error is detected in the operation of the target process 160, it is not possible to restart the target process 160 , the operation of the target process 160 is stopped.

In another particular embodiment, when an error is detected in the operation of the target process 160, the user is additionally notified about the malfunction of the target process 160 .

In FIG. 7 illustrates a computer system on which various embodiments of the systems and methods disclosed herein can be implemented. The computer system 20 may be a system configured to implement the present invention and may be in the form of a single computing device or multiple computing devices, such as a desktop computer, a laptop computer, a notebook computer, a mobile computing device, a smartphone, a tablet computer, a server, mainframe, embedded device and other forms of computing devices.

As shown in FIG. 7 , the computer system 20 includes: a central processing unit 21 , a system memory 22 , and a system bus 23 that communicates various system components, including memory, coupled to the central processing unit 21 . System bus 23 is implemented like any bus structure known in the art, which in turn includes a bus memory or bus memory controller, a peripheral bus, and a local bus capable of interfacing with any other bus architecture. Examples of buses include: PCI, ISA, PCI-Express, HyperTransport™, InfiniBand™, Serial ATA, I2C and other suitable connections between computer system components 20 . The central processing unit 21 contains one or more processors having one or more cores. The central processing unit 21 executes one or more sets of computer-readable instructions implementing the methods presented herein. System memory 22 may be any memory for storing data and/or computer programs executable by the central processing unit 21 . System memory can contain both read-only memory (ROM) 24 and random access memory (RAM) 25 . The basic input/output system (BIOS) 26 contains the basic procedures that ensure the transfer of information between elements of the computer system 20 , for example, when the operating system is loaded using ROM 24 .

Computer system 20 includes one or more data storage devices, such as one or more removable storage devices 27 , one or more non-removable storage devices 28 , or combinations of removable and non-removable devices. One or more removable storage devices 27 and/or non-removable storage devices 28 are connected to the system bus 23 via an interface 32 . In one embodiment, removable storage devices 27 and associated computer-readable storage media are nonvolatile modules for storing computer instructions, data structures, program modules, and other computer system 20 data. System memory 22 , removable storage devices 27 , and non-removable storage devices 28 may use various computer readable media. Examples of computer-readable storage media include computer memory such as cache memory, SRAM, DRAM, capacitorless RAM (Z-RAM), thyristor memory (T-RAM), eDRAM, EDO RAM, DDR RAM, EEPROM, NRAM , RRAM, SONOS, PRAM; flash memory or other memory technologies such as solid-state drives (SSDs) or flash drives; magnetic cassettes, magnetic tapes and magnetic disks such as hard disks or floppy disks; optical media such as compact discs (CD-ROMs) or digital versatile discs (DVDs); and any other media that can be used to store the desired data and that can be accessed by the computer system 20 .

System memory 22 , removable storage devices 27 , and non-removable storage devices 28 contained in the computer system 20 are used to store the operating system 35 , applications 37 , other program modules 38 , and program data 39 . Computer system 20 includes a peripheral interface 46 for transmitting data from input devices 40 such as a keyboard, mouse, stylus, game controller, voice input device, touch input device, or other peripheral devices such as a printer or scanner through one or more input/output ports, such as a serial port, parallel port, universal serial bus (USB), or other peripheral interface. A display device 47 , such as one or more monitors, projectors, or embedded displays, is also connected to the system bus 23 through an output interface 48 , such as a video adapter. In addition to the display devices 47 , the computer system 20 is equipped with other peripheral output devices (not shown in FIG. 7 ), such as speakers and other audiovisual devices.

The computer system 20 may operate in a networked environment using a network connection to one or more remote computers 49 . The remote computer(s) 49 is a working personal computer or server that contains most or all of the components noted previously in describing the nature of the computer system 20 shown in FIG. 7 . There may also be other devices in the network environment, such as routers, network stations or other network nodes. The computer system 20 may include one or more network interfaces 51 or network adapters for communicating with remote computers 49 over one or more networks, such as a local area network (LAN) 50 , a wide area network (WAN), an intranet, and the Internet. Examples of network interface 51 are an Ethernet interface, a Frame Relay interface, a SONET interface, and wireless interfaces.

Embodiments of the present invention may be a system, a method, or a computer-readable medium (or storage medium).

A computer-readable storage medium is a tangible device that stores and stores program code in the form of machine-readable instructions or data structures that are accessible by the central processing unit 21 of a computer system 20 . A computer-readable medium may be an electronic, magnetic, optical, electromagnetic, semiconductor storage device, or any suitable combination thereof. By way of example, such a computer-readable storage medium may include random access memory (RAM), read-only memory (ROM), EEPROM, compact disc read-only memory (CD-ROM), digital versatile disk (DVD) ), flash memory, hard drive, portable computer floppy disk, memory card, floppy disk, or even a mechanically encoded device such as punch cards or embossed structures with instructions written on them.

The system and method of the present invention can be thought of in terms of means. The term "device" as used herein refers to an actual device, component, or group of components implemented in hardware, such as an application-specific integrated circuit (ASIC) or FPGA, or a combination of hardware and software. providing, for example, using a microprocessor system and a set of machine-readable instructions to implement the functionality of a tool that (in the process of execution) turns the microprocessor system into a special-purpose device. The tool may also be implemented as a combination of these two components, with some functions being implemented by hardware alone and other functions being implemented by a combination of hardware and software. In some embodiments, at least a portion, and in some cases all, of the means may be executed on the central processing unit 21 of the computer system 20 . Accordingly, each means may be implemented in various suitable configurations and should not be limited to any particular embodiment set forth herein.

In conclusion, it should be noted that the information given in the description are examples that do not limit the scope of the present invention as defined by the formula. One skilled in the art will appreciate that in developing any actual embodiment of the present invention, many decisions specific to the particular implementation must be made to achieve specific objectives, and these specific objectives will differ from one embodiment to another. It is understood that such development efforts can be complex and time-consuming, but would nevertheless be a routine engineering task for those of ordinary skill in the art using the present disclosure.

Claims (23)

1. A method for monitoring the performance of processes in an operating system to identify errors in the operation of processes, wherein the method is carried out in a hardware processor, and the method contains the steps according to which: converting the program source code into equivalent machine code; build control flow graphs that reflect the structure and syntax of the program source code; converting control flow graphs into call flow graphs using target instructions received from the operating system; generating at least one control flow signature based on the generated call flow graphs; launching a control flow monitor, after which a target process corresponding to the specified program is launched in the operating system; intercept, using a control flow monitor, instructions of the specified program executed by the target process; checking the intercepted instructions against at least one control flow signature of the specified target process; if the instructions of the target process do not match the control flow signature, an error in the operation of the target process is determined. 2. The method according to claim 1, in which the source code of the program is converted into equivalent machine code using a compiler or interpreter. 3. The method according to claim 1, in which, when an error is detected in the operation of the target process, the target process is restarted. 4. The method according to claim 1, in which an error in the target process associated with one of: failure to carry out at least one of the expected instructions; executing an illegal instruction. 5. The method according to claim 4, in which the expected instructions are instructions received from the operating system, within which the health control of the target process is implemented. 6. The method of claim 4, wherein an invalid instruction is any instruction that is not contained in the source code. 7. The method according to claim 1, in which the control flow graph is converted into call flow graphs by excluding from blocks of program code the control flow graphs of instructions that are not target instructions or function calls. 8. The method according to claim 1, in which a control flow signature is generated in at least one of the ways: combining call flow graphs and call flow graph connections; by transforming the at least one call flow graph to a state machine view of the calls of the target instructions. 9. The method according to claim 1, in which a control flow signature is generated by generating a set of call flow graphs, wherein the set of call flow graphs is formed by sequentially replacing basic blocks with target instructions with their corresponding call flow graphs. 10. The method according to claim 1, in which, if an error is detected in the operation of the target process, the operation of the process is stopped. 11. A system for monitoring the performance of processes in the operating system to identify errors in the operation of processes, consisting of a computer device with a hardware processor configured to perform the method according to any one of paragraphs. 1-10.