LU500621B1 - Enhancing shadow stack enforcement after process creation - Google Patents

Abstract

Enhancing shadow stack enforcement after process creation. A process is created from an application binary, including creating a call stack associated with the process. A shadow stack enhancement application programming interface (API) is then exposed to the process. After creating the process from the application binary, a shadow stack enhancement API call is received from the process. Based at least on receiving the shadow stack enhancement API call, a shadow stack enforcement mode for the process is enhanced. Examples of enhancement include upgrading the shadow stack enforcement mode for the process from an off mode to one of an audit mode, a compatibility mode, or a strict mode; upgrading the shadow stack enforcement mode for the process from the audit mode to the compatibility mode or the strict mode; or upgrading the shadow stack enforcement mode for the process from the compatibility mode to the strict mode. 

Description

410345-LU-NP

ENHANCING SHADOW STACK ENFORCEMENT AFTER PROCESS CREATION LU500621

TECHNICAL FIELD

[001] The present disclosure relates to systems, methods, and devices that use a shadow stack to protect a procedure’s stored return address from malicious or unintentional modification.

BACKGROUND

[002] In most modern computing architectures, an operating system allocates a portion of system memory for each thread to a call stack (sometimes referred to as a machine stack or an execution stack). This call stack is used to facilitate procedure calls (e.g., functions, routines, etc.) within application code. In general, each time a procedure is called during execution of a thread, a new stack frame is added or “pushed” to the call stack for that thread. This stack frame is generally created by a “prologue” to the procedure which saves state information needed to return to the calling context (e.g., a return address of the next instruction to be executed after the procedure exits, saved register values, etc.), allocates memory for any parameters passed to the procedure, allocates memory for local variables to the procedure, etc. The particular data that is included in a stack frame, and the layout of that data, varies depending on processor architecture, operating system, etc. When a procedure exits, its stack frame is removed or “popped” from the call stack for the thread. This stack frame is generally removed by an “epilogue” to the procedure, which restores saved information needed by the calling context (e.g., by placing a saved return address in a program counter or instruction counter, restore saved register values, etc.) and deallocates any call stack memory that was occupied by the stack frame.

[003] Recently, some operating systems and/or processors have included support for shadow stacks, which protect a procedure’s stored return address from malicious or unintentional modification. As examples, recent processors designed by INTEL CORPORATION of Santa Clara,

California implement shadow stack hardware support known as control-flow enforcement technology (CET), and recent builds of the WINDOWS operation system from MICROSOFT CORPORATION of

Redmond Washington interoperate with CET to implement shadow stack software support known as hardware-enforced stack protection (HSP). A shadow stack for a thread is a second, separate stack that “shadows” a thread’s normal call stack. When shadow stack enforcement is enabled for an application’s thread, execution of each of that application’s procedure prologues causes a return address to be stored in both the thread’s call stack and the thread’s shadow stack. Execution of each of that application’s procedure epilogues, on the other hand, causes a return address to be loaded from both the thread’s call stack and the thread’s shadow stack, and then compared. If the two records of the return address differ between the call stack and the shadow stack, then a shadow stack violation 1

410345-LU-NP may be detected, and a remedial action is taken—such as termination of the thread (or the process to LU500621 which the thread belongs).

[004] As an example, shadow stacks are useful for detecting a call stack “buffer overflow” that causes a procedure to return to an address other than its proper return address when it completes.

For example, a malicious actor may attempt to carry out a stack buffer overflow attack by taking advantage of flaws in a procedure’s bounds checking to write beyond the amount of memory allocated to parameters and/or local variables in the procedure’s stack frame, and into stack memory in the procedure’s stack frame that is used to store the procedure’s return address. Thus, the malicious actor may use a stack buffer overflow to replace a procedure’s stored memory address with a new memory address (e.g., within the malicious actor's own code, or within code that was not otherwise intended to be executed upon return from the procedure). Without shadow stack enforcement, execution of the procedure’s epilogue results in a “return” to this new memory address and execution of malicious/unintended code at that memory address. With shadow stack enforcement, on the other hand, execution of the procedure’s epilogue results in detection of a return address mismatch between the call stack and the shadow stack and a remedial action such as termination of the application’s thread/process.

[005] Because shadow stacks are a relatively recent development, there may be compatibility concerns when enabling shadow stack enforcement for many applications. In particular, enabling shadow stack enforcement on applications that were authored without consideration of shadow stack technology often triggers shadow stack violations by those applications—even in the absence of malicious activity, which can lead to unwanted and/or unintended termination of those application’s threads/processes. These incompatibilities can arise due to an entity (e.g., the application, a module loaded by the application, or some other application) non-maliciously tampering with a return address in the application’s call stack. This tampering could be intentional (e.g., due to an antivirus application or anti-cheating software using return-oriented programming (ROP) techniques to inject code into the application), or unintentional (e.g., due to a bug). Due to these compatibility concerns, most operating environments make shadow stack enforcement “opt-in” for user-mode applications and enable shadow stack enforcement for a given process at process creation. For example, when creating a new process from an application binary, the WINDOWS operating system enables HSP for the new process based on the presence of particular flag in the binary’s portable executable (PE) header (which, for example, is present based on the application having been compiled with the /CETCOMPAT linker flag), or based on an express request for HSP enablement when calling a CreateProcess application programming Interface (API). 2

410345-LU-NP

BRIEF SUMMARY LUS00621

[006] Making make shadow stack enforcement opt-in for user-mode application at process creation gives application developers the flexibility to enable shadow stack enforcement for an application only when the application has been deemed to be sufficiently compatible with shadow stacks. However, the inventors have observed that adoption of shadow stack enforcement has been unsatisfactorily slow. This is because once an application binary has been configured to opt-in to shadow stack enforcement, processes created from that application binary always execute with shadow stack enforcement enabled. However, even though an application may be able to compatibly execute with shadow stack enforcement enabled in many scenarios, there may be other scenarios in which this is not the case (e.g., due to the presence of antivirus or anti-cheating software, due to the loading of certain modules withing the application, etc.). So long as one or more of these other scenarios exist, an application developer may be unable or unwilling to opt-in to shadow stack enforcement for the application.

[007] Additionally, some operating environments, such as WINDOWS, enable varying shadow stack enforcement modes that take different remedial actions when shadow stack violations are detected.

In embodiments, these enforcement modes include, as a most restrictive mode, a strict mode that terminates an opted-in application’s process when there is any shadow stack violation by the process.

In embodiments these enforcement modes also include, as a moderately restrictive mode, a compatibility mode that terminates an opted-in application’s process when the application itself, or a shadow stack compatible module loaded by the process, causes a shadow stack violation; however, compatibility mode gracefully handles shadow stack violations by non-shadow stack compatible modules loaded by the process, or by dynamic code, without terminating the process. In embodiments these enforcement modes also include, as a least restrictive mode, an audit mode that logs shadow stack violations associated with an opted-in application’s process without terminating the process. In these operating environments, a process that is created from an opted-in application binary operates under one of those enforcement modes from its creation to its termination; thus, it is not impossible to upgrade the enforcement mode for that process after its creation. Even though an application may be able to compatibly execute with shadow stack enforcement enabled in a more restrictive mode (e.g., compatibility mode or strict mode), there may be other scenarios in which this is not the case— such as due to the presence of antivirus or anti-cheating software, due to the loading of certain modules within the application, etc. So long as one or more of these other scenarios exist, an application developer may be unable or unwilling to opt-in to a more restrictive enforcement mode.

[008] At least some embodiments described herein provide additional flexibility in use of shadow stacks by enabling the enhancing of shadow stack enforcement for a process after the process’ 3

410345-LU-NP creation. As used herein, “enhancing shadow stack enforcement” for a process comprises at least one LU500621 of (i) transitioning from executing a process with shadow stack enforcement disabled to executing the process with shadow stack enforcement enabled without terminating the process, or (ii) transitioning the shadow stack enforcement mode for the process from a less restrictive enforcement mode to a more restrictive enforcement mode without terminating the process (e.g., from audit mode to compatibility mode or strict mode, or from compatibility mode to strict mode). Notably, embodiments specifically prohibit diminishing/downgrading shadow stack enforcement (e.g., from shadow stack enforcement being enabled to shadow stack enforcement being disabled, or from a more restrictive enforcement mode to a less restrictive enforcement mode), since doing so could provide a mechanism for malware or malicious actors to circumvent or subvert shadow stack protections.

[009] It will be appreciated that enabling the enhancing of shadow stack enforcement for a process after the process’ creation promotes system security, since it enables applications to selectively opt- in to using shadow stacks, and/or to selectively select a more restrictive enforcement mode. This means that, even if situations exist in which an application cannot execute compatibly when using shadow stacks, an application can choose to enable shadow stacks or use a more restrictive enforcement mode in situations in which the application can execute compatibly when using shadow stacks. Previously, if there were situations in which an application could not execute compatibly when using shadow stacks, an application developer would be forced to not opt-in to shadow stack enforcement globally. Using the techniques herein, however, an application is able to begin execution with shadow stack support disabled, or with shadow stack support enabled but in a less restrictive enforcement mode, and then request enhanced shadow stack enforcement after it is determined that the application is executing in a compatible environment. As such, the embodiments herein promote/encourage the adoption of shadow stacks by a greater number of applications. For every application that adopts shadow stacks, a potential attack surface at a computer system is reduced, thereby promoting system security.

[010] Some embodiments are directed to methods, systems, and computer program products that enhance shadow stack enforcement after process creation. These embodiments create a process from an application binary, which includes creating a call stack associated with the process. These embodiments also expose a shadow stack enhancement API to the process. After creating the process from the application binary, these embodiments receive a shadow stack enhancement API call from the process. Then, based at least on receiving the shadow stack enhancement API call, these embodiments enhance a shadow stack enforcement mode for the process.

[011] In embodiments, when shadow stack enforcement mode for the process is an off mode upon creating the process, the enhancement includes seeding a shadow stack for the process and upgrading 4

410345-LU-NP the shadow stack enforcement mode for the process from the off mode to one of an audit mode, a LU500621 compatibility mode, or a strict mode. In embodiments, when shadow stack enforcement mode for the process is the audit mode (either upon creating the process or based on a prior enhancement), the enhancement includes upgrading the shadow stack enforcement mode for the process from the audit mode to the compatibility mode or the strict mode. In embodiments, when shadow stack enforcement mode for the process is the compatibility mode (either upon creating the process or based on a prior enhancement), the enhancement includes upgrading the shadow stack enforcement mode for the process from the compatibility mode to the strict mode.

[012] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

[013] In order to describe the manner in which the above-recited and other advantages and features of the invention can be obtained, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered to be limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[014] Figure 1A illustrates an example computer system that facilitates enhancing shadow stack enforcement after process creation;

[015] Figure 1B illustrates example operating system components that facilitate enhancing shadow stack enforcement after process creation;

[016] Figure 2 illustrates a flow chart of an example method for seeding a shadow stack;

[017] Figures 3A and 3B illustrate an example of seeding a shadow stack; and

[018] Figure 4 illustrates a flow chart of an example method for enhancing shadow stack enforcement after process creation.

DETAILED DESCRIPTION

[019] Figure 1A illustrates an example computer system 101 that facilitates enhancing shadow stack enforcement after process creation. Computer system 101 comprises or utilizes a special-purpose or general-purpose computer hardware, such as, for example, a processor 102 (or a plurality of

410345-LU-NP processors), system memory 103, and durable storage 104, which are communicatively coupled using LU500621 one or more communications buses 105.

[020] Embodiments within the scope of the present invention include physical and other computer- readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that are accessible by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions and/or data structures are computer storage media. Computer-readable media that carry computer-executable instructions and/or data structures are transmission media. Thus, by way of example, and not limitation, embodiments of the invention can comprise at least two distinctly different kinds of computer-readable media: computer storage media and transmission media.

[021] Computer storage media are physical storage media (e.g., system memory 103 and/or durable storage 104) that store computer-executable instructions and/or data structures. Physical storage media include computer hardware, such as RAM, ROM, EEPROM, solid state drives ("SSDs"), flash memory, phase-change memory ("PCM"), optical disk storage, magnetic disk storage or other magnetic storage devices, or any other hardware storage device(s) which can be used to store program code in the form of computer-executable instructions or data structures, which can be accessed and executed by a general-purpose or special-purpose computer system to implement the disclosed functionality of the invention.

[022] Transmission media can include a network and/or data links which can be used to carry program code in the form of computer-executable instructions or data structures, and which can be accessed by a general-purpose or special-purpose computer system. À "network" is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer system, the computer system may view the connection as transmission media.

Combinations of the above should also be included within the scope of computer-readable media.

[023] Further, upon reaching various computer system components, program code in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to computer storage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module, and then eventually transferred to computer system RAM (e.g., system memory 103) and/or to less volatile computer storage media (e.g., durable storage 104) at the computer system. Thus, it should be understood that computer storage media can be included in computer system components that also (or even primarily) utilize transmission media. 6

410345-LU-NP

[024] Computer-executable instructions comprise, for example, instructions and data which, when LU500621 executed at one or more processors, cause a general-purpose computer system, special-purpose computer system, or special-purpose processing device to perform a certain function or group of functions. Computer-executable instructions may be, for example, machine code instructions (e.g., binaries), intermediate format instructions such as assembly language, or even source code.

[025] Those skilled in the art will appreciate that the invention may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi- processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, and the like. The invention may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. As such, in a distributed system environment, a computer system may include a plurality of constituent computer systems. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

[026] Those skilled in the art will also appreciate that the invention may be practiced in a cloud computing environment. Cloud computing environments may be distributed, although this is not required. When distributed, cloud computing environments may be distributed internationally within an organization and/or have components possessed across multiple organizations. In this description and the following claims, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of "cloud computing" is not limited to any of the other numerous advantages that can be obtained from such a model when properly deployed.

[027] Acloud computing model can be composed of various characteristics, such as on-demand self- service, broad network access, resource pooling, rapid elasticity, measured service, and so forth. A cloud computing model may also come in the form of various service models such as, for example,

Software as a Service ("SaaS"), Platform as a Service ("PaaS"), and Infrastructure as a Service ("laaS").

The cloud computing model may also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, and so forth.

[028] Some embodiments, such as a cloud computing environment, may comprise a system that includes one or more hosts that are each capable of running one or more virtual machines. During operation, virtual machines emulate an operational computing system, supporting an operating system and perhaps one or more other applications as well. In some embodiments, each host includes 7

410345-LU-NP a hypervisor that emulates virtual resources for the virtual machines using physical resources that are LU500621 abstracted from view of the virtual machines. The hypervisor also provides proper isolation between the virtual machines. Thus, from the perspective of any given virtual machine, the hypervisor provides the illusion that the virtual machine is interfacing with a physical resource, even though the virtual machine only interfaces with the appearance (e.g., a virtual resource) of a physical resource. Examples of physical resources including processing capacity, memory, disk space, network bandwidth, media drives, and so forth.

[029] As shown in Figure 1A, the processor 102 includes (among other things) a processing unit 106 (or a plurality of processing units), which loads and executes machine code instructions from system memory 103 (usually via one or more processor caches, not shown). In some embodiments, the processor 102 includes gate logic and/or microcode that provide shadow stack support 107 (e.g., INTEL

CET) by the processor 102. The particular functionality of shadow stack support 107 can vary depending on design choices, but example functionality includes the ability to allocate and protect memory regions for shadow stack use (e.g., via page table mappings), the ability to “push” return addresses onto a shadow stack during execution of a procedure prologue (e.g., as part of execution of a “call” instruction), the ability to “pop” a return address from a shadow stack during execution of a procedure epilogue (e.g., as part of execution of a “return” instruction), the ability to compare one return address popped from a call stack with another return address popped from a shadow stack (e.g., as part of execution of a “return” instruction), and/or the ability to trigger an exception when there is a mismatch between the return address popped from a call stack and the return address popped from a shadow stack. However, it will be appreciated that the embodiments herein could be implemented without shadow stack support 107. For example, shadow stack support 107 could instead be provided by an operating system and/or could be complied into procedure prologues and epilogues of an application binary.

[030] As illustrated, the durable storage 104 stores computer-executable instructions and/or data structures representing executable software components; correspondingly, during execution of this software at the processor 102, one or more portions of these computer-executable instructions and/or data structures are loaded into system memory 103. For example, the durable storage 104 is shown as storing computer-executable instructions and/or data structures corresponding to an operating system 108 and an application 109 (or a plurality of applications), and as potentially storing computer-executable instructions and/or data structures corresponding to a module 110 (or a plurality of modules). The durable storage 104 can also store data, such as rules 111 and logs 112.

[031] The system memory 103 is capable of storing a broad variety of data, but for purposes of illustrating the embodiments herein, the system memory 103 is shown as storing at least a portion of 8

410345-LU-NP code of at least one running application (i.e., application code 109a), and potentially at least a portion LU500621 of code of a module called by that running application (i.e., module code 110a). The system memory 103 is shown as storing memory allocated to a call stack 113 (or a plurality of call stacks) and a shadow stack 114 (or a plurality of shadow stacks). In embodiments, the call stack 113 is a call stack for application 109, and the shadow stack 114 is a shadow stack for application 109.

[032] Figure 1B illustrates details of operating system 108, including example components that facilitate enhancing shadow stack enforcement after process creation, according to some embodiments. It will be appreciated that the depicted components—including their identity, sub- components, and arrangement—are presented merely as an aid in describing various embodiments of operating system 108 described herein, and that these components are non-limiting to how software and/or hardware might implement various embodiments described herein, or of the particular functionality thereof.

[033] Operating system 108 is illustrated as including kernel 115 that includes a task manager 116 which is responsible for creating and managing execution of processes (including one or more threads) at the processor 102, based on binary images corresponding to operating system 108 and application 109. Based upon a request for creation of a process from an identified binary image, the task manager 116 creates a process from the identified binary image, including creating at least one traditional call stack for at least one thread executing as part of the newly-created process. In some situations, the task manager 116 determines that a newly-created process should have shadow stack enforcement enabled from its inception. In these situations, the task manager 116 interfaces with a shadow stack enabler 119, which uses a shadow stack creation component 120 to also create at least one shadow stack for at least one thread executing as part of the newly-created process prior to executing code from the identified binary image. In order to determine if a newly-created process should have shadow stack enforcement enabled, the task manager 116 is illustrated as including one or both of a binary flag checker 117 or a shadow stack enablement API 118 (enablement API 118). In embodiments, the binary flag checker 117 checks for the presence of an indicator (e.g., a flag, a value, etc.) within a binary image which, when present, signals that the binary image is opted-in to shadow stack enforcement. In some embodiments, such as when implemented within the WINDOWS operating system, the binary flag checker 117 checks for a flag within a PE header of the binary image, with the flag being present based on use of a /CETCOMPAT linker flag during compilation; however, the binary flag checker 117 could also check for a flag (or other indicator) within a variety of binary image types, such as executable and linkable format (ELF), Mach object file (MACH-O), and the like.

[034] In embodiments, the enablement API 118 enables an opt-in to shadow stack enforcement during process creation, even if the corresponding image lacks an indicator (e.g., a flag, a value, etc.) 9

410345-LU-NP requesting shadow stack enforcement. Thus, a parent process that initiates a request for creation of LU500621 a new process can utilize the enablement API 118 to enable shadow stack enforcement for the new process.

[035] The kernel 115 is also shown as including a shadow stack violation exception handler 125 (exception handler 125) that processes exceptions when a shadow stack violation is detected during execution of a thread within a process for which shadow stack enforcement is enabled. In some embodiments, execution of the exception handler 125 is triggered by the shadow stack support 107 of the processor 102 (e.g., via a hardware interrupt) when a mismatch between a call stack return address and a shadow stack return address is detected. However, the exception handler 125 could be triggered in other ways (e.g., via a software interrupt or exception), such as by code that executes as part of a procedure epilogue and that determines whether or not a call stack return address and a shadow stack return address match. Thus, the exception handler 125 is capable of being utilized in a wide variety of environments, including those that include hardware support for shadow stacks (e.g., shadow stack support 107), and those that lack hardware shadow stack support (e.g., in which shadow stack functionality is implemented entirely in software, such as via specially-configured procedure prologues and epilogues).

[036] The exception handler 125 is illustrated as supporting enforcement modes 126. In embodiments, each enforcement mode in enforcement modes 126 takes different remedial actions when shadow stack violations are detected. In embodiments the enforcement modes 126 include, as a most restrictive mode, a “strict mode” that terminates a process executing with shadow stack support enabled when any shadow stack violation occurs in connection with execution of the process (i.e., when a return address popped off of a call stack associated with the process does not match a return address in a corresponding shadow stack). In embodiments, this strict mode also logs shadow stack violations in the logs 112. For example, the operating system 108 is shown as including a logger 127, which generates log entries (e.g., stored in logs 112) in connection with operation of the exception handler 125.

[037] In embodiments the enforcement modes 126 also include, as a moderately restrictive mode, a “compatibility mode” that terminates a process executing with shadow stack support enabled when the application, itself, causes a shadow stack violation, or when a shadow stack compatible module loaded by the process causes a shadow stack violation. In embodiments, however, the compatibility mode gracefully handles shadow stack violations caused by execution of modules loaded by the process that are shadow stack incompatible without terminating the process, and/or gracefully handles shadow stack violations by execution of dynamic code without terminating the process. In embodiments, a module loaded by the process is determined to be shadow stack compatible or

410345-LU-NP incompatible based on the presence or absence of an indicator (e.g., a flag, a value, etc.) within a LU500621 binary image corresponding to the module. In embodiments, the compatibility mode logs shadow stack violations in the logs 112, whether they occur during execution of any of the application, a shadow stack compatible module, a shadow stack incompatible module, or dynamic code.

[038] In embodiments the enforcement modes 126 also include, as a least restrictive enforcement mode, an “audit mode” that logs shadow stack violations associated with a process executing with shadow stack support enabled, without actually terminating the process when a shadow stack violation occurs. In embodiments, the audit mode is usable to run an application with shadow stack support enabled in order to identify behaviors that cause shadow stack violations (e.g., as part of testing compatibility, in order to identify bugs, etc.), without actually terminating the application when those shadow stack violations occur.

[039] The kernel 115 is also shown as including a shadow stack enhancer 122 (enhancer 122) that enables a process to request an enhancement of its shadow stack enforcement mode post-creation (i.e., after the process has had a chance to begin execution). In embodiments, the enhancer 122 recognizes an off mode (i.e., shadow stack enforcement is disabled for a process, and there are thus no shadow stacks associated with the process), as well as one or more of the enforcement modes 126 discussed previously (each of which is more restrictive than the off mode). In embodiments the enhancer 122 only permits a transition to an enforcement mode that is more restrictive than a mode in which a requesting process is operating, which prevents a diminishing/downgrading shadow stack enforcement mode.

[040] Figure 1B shows that the enhancer 122 includes a shadow stack enhancement API 123 (enhancement API 123) that the kernel 115 exposes to one or more processes created by the task manager 116, and through which the process can request an enhancement of its shadow stack enforcement. In some embodiments, the enhancement API 123 simply upgrades the enforcement mode for the process from a previous mode (e.g., off mode, audit mode, etc.) to a predetermined target enforcement mode (e.g., compatibility mode or strict mode). In other embodiments, the enhancement API 123 accepts a parameter that specifies a desired target enforcement mode (e.g., audit, compatibility, or strict).

[041] Figure 1B shows that, in at least some embodiments, the enhancer 122 includes a compatibility state checker 124, which performs a compatibility check when a given process requests an enhancement of its shadow stack enforcement. For example, in embodiments, the compatibility state checker 124 determines (e.g., based on rules 111) if there is any application (e.g., antivirus software, anti-cheating software, etc.) or module loaded in the system memory 103 that is known to cause shadow stack violations for an application corresponding to the process, and/or if the logs 112 11

410345-LU-NP contain a record of any shadow stack violations that may be relevant to an application corresponding LU500621 to the process. In some embodiments, the enhancer 122 disallows enhancement of shadow stack enforcement for a process if the compatibility state checker 124 determines that there is an incompatibility with the process generally, or if there is an incompatibility with a target enforcement mode requested by the process. In embodiments, the enhancement API 123 enables a process to unconditionally request an enhancement of its shadow stack enforcement—without any checks by the compatibility state checker 124. In additional, or alternative embodiments, the enhancement API 123 enables a process to request an enhancement of its shadow stack enforcement conditioned on the compatibility state checker 124 determining that the enhancement would be compatible with the process.

[042] In embodiments, if a process that has shadow stack support disabled (i.e., it is operating in the off mode) requests an enhancement of its shadow stack enforcement, then the enhancer 122 calls the shadow stack enabler 119 for creation of one or more shadow stacks for the process. The shadow stack enabler 119, in turn, creates at least one shadow stack using the shadow stack creation component 120, and uses a shadow stack seeding component 121 to seed that shadow stack from a corresponding call stack. In general, the shadow stack seeding component 121 parses through a call stack and copies identified return addresses to a corresponding shadow stack. In embodiments, this is accomplished by performing a stack walk of the call stack (e.g., one stack frame at a time) to find the return address of each stack frame. In embodiments, this stack walk includes using unwind metadata to determine where each stack frame begins and ends, and where the return address lies inside each stack frame. In embodiments, this unwind metadata is compiled into the application binary, and contains the stack frame layout for each function.

[043] In some embodiments, the shadow stack seeding component 121 copies only return addresses that are deemed to be “safe” to the shadow stack. In embodiments, return addresses are deemed to be safe when they point to operating system binaries and/or binaries that have been compiled as shadow stack compatible. By copying only these safe return addresses to the shadow stack, the shadow stack seeding component 121 may avoid seeding any “unsafe” return addresses to the shadow stack that point to unsafe and potentially malicious binaries. Thus, after the shadow stack enforcement enhancement is complete, any attempt to return to an unsafe binary (e.g., based on a function epilogue popping a return address pointing to the unsafe binary from the call stack) will result in a shadow stack violation.

[044] In order to further describe operation of the shadow stack seeding component 121, Figure 2 illustrates a flowchart of an example method 200 for seeding a shadow stack, while Figures 3A and 3B illustrate an example of seeding a shadow stack. Referring initially to Figure 3A, illustrated is an 12

410345-LU-NP example 300a that depicts a representation of an application binary 301 and a module 302. While LU500621 binary formats vary from operating system to operating system, in general, a binary includes a header describing properties and layout of the binary, and a body that comprises application code and data (e.g., in the form a text segment, a data segment, etc.). Thus, in example 300a, application binary 301 is shown as including a header 301a portion and a body 301b portion. The header 301a is shown as including a checkbox containing a checkmark. In example 300a, this checkbox represents a flag or other indicator of whether or not the application binary 301 was compiled with support for and/or requests enforcement of, shadow stacks. Since header 301a includes the checkmark, the application binary 301 is shadow stack compatible example 300a. Module 302 is also shown as including a header 302a portion and a body 302b portion. The header 302a is shown as including a checkbox not containing a checkmark. Since header 302a lacks a checkmark, the module 302 is not shadow stack compatible in example 300a.

[045] Example 300a shows that the task manager 116 has initiated execution of at least one thread for application binary 301, since it has created a call stack 303 corresponding to that initiated thread.

It is noted that, in example 300a, call stack 303 “grows” downward; that is, new information is pushed onto the bottom of these stacks, such that the “top” item on the stack is visually shown at the bottom of the call stack 303. The call stack 303 is shown as including five stack frames (i.e., stack frame 3033, stack frame 303b, stack frame 303c, stack frame 303d, and stack frame 303g) that were created in connection with execution of code of application binary 301 (e.g., due to internal procedure calls within that binary), as well as two stack frames (i.e., stack frame 303e and stack frame 303f) that were created in connection with execution of code of module 302 (e.g., due to the application binary 301 calling a procedure within module 302, and due to module 302 calling an internal procedure). Each of stack frames 303a-303g is illustrated as storing a corresponding return address (i.e., return address 305a for stack frame 303a, return address 305b for stack frame 303b, return address 305c for stack frame 303c, return address 305d for stack frame 303d, return address 305e for stack frame 303e, return address 305f for stack frame 303f, and return address 305g for stack frame 303g).

[046] Referring now to Figure 2, method 200 comprises an act 201 of identifying a return address from a call stack frame. In an example, starting at the “bottom” stack frame (i.e., stack frame 303a), the shadow stack seeding component 121 identifies return address 305a.

[047] Method 200 also comprises a decision block (act 202) for determining whether a return address points to a “safe” binary. In an example, the shadow stack seeding component 121 identifies a binary that return address 305a points to, and determines if that binary is an operating system binary and/or if that binary is compiled as being shadow stack compatible. Since the application binary 301 13

410345-LU-NP includes a checkmark in example 300a, the stack seeding component 121 can determine that return LU500621 address 305a points to a safe binary.

[048] In Figure 2, if a return address points to a safe binary (“Yes” from act 202), then method 200 comprises an act 204 of seeding the return address to the shadow stack. Thus, in the example, the stack seeding component 121 can seed return address 305a onto a shadow stack.

[049] In Figure 2, if the return address does not point to a safe binary (“No” from act 202), then method 200 comprises an act 203 of ignoring the return address. For instance, if act 202 instead considered return address 305e or return address 305f, the shadow stack seeding component 121 would determine that module 302 is not shadow stack compatible (i.e., header 302a lacks a checkmark) and determine that the return address does not point to a safe binary. Thus, the shadow stack seeding component 121 would ignore the return address for seeding purposes.

[050] In either case, as is indicated by a broken arrow that connects act 203 with act 201, and a broken arrow thar connects act 204 with act 201, method 200 can repeat to identify and potentially seed an additional return address. Thus, method 200 can proceed to stack frame 303b, then stack frame 303c, and so on until stack frame 303g. It is noted that, while method 200 has been described as proceeding from a bottom stack frame to a top stack frame, it could alternatively proceed from the top stack frame to the bottom stack frame.

[051] Referring now to Figure 3B, illustrated is an example 300b that depicts the application binary 301, the module 302, and the call stack 303 from example 300a. However, example 300b also illustrates a shadow stack 304 as seeded by method 200, and which includes return address 305a, return address 305b, return address 305c, return address 305d, and return address 305g—each corresponding to “safe” application binary 301.

[052] Further operation of the components of Figure 1B are now described in connection with

Figure 4, which illustrates a flowchart of a method 400 for enhancing shadow stack enforcement after process creation. As shown in Figure 4, method 400 comprises an act 401 of creating a process from an application binary. In an example, the task manager 116 initiates execution of application 109 at the processor 102. In embodiments, initiating execution includes the task manager 116 creating a process from an application binary for application 109, causing application code 109a for the application 109 to be loaded into system memory 103, and creating kernel data structures supporting execution of one or more threads at processing unit 106.

[053] In embodiments, act 401 also comprises creating a call stack associated with the process, and potentially creating a shadow stack associated with the process. For example, act 401 is illustrated as including a decision block (act 402) for determining whether shadow stacks are requested. In an example, the task manager 116 determines if shadow stack enforcement is requested for the process 14

410345-LU-NP upon its creation. In embodiments, the task manager 116 determines this from an indicator (e.g., a LU500621 flag) that is part of the applications’ binary (i.e., using binary flag checker 117), or from an API call used to request creation of the process (i.e., using enablement API 118).

[054] If, in act 402, shadow stacks are determined to be requested (“Yes” from act 402), act 401 includes an act 403 of creating a call stack and a shadow stack. In an example, the task manager 116 creates a call stack 113 for at least one thread for the process, and invokes the shadow stack enabler 119 to create a shadow stack 114 for that thread. In some embodiments, shadow stacks are determined to be requested in act 402 when the application binary includes an indicator requesting enforcement of shadow stacks. In one example specific to the WINDOWS operating system, the binary flag checker 117 identifies a flag within a PE header of the application binary, with the flag being present based on use of a /CETCOMPAT linker flag during compilation. In these embodiments, based at least on the indicator, act 402 determines that shadow stacks are requested, and creating the process from the application binary in act 401 also includes creating a shadow stack associated with the process in act 403 (i.e., in addition to creating the call stack in act 403). In other embodiments, creating the process from the application binary is caused by a call to a process creation API that enables shadow stack enforcement at process creation time, and shadow stacks are determined to be requested in act 402 based on this call. In one example specific to the WINDOWS operating system, a

CreateProcess API is called with appropriate parameters and/or flags so that it functions as the enablement API 118, in order to request creation of the process from the application binary. In these embodiments, based at least on the call to the process creation API, act 402 determines that shadow stacks are requested, and creating the process from the application binary in act 401 also includes creating a shadow stack associated with the process in act 403 (i.e., in addition to creating the call stack in act 403).

[055] After act 403, act 401 also includes an act 404 of turning on shadow stack enforcement. In an example, the shadow stack enabler 119 sets an enforcement mode (e.g., audit, compatibility, or strict) used by the exception handler 125 for the process. Thus, upon creating the process from the application binary, the shadow stack enforcement mode for the process is one of an audit mode, a compatibility mode, or a strict mode. This has a technical effect of increasing security of the process started in act 401, since any shadow stack violations will be handled by the exception handler 125.

[056] If, in act 402, shadow stacks are determined to not be requested (“No” from act 402), act 401 includes an act 405 of creating a call stack. In an example, the task manager 116 creates a call stack 113 for at least one thread for the process. This time, however, the task manager 116 does not invoke the shadow stack enabler 119 to create a shadow stack 114 for that thread. Thus, in some embodiments, act 401 comprises creating a process from an application binary, including creating a

410345-LU-NP call stack associated with the process and in which, upon creating the process from the application LU500621 binary, the shadow stack enforcement mode for the process is an off mode. In embodiments, shadow stacks are determined to not be requested in act 402 based on the application binary lacking an indicator requesting enforcement of shadow stacks, and/or based on the lack of a call to a process creation API that enables shadow stack enforcement at process creation time.

[057] After act 401, method 400 also comprises an act 406 of exposing a shadow stack enhancement

API to process. In an example, the kernel 115 exposes the enhancement API 123 to the process created in act 401, enabling the process to make a call to the enhancement API 123. This has a technical effect of providing a novel ability for the process to request an enhancement to its shadow stack enforcement after its creation.

[058] Notably, exposing a shadow stack enhancement API to the process enables the process to execute without shadow stack enforcement enabled (e.g., an off mode), or to execute with shadow stack enforcement enabled but in a less restrictive mode (e.g., audit or compatibility). This enables the process to execute logic of its own to determine if enhanced stack enforcement would be appropriate or desired. In one example, such logic could include logic that receives user input requesting enhanced stack enforcement be enabled for the process. In another example, such logic could include the process, itself, conducting a compatibility check—such as by determining which other processes are loaded, which modules are loaded into the process, checking for shadow stack violations in the logs 112, etc. Thus, technical effects also include providing a process an execution window during which the process, itself, can execute logic to decide if enhanced stack enforcement is desired, and potentially what mode of enhanced stack enforcement is desired.

[059] Method 400 also comprises an act 407 of receiving a shadow stack enhancement API call from the process. In embodiments, act 407 comprises, after creating the process from the application binary, receiving a shadow stack enhancement API call from the process. In an example, after launching a process for application 109, that process makes an API call to the enhancement API 123, which requests an enhancement of shadow stack enforcement for the process. In some embodiments, receiving the shadow stack enhancement API call comprises receiving a requested shadow stack enforcement mode (e.g., audit, compatibility, or strict).

[060] In some embodiments, receiving the shadow stack enhancement API call comprises receiving a request for an unconditional enhancement of the shadow stack enforcement mode (e.g., without any compatibility checks by the compatibility state checker 124). In an example, the process created for application 109 in act 401 calls enhancement API 123 in a way that enhances shadow stack support without any compatibility check by the compatibility state checker 124, or without regard for the result of a compatibility check by the compatibility state checker 124. 16

410345-LU-NP

[061] In some embodiments, receiving the shadow stack enhancement API call comprises receiving LU500621 a request for a conditional enhancement of the shadow stack enforcement mode that is conditioned on a computer system compatibility state check by the compatibility state checker 124. In an example, the process created for application 109 in act 401 calls enhancement API 123 in a way that requests a compatibility check by the compatibility state checker 124, and that enhances shadow stack support for the process only if suitable compatibility is found. In these embodiments, the method 400 also comprises, based at least on receiving the shadow stack enhancement API call, determining whether a compatibility state at the computer system is satisfied, and, based on the compatibility state at the computer system being satisfied, determining that the shadow stack enforcement mode for the process can be enhanced. In an example, the compatibility state checker 124 determines (e.g., based on rules 111) if there is any application (e.g., antivirus software, anti-cheating software, etc.) or module loaded in the system memory 103 that is known to cause shadow stack violations for an application corresponding to the process, and/or if the logs 112 contain a record of any shadow stack violations that may be relevant to an application corresponding to the process. Then, the enhancer 122 disallows enhancement of shadow stack enforcement for the process if the compatibility state checker 124 determines that there is an incompatibility with the process generally, or if there is an incompatibility with a target enforcement mode requested by the process.

[062] Method 400 also comprises an act 408 of enhancing a shadow stack enforcement mode. In embodiments, act 408 comprises, based at least on receiving the shadow stack enhancement API call, enhancing a shadow stack enforcement mode for the process. In an example, the enhancer 122 turns on shadow stack enforcement if it was previously off, or upgrades an enforcement mode used by the exception handler 125 for the process if shadow stack enforcement if it was previously on. This has a novel technical effect of modifying the level of shadow stack enforcement for a process after the process has already been launched and had a chance to execute.

[063] Inembodiments, enhancing the shadow stack enforcement mode in act 408 comprises turning on shadow stack enforcement, including seeding a shadow stack, or upgrading an enforcement mode used for the process by the exception handler 125. For example, act 401 is illustrated as including a decision block (act 409) for determining whether shadow stacks are enabled. In an example, the enhancer 122 determines if the process that requested the enhancement in act 407 is already executing with the presence of shadow stacks. In various examples, the enhancer 122 consults the task manager 116, the exception handler 125, or some data structure maintained by the kernel 115.

[064] If, in act 409, shadow stacks are determined to be enabled (“Yes” from act 409), act 408 includes an act 410 of upgrading a shadow stack enforcement mode. In examples, shadow stacks could be determined to be enabled because the process had shadow stack support enabled at launch, or 17

410345-LU-NP due to a prior instance of method 400. In these embodiments, upgrading the shadow stack LU500621 enforcement mode for the process includes changing the mode that the exception handler 125 applies when receiving shadow stack exceptions for the process. In embodiments, upon creating the process from the application binary in act 401, the shadow stack enforcement mode for the process is one of an audit mode, a compatibility mode, or a strict mode, and in act 410 enhancing the shadow stack enforcement mode for the process comprises at least one of upgrading the shadow stack enforcement mode for the process from the audit mode to the compatibility mode, upgrading the shadow stack enforcement mode for the process from the audit mode to the strict mode, or upgrading the shadow stack enforcement mode for the process from the compatibility mode to the strict mode.

[065] If, in act 409, shadow stacks are determined to not be enabled (“No” from act 409), act 408 includes an act 411 of seeding a shadow stack. For example, the enhancer 122 calls the shadow stack enabler 119, which creates a shadows stack (shadow stack creation component 120), and uses the shadow stack seeding component 121 to seed that shadow stack, as described supra in connection with method 200. Thus, in embodiments, upon creating the process from the application binary in act 401, the shadow stack enforcement mode for the process is an off mode. Then, as will be appreciated in view of the discussion of method 200, in act 411 enhancing the shadow stack enforcement mode for the process comprises seeding a shadow stack associated with the process from the call stack. In embodiments, when enhancing the shadow stack enforcement mode for the process, seeding the shadow stack from call stack comprises identifying a set of return addresses on the call stack, and copying the set of return addresses to the shadow stack. In embodiments, when identifying the set of return addresses on the call stack, each return address in the set of return addresses is identified from a different stack frame. In embodiments, as part of identifying the set of return addresses on the call stack, each return address in the set of return addresses is verified to point to a binary in a set of safe binaries. In embodiments, the set of safe binaries include one or more of: one or more operating system binaries, or one or more binaries having an indicator requesting enforcement of shadow stacks.

[066] After act 411, act 412 turns on shadow stack enforcement. For example, shadow stack enabler 119 sets an enforcement mode (e.g., audit, compatibility, or strict) used by the exception handler 125 for the process. In embodiments, upon creating the process from the application binary in act 401, the shadow stack enforcement mode for the process is an off mode, and in act 412 enhancing the shadow stack enforcement mode for the process comprises at least one of upgrading the shadow stack enforcement mode for the process from the off mode to an audit mode, upgrading the shadow stack enforcement mode for the process from the off mode to a compatibility mode, or upgrading the shadow stack enforcement mode for the process from the off mode to a strict mode. This has a 18

410345-LU-NP technical effect of increasing security of the process since any shadow stack violations will be handled LU500621 in a more restrictive manner by the exception handler 125.

[067] Accordingly, the embodiments herein provide flexibility in use of shadow stacks by enabling the enhancing of shadow stack enforcement for a process after the process’ creation. This enables a process to make an API call that accomplishes at least one of (i) transitioning from executing the process with shadow stack enforcement disabled to executing the process with shadow stack enforcement enabled without terminating the process, or (ii) transitioning the shadow stack enforcement mode for the process from a less restrictive enforcement mode to a more restrictive enforcement mode without terminating the process (e.g., from audit mode to compatibility mode or strict mode, or from compatibility mode to strict mode). Using the techniques herein, an application is able to begin execution with shadow stack support disabled, or with shadow stack support enabled but in a less restrictive enforcement mode, and then choose to enhance shadow stack enforcement when it is determined that the application is executing in a compatible environment. As such, the embodiments herein have a technical effect of promoting/encouraging the adoption of shadow stacks. Since every application that adopts shadow stacks reduces a potential attack surface at a computer system, the embodiments herein also have a technical effect of promoting system security.

[068] Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above, or the order of the acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims. Additionally, although methodological acts were discussed in a certain order, or were illustrated in a flow chart as occurring in a particular order, no particular ordering is required unless specifically stated, or required because an act is specifically described as being dependent on another act being completed prior to the act being performed.

[069] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. Such embodiments may include a data processing device comprising means for carrying out one or more of the methods described herein; a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out one or more of the methods described herein; and/or a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out one or more of the methods described herein. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope. When introducing elements in the 19

410345-LU-NP appended claims, the articles “a,” “an,” “the,” and “said” are intended to mean there are one or more LU500621 of the elements.

The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements.

Unless otherwise specified, the terms “set,” “superset,” and “subset” are intended to exclude an empty set, and thus “set” is defined as a non-empty set, “superset” is defined as a non-empty superset, and “subset” is defined as a non-empty subset.

Unless otherwise specified, the term “subset” excludes the entirety of its superset (i.e., the superset contains at least one item not included in the subset). Unless otherwise specified, a “superset” can include at least one additional element, and a “subset” can exclude at least one element.

Claims (15)

410345-LU-NP CLAIMS LU500621 What is claimed:

1. A method (400), implemented at a computer system (101) that includes a processor (102), for enhancing shadow stack enforcement after process creation, the method comprising: creating (116) a process from an application binary (109), including creating (120) a call stack (113) associated with the process; exposing (115) a shadow stack enhancement application programming interface (API) (123) to the process; after creating the process from the application binary, receiving (122) a shadow stack enhancement API call from the process; and based at least on receiving the shadow stack enhancement API call, enhancing (122) a shadow stack enforcement mode for the process.

2. The method of claim 1, wherein, upon creating the process from the application binary, the shadow stack enforcement mode for the process is an off mode.

3. The method of claim 2, wherein enhancing the shadow stack enforcement mode for the process comprises at least one of: upgrading the shadow stack enforcement mode for the process from the off mode to an audit mode; upgrading the shadow stack enforcement mode for the process from the off mode to a compatibility mode; or upgrading the shadow stack enforcement mode for the process from the off mode to a strict mode.

4. The method of any of claims 2 or 3, wherein enhancing the shadow stack enforcement mode for the process comprises seeding (121) a shadow stack (114) associated with the process from the call stack.

5. The method of claim 4, wherein seeding the shadow stack from call stack comprises: identifying a set of return addresses on the call stack; and copying the set of return addresses to the shadow stack.

6. The method of claim 5, wherein each return address in the set of return addresses is identified from a different stack frame.

7. The method of any of claim 5 or 6, wherein each return address in the set of return addresses is verified to point to a binary in a set of safe binaries.

8. The method of claim 7, wherein the set of safe binaries includes one or more of: one or more operating system binaries; or 21

410345-LU-NP one or more binaries having an indicator requesting enforcement of shadow stacks. LU500621

9. The method of claim 1, wherein, upon creating the process from the application binary, the shadow stack enforcement mode for the process is one of an audit mode, a compatibility mode, or a strict mode.

10. The method of claim 9, wherein enhancing the shadow stack enforcement mode for the process comprises at least one of: upgrading the shadow stack enforcement mode for the process from the audit mode to the compatibility mode; upgrading the shadow stack enforcement mode for the process from the audit mode to the strict mode; or upgrading the shadow stack enforcement mode for the process from the compatibility mode to the strict mode.

11. The method of claim 9, wherein, the application binary includes an indicator requesting enforcement of shadow stacks; and based at least on the indicator, creating the process from the application binary also includes creating a shadow stack associated with the process.

12. The method of claim 9, wherein, creating the process from an application binary is caused by a call to a process creation API that enables shadow stack enforcement at process creation time; and based at least on the call to a process creation API, creating the process from the application binary also includes creating a shadow stack associated with the process.

13. The method of any preceding claim, wherein receiving the shadow stack enhancement API call comprises receiving a requested shadow stack enforcement mode.

14. The method of any preceding claim, wherein receiving the shadow stack enhancement API call comprises receiving a request for an unconditional enhancement of the shadow stack enforcement mode.

15. The method of any of claims 1 to 13, wherein, receiving the shadow stack enhancement API call comprises receiving a request for a conditional enhancement of the shadow stack enforcement mode that is conditioned on a computer system compatibility state check; and the method further comprises: based at least on receiving the shadow stack enhancement API call, determining whether a compatibility state at the computer system is satisfied; and 22

410345-LU-NP based on the compatibility state at the computer system being satisfied, determining LU500621 that the shadow stack enforcement mode for the process can be enhanced. 23