EP3535681A1 - System und verfahren zur detektion von und zur warnung vor exploits in computerisierten systemen - Google Patents

System und verfahren zur detektion von und zur warnung vor exploits in computerisierten systemen

Info

Publication number
EP3535681A1
EP3535681A1 EP17867010.5A EP17867010A EP3535681A1 EP 3535681 A1 EP3535681 A1 EP 3535681A1 EP 17867010 A EP17867010 A EP 17867010A EP 3535681 A1 EP3535681 A1 EP 3535681A1
Authority
EP
European Patent Office
Prior art keywords
memory
address
exploit
processor
cpu
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP17867010.5A
Other languages
English (en)
French (fr)
Other versions
EP3535681A4 (de
EP3535681B1 (de
Inventor
Shlomi Levin
Michael Aminov
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Perception Point Ltd
Original Assignee
Perception Point Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Perception Point Ltd filed Critical Perception Point Ltd
Publication of EP3535681A1 publication Critical patent/EP3535681A1/de
Publication of EP3535681A4 publication Critical patent/EP3535681A4/de
Application granted granted Critical
Publication of EP3535681B1 publication Critical patent/EP3535681B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/50Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
    • G06F21/57Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
    • G06F21/577Assessing vulnerabilities and evaluating computer system security
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/30Authentication, i.e. establishing the identity or authorisation of security principals
    • G06F21/44Program or device authentication
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/50Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
    • G06F21/52Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/50Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
    • G06F21/55Detecting local intrusion or implementing counter-measures
    • G06F21/552Detecting local intrusion or implementing counter-measures involving long-term monitoring or reporting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/50Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
    • G06F21/55Detecting local intrusion or implementing counter-measures
    • G06F21/554Detecting local intrusion or implementing counter-measures involving event detection and direct action
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/50Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
    • G06F21/55Detecting local intrusion or implementing counter-measures
    • G06F21/56Computer malware detection or handling, e.g. anti-virus arrangements
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/50Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
    • G06F21/55Detecting local intrusion or implementing counter-measures
    • G06F21/56Computer malware detection or handling, e.g. anti-virus arrangements
    • G06F21/566Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2221/00Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/03Indexing scheme relating to G06F21/50, monitoring users, programs or devices to maintain the integrity of platforms
    • G06F2221/033Test or assess software

Definitions

  • the present invention generally relates to detection of exploits in computerized systems. More particularly, the present invention relates to issuing alerts upon detection of attacks that exploit vulnerabilities to take control of the central processing unit (CPU).
  • CPU central processing unit
  • Zero-day vulnerability e.g. time of identifying bugs in the system
  • Such vulnerabilities have become common in targeted attacks against organizations, for instance for corporate espionage and/or economical motives.
  • Cloud providers run hundreds of different services and programs to serve their clients, situating them as the main target for such targeted attacks.
  • an attacker can attack a single client by breaching the services running within that specific client' s virtual machines, for example a Secure-Shell (SSH) service that contains vulnerability that only the attacker knows about.
  • SSH Secure-Shell
  • Data centers e.g., servers of FACEBOOK®
  • the second stage usually includes more advanced malicious logic such as installing a backdoor program which gives the attacker persistent access to the system, thereby giving attackers the power to operate independently and gain control of the system.
  • advanced malicious logic such as installing a backdoor program which gives the attacker persistent access to the system, thereby giving attackers the power to operate independently and gain control of the system.
  • various monitoring units can also be targeted and attacked in a similar way.
  • Security solutions can be implemented either at the hardware level or the software level.
  • Hardware level implementations such as NX-bit, TPM and SGX
  • their long integration cycles mean that attackers have enough time to create new exploitation techniques keeping them at an advantage point.
  • Software level security solutions have short integration cycles and update frequently but have limited visibility of the system.
  • all security software is essentially based on "hooking" into applications to gain visibility which greatly affects the performance, stability and integrity of the application, for instance running in the data center.
  • the host based deployment uses a variety of techniques to detect malware.
  • the most common being static signatures that are continuously compared against the file system, however the continuous scanning causes performance reduction and can easily be bypassed by simply performing small changes in the malware.
  • Dynamic behavior based signatures were developed but due to the nature of implementation (i.e. memory hooks etc.), the stability of the system is compromised as well as resulting in high false positive rates. Consequently, data center operators prefer to deploy network based security solutions.
  • network security solutions scan network streams for known malware signatures which eliminate the high performance penalty of host based scanning, however still easily bypassable for the same reasons.
  • Dynamic behavior based signatures are also not applicable to networks as they require executing binary code which does not comply with the performance requirements of a data center network.
  • malware Regardless of the attack vector and exploitation method employed, the ultimate goal of an attacker is to perform malicious computations on the target system by executing machine instructions that are under control of the attacker.
  • malicious computations are caused by illegitimate code that was not provided or intended to be executed by the developer of the exploited service (e.g. SSH).
  • SSH the developer of the exploited service
  • the malicious code is usually introduced into the target system using external network data or application files.
  • a method of detecting an exploit of a vulnerability of a computing device including receiving an execution flow of at least one process running in a processor of the computing device, wherein the execution flow is received from a performance monitoring unit (PMU) of the processor, receiving memory pages from a memory of the computing device, reconstructing the execution flow of the process on another processor based on PMU data and the memory pages, running at least one exploit detection algorithm on the reconstructed process in order to identify an exploit attempt, and issuing an alert.
  • PMU performance monitoring unit
  • the process running on the processor of the computing device may be interrupted when an exploit is detected.
  • a map of addresses may be maintained in memory of the processor.
  • the structured exception handler (SEH) of the operating system of the processor may be mapped.
  • memory address of SEH may be added.
  • structured exception handler may be checked if registered.
  • at least one of memory allocation addresses and memory deallocation addresses may be mapped.
  • at least one of memory allocation function and memory deallocation function may be checked is called.
  • At least one of allocation counter and deallocation counter may be increased. In some embodiments, difference of allocation counter and deallocation counter may be checked if greater than a predefined value. In some embodiments, at least one instruction may be received. In some embodiments, a shadow stack may be maintained. In some embodiments, a call instruction in the execution flow may be checked if it has been executed.
  • expected return address may be pushed to the shadow stack.
  • a return instruction may be checked if it has been executed.
  • target address may be checked if it is different from top address on shadow stack.
  • an address may be popped from the stack.
  • a database of legal addresses may be maintained to transfer control indirectly to, and instructions may be received for indirect branch from the reconstructed execution flow.
  • the received instruction may be checked if it corresponds to the database of legal addresses to transfer control indirectly to.
  • FIG. 1 shows a block diagram of an exemplary computing device, according to some embodiments of the invention
  • FIG. 2 schematically illustrates a block diagram of an exploit detection system, according to some embodiments of the invention.
  • FIG. 3 shows a fiowchart for a stack injection exploit detection algorithm, according to some embodiments of the invention.
  • FIG. 4A shows a flowchart for a structured exception handling (SEH) exploit detection algorithm, according to some embodiments of the invention
  • FIG. 4B shows a continuation of the fiowchart from Fig. 4A, according to some embodiments of the invention.
  • FIG. 5A shows a fiowchart for a use of memory allocation tracking exploit detection algorithm, according to some embodiments of the invention
  • FIG. 5B shows a continuation of the fiowchart from Fig. 5A, according to some embodiments of the invention.
  • FIG. 6A shows a fiowchart for a first return oriented programming exploit detection algorithm, according to some embodiments of the invention.
  • FIG. 6B shows a continuation of the fiowchart from Fig. 6A, according to some embodiments of the invention.
  • FIG. 6C shows a flowchart for a second return oriented programming exploit detection algorithm, according to some embodiments of the invention.
  • Fig. 6D shows a fiowchart for a continuation of the fiowchart of Fig. 6C, according to some embodiments of the invention
  • Fig. 7 shows a fiowchart for illegal indirect control transfer detection algorithm, according to some embodiments of the invention
  • Fig. 8 shows a flowchart for a method of detecting an exploit of a vulnerability of a computing device, according to some embodiments of the invention
  • FIG. 9A shows a flowchart for combined algorithm operation, according to some embodiments of the invention.
  • FIG. 9B shows a flowchart for a continuation of the flowchart of Fig. 9 A, according to some embodiments of the invention.
  • the terms “plurality” and “a plurality” as used herein may include, for example, “multiple” or “two or more”.
  • the terms “plurality” or “a plurality” may be used throughout the specification to describe two or more components, devices, elements, units, parameters, or the like.
  • the method embodiments described herein are not constrained to a particular order or sequence. Additionally, some of the described method embodiments or elements thereof can occur or be performed simultaneously, at the same point in time, or concurrently.
  • Computing device 100 may include a controller 105 that may be, for example, a central processing unit processor (CPU), a chip or any suitable computing or computational device, an operating system 115, a memory 120, a storage 130, an input devices 135 and an output devices 140.
  • controller 105 may be, for example, a central processing unit processor (CPU), a chip or any suitable computing or computational device, an operating system 115, a memory 120, a storage 130, an input devices 135 and an output devices 140.
  • Operating system 115 may be or may include any code segment designed and/or configured to perform tasks involving coordination, scheduling, arbitration, supervising, controlling or otherwise managing operation of computing device 100, for example, scheduling execution of programs. Operating system 115 may be a commercial operating system.
  • Memory 120 may be or may include, for example, a Random Access Memory (RAM), a read only memory (ROM), a Dynamic RAM (DRAM), a Synchronous DRAM (SD-RAM), a double data rate (DDR) memory chip, a Flash memory, a volatile memory, a non- volatile memory, a cache memory, a buffer, a short term memory unit, a long term memory unit, or other suitable memory units or storage units.
  • Memory 120 may be or may include a plurality of, possibly different memory units.
  • Executable code 125 may be any executable code, e.g., an application, a program, a process, task or script. Executable code 125 may be executed by controller 105 possibly under control of operating system 115. For example, executable code 125 may be an application for managing power consumption data. Where applicable, executable code 125 may carry out operations described herein in real-time. Computing device 100 and executable code 125 may be configured to update, process and/or act upon information at the same rate the information, or a relevant event, are received. In some embodiments, more than one computing device 100 may be used. For example, a plurality of computing devices that include components similar to those included in computing device 100 may be connected to a network and used as a system. For example, managing power consumption data may be performed in realtime by executable code 125 when executed on one or more computing devices such computing device 100.
  • Storage 130 may be or may include, for example, a hard disk drive, a floppy disk drive, a Compact Disk (CD) drive, a CD-Recordable (CD-R) drive, a universal serial bus (USB) device or other suitable removable and/or fixed storage unit.
  • Content may be stored in storage 130 and may be loaded from storage 130 into memory 120 where it may be processed by controller 105.
  • memory 120 may be a non- volatile memory having the storage capacity of storage 130. Accordingly, although shown as a separate component, storage 130 may be embedded or included in memory 120.
  • Input devices 135 may be or may include a mouse, a keyboard, a touch screen or pad or any suitable input device. It will be recognized that any suitable number of input devices may be operatively connected to computing device 100 as shown by block 135.
  • Output devices 140 may include one or more displays, speakers and/or any other suitable output devices. It will be recognized that any suitable number of output devices may be operatively connected to computing device 100 as shown by block 140.
  • Any applicable input/output (I/O) devices may be connected to computing device 100 as shown by blocks 135 and 140.
  • NIC network interface card
  • modem modem
  • printer or facsimile machine a universal serial bus (USB) device or external hard drive
  • USB universal serial bus
  • Embodiments of the invention may include an article such as a computer or processor non-transitory readable medium, or a computer or processor non-transitory storage medium, such as for example a memory, a disk drive, or a USB flash memory, encoding, including or storing instructions, e.g., computer-executable instructions, which, when executed by a processor or controller, carry out methods disclosed herein.
  • a storage medium such as memory 120
  • computer-executable instructions such as executable code 125
  • controller such as controller 105.
  • Some embodiments may be provided in a computer program product that may include a non-transitory machine -readable medium, with instructions stored thereon, which may be used to program a computer, or other programmable devices, to perform methods as disclosed herein.
  • Embodiments of the invention may include an article such as a computer or processor non-transitory readable medium, or a computer or processor non- transitory storage medium, such as for example a memory, a disk drive, or a USB flash memory, encoding, including or storing instructions, e.g., computer-executable instructions, which when executed by a processor or controller, carry out methods disclosed herein.
  • the storage medium may include, but is not limited to, any type of disk including floppy disks, optical disks, compact disk read-only memories (CD-ROMs), rewritable compact disk (CD-RWs), and magneto-optical disks, semiconductor devices such as read-only memories (ROMs), random access memories (RAMs), such as a dynamic RAM (DRAM), erasable programmable read-only memories (EPROMs), flash memories, electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, or any type of media suitable for storing electronic instructions, including programmable storage devices.
  • ROMs read-only memories
  • RAMs random access memories
  • DRAM dynamic RAM
  • EPROMs erasable programmable read-only memories
  • EEPROMs electrically erasable programmable read-only memories
  • magnetic or optical cards or any type of media suitable for storing electronic instructions, including programmable storage devices.
  • a system may include components such as, but not limited to, a plurality of central processing units (CPUs) or any other suitable multi-purpose or specific processors or controllers, a plurality of input units, a plurality of output units, a plurality of memory units, and a plurality of storage units.
  • a system may additionally include other suitable hardware components and/or software components.
  • a system may include or may be, for example, a personal computer, a desktop computer, a mobile computer, a laptop computer, a notebook computer, a terminal, a workstation, a server computer, a Personal Digital Assistant (PDA) device, a tablet computer, a network device, or any other suitable computing device.
  • PDA Personal Digital Assistant
  • FIG. 2 schematically illustrates a block diagram of an exploit detection system 200, according to some embodiments of the invention. It should be noted that the direction of arrows in Fig. 2 may indicate the direction of information flow. Exploit detection system 200 may be utilized to issue an alert upon detection of exploits in a computerized device 210 such as computing device 100 (e.g., as shown in Fig. 1), having a processor 201 operably coupled to a memory unit 202.
  • a computerized device 210 such as computing device 100 (e.g., as shown in Fig. 1), having a processor 201 operably coupled to a memory unit 202.
  • computerized device 210 may also refer to an endpoint of a data center of a computerized system.
  • computerized device 210 for example a personal computer (PC) or a smartphone, may further include a performance monitoring unit (PMU) 204 that may be configured to monitor the operation of processor 201.
  • PMU performance monitoring unit
  • gathering data of actual computer processes may allow access to logs of code execution such that exploit or breach detection may be allowed, as further described hereinafter.
  • exploit detection system 200 may include an observer CPU 203 configured to receive data corresponding to the processing execution flow from computerized device 210.
  • Observer CPU 203 may receive execution flow data from PMU 204 and/or processor 201.
  • observer CPU 203 may further receive memory logs (e.g. RAM) from processor 201 and/or memory unit 202. It should be noted that while processor 201 executes code and/or specific programs the observer CPU 203 may be configured to reconstruct the execution and constantly check the instructions being executed. It may then be possible to act upon these instructions and detect violations.
  • observer CPU 203 may be implemented in a dedicated hardware component (e.g., on a separate computer chip) and/or as software algorithm.
  • observer CPU 203 may receive trace data (with information regarding executed instructions) from the processor 201, for instance using branch tracing and/or processor tracing techniques.
  • memory logs as used hereinafter may refer to memory pages and/or memory images.
  • data received by observer CPU 203 may be at least temporarily stored at a database 205.
  • observer CPU 203 may reconstruct the execution flow from the received data and memory pages so as to allow comparison (e.g., with data stored at database 205) to expected behavior and thereby detect exploits therein. In some embodiments, observer CPU 203 may further be configured to verify that the sequence of opcodes does not include malicious content attempting to exploit vulnerabilities of the process. In some embodiments, observer CPU 203 may monitor execution of code from the stack indicating a possible attack.
  • exploit detection system 200 may automatically issue an alert and/or disable that endpoint (or node) so as to maintain normal operation of the remaining system, and/or prevent spreading of the attack to other endpoints.
  • a crash in the system may indicate that the detected exploit is not an attack since attackers desire to prevent crashes.
  • At least two different exploit detection algorithms may be implemented on observer CPU 203 (and stored at database 205), where each exploit detection algorithm may be dedicated to detect a different exploit.
  • the at least two exploit detection algorithms may run simultaneously until at least one exploit detection algorithm detects an exploit, and for instance issues an alert.
  • FIGs. 3, 4A-4B, 5A-5B and 6A-6B show flowcharts with various examples of exploit detection algorithms. It should be appreciated that upon detection of an exploit by at least one such algorithm an alert may be issued by the processor 201 and/or by observer CPU 203, e.g. to the user, so as to prevent harm to the system.
  • Fig. 3 shows a flowchart for a stack injection exploit detection algorithm, according to some embodiments of the invention.
  • Stack injection exploit detection algorithm may include observer CPU 203 to maintain 301 stack addresses in memory (e.g., in database 205), and receive instructions 302. It should be appreciated that as used herein stack memory may also refer to dynamic (or heap) memory.
  • Observer CPU 203 may receive an instruction pointer (IP) from PMU 204, check 303 that the IP includes a value memory that is a stack, such that observer CPU 203 may issue 304 an alert. In case that value memory is not a stack 303, observer CPU 203 may continue iteration until a stack is detected.
  • IP instruction pointer
  • Figs. 4A-4B show a flowchart for a structured exception handling (SEH) exploit detection algorithm, according to some embodiments of the invention.
  • SEH exploit detection algorithm may include observer CPU 203 to map 401 the SEH handler of the operating system, e.g. of computerized device 210, and receive instructions 402.
  • Observer CPU 203 may check 403 that if a new SEH handler is registered, the corresponding handler address may be added to a shadow SEH linked list 404. It should be noted that the shadow SEH keeps track of the SEH handler registration and removal in order to detect an exploitation attempt.
  • observer CPU 203 may check 407 if exception handler of the operating system has been invoked. In case that exception handler of the operating system has not been executed 407, observer CPU 203 may return to check 402 the next instruction to be processed. It should be noted that due to the (circular) flow structure of the algorithm, for each executed instruction the algorithm may perform checks 403 and/or 405 and/or 407 repeatedly. In some embodiments, only when check 407 is negative then the next instruction may be processed.
  • observer CPU 203 may continue to check 408 if invoked handler address is in the shadow SEH linked list. In case that invoked handler address is in the shadow SEH linked list 408, observer CPU 203 may return to check 402 the next instruction to be processed. In case that invoked handler address is not in the shadow SEH linked list 408, observer CPU 203 may issue 409 an alert.
  • FIGs. 5A-5B show a flowchart for a use of memory allocation tracking exploit detection algorithm, according to some embodiments of the invention.
  • Memory allocation tracking exploit detection algorithm may include observer CPU 203 to 501 the memory allocation and memory deallocation functions, receive instructions 502, and may also check 503 that an allocation function is called.
  • observer CPU 203 may add 1 to the allocation counter 504. In case that an allocation function has not been called 503, observer CPU 203 may check 505 if a deallocation has been called. In case that a deallocation function has been called 505, observer CPU 203 may add 1 to the deallocation counter 506.
  • observer CPU 203 may check 507 if the IP contains a value that is mapped as heap memory. In case that IP does not contain a value that is mapped as heap memory 507, observer CPU 203 may return to also check 502 the next instruction to be processed. It should be noted that due to the (circular) flow structure of the algorithm, for each executed instruction the algorithm may perform checks 503 and/or 505 and/or 507 repeatedly. In some embodiments, only when check 507 is negative then the next instruction may be processed.
  • observer CPU 203 may check 508 if difference between allocation and deallocation counters is greater than a predefined value (e.g., greater than ten). In case that the difference is not greater than the predefined value 508, observer CPU 203 may return to also check 502 the next instruction to be processed. In case that the difference is greater than the predefined value 508, observer CPU 203 may issue 509 an alert.
  • a predefined value e.g., greater than ten
  • Figs. 6A-6B show a flowchart for a first return oriented programming (ROP) exploit detection algorithm, according to some embodiments of the invention.
  • the first return oriented exploit detection algorithm may include observer CPU 203 to maintain a shadow stack 601 , and receive instructions 602. Observer CPU 203 may then check 603 if a 'call' (or return) instruction in the execution flow has been executed for verification.
  • observer CPU 203 may push 604 expected return address to the shadow stack. In case that a call instruction has not been executed 603, observer CPU 203 may check 605 if a return (RET) instruction has been executed. In case that a return instruction has not been executed 605, observer CPU 203 may return to check 602 the next instruction to be processed.
  • RET return
  • observer CPU 203 may check 606 if target address is different from top address on shadow stack. In case that target address is different from top address on shadow stack 606, observer CPU 203 may issue 607 an alert. It should be noted that due to the (circular) flow structure of the algorithm, for each executed instruction the algorithm may perform checks 603 and/or 605 and/or 606 repeatedly. In some embodiments, only when check 606 is negative then the next instruction may be processed.
  • target address is not different from top address on shadow stack 606
  • observer CPU 203 may pop 608 an address (e.g., most recent address which correctly matched the anticipated address) from the stack, and then return to check 602 the next instruction to be processed.
  • Figs. 6C-6D show a flowchart for a second return oriented programming (ROP) exploit detection algorithm, according to some embodiments of the invention.
  • the second return oriented exploit detection algorithm may include observer CPU 203 to maintain a shadow stack 621 , and receive instructions 622. Observer CPU 203 may then check 623 if a 'call' (or return) instruction in the execution flow has been executed for verification. [068] In case that a call instruction has been executed 623, observer CPU 203 may add 624 the expected return address to the shadow set and update the counter (e.g., initialize counter to T) and/or add T to the existing counter.
  • the counter e.g., initialize counter to T
  • CPU 203 may check if the expected return address already exists in the shadow set 625. In case that the address already exists in the shadow set 625, observer CPU 203 may update the counter (e.g., add T to the existing counter) 626a. In case that the address is not in the shadow set 625, observer CPU 203 may update the counter (e.g., initialize the counter to ⁇ ') 626b.
  • the counter e.g., add T to the existing counter
  • observer CPU 203 may check 627 if a return (RET) instruction has been executed. In case that a return instruction has not been executed 625, observer CPU 203 may return to check 623 the next instruction to be processed.
  • RET return
  • observer CPU 203 may check 628 if target address is present in the shadow set. In case that target address is not present in the shadow set 628, observer CPU 203 may issue 629 an alert. It should be noted that due to the (circular) flow structure of the algorithm, for each executed instruction the algorithm may perform checks 623 and/or 627 and/or 628 repeatedly. In some embodiments, only when check 628 is positive then the next instruction may be processed.
  • observer CPU 203 may check 630 if the counter is ⁇ '. In case that the counter is '0' 629, observer CPU 203 may issue 629 an alert. In case that the counter is not '0' 629, observer CPU 203 may update the counter 631 (e.g., reduce the counter by T), and then return to check 622 the next instruction to be processed.
  • Indirect program control instructions may calculate the target address based on an argument or variable that resides in a register or memory location. For every indirect branch in the reconstructed execution flow, the target address may be validated against a preprocessed database.
  • the preprocessed database may include legitimate target addresses that the processor is allowed to transfer control to and/or legitimate call-site source addresses to "critical functions" that the processor is allowed to transfer control from.
  • a critical function may be any function (e.g., of the operating system) that an attacker has interest to invoke in order to successfully carry out the attack. For instance, functions that control memory permissions are "critical" because the attacker may work more freely when the permissions of memory are under his control. In some embodiments, an attacker may enable the execution flag via such critical function.
  • a target address (e.g., collected by parsing memory images of executable code) may be classified as safe if it meets at least one of the following: the address may be in a relocation table of a module, so that the pointer may be fixed at load time and therefore points to an address that is legal to transfer control to, and/or the address may be the entry point of a module, and/or the address may be in the import table/dynamic symbol table of a module so that control may be transferred to these addresses from foreign modules during run time, and/or the address may be in the export table/symbol table of a module so that control may be transferred to these addresses from the current module, and/or the address may be a relative call so that a module may transfer the control flow within itself, and/or the address may be preceded by a 'CALL' instruction in the modules.
  • Illegal indirect control transfer detection algorithm may include maintaining 701 a database of legal addresses to transfer control indirectly to.
  • illegal indirect control transfer detection algorithm may further include receiving 702 instructions for next indirect branch, for instance receiving instructions from the reconstructed execution flow. If the address is not in the database 703, an alarm may be issued 704. If the address is in the database 703, then the illegal indirect control transfer detection algorithm may wait until instructions for next indirect branch may be received. In some embodiments, an alarm may be issued if the source of the address of the indirect branch instruction is in the database of legal addresses to transfer control indirectly to.
  • the method of detecting an exploit of a vulnerability of a computing device may include receiving 801 an execution flow of at least one process running in a processor 201 of the computing device 210, wherein the execution flow is received from a performance monitoring unit (PMU) 204 of the processor 201.
  • PMU performance monitoring unit
  • the method may further include receiving 802 memory pages from a memory of the computing device 210.
  • the method may further include reconstructing 803 the process on another processor based on the execution flow and the memory pages.
  • the method may further include running 804 at least one exploit detection algorithm on the reconstructed process in order to identify an exploit attempt, and issuing 805 and alert for instance upon detection of an exploit.
  • the alert (e.g., to the user) may be issued by processor 201 and/or by observer CPU 203.
  • the method may further include interrupting the process running on the processor of the computing device when an exploit is detected.
  • a decoder may include a number of abstract layers such as packet layers, event layers, instruction flow layers and block layers. While the trace data that may be generated by the CPU includes packets with logical binary data units that represent different types of events occurring throughout the lifecycle of execution, the decoding process may add information to the context at each layer until the final reconstruction is achieved. Therefore, matching the detection algorithm to the correct layer may allow maximizing performance and visibility of the system. In some embodiments, the performance order of the layers may be packet, event, instruction flow and block layer where each layer may have different visibility.
  • the algorithms described in Figs. 3 and 7 may correspond to decoder with an event layer.
  • the algorithms described in Figs. 4A- 4B and 6A-6D may correspond to decoder with a block layer.
  • shadow set algorithms may detect abused return instructions that transfer control to attacker chosen addresses. In order to be able to have the visibility it needs, shadow set may be matched with the block layer. This may be due to the fact that only from the instruction flow layer and above (e.g., the block layer) the context may include the type of the instruction executed by the CPU.
  • Figs. 9A-9B show a flowchart for combined algorithm operation, according to some embodiments of the invention.
  • at least two different algorithms may be operated simultaneously and/or in combination (e.g., in hybrid mode) such that each algorithm may be activated and/or switched when needed. For example, detecting an indirect branch target address of a 'critical function' and switching to scan the indirect branch source address of the corresponding function for suspect indirect calls.
  • the combined algorithm may include maintaining 901 a database of legal addresses to indirectly transfer control to and/or maintaining 901 a database of critical functions.
  • the combined algorithm may further include receiving 902 instructions for next indirect branch, for instance receiving instructions from the reconstructed execution flow. If the indirect branch target address is not in the database 903 (e.g., database of legal addresses to indirectly transfer control to), the combined algorithm may issue an alarm 904. If the indirect branch target address is in the database 903 (e.g., database of legal addresses to indirectly transfer control to), the combined algorithm may check 905 if the indirect target address is a critical function address.
  • the combined algorithm may return to receiving 902 instructions for the next indirect branch.
  • the combined algorithm may rescan 906 the indirect branch to obtain the source address of the indirect branch.
  • the combined algorithm may check 907 if the source address corresponds to the critical function address (e.g., in the critical function address database). In case that the source address does not correspond to the critical function address 907, an alarm may be issued 908. In case that the source address corresponds to the critical function address 907, the combined algorithm may return to receiving 902 instructions for the next indirect branch.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Virology (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Debugging And Monitoring (AREA)
  • Spinning Or Twisting Of Yarns (AREA)
  • Tents Or Canopies (AREA)
  • Telephone Function (AREA)
EP17867010.5A 2016-11-07 2017-11-05 System und verfahren zur detektion von und zur warnung vor exploits in computerisierten systemen Active EP3535681B1 (de)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201662418294P 2016-11-07 2016-11-07
US201762516126P 2017-06-07 2017-06-07
PCT/IL2017/051206 WO2018083702A1 (en) 2016-11-07 2017-11-05 System and method for detecting and for alerting of exploits in computerized systems

Publications (3)

Publication Number Publication Date
EP3535681A1 true EP3535681A1 (de) 2019-09-11
EP3535681A4 EP3535681A4 (de) 2020-05-27
EP3535681B1 EP3535681B1 (de) 2023-07-26

Family

ID=62076446

Family Applications (1)

Application Number Title Priority Date Filing Date
EP17867010.5A Active EP3535681B1 (de) 2016-11-07 2017-11-05 System und verfahren zur detektion von und zur warnung vor exploits in computerisierten systemen

Country Status (5)

Country Link
US (1) US11899797B2 (de)
EP (1) EP3535681B1 (de)
ES (1) ES2960375T3 (de)
IL (1) IL266459B2 (de)
WO (1) WO2018083702A1 (de)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10885183B2 (en) * 2017-09-28 2021-01-05 International Business Machines Corporation Return oriented programming attack protection
US10984096B2 (en) * 2018-03-28 2021-04-20 Intel Corporation Systems, methods, and apparatus for detecting control flow attacks

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7603704B2 (en) 2002-12-19 2009-10-13 Massachusetts Institute Of Technology Secure execution of a computer program using a code cache
US20090328185A1 (en) 2004-11-04 2009-12-31 Eric Van Den Berg Detecting exploit code in network flows
US8707433B1 (en) * 2011-05-03 2014-04-22 Symantec Corporation Fake exception handler detection
US9256730B2 (en) * 2012-09-07 2016-02-09 Crowdstrike, Inc. Threat detection for return oriented programming
US9292686B2 (en) * 2014-01-16 2016-03-22 Fireeye, Inc. Micro-virtualization architecture for threat-aware microvisor deployment in a node of a network environment
US10284591B2 (en) * 2014-01-27 2019-05-07 Webroot Inc. Detecting and preventing execution of software exploits
US9438623B1 (en) 2014-06-06 2016-09-06 Fireeye, Inc. Computer exploit detection using heap spray pattern matching
US9961102B2 (en) * 2014-07-16 2018-05-01 Mcafee, Llc Detection of stack pivoting
US9858411B2 (en) * 2014-12-19 2018-01-02 Intel Corporation Execution profiling mechanism
US10007784B2 (en) * 2015-03-27 2018-06-26 Intel Corporation Technologies for control flow exploit mitigation using processor trace
US10803165B2 (en) * 2015-06-27 2020-10-13 Mcafee, Llc Detection of shellcode
US20170091454A1 (en) * 2015-09-25 2017-03-30 Vadim Sukhomlinov Lbr-based rop/jop exploit detection
US9965620B2 (en) * 2015-12-24 2018-05-08 Intel Corporation Application program interface (API) monitoring bypass

Also Published As

Publication number Publication date
ES2960375T3 (es) 2024-03-04
IL266459B1 (en) 2023-06-01
EP3535681A4 (de) 2020-05-27
WO2018083702A1 (en) 2018-05-11
US20190258806A1 (en) 2019-08-22
IL266459B2 (en) 2023-10-01
EP3535681B1 (de) 2023-07-26
IL266459A (en) 2019-06-30
US11899797B2 (en) 2024-02-13

Similar Documents

Publication Publication Date Title
US11063974B2 (en) Application phenotyping
JP6142027B2 (ja) ハイパーバイザ環境においてカーネルルートキットに対する保護を実行するシステムおよび方法
KR101880375B1 (ko) 네트워크 활동을 보이는 실행파일들의 분리
US9197662B2 (en) Systems and methods for optimizing scans of pre-installed applications
AU2006235058B2 (en) System and method for foreign code detection
RU2724790C1 (ru) Система и способ формирования журнала при исполнении файла с уязвимостями в виртуальной машине
US9064120B2 (en) Systems and methods for directing application updates
AU2021319159B2 (en) Advanced ransomware detection
US20080028180A1 (en) Inappropriate access detector based on system segmentation faults
US11775649B2 (en) Perform verification check in response to change in page table base register
US20240028724A1 (en) Control flow integrity monitoring for applications running on platforms
US9483643B1 (en) Systems and methods for creating behavioral signatures used to detect malware
US9785492B1 (en) Technique for hypervisor-based firmware acquisition and analysis
US11899797B2 (en) System and method for detecting and for alerting of exploits in computerized systems
US10846405B1 (en) Systems and methods for detecting and protecting against malicious software
CN111125793B (zh) 一种访问控制中客体内存可信验证方法及系统
US20230401083A1 (en) Information processing apparatus and information processing method
Wang et al. IRePf: An Instruction Reorganization Virtual Platform for Kernel Stack Overflow Detection
CN113127148A (zh) 一种虚拟化环境主动动态度量方法和系统

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20190607

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
RIN1 Information on inventor provided before grant (corrected)

Inventor name: AMINOV, MICHAEL

Inventor name: LEVIN, SHLOMI

REG Reference to a national code

Ref document number: 602017071931

Country of ref document: DE

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G06F0021440000

Ipc: G06F0021520000

A4 Supplementary search report drawn up and despatched

Effective date: 20200423

RIC1 Information provided on ipc code assigned before grant

Ipc: G06F 21/55 20130101ALI20200417BHEP

Ipc: G06F 21/57 20130101ALI20200417BHEP

Ipc: G06F 21/56 20130101ALI20200417BHEP

Ipc: G06F 21/52 20130101AFI20200417BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20211027

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20230223

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602017071931

Country of ref document: DE

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20230726

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20231110

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231027

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231126

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231127

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231026

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231126

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231027

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20230912

Year of fee payment: 7

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2960375

Country of ref document: ES

Kind code of ref document: T3

Effective date: 20240304

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602017071931

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

26N No opposition filed

Effective date: 20240429

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231105

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231130

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20231105

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231105

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231130

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20231130

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231105

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231105

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231130