JP2022541250A - Inline malware detection - Google Patents

Abstract

Malicious file detection is disclosed. A set containing one or more sample classification models is stored on the network device. N-gram analysis is performed on a sequence of received packets associated with a received file Performing n-gram analysis includes using at least one stored sample classification model. A determination is made that the received file is malicious based at least in part on an n-gram analysis of the sequence of received packets. Propagation of the received file is prevented in response to determining that the file is malicious.

Description

Malware is a general term referring to malicious software (eg, including various hostile, intrusive, and/or unwanted software). Malware can be in the form of code, scripts, active content, and/or other software. Examples of malware uses include disrupting computer and/or network operations, stealing proprietary information (e.g., confidential information, such as identity, financial, and/or intellectual property related information), and/or including gaining access to private/proprietary computer systems and/or computer networks. Unfortunately, as techniques are developed to help detect and mitigate malware, malicious writers find ways to circumvent such efforts. Accordingly, there is a continuing need for improved techniques for identifying and mitigating malware.

Various embodiments of the present invention are disclosed in the following detailed description and accompanying drawings.
FIG. 1 illustrates one embodiment of an environment in which malicious applications are detected and prevented from causing harm. FIG. 2A shows one embodiment of a data appliance. Figure 2B is a functional diagram of the logical components of one embodiment of a data appliance. FIG. 3 shows one example of logical components that can be included in a system for analyzing samples. FIG. 4 shows portions of one exemplary embodiment of a threat engine. FIG. 5 shows one example of a portion of the tree. FIG. 6 shows one embodiment of a process for performing inline malware detection on data appliances. FIG. 7A shows one exemplary hash table for files. FIG. 7B shows one exemplary threat signature for the sample. FIG. 8A shows one embodiment of a process for performing feature extraction. FIG. 8B shows one embodiment of the process for generating the model.

The present invention can be implemented in numerous ways, including processes, devices, systems, compositions of matter, computer program products embodied on computer readable storage media, and/or processors. A processor configured to execute instructions stored in and/or provided by a memory coupled to the processor. In this specification, these embodiments, or other forms that the invention may take, are referred to as techniques. In general, the order of steps of disclosed processes may be altered within the scope of the invention. Unless otherwise indicated, a component such as a processor or memory described as being configured to perform a task is a generic component that is temporarily configured to perform a task at any given time; Alternatively, it can be implemented as a specific component manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions. .

A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. Although the invention is described in connection with such embodiments, the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention includes numerous alternatives, modifications and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. These details are provided for the purpose of example, and the invention may be practiced according to the claims without some or all of these specific details. For the sake of clarity, technical material that is well known in the technical fields related to the invention has not been described in detail so as not to unnecessarily obscure the invention.

I.Overview

Firewalls generally allow authorized communications to pass through the firewall while protecting the network from unauthorized access. A firewall is typically a device, set of devices, or software running on a device that provides firewall functionality for network access. For example, a firewall can be integrated into the operating system of a device (eg, computer, smart phone, or other type of network communication enabled device). Firewalls can also be devices of various types, such as computer servers, gateways, network/routing devices (e.g., network routers), or data appliances (e.g., security appliances, or other types of special purpose devices). or integrated or executed as a software application on the security device, and in some implementations specific operations may be implemented in special purpose hardware, such as ASICs or FPGAs.
be.

A firewall typically denies or permits network transmissions based on a set of rules. These rule sets are often referred to as policies (eg, network policies or network security policies). For example, a firewall can filter inbound traffic by applying a set of rules or policies to prevent unwanted external traffic from reaching a protected device. Firewalls can also filter outbound traffic by applying a set of rules or policies (e.g., allow, block, monitor, notify, or log ), and/or other actions that may be specified in firewall rules or policies, which may be triggered based on various criteria, as described herein). A firewall can also filter local network (eg, intranet) traffic by applying a set of rules or policies as well.

Security devices (eg, security appliances, security gateways, security services, and/or other security devices) may perform various security operations (eg, firewall, anti-malware, intrusion prevention/detection, proxy, and/or other security functions), network functions (e.g., routing, quality of service (QoS), workload balancing of network-related resources, and/or other network functions), and/or other security and/or network functions It can perform related functions. For example, routing can be performed based on source information (eg, IP address and port), destination information (eg, IP address and port), and protocol information.

A basic packet-filtering firewall filters network communication traffic by inspecting each individual packet sent over the network (e.g., a stateless packet-filtering firewall, a packet-filtering firewall or 1st generation firewall). Stateless packet-filtering firewalls typically inspect individual packets themselves, and based on the inspected packet (e.g., a combination of the packet's source and destination address information, protocol information, and port number). ) to apply the rule.

An application firewall can also perform application layer filtering (eg, using an application layer filtering firewall or a second generation firewall that works at the application level of the TCP/IP stack). Application-layer filtering firewalls, or application firewalls, are generally used to protect certain applications and protocols (e.g., web browsing using Hypertext Transfer Protocol (HTTP), Domain Name System (DNS) requests, File Transfer Protocol (FTP) and various other types of applications and other protocols such as Telnet, DHCP, TCP, UDP, and TFTP (GSS)). For example, application firewalls can block unauthorized protocols that attempt to communicate on standard ports (e.g., sneak through by using a non-standard port for that protocol). Unauthorized/outside policy protocols attempting to through) can generally be identified using an application firewall).

Stateful firewalls can also perform stateful-based packet inspection, where each packet is viewed within the context of a set of packets associated with that network transmission's packets/packet flow. be inspected. This firewall technology is commonly referred to as stateful packet inspection. Keeps a record of all connections that pass through the firewall and can determine whether a packet is the start of a new connection, part of an existing connection, or an invalid packet It is from. For example, the state of a connection can itself be one of the criteria that trigger a rule in a policy.

Advanced or next-generation firewalls can perform stateless and stateful packet filtering and application layer filtering, as described above. Next-generation firewalls can also implement additional firewall technologies. For example, certain newer firewalls, often referred to as advanced or next-generation firewalls, can also identify users and content. In particular, certain next-generation firewalls extend the list of applications that these firewalls can automatically identify to thousands of applications. Examples of such next-generation firewalls are commercially available from Palo Alto Networks (eg, Palo Alto Networks' PA series firewalls). For example, Palo Alto Networks next-generation firewalls use a variety of identification technologies to help enterprises and service providers identify applications, users, and content—not just ports, IP addresses, and packets—and , allows you to control Various identification technologies include App-ID (e.g., App-ID) for precise application identification, User-ID (e.g., User-ID) for user identification, and real-time Content-ID for content scanning (eg, Content-ID) (eg, controlling web surfing and restricting data and file transfers). These identification techniques allow enterprises to use business-related concepts to securely enable application usage instead of following the traditional approach provided by traditional port-blocking firewalls. Also, special-purpose hardware for next-generation firewalls (e.g., implemented as dedicated equipment) generally provides higher performance levels for application inspection than software running on general-purpose hardware (e.g., , security appliances from Palo Alto Networks, Inc., which utilize dedicated, function-specific processing that is tightly integrated with a single-pass software engine to reduce latency for Palo Alto Networks' PA Series Next-Generation Firewalls. maximizing network throughput while minimizing (latency)).

Advanced or next generation firewalls can also be implemented using virtualized firewalls. Examples of such next-generation firewalls are commercially available from Palo Alto Networks (Palo Alto Networks' firewalls include VMware(R) ESXi ^TM and NSX ^TM , Citrix(R) Netscaler SDX ^TM , KVM/OpenStack (Centos/RHEL , Ubuntu(R)), and Amazon Web Services (AWS)). For example, virtualized firewalls can support similar or even identical next-generation firewalls and advanced threat protection capabilities available in physical form factor devices, enabling enterprises to deploy private, public, and securely enable the influx of applications to hybrid cloud computing environments. Automation features such as VM monitoring, dynamic address groups, and REST-based APIs allow enterprises to dynamically monitor VM changes and reflect that context in security policies, thereby enabling Eliminates possible policy lag.

II. Environmental Examples

Figure 1 shows an example of an environment in which a malicious application (“malware”) is detected and does not cause damage. As described in further detail below, malware taxonomies (eg, created by security platform 122) can be shared and/or refined in various ways among various entities included in the environment shown in FIG. , the techniques described herein can be used to protect devices, such as endpoint client devices 104-110, from such malware.

The term "application" is used throughout this specification to refer generically to programs, bundles of programs, manifests, packages, etc., regardless of format/platform. An "application" (also referred to herein as a "sample") may be a standalone file (eg, a calculator application with the file name "calculator.apk" or "calculator.exe"), or another application (e.g., a mobile advertising SDK or a library embedded within a computational application).

"Malware," as used herein, means any behavior, whether covert or not (and illegal or not), that the user would not/would not approve of if fully informed. Involved. Examples of malware include Trojan horses, viruses, rootkits, spyware, hacking tools, keyloggers, and the like. One example of malware is a desktop application that collects and reports the end-user's location to a remote server (but does not provide users with location-based services, such as mapping services). ). Another example of malware is a malicious Android application package .apk (APK) that appears to be a free game to the end user but secretly sends SMS premium messages (e.g. , cost $10 each) and inflate the end user's phone bill. Another example of malware is Apple's iOS Flashlight application, which secretly collects users' contacts and sends them to spammers. Other forms of malware can also be detected/blocked using the techniques described herein (eg, ransomware). Furthermore, although n-grams/feature vectors/output accumulation variables are described herein as being generated for malicious applications, the techniques described herein are also applicable to other types of applications. Various embodiments can also be used to generate profiles for (eg, adware profiles, goodware profiles, etc.).

The techniques described herein can be used on different platforms (eg, desktops, mobile devices, gaming platforms, embedded systems, etc.) and/or different types of applications (eg, Android apk files, iOS applications, Windows PE files, Adobe Acrobat PDF files, etc.). In the exemplary environment shown in FIG. 1, client devices 104 - 108 are laptop computers, desktop computers, and tablets residing on enterprise network 140 . Client device 110 is a laptop computer that resides outside enterprise network 140 .

Data appliance 102 is configured to enforce policies regarding communications between client devices, such as client devices 104 and 106, and nodes outside enterprise network 140 (eg, reachable via external network 118). . Examples of such policies include policies governing traffic shaping, quality of service, and routing of traffic. Other examples of policies include scanning for threats in incoming (and outgoing) email attachments, website content, files exchanged via instant messaging programs, and/or other file transfers. including security policies, such as requiring In some embodiments, data appliance 102 is also configured to enforce policies regarding traffic remaining within enterprise network 140 .

One embodiment of a data appliance is shown in FIG. 2A. The examples shown are representations of physical components included in data appliance 102, in various embodiments. Specifically, data appliance 102 includes a high performance multi-core central processing unit (CPU) 202 and random access memory (RAM) 204 . Data appliance 102 also includes storage 210 (such as one or more hard disks or solid-state storage units). In various embodiments, data appliance 102 stores information (RAM 204, storage 210, and/or other suitable locations) used to monitor enterprise network 140 and implement the disclosed techniques. , or ). Examples of such information include application identifiers, content identifiers, user identifiers, requested URLs, IP address mappings, policy and other configuration information, signatures, hostname/URL categorization information, malware profiles, and machine learning models. include. Data appliance 102 may also include one or more optional hardware accelerators. For example, the data appliance 102 may execute a cryptographic engine 206 configured to perform encryption and decryption operations, a matcher, act as a network processor, and/or perform other tasks. It may include one or more field programmable gate arrays 208 configured to.

The functionality described herein as being performed by data appliance 102 may be provided/implemented in a variety of ways. For example, data appliance 102 may be a dedicated device or set of devices. The functionality provided by data appliance 102 may be integrated or executed as software on general purpose computers, computer servers, gateways, and/or network/routing devices. In some embodiments, at least some of the services described as being provided by the data appliance 102 are alternatively (or in addition) provided by software executing on the client device (e.g., provided to client device 104 or client device 110).

Whenever data appliance 102 is described as performing a task, a single component, a subset of components, or all components of data appliance 102 can cooperate to perform the task. Similarly, whenever a component of data appliance 102 is described as performing a task, a subcomponent can perform the task and/or the component can perform the task in conjunction with other components. can be done. In various embodiments, portions of data appliance 102 are provided by one or more third parties. Depending on factors such as the amount of computing resources available to the data appliance 102, various logical components and/or features of the data appliance 102 may be omitted and the techniques described herein adapted accordingly. . Similarly, additional logical components/features may be included in embodiments of data appliance 102 as applicable. One example of a component included in data appliance 102 in various embodiments is configured to identify applications (eg, using various application signatures to identify applications based on packet flow analysis). is an application identification engine that For example, the application identification engine can determine the type of traffic the session will involve. Web Browsing - Social Networking, Web Browsing - News, SSH, etc.

Figure 2B is a functional diagram of the logical components of one embodiment of a data appliance. The example shown is a representation of logical components that may be included in data appliance 102 in various embodiments. Unless otherwise specified, the various logical components of the data appliance 102 generally run a set of one or more scripts (eg, written in Java, python, etc., as applicable). can be implemented in a variety of ways, including

As shown, data appliance 102 includes a firewall and includes management plane 232 and data plane 234 . The management plane is responsible for managing user interaction, such as by setting policies and providing a user interface for viewing log data. The data plane is responsible for data management, such as by performing packet processing and session processing.

Network processor 236 is configured to receive packets from client devices, such as client device 108, and provide them to data plane 234 for processing. Flow module 238 creates a new session flow whenever it identifies a packet as part of a new session. Subsequent packets are identified as belonging to the session based on the flow lookup. If applicable, SSL decryption is applied by SSL decryption engine 240 . Otherwise, processing by the SSL decryption engine 240 is skipped. Decryption engine 240 helps data appliance 102 inspect and control SSL/TLS and SSH encrypted traffic, thus stopping threats that may otherwise remain hidden within encrypted traffic. help Decryption engine 240 can also help prevent sensitive content from leaving enterprise network 140 . Decryption can be selectively controlled (eg, enabled or disabled) based on parameters such as URL category, traffic source, traffic destination, user, user group, and port. In addition to decryption policies (eg, those that specify sessions to decrypt), decryption profiles can be assigned to control various options of sessions controlled by the policy. For example, the use of specific cipher suites and encryption protocol versions may be required.

Application identification (APP-ID) engine 242 is configured to determine the type of traffic the session involves. As one example, application identification engine 242 can recognize a GET request in the received data and conclude that the session requires an HTTP decoder. In some cases, for example, during a web browsing session, the identified application can be changed, and such changes are noted by the data appliance 102 . For example, a user may first browse a company's wiki (classified as "Web Browsing-Productivity" based on the URL visited), then a social networking site (visited Based on the URL, classified as "Web Browsing-Social Networking") can be browsed. Different types of protocols have corresponding decoders.

Based on decisions made by application identification engine 242, threat engine 244 sends packets to appropriate decoders configured to assemble packets into the correct order, perform tokenization, and extract information. sent. Threat engine 244 also performs signature matching to determine what should happen to the packet. If necessary, SSL encryption engine 246 can re-encrypt the decrypted data. Packets are forwarded using a forwarding module 248 for forwarding (eg, to a destination).

Also shown in FIG. 2B, policy 252 is received and stored in management plane 232 . A policy can contain one or more rules, which can be specified using domain names and/or host/server names, and the rules can be extracted from various monitored session traffic flows. Based on the parameters/information obtained, one or more signatures or other verification criteria or heuristics can be applied, such as for security policy enforcement on subscribers/IP flows. An interface (I/F) communicator 250 is provided for management communication (eg, via (REST) API, messages, or network protocol communication, or other communication mechanism).

III. Security Platform

Returning to FIG. 1, assume that a malicious individual (using system 120) created malware 130. A malicious individual wants a client device, such as client device 104, to run a copy of malware 130, compromising the client device and, for example, turning the client device into a bot in a botnet. ). The compromised client device then performs a task (e.g., mining cryptocurrency or participating in a denial of service attack) and provides information to an external entity, such as a command and control (C&C) server 150. It can be instructed to report and, if necessary, to receive commands from C&C server 150 .

Assume that data appliance 102 intercepts an email sent to user “Alice” operating client device 104 . A copy of malware 130 is attached to the message by system 120 . Alternatively, as a similar scenario, data appliance 102 may intercept an attempt by client device 104 to download malware 130 (eg, from a website). In either scenario, data appliance 102 determines whether a file signature (eg, email attachment or website download of malware 130 ) exists on data appliance 102 . A signature, if present, can indicate that the file is known to be safe (e.g., is on a whitelist) and also that the file is known to be malicious. You can also indicate that you are on a blacklist (for example).

In various embodiments, data appliance 102 is configured to operate in cooperation with security platform 122 . As one example, security platform 122 may provide data appliance 102 with a set of known malicious file signatures (eg, as part of a subscription). If a signature for malware 130 is included in the set (eg, an MD5 hash of malware 130), data appliance 102 responds accordingly (eg, an MD5 hash of an email attachment sent to client device 104 to an MD5 hash of malware 130). By detecting a match), transmission of malware 130 to client device 104 can be prevented. The security platform 122 can also provide the data appliance 102 with a list of known malicious domains and/or IP addresses so that the data appliance 102 can identify the enterprise network 140 and the C&C server 150 (e.g., if the C&C server 150 is malicious). allows you to block traffic to and from A list of malicious domains (and/or IP addresses) also helps the data appliance 102 determine when one of its nodes has been compromised. For example, if the client device 104 attempts to contact the C&C server 150, such attempts indicate that the client 104 has been compromised by malware (and thus prevent the client device 104 from communicating with other nodes within the enterprise network 140). is a strong indicator that corrective action, such as isolating the As described in more detail below, security platform 122 also provides other types of information to data appliance 102, such as a set of machine learning models that can be used by data appliance 102 to perform inline analysis of files. (e.g., as part of a reservation).

In various embodiments, data appliance 102 may take various actions if a signature for an attachment is not found. As a first example, the data appliance 102 fails by blocking transmission of attachments that are not whitelisted as benign (e.g., do not match signatures of known good files). safe). A drawback of this approach is that there can be many legitimate attachments that are unnecessarily blocked as potentially malware, even when they are in fact benign. As a second example, the data appliance 102 can prevent failure by allowing the transmission of attachments that are not blacklisted as malicious (e.g., those that do not match signatures of known malicious files). can pose a fail-danger. A drawback of this approach is that newly created malware (previously invisible to the platform 122) is not prevented from causing harm.

As a third example, data appliance 102 provides a file (eg, malware 130) to security platform 122 for static/dynamic analysis to determine whether it is malicious, and/or It can be configured to sort. Various actions can be taken by the data appliance 102 while analysis is performed by the attached security platform 122 (where the signature does not yet exist). As a first example, data appliance 102 can prevent the email (and attachments) from being delivered to Alice until a response is received from security platform 122 . Assuming that platform 122 takes approximately 15 minutes to fully analyze the sample, this means that the received message to Alice will be delayed by 15 minutes. In this example, the attachment is malicious, so such a delay does not negatively impact Alice. In another example, suppose someone sent Alice a time sensitive message with a benign attachment for which there is no signature. Delaying the delivery of the message to Alice by 15 minutes is likely to be considered unacceptable (eg by Alice). As will be described in more detail below, an alternative approach is to perform at least some real-time analysis of attachments at data appliance 102 (eg, while awaiting adjudication from platform 122). If the data appliance 102 can independently determine whether an attachment is malicious or benign, it can take initial action (e.g., block or allow delivery to Alice); Then, after receiving the verdict from the security platform 122, additional actions can be adjusted/performed as needed.

Security platform 122 stores a copy of the received sample in storage 142, and analysis is initiated (or scheduled, as appropriate). One example of storage 142 is an Apache Hadoop cluster. The results of the analysis (and additional information related to the application) are stored in database 146 . If the application is determined to be fraudulent, the data appliance can be set to automatically block file downloads based on the analysis results. Additionally, to automatically block future file transfer requests that download files determined to be malicious, signatures are generated for the malware, and (e.g., data devices such as data devices 102, 136, 148 against) can be distributed.

In various embodiments, security platform 122 includes one or more dedicated off-the-shelf hardware servers (eg, multi-core processors) running typical server-class operating systems (eg, Linux®). , 32G+ of RAM, a Gigabit network interface adapter, and a hard drive). Security platform 122 may be implemented across a scalable infrastructure that includes multiple such servers, solid state drives, and/or other applicable high performance hardware. Security platform 122 may have multiple distributed components, including components provided by one or more third parties. For example, part or all of security platform 122 may be implemented using Amazon Elastic Compute Cloud (EC2) and/or Amazon Simple Storage Service (S3). Further, as with the data appliance 102, whenever the security platform 122 is referred to as performing a task, such as storing data or processing data, the subcomponent or subcomponents of the security platform 122 are , can cooperate (individually or in cooperation with third party components) to perform its tasks. As one example, security platform 122 can optionally cooperate with one or more virtual machine (VM) servers, such as VM server 124, to perform static/dynamic analysis.

One example of a virtual machine server is off-the-shelf server-class hardware (e.g., multi-core processor, 32G+ of RAM, and A physical machine that contains one or more gigabit network interface adapters). In some embodiments, virtual machine servers are omitted. Further, virtual machine servers may be under the control of the same entity that manages security platform 122, but may also be provided by a third party. As one example, a virtual machine server can rely on EC2, with the rest of the security platform 122 provided by dedicated hardware owned and under the control of the operator of the security platform 122. . VM server 124 is configured to provide one or more virtual machines 126-128 to emulate client devices. Virtual machines can run various operating systems and/or versions thereof. Observed behavior resulting from running the application in the virtual machine is logged and analyzed (eg, if it indicates that the application is malicious). In some embodiments, log analysis is performed by a VM server (eg, VM server 124). In other embodiments, analysis is performed, at least in part, by other components of security platform 122 , such as coordinator 144 .

In various embodiments, security platform 122 makes the results of sample analysis available to data appliance 102 via a list of signatures (and/or other identifiers) as part of a subscription. to enable. For example, security platform 122 may periodically transmit content packages that identify malware applications (eg, daily, hourly, or other intervals and/or on events configured by one or more policies). based on). An example content package includes a list of identified malware applications, with a package name, a hash value to uniquely identify the application, and a malware name (and/or malware family name) for each identified malware application. accompanied by information such as A subscription may only cover analysis of files intercepted by data equipment 102 and transmitted by data equipment 102 to security platform 122, and may also cover security platform 122 (or a subset thereof, mere mobile malware). However, it also covers all known malware signatures for non-other forms of malware (e.g. PDF malware). As described in more detail below, platform 122 may also make available other types of information, such as machine learning models that can help data appliance 102 detect malware.

In various embodiments, security platform 122 is configured to provide security services to various entities in addition to (or instead of, where applicable) operators of data appliance 102 . For example, other enterprises that have their own respective enterprise networks 114 and 116 and their own respective data appliances 136 and 148 may contract with the security platform 122 operator. Other types of entities may also utilize the services of security platform 122 . For example, an Internet service provider that provides Internet service to client device 110 may contract with security platform 122 to analyze applications that client device 110 attempts to download. As another example, an owner of client device 110 can install software on client device 110 that communicates with security platform 122 (e.g., receive a content package from security platform 122 and use the received content package). , check the attachment according to the techniques described herein, and send the application to security platform 122 for analysis).

IV. SAMPLE ANALYSIS USING STATIC/DYNAMIC ANALYSIS

Figure 3 shows an example of the logical components that can be included in a system for analyzing samples. Analysis system 300 can be implemented using a single device. For example, the functionality of analysis system 300 may be implemented in malware analysis module 112 embedded within data appliance 102 . The analysis system 300 can also be implemented collectively across multiple separate devices. For example, the functionality of analysis system 300 may be provided by security platform 122 .

In various embodiments, analysis system 300 uses a list, database, or other collection of known safe content and/or known bad content (collectively shown in FIG. 3 as collection 314). . Collections 314 may be obtained through subscription services (e.g., provided by third parties) and/or as a result of other processing (e.g., performed by data appliance 102 and/or security platform 122). can be obtained in various ways. Examples of information included in collection 314 are known malicious server URLs, domain names and/or IP addresses, known good server URLs, domain names and/or IP addresses, known commands. and control (C&C) domain URLs, domain names, and/or IP addresses; signatures, hashes, and/or other identifiers of known malicious applications; signatures, hashes, and/or other identifiers of known good applications; or other identifiers, signatures, hashes, and/or other identifiers of known malicious files (e.g., Android exploit files), signatures, hashes, and/or other identifiers of known safe libraries; and signatures, hashes, and/or other identifiers of known malicious libraries.

A. ingestion

In various embodiments, when a new sample is received for analysis (eg, no existing feature associated with the sample exists in analysis system 300), it is added to queue 302. FIG. As shown in FIG. 3, application 130 is received by system 300 and added to queue 302 .

B. Static analysis

Coordinator 304 monitors queue 302, and when resources (e.g., static analysis workers) become available, coordinator 304 retrieves samples from queue 302 for processing (e.g., fetch a copy). In particular, coordinator 304 first provides samples to static analysis engine 306 for static analysis. In some embodiments, one or more static analysis engines are contained within analysis system 300, where analysis system 300 is a single device. In other embodiments, static analysis is performed by a separate static analysis server that includes multiple workers (ie, multiple instances of static analysis engine 306).

The static analysis engine obtains general information about the sample and includes it in the static analysis report 308 (along with heuristics and other information as appropriate). The report may be generated by the static analysis engine or by coordinator 304 (or another suitable component), which may be configured to receive information from static analysis engine 306 . In some embodiments, the collected information may be used as sample database records ( For example, stored in database 316). In some embodiments, the static analysis engine also identifies applications (e.g., "safe", "suspicious", or "malicious" ). As an example, even if one "malicious" static feature is present in the application (e.g., the application contains hardlinks to known malicious domains), the adjudication is "malicious". can be As another example, points can be assigned to each feature (e.g., based on the severity when discovered, based on the feature's reliability for predicting malice, etc.), and the adjudication is static. It can be assigned by static analysis engine 306 (or coordinator 304, if applicable) based on the number of points associated with the analysis result.

C. Dynamic analysis

Once static analysis is complete, coordinator 304 deploys available dynamic analysis engines 310 to perform dynamic analysis on the application. As with static analysis engine 306, analysis system 300 may directly include one or more dynamic analysis engines. In other embodiments, dynamic analysis is performed by a separate dynamic analysis server that includes multiple workers (ie, multiple instances of dynamic analysis engine 310).

Each dynamic analysis worker manages a virtual machine instance. In some embodiments, static analysis results (eg, those performed by static analysis engine 306) are in report form (308) and/or stored in database 316, or , or otherwise stored, and provided as input to the dynamic analysis engine 310 . For example, use static report information to help select/customize the virtual machine instance (e.g., Microsoft Windows7 SP2 vs. Microsoft Windows10 Enterprise, or iOS 11.0 vs. iOS 12.0) used by the dynamic analysis engine 310 can do. If multiple virtual machine instances run concurrently, a single dynamic analysis engine can manage all instances, or multiple dynamic analysis engines can be used (e.g., their own (with each management of virtual machine instances). During the dynamic portion of the analysis, actions taken by the application (including network activity) are analyzed, as described in more detail below.

In various embodiments, static analysis of samples is omitted or, where applicable, performed by a separate entity. As one example, conventional static and/or dynamic analysis may be performed on the file by the first entity. Once it is determined (e.g., by a first entity) that a given file is malicious, that file may be used for additional analysis (e.g., by the dynamic analysis engine 310) to a second entity (eg, an operator of the security platform 122).

The environment used by the analysis system 300 is configured so that behaviors observed while the application is running are logged when they occur (e.g., hooking and logcat). ), instrumented/hooked. Network traffic associated with the emulator is also captured (eg, using pcap). Log/network data can be stored as temporary files on the analysis system 300 and also more permanently (e.g. HDFS or other suitable storage technology or combination of technologies such as MongoDB). can be used) and stored. A dynamic analysis engine (or another appropriate component) compares connections made by the sample to a list of domains, IP addresses, etc. (314) and determines whether the sample communicated with a malicious entity ( Alternatively, it can determine whether or not communication has been attempted.

Like static analysis engines, dynamic analysis engines store the results of their analysis in database 316 in records associated with the tested application (and/or include the results in report 312, if applicable). . In some embodiments, the dynamic analysis engine also forms a verdict on the application (eg, "safe," "suspicious," or "malicious"). As an example, even if one "malicious" action is taken by the application (e.g., an attempt to contact a known malicious domain, or an attempt to remove sensitive information is observed ), the ruling may be “malicious”. As another example, points can be assigned for actions taken (e.g., based on the severity if discovered, based on the reliability of the action for predicting malice, etc.), and , by the dynamic analysis engine 310 (or coordinator 304, if applicable), based on the number of points associated with the dynamic analysis results. In some embodiments, a final adjudication relating to the samples is made (eg, by coordinator 304) based on a combination of reports 308 and 312.

V. Inline Malware Detection

Returning to the environment of Figure 1, millions of new malware samples can be generated each month (e.g., by rogue individuals, such as operators of system 120, either making subtle changes to existing malware or creating new malware samples). either by creating malware). Therefore, there are many malware samples that security platform 122 does not (at least initially) have signatures for. Furthermore, even if the security platform 122 generates signatures for newly created malware, due to resource constraints, a data appliance, such as the data appliance 102, at any given time, may have a list of all known signatures (e.g., platform 122).

Occasionally, malware, such as malware 130 , successfully infiltrates network 140 . One reason for this is that the data appliance 102 operates on a "first-time allow" principle. If data appliance 102 does not have a signature on a sample (e.g., sample 130) and submits it to security platform 122 for analysis, the verdict (e.g., "benign", "malicious , "unknown", etc.), it takes approximately 5 minutes for the security platform 122 to return. Instead of blocking communication between system 120 and client device 104 during that five minute period, communication is allowed under the first-allow principle. If the adjudication is returned (eg, after 5 minutes), data appliance 102 can use the adjudication to block subsequent transmission of malware 130 to network 140, thereby preventing system 120 from communicating with network 140. can block communication between In various embodiments, if a second copy of sample 130 arrives at data appliance 102 while data appliance 102 is awaiting adjudication from security platform 122, the second copy of sample 130 (and any subsequent ) is held by system 120 while waiting for a response from security platform 122 .

Unfortunately, during the five minutes that data appliance 102 waits for a verdict from security platform 122, a user of client device 104 could execute malware 130 and compromise client device 104 or other nodes in network 140. rice field. As noted above, in various embodiments, data appliance 102 includes malware analysis module 112 . One of the tasks that the malware analysis module 112 can perform is in-line malware detection. In particular, machine learning techniques are applied to perform efficient parsing of files (such as sample 130) on data appliance 102 as they pass through data appliance 102, as described in further detail below. (eg, in parallel with other processing performed on the file by the data appliance 102), and an initial malicious adjudication may be performed by the data appliance 102 (eg, while waiting for a minimum from the security platform 122). can be determined by

Various difficulties can arise in performing such analysis on a resource constrained device, such as data device 102 . One major resource in device 102 is session memory. A session is a network transfer of information, including files that device 102 parses according to the techniques described herein. A single device may have millions of concurrent sessions, and the memory that can last during a given session is extremely limited. A first difficulty in performing inline analysis on a data appliance, such as data appliance 102, is that due to such memory constraints, data appliance 102 typically cannot process an entire file at once. Instead, it receives a series of packets that need to be processed packet by packet. Therefore, the machine learning approach used by data appliance 102 should accommodate packet streams in various embodiments. A second problem is that in some cases the data appliance 102 cannot determine where the end of a given file being processed occurs (eg, the end of the samples 130 in the stream). The machine learning approach used by the data appliance 102 is therefore potentially midstream (e.g., during receipt/processing of samples 130, or otherwise prior to the actual end of file) in various embodiments. ) for a given file.

A. Machine learning model

As described in further detail below, in various embodiments, security platform 122 provides a set of machine learning models to data appliance 102 for use with in-line malware detection. The model incorporates features (eg, n-grams or other features) determined by security platform 122 that correspond to malicious files. Examples of two types of such models include linear classification models and non-linear classification models. Examples of linear classification models that can be used by data appliance 102 include logistic regression and linear support vector machines. One example of a non-linear classification model that may be used by data appliance 102 includes gradient boosting trees (eg, eXtreme Gradient Boosting (XGBoost)). The non-linear model is more accurate (and can better detect obfuscated/disguised malware), but the linear model uses significantly less resources on the device 102 (and does not require JavaScript or more suitable for efficient parsing of similar files).

As described in further detail below, the type of classification model used for a given file being parsed can be based on the file type associated with that file (and determined, for example, by a magic number). can do).

1. Additional details about the threat engine

In various embodiments, data appliance 102 includes threat engine 244 . The threat engine incorporates both protocol decoding and threat signature matching during the respective decoder and pattern match stages. The results of the two stages are merged by the detector stage.

When data appliance 102 receives a packet, data appliance 102 performs a session match to determine which session the packet belongs to (allowing data appliance 102 to support concurrent sessions). Each session has a session state which refers to a specific protocol decoder (eg web browsing decoder, FTP decoder or SMTP decoder). When a file is sent as part of a session, the applicable protocol decoder can use an appropriate file-specific decoder (eg PE file decoder, JavaScript decoder, or PDF decoder).

Portions of one exemplary embodiment of threat engine 244 are shown in FIG. For a given session, the decoder 402 walks the traffic byte stream according to the corresponding protocol and marking context. One example of context is end-of-file context (eg, encountering </script> while processing a JavaScript file). The decoder 402 can mark the end-of-file context within the packet and then use the observed characteristics of the file to trigger execution of the appropriate model. In some cases (eg, FTP traffic), there may be no explicit protocol-level tags for the decoder 402 that identify/mark the context. As will be described in more detail below, in various embodiments, the decoder 402 uses other information (e.g., the file size reported in the header) to determine when feature extraction for the file should end (e.g., , start the overlay section) and determine if execution using the appropriate model should begin.

Decoder 402 consists of two parts. A first part of the decoder 402 is a virtual machine part (404) that can be implemented as a state machine using the state machine language. The second part of decoder 402 is a set of tokens 406 for triggering state machine transitions and actions when traffic matches. The threat engine 244 also includes a threat pattern matcher 408 (eg, using regular expressions) that performs pattern matching (eg, against threat patterns). As one example, threat pattern matcher 408 includes a table of strings (either exact strings or wildcard strings) to match (e.g., by security platform 122) and a matching It can have a corresponding action to take if a string that matches is found. Detector 410 processes the output provided by decoder 402 and threat pattern matcher 408 to perform various actions.

2. n-grams

Data within a session can be divided into a series of n-grams - a series of byte strings. As an example, assume that a portion of the hexadecimal data in the session is "1023ae42f6f28762aab". Then the 2-grams in the sequence are all pairs of adjacent characters, such as "1023", "23ae", "ae42", "42f6", and so on. In various embodiments, threat engine 244 is configured to parse files using 8-grams. Other n-grams can also be used, such as 7-grams or 4-grams. In the string example above, "1023ae42f6f28762" is 8 grams, "23ae42f6f28762aa" is 8 grams, and so on. The total number of different 8-grams possible in a byte sequence is 2 to the 64th power (18,446,744,073,709,551,616). Searching for all possible 8-grams within a byte sequence would easily exceed the resources of the data appliance 102 . Instead, security platform 122 provides a significantly reduced set of 8 grams to data appliance 102 for use by threat engine 244, as described in more detail below.

When a session packet corresponding to a file is received by threat engine 244, threat pattern matcher 408 parses the packet for matches against strings in the table (e.g., performs regular expression and/or exact string matching). by doing). A list is generated of matches (eg, each instance of a match identified by the corresponding pattern ID) and at which offset each match occurred. Actions on these matches are performed in offset order (eg, bottom to top). For a given match (i.e., corresponding to a particular pattern ID), a set of one or more actions to be taken is specified (e.g., via an action table mapping actions to pattern IDs) .

The 8-gram set provided by the security platform 122 matches what the threat pattern matcher 408 has already performed (e.g., where a JavaScript file accesses password storage, or where a PE file accesses the Local Security Authority Subsystem Service (LSASS ) can be added (eg, as exact string matches) to a table of heuristic matches looking for specific indicators of malware, such as where to call an API. One advantage of this approach is that instead of performing multiple passes through the packet (e.g., first evaluating heuristic matches and then evaluating 8-gram matches), threat pattern matcher 408 8grams can be searched in parallel with other searches being performed.

As described in more detail below, 8-gram matching is used by both linear and non-linear classification models in various embodiments. Examples of actions that can be specified for n-gram matching are (e.g. , for linear classifiers) and storing matches in feature vectors (eg, for non-linear classifiers). Which action to take may be specified based on the file type associated with the packet (which determines which type of model to use).

3. Model selection

In some cases, a particular file type is specified in the file's header (eg, as a magic number that appears within the first 7 bytes of the file itself). In such scenarios, threat engine 244 may select the appropriate model corresponding to the specified file type (eg, based on a table provided by security platform 122 listing file types and corresponding models). . In other cases, such as JavaScript, the magic number or other file type identifier (if present in the header) does not prove which classification model should be used. As an example, JavaScript would have a file type of "textfile". To identify file types such as JavaScript, decoder 402 is used to perform deterministic finite state automaton (DFA) pattern matching and heuristics (e.g., to detect that a file is JavaScript). identifying <script> and other indicators) can be applied. The determined file type and/or the selected classification model are saved in the session state. File types associated with a session can be updated as the session progresses. For example, in a text stream, a JavaScript file type can be assigned to a session when a <script> tag is encountered. The file type can be changed (e.g. back to plaintext) when a corresponding </script> is encountered.

4. Linear classification model

One way to express the linear model is to use the following linear equations.

Σ(β _i x _i )<C, i=1,2,3...,P
where P is the total number of features, x _i is the ith feature, β _i is the coefficient (weighting) of feature x _i and C is the threshold constant. In this example, C is the threshold for bad faith verdicts, and if the sum is less than C for a given file, then the file is assigned a benign verdict, and if the sum is greater than or equal to C, then the Means that the file is assigned a verdict of bad faith.

One approach for using a linear classification model by data appliance 102 is as follows. A single float (d) is used to track the score of the input file, and a hash table is used to store the observed n-grams and the corresponding coefficients (i.e., x _i and β _i ). be. For each incoming packet, each n-gram feature (eg, such as provided by security platform 122) is checked. Whenever a match is found for a hashtable feature (x _i ), a single float (β _i ) matching that feature in the hashtable is added (eg, to d). When the file end is reached, the single float (d) is compared against a threshold (C) to determine the verdict for the file.

For n-gram counts, feature x _i is equal to the number of times the i-th n-gram is observed. Suppose the i-th n-gram was observed 4 times for a particular file. 4*β _i can be rewritten as β _i +β _i +β _i +β _i . Instead of counting how many times the ith n-gram is observed (i.e. 4 times) and multiplying by β _i , another approach is to use β adding _i . Further assume that the jth n-gram was observed 3 times for the file. 3*β _i can similarly be written as β _i +β _i +β _i , and instead of counting how many times β _i is observed, add β _i each time and then add at the end .

To find Σ(β _i x _i ), each of β _i x _i , β _j x _j , ... is added (where ... corresponds to all other features/weightings). This can be rewritten as β _i +β _i +β _i +β _j +β _j +β _j +β _j . Additions are cumulative, so the addition of values can be added in any order (e.g., _{βi + βj + βi + βj + βi + βi + βj , etc. ) and accumulated into} a single float. . Now assume that float (d) starts at 0.0. Each time feature x _i is observed, β _i can be added to float d, and each time x _j is observed, β _j can be added to float d. This approach allows a 4-byte float to be used as the total per-session memory, as opposed to approaches where the per-session memory is proportional to the number of features. Here, the entire feature vector is stored in memory to be multiplied by the weighting vector. Using the 4K byte feature example of 4 bytes * 1,000, 4K of storage would be required (compared to a single 4 byte float). This is 1,000 times more expensive.

5. Nonlinear Classification Model

Various non-linear classification approaches can be used with the techniques described herein. One example of a non-linear classification model is a gradient boosted tree. In this example, the feature vector is initialized to an all-zero vector. Unfortunately, in nonlinear models (unlike linear models) the entire set of features whose presence is being detected (eg, 1000 features) persists for the entire duration of the session. This is not as efficient as in a linear approach, but some efficiency can still be gained by downsampling the features to be a 1-byte (0-255) float instead of a full 4-byte float. (can be used on devices that are not memory constrained).

As the data appliance 102 scans the entire file, each time a feature is observed, the value of that feature is incremented by one in the feature vector. Once the end of the file is reached (or the end of feature observation occurs otherwise), the constructed feature vector is fed into the gradient boosting tree model (eg, received from security platform 122). . As described in more detail below, non-linear classification models can be built using both n-gram (eg, 8-gram) and non-n-gram features. One example of a non-n-gram feature is the file's intended size (which can be read as a value from the packet containing the file's header). File data that appears after the intended end-of-file (eg, based on the file size specified in the header) is called an overlay. In addition to serving as a feature, the intended file length can be used as a proxy for how long the file is expected to be. A non-linear classifier may be run on the file's packet stream until the intended file length is reached. A verdict can then be formed for the file regardless of whether the end of the file has actually been reached. The inclusion of overlays in a given file is also an example of a feature that can be used as part of a non-linear classification model. In various embodiments, the overlay portion of the file is not parsed, and again - the parsing can occur prior to the actual end of the file. In other embodiments, feature extraction is performed and verdicts of malice are not formed until the actual file end is reached.

In one exemplary embodiment, the tree model contains 5000 binary trees. Every node on each tree contains features and corresponding thresholds. An example of part of the tree is shown in Figure 5. In the example shown in FIG. 5, if the value of a feature (eg, feature F4) is less than its threshold (eg, 30), the left branch is taken (502). If the value of the feature is greater than or equal to the threshold, the right branch is taken (504). The tree progresses until it reaches a leaf node (eg, node 506) that has an associated value (eg, 0.7). The values of each leaf reached (for each tree) are summed (rather than multiplied) to get the final score for computing the adjudication. If the score is below the threshold, the file is considered benign, and above the threshold the file is considered malicious. The lack of multiplication in obtaining the final score helps the model to be used more efficiently in resource constrained environments of data appliance 102 .

In various embodiments, the tree itself may be stored in a shared memory that is fixed in the data appliance 102 (until an updated model is received) and can be accessed by multiple sessions at the same time. The cost per session is the cost of storing the session's feature vector and can be zero once the session's analysis is complete.

6. Example Process

FIG. 6 shows one example of a process for performing inline malware detection on a data appliance. In various embodiments, process 600 is performed by data appliance 102 and, in particular, threat engine 244 . Threat engine 244 can be implemented using a script (or set of scripts) written in a suitable scripting language (eg, Python). Process 600 may also be performed at an endpoint, such as client device 110 (eg, by an endpoint protection application running on client device 110).

Process 600 begins at 602 when an indication is received by device 102 that a file is being sent as part of a session. As one example of the processing performed at 602, for a given session, the associated protocol decoder may call the appropriate file-specific decoder when the start of a file is detected by the protocol decoder, or You can use it if you don't have it. As described above, the file type is determined (eg, by decoder 402) and associated with the session (eg, subsequent file type analysis is performed until the file type changes or no file packets are sent). so you don't have to).

At 604, n-gram analysis is performed on the sequence of received packets. As noted above, n-gram parsing can be done in-line with other parsing being performed by device 102 in a session. For example, while device 102 is performing analysis on a particular packet (eg, to check for the existence of a particular heuristic), it also It can also determine whether it matches the 8 grams During the processing performed at 604, when an n-gram match is found, the corresponding pattern ID is used to map conditions to actions based on filetype. This action either increments a weighted counter (e.g. if the file type is associated with a linear classifier) or updates the feature vector to account for the match (e.g. if the file type is non-linear associated with a classifier).

N-gram parsing continues, packet by packet, until either an end-of-file condition or a checkpoint is reached. At that point (606), the appropriate model is used to determine the file's verdict (ie, the final value obtained using the model is compared to the bad faith threshold). As noted above, the model incorporates n-gram features and may also incorporate other features (eg, in the case of non-linear classifiers).

Finally, at 608, action is taken in response to the decision made at 606. One example of a response action is terminating a session. Another example of a response action is to allow the session to continue, but prevent the file from being sent (instead put it in quarantine). In various embodiments, device 102 is configured to share its verdicts (either benign verdicts, bad verdicts, or both) with security platform 122 . Once the security platform 122 has completed its independent analysis of the file, it can use the adjudications reported by the device 102 for various purposes, including evaluating the performance of the model that formed the adjudication.

An example threat signature for the sample is shown in FIG. 7B. Specifically, for a sample with a SHA-256 hash of "4d73f42438fb5a8579219cdfa9cbbb4ce3f771ffed93af81b052831e4813f8", the first value in each pair corresponds to the feature and the second value corresponds to the count. In the example shown in FIG. 7B, features containing numbers (e.g., feature "3905") correspond to n-gram features, and features containing "J" and numbers (e.g., feature "J18") correspond to non-n-gram features. It corresponds to n-gram features.

In one exemplary embodiment, security platform 122 targets a particular false positive ratio (e.g., 0.001) when generating models for use by devices such as data device 102. is configured as Therefore, in some cases (eg, 1 out of 1000 files), while performing inline analysis using a model according to the techniques described herein, the data appliance 102 may detect a benign A file can be falsely determined to be malicious. In such scenarios, if the security platform 122 subsequently determines that the file is in fact benign, it may be whitelisted so that it is not later flagged as malicious (e.g., by another device). can be added to

One approach to whitelisting is to instruct security platform 122 to add the file to a whitelist stored on device 102 . Another approach is to instruct security platform 122 to whitelist system 154 of false positives, and whitelist system 154, in turn, to keep devices, such as device 102, up to date with false positive information. As mentioned above, one problem with devices such as device 102 is that they are resource constrained. One approach to minimizing the resources used to maintain the whitelist on the device is to use a Least Recently Used (LRU) cache to maintain the whitelist. The whitelist can include file hashes and can also be based on other factors such as feature vectors or hashes of feature vectors.

VI. Model building

Returning to the environment shown in FIG. 1, security platform 122 is configured to perform static and dynamic analysis on received samples, as described above. Security platform 122 can receive samples for analysis from a variety of sources. As noted above, one exemplary type of sample source is a data appliance (eg, data appliances 102, 136, and 148). Other sources (eg, one or more third party providers of samples, such as other security device vendors, security researchers, etc.) can also be used as appropriate. As described in more detail below, the security platform 122 can use the corpus of samples it receives to build a model (e.g., the model can be a model of the techniques described herein). may then be used by the security device 102, according to an embodiment).

In various embodiments, static analysis engine 306 is configured to perform feature extraction on the received samples (e.g., while performing other static analysis functions as described above). and). One exemplary process for performing feature extraction (eg, by security platform 122) is shown in FIG. 8A. Process 800 begins at 802 when static analysis of a sample is initiated. During feature extraction (804), all 8-grams (or other applicable n-grams in embodiments where 8-grams are not used) from the sample being processed (eg, sample 130 in FIG. 3) is extracted. In particular, a histogram of the 8-grams within the sample being analyzed is extracted (eg, into a hash table), indicating the number of times a given 8-gram was observed within the sample being processed. One advantage of extracting 8-grams during feature analysis by the static analysis engine 306 is the potential privacy and It can reduce contractual issues. This is because the original file cannot be reconstructed from the resulting histogram. The extracted histogram is saved at 806.

In various embodiments, static analysis engine 306 stores the extracted histogram (eg, represented using a hash table) for a given sample along with histograms extracted from other samples in storage 142 (eg, , Hadoop cluster). The data in the hadoop is compressed, and the required data is decompressed on the fly when operations are performed on the hadoop data. An example of one exemplary hash table (represented in JSON) for a file is shown in Figure 7A. Line 702 shows the SHA-256 hash of the file. Row 704 indicates the UNIX time that sample 130 arrived at security platform 122 . Row 706 shows the count of n-grams in the overlay section (eg 'd00fbf4e08bc366':1 indicates that one instance of 'd00fbf4e08bc366' was found in the overlay section). Line 708 shows the count of each 8-gram present in the file. Line 710 indicates that the file has overlays. Line 712 indicates that the file type of the file is ".exe". Line 714 indicates the UNIX time when security platform 122 finished processing sample 130 . Line 716 shows the count for each non-8-gram feature hit by the file. Finally, line 718 indicates that the file has been determined (eg, by security platform 122) to be malicious.

In one exemplary embodiment, the set of 8-gram histograms stored in the hadoop cluster grows by approximately 3 terabytes of 8-gram histogram data per day. Histograms accommodate both malicious and benign samples (e.g., labeled as such based on the results of other static and dynamic analyzes performed by security platform 122 as described above). is done.)

An 8-gram histogram extracted from the analyzed sample is approximately 10% larger than the file itself, and a typical sample has a histogram containing approximately 1 million different 8-grams. The total number of different possible 8-grams is 2 to the 64th power (2 ⁶⁴ ). As noted above, by contrast, a classification model transmitted by security platform 122 to a device such as data appliance 102 (e.g., as part of a subscription) may, in various embodiments, consist of thousands of features (e.g., 1000 features) only. One exemplary way to reduce the set of potentially maximum ²⁶⁴ features to the 1000 most important features for use in the model is to use mutual information technology. Other approaches (eg chi-squared scores) are also applicable. The four required parameters include the number of malicious samples with a given function, the number of benign samples with a given function, the total number of malicious samples, and the total number of benign samples. One of the advantages of mutual information is that it can be used efficiently on very large datasets. In Hadoop, the mutual information approach distributes the task over multiple mappers in a single pass (i.e., an 8-gram histogram stored in the Hadoop cluster dataset for a given file type). through all), each of which is responsible for handling a particular function. Those features with the highest mutual information can be selected as the set of features that are most malicious and/or most benign, as applicable. The resulting 1000 features can then be used to build models (eg, linear and non-linear classification models) where applicable. For example, to build a linear classification model, the model builder 152 (implemented using a set of open source tools and/or scripts written in a suitable language such as python) uses the top 1000 and applicable weightings as a set of n-gram features for device 102 to check (eg, as described in section VA4 above).

In some embodiments, a non-linear classification model is also built by model builder 152 using the top 1000 (or other desired number) of features. In other embodiments, the non-linear classification model is built primarily using top features (e.g., 950), but other features that may be detected during per-packet feature extraction and analysis. Non-gram features (eg, 50 such features) are also incorporated. Some examples of non-n-gram features that can be incorporated into non-linear classification models are (1) the size of the header, (2) the presence or absence of checksums in the file, (3) the number of sections in the file, (4) The intended length of the file (as indicated in the header of the PE file), (5) whether the file contains overlay parts, and (6) whether the file requires the Windows EFI subsystem to run the PE. and whether or not.

In some embodiments, rather than using mutual information to select the top 1000 features, a larger set of features (the set of over-generated features) is determined. As an example, the top 5000 features can be selected first using mutual information. The 5000 sets can then be used as input to conventional feature selection techniques (eg, bagging). It does not scale well for very large datasets (eg the entire Hadoop dataset), but is more effective for reduced sets (eg 5000 features). Conventional feature selection techniques can be used to select the final 1000 features from the set of 5000 features identified using mutual information.

Once the final 1000 features are selected, one exemplary method for building the nonlinear model is to use open source tools such as scikit-learn or XGBoost. If applicable, parameter tuning can be performed, such as by using cross-validation.

One exemplary process for generating models is shown in FIG. 8B. In various embodiments, process 850 is performed by security platform 122 . Process 850 begins at 852 when a set of extracted features (eg, including n-gram features) is received. One exemplary way in which the feature set can be received is by reading the features stored as a result of process 800 . At 854 a reduced set of features is determined from the features received at 852 . As mentioned above, one exemplary method of determining the reduced set of features is by using mutual information. Other approaches (eg, chi-squared scores) can also be used. Furthermore, also as described above, combinations of techniques such as using mutual information to select an initial set of features and refining the initial set of features using bagging or other suitable techniques may also Can be used with 852/854. Ultimately, as described above, once the features are selected (e.g., at 854), an appropriate model is built at 856 (e.g., using open source or other tools and, if applicable, performs parameter tuning). A model (eg, generated by model builder 152 using process 850) is provided to data appliance 102 and other applicable recipients (eg, data appliances 136 and 148) (eg, subscription services). as a part).

In various embodiments, model builder 152 generates models (eg, linear and non-linear classification models) on a daily (or other applicable) basis. By executing process 850 or otherwise periodically generating models, security platform 122 ensures that the models used by devices, such as device 102, are current types of malware threats (e.g., malicious can help ensure that the latest deployed threats by some individuals are detected.

Whenever a newly generated model is determined to be better than an existing model (e.g., as determined based on a set of quality assessment metrics exceeding a threshold), the updated model is It may be transmitted to a data device such as device 102 . In some cases, such updates adjust the weightings assigned to features. Such updates are easily deployed to and adopted by the device (eg, as real-time updates). In other cases, such updates adjust the features themselves. Such updates may require patches to equipment components, such as decoders, which may make deployment more complex. One advantage of using overtraining during model generation is that the model can take into account whether the decoder can detect certain features.

In various embodiments, a device is required (eg, by security platform 122) to deploy updates to the model as they are received. In other embodiments, the device can selectively deploy updates (at least for a period of time). As one example, when a new model is received by the device 102, both the existing model and the new model can be run in parallel on the device 102 for a period of time (e.g., if the existing model is used in production). , and the new model reports on actions that would be performed without actually executing them). A device administrator can indicate whether an existing model or a new model should be used to handle traffic on the device (e.g., based on which model shows better performance). ). In various embodiments, the device 102 provides telemetry that indicates information such as which model is running on the device 102 and how valid that model is (eg, false positive statistics). back to the security platform 122.

Although the above embodiments have been described in some detail for clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are exemplary and non-limiting.

Claims (40)

a system,
is a processor;
storing a set containing one or more sample classification models on a network device;
performing n-gram analysis on a sequence of received packets associated with a received file, said performing n-gram analysis comprising using at least one stored sample classification model;
determining, based at least in part on the n-gram analysis of the sequence of received packets, that the received file is malicious; and in response to determining that the file is malicious, prevent propagation of received files,
a processor configured to;
a memory coupled to the processor and configured to provide instructions to the processor;
system, including The processor
performing the n-gram analysis, at least in part, by comparing n-grams in the received packet against a predefined list of n-grams;
2. The system of claim 1, configured to: the default list of n-grams was generated using a plurality of pre-collected malware samples;
3. The system of claim 2. The processor further
determining a file type associated with said file;
2. The system of claim 1, configured to: The processor
selecting a linear classification model from the set of one or more sample classification models based on the determined file type associated with the file;
5. The system of claim 4, configured to: performing the n-gram analysis includes accumulating a set of weights corresponding to observed n-grams;
6. The system of claim 5. the weighting is accumulated over a single float value;
7. A system according to claim 6. The processor
selecting a nonlinear classification model from the set of one or more sample classification models based on the determined file type associated with the file;
5. The system of claim 4, configured to: the non-linear classification model includes n-gram features and non-n-gram features;
9. System according to claim 8. at least one non-n-gram feature is associated with file size;
10. System according to claim 9. at least one non-n-gram feature is associated with the presence of the overlay;
10. System according to claim 9. performing the n-gram analysis includes updating values for features in a feature vector whenever the features match;
9. System according to claim 8. using the at least one stored sample classification model includes running a non-linear classifier on the packet stream until an intended file length is reached;
The system of claim 1. the intended file length is not the actual file length, and the ruling is determined before reaching the actual end of the file;
14. The system of claim 13. The processor further
receiving at least one updated classification model;
2. The system of claim 1, configured to: the n-gram analysis is performed inline with other packet analysis as a single pass analysis of the traffic stream;
The system of claim 1. The processor further
using a whitelisted set of n-grams when performing the n-gram analysis;
2. The system of claim 1, configured to: The processor further
sending a copy of the received file to a security platform and performing the n-gram analysis while awaiting adjudication from the security platform;
2. The system of claim 1, configured to: storing a set containing one or more sample classifications on a network device;
performing n-gram analysis on a sequence of received packets associated with a received file, said performing n-gram analysis comprising using at least one stored sample classification model;
determining, based at least in part on the n-gram analysis of the sequence of received packets, that the received file is malicious; and in response to determining that the file is malicious, preventing propagation of the received file;
A method, including A computer program stored on a tangible computer-readable storage medium and comprising a plurality of computer instructions,
When the computer instructions are executed, the computer
storing a set containing one or more sample classifications on a network device;
performing n-gram analysis on a sequence of received packets associated with a received file, said performing n-gram analysis comprising using at least one stored sample classification model;
determining, based at least in part on the n-gram analysis of the sequence of received packets, that the received file is malicious; and in response to determining that the file is malicious, preventing propagation of the received file;
A computer program that causes the a system,
is a processor;
receiving a set of features, including a plurality of n-grams, extracted from a set of files;
determining a reduced set of features that includes at least some of the plurality of n-grams; and
using said reduced set of features to generate a model usable by a data appliance to perform inline malware analysis;
a processor;
a memory coupled to the processor and configured to provide instructions to the processor;
system, including the set of features includes features extracted from a set of known malicious files;
The system of claim 1. the set of features includes features extracted from a set of known benign files;
The system of claim 1. the reduced set of features is determined using mutual information;
The system of claim 1. the reduced set of features is determined using a chi-square score;
The system of claim 1. the generated model includes n-gram features;
22. The system of claim 21. the generated model further includes non-n-gram features;
27. The system of claim 26. at least one non-n-gram feature is associated with file size;
8. The system of claim 7. at least one non-n-gram feature is associated with a header size;
8. System according to claim 7. the at least one non-n-gram feature is associated with at least one of the presence or absence of a checksum in the file;
8. The system of claim 7. at least one non-n-gram feature is associated with the number of sections in the file;
8. The system of claim 7. at least one non-n-gram feature is associated with the length of the claimed file;
8. The system of claim 7. at least one non-n-gram feature is associated with whether the file contains an overlay;
8. System according to claim 7. the model is a linear model,
22. The system of claim 21. wherein the model is a non-linear model;
22. The system of claim 21. the plurality of n-grams are extracted during static analysis of the set of files;
22. The system of claim 21. a system in which the model is transmitted to a first data appliance;
22. The system of claim 21. In response to false positive results reported by a second data appliance, the processor is configured to generate an updated model and send the updated model to the first data appliance. ,
38. The system of claim 37. receiving a feature set comprising a plurality of n-grams extracted from a set of files;
determining a reduced set of features that includes at least some of the plurality of n-grams;
using said reduced set of features to generate a model usable by a data appliance to perform inline malware analysis;
A method, including A computer program stored on a tangible computer-readable storage medium and comprising a plurality of computer instructions,
When the computer instructions are executed, the computer
receiving a feature set comprising a plurality of n-grams extracted from a set of files;
determining a reduced set of features that includes at least some of the plurality of n-grams;
using said reduced set of features to generate a model usable by a data appliance to perform inline malware analysis;
A computer program that causes the

Images