WO2014110293A1 - Procédé de streaming amélioré et système de traitement de métadonnées de réseau - Google Patents

Procédé de streaming amélioré et système de traitement de métadonnées de réseau Download PDF

Info

Publication number
WO2014110293A1
WO2014110293A1 PCT/US2014/010932 US2014010932W WO2014110293A1 WO 2014110293 A1 WO2014110293 A1 WO 2014110293A1 US 2014010932 W US2014010932 W US 2014010932W WO 2014110293 A1 WO2014110293 A1 WO 2014110293A1
Authority
WO
WIPO (PCT)
Prior art keywords
network
metadata
traffic
information
cloud
Prior art date
Application number
PCT/US2014/010932
Other languages
English (en)
Inventor
Igor Balabine
Alexander VELEDNITSKY
Original Assignee
Netflow Logic Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US13/830,924 external-priority patent/US20140075557A1/en
Application filed by Netflow Logic Corporation filed Critical Netflow Logic Corporation
Priority to CN201480012616.9A priority Critical patent/CN105051696A/zh
Priority to CA2897664A priority patent/CA2897664A1/fr
Priority to KR1020157021506A priority patent/KR20150105436A/ko
Priority to JP2015552783A priority patent/JP2016508353A/ja
Priority to RU2015132628A priority patent/RU2015132628A/ru
Publication of WO2014110293A1 publication Critical patent/WO2014110293A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/04—Processing captured monitoring data, e.g. for logfile generation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
    • G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
    • G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
    • G06F11/3079—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting the data filtering being achieved by reporting only the changes of the monitored data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466—Performance evaluation by tracing or monitoring
    • G06F11/3476—Data logging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/86—Event-based monitoring
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/875—Monitoring of systems including the internet

Definitions

  • the present invention relates to network monitoring and event management. More specifically it relates to processing of network metadata obtained through network monitoring activities and a subsequent processing of the metadata, which may efficiently result in useful information being reported in a timely manner to a consumer of the metadata.
  • Network monitoring is a critical information technology (IT) function often used by Enterprises and Service Providers, which involves watching the activities occurring on an internal network for problems related to performance, misbehaving hosts, suspicious user activity, etc.
  • IT critical information technology
  • Network monitoring is made possible due to the information provided by various network devices.
  • the information has been generally referred to as network metadata, i.e., a class of information describing activity on the network which is supplemental and complimentary to the rest of information transmitted over the network.
  • Syslog is one type of network metadata commonly used for network monitoring. Syslog is a standard for logging program messages and provides devices which would otherwise be unable to communicate a means to notify administrators of problems or performance. Syslog is often used for computer system management and security auditing as well as generalized informational, analysis, and debugging messages. It is supported by a wide variety of devices (like printers and routers) and receivers across multiple platforms. Because of this, syslog can be used to integrate log data from many different types of systems into a central repository.
  • NetFlow is a network protocol for collecting IP traffic information that has become an industry standard for traffic monitoring.
  • NetFlow can be generated by a variety of network devices such as routers, switches, firewalls, intrusion detection systems (IDS), intrusion protection systems (IPS), network address translation (NAT) entities and many others.
  • IDS intrusion detection systems
  • IPS intrusion protection systems
  • NAT network address translation
  • NetFlow metadata can generally be attributed to the high volume and high delivery rate of information produced by the network devices, the diversity of the information sources and an overall complexity of integrating additional information streams into existing event analyzers. More particularly, NetFlow metadata producers have typically generated more information than consumers could analyze and use in a real time setting. For example, a single medium to large switch or router on a network might generate 400,000 NetFlow records per second.
  • SIM security information management
  • SEM security event management
  • SIEM security information and event management
  • Big Data Network managers and network security professionals continuously confront and struggle with a problem often referred to in the industry as "Big Data”.
  • Some of the issues created by the Big Data problem include an inability to analyze and store massive amounts of machine-generated data that often exists in different formats and structures.
  • the problems commonly experienced can be summarized as follows:
  • the present invention provides a system and method capable of addressing all of the above-identified problems associated with Big Data by providing the ability to analyze large volumes of metadata in real time, convert large volumes of metadata into a common format that allows ready correlation with other data within a single monitoring system, and dramatic reduction in the volume of the incoming data through real time data reduction techniques such as packet validation, filtering, aggregation and de-duplication.
  • Embodiments of the present invention are able to check the validity of incoming packets of network metadata and discard malformed or improper messages. Embodiments are also able to examine and filter incoming packets of network metadata in real time to identify relevant aspects of their information content and segment or route different streams of incoming network metadata for differing processing within the processing engine of the present invention. Included in such differing processing is the opportunity to reduce output metadata traffic by dropping particular messages or selected streams of messages based upon criteria that can be configured by a network manager and determined during the early examination of incoming messages. This enables a network manager to focus the network analysis, either on an ongoing basis or temporarily in response to a particular network condition. As an example, a network manager can elect to focus attention upon network metadata within the system that is generated only be the edge devices on the network to investigate possible intrusion events.
  • Embodiments of the present invention are further able to aggregate the information content contained in incoming packets of network metadata and replace a large quantity of related packets with one or a much smaller number of other packets that capture the same information but generate a much smaller downstream display, analysis and storage requirement than the original metadata flow.
  • Embodiments of the present invention are further able to de-duplicate the content of the normal metadata flow generated by the network devices. Because incoming traffic is typically routed within the network through a sequence of network devices to its destination device, and because each network device typically generates network metadata for each flow that traverses it, a significant amount of redundant metadata is generated that contributes to the Big Data problem in the industry. [0015]
  • the present invention relates to a system and method capable of receiving arbitrary structured data, e.g., network or machine-generated metadata, in a variety of data formats (hereafter network metadata), efficiently processing the network metadata and forwarding the received network metadata and/or network metadata derived from the original network metadata in a variety of data formats.
  • Network metadata could be generated by a variety of network devices such as routers, switches, firewalls, intrusion detection systems (IDS), intrusion protection systems (IPS), network address translation (NAT) entities and many others.
  • the network metadata information is generated in a number of formats including but not limited to NetFlow and its variants, (e.g., jFlow, cflowd, sFlow, IPFIX), SNMP, SMTP, syslog, etc.
  • the method and system described herein is able to output network metadata information in a number of formats including but not limited to NetFlow and its versions, O ' Flow, cflowd, sFlow, IPFIX,) SNMP, SMTP, syslog, OpenFlow, etc.
  • embodiments of the invention are able to output selected types of network metadata information at a rate sufficient to allow real-time or near-real- time network services to be provided.
  • the system is capable of providing meaningful services in deployments with N (N > 1) producers of the network metadata and M (M > 1) consumers of the original or derived network metadata. It may be appreciated that a particular embodiment of this invention aligns with a definition of IPFIX Mediator as reflected in RFC 5982.
  • An embodiment of the present invention provides a method and system for identifying the nature, character and/or type ("class") of received network metadata and organizing received information into categories or classes. This may be of particular usefulness when used in association with NetFlow v9 and similar messages that are template -based and can be of widely varied content and purpose. Once categorized or classified, each individual class member instance can be further processed according to zero, one or a plurality of class specific processing rules or according to a default processing rule ("policies"). This aspect of the invention enables fine grain processing of an unlimited variety of network metadata types.
  • the embodiment is able to efficiently organize the processing of network metadata, and in appropriate circumstances, reduce the amount of processing required by filtering, consolidating and/or eliminating portions of the network metadata that is of limited interest to the system administrator, thereby contributing to the real-time or near-real-time operation of the system and potentially reducing storage requirements at a network metadata collector.
  • network metadata may be generated from each traversed device that contains redundant information.
  • Policies can be introduced that remove redundancies from certain classes of network metadata that are directed to the SIEM system, while at the same time preserving all such metadata for the flow that is directed to a collector.
  • policies implemented by embodiments of the invention can be defined in a manner that supports and/or are coordinated with policies or areas of focus of a SIEM system and/or metadata collector that is operating within the network.
  • Policies can be introduced for the purpose of detecting important or unusual network events that might be indicative of security attacks, reporting traffic spikes on the network, detecting attacks on the network, fostering better usage of network resources, and/or identifying applications running on the network, for network management and security purposes.
  • Policies can be general purpose or time-based, and can be applied to a specific class or a subset of the network metadata passing through the network.
  • An embodiment of the invention contemplates the provision of multiple working threads that operate in cooperation with multiple policy modules to increase system throughput and performance.
  • Working threads can be introduced that are specialized or tuned for use with a particular class or subclass of network metadata to further enhance system performance and throughput.
  • Such specialized working threads and policy modules can perform processing operations on different portions of the stream of network metadata in parallel to enhance system performance and throughput.
  • multiple instances of the specialized working thread and/or policy module can be instantiated to operate in parallel to further enhance system performance and throughput.
  • an embodiment of this invention provides a unique capability of detecting externally controlled network hosts (“botnet member”) residing on an internal network.
  • botnet master operated by a central controller
  • detection of malicious content on a network host requires installing a dedicated plug-in module on that host.
  • This method does not work against sophisticated malicious agents (“rootkit”) which are undetectable by any host-based means.
  • rootkit sophisticated malicious agents
  • An embodiment of the present invention introduces a policy which is able to identify and notify a security system about an act of communication between a botnet master and a botnet member on the internal network.
  • intelligence provided by the present invention achieves a higher degree of trustworthiness than intelligence provided by similar-in-purpose devices exposed to the network traffic.
  • IDS Intrusion Detection System
  • IPS Intrusion Detection System
  • DoS Denial of Service
  • the present invention enables transforming network metadata which makes it suitable for deployments which require network metadata obfuscation.
  • the method and system may be implemented in a streaming fashion, i.e., processing the input network metadata as it arrives ("in real-time or near-real-time") without the need to resort to persistent storage of the network metadata.
  • This embodiment of the invention allows deployment of the system and method on a computer with limited memory and storage capacity, which makes the embodiment especially well suited for deployments in a computing cloud.
  • an embodiment of the present invention may provide an efficient method for converting the results of the policies' application into zero, one or more representations ("converter") suitable for further processing by recipients of the converted network metadata or the original network metadata.
  • converter representations
  • An embodiment of the invention provides a plurality of converters that may be customized for a particular class or classes of network metadata and/or output format, thereby increasing throughput of the system to better enable real-time or near-real-time services on the network. Further, in response to a heavy volume of a particular class or subclass of network metadata, multiple instances of the customized working thread and/or conversion modules can be instantiated to operate in parallel to further enhance system performance and throughput.
  • an embodiment of the present invention is able to ensure integrity of the converted network metadata by appending message authentication codes.
  • This embodiment of the invention enables sophisticated network metadata recipients to verify authenticity of the received information.
  • Yet another embodiment of this invention is the ability to deploy the system and method in a fashion transparent to the existing network ecosystem. This embodiment does not require any change in the existing network components' configuration.
  • Another embodiment of the present invention provides a method and apparatus for describing network metadata processing and conversion rules either in visual or in textual terms or a combination thereof.
  • FIG. 1 provides a simplified schematic diagram of a software-defined network system including a variety of network devices that generate metadata that can be analyzed in accordance with an embodiment of the present invention
  • FIG. 2 provides a simplified schematic diagram of a software-defined network system including a variety of network devices that generate metadata and a system in accordance with an embodiment of the present invention for managing the network while analyzing such metadata;
  • FIG. 3 provides a simplified schematic diagram of a cloud-based network system including a variety of network devices that generate metadata that can be analyzed in accordance with an embodiment of the present invention;
  • FIG. 4 provides a simplified schematic diagram of a cloud-based network system including a variety of processing modules that cooperate to automate the network while analyzing metadata in accordance with an embodiment of the present invention
  • FIG. 5 provides a somewhat simplified schematic diagram of a software-defined network and cloud-based computing environment, including modules that cooperate to analyze metadata in accordance with an embodiment of the present invention
  • FIG. 6 is a simplified schematic diagram that illustrates an embodiment of the present invention in which short term storage is incorporated in order to provide on-demand NetFlow information;
  • FIG. 7 provides another simplified schematic diagram that illustrates an alternative embodiment of the present invention in which short term storage is incorporated in order to provide on-demand NetFlow information
  • FIG. 8 provides a simplified schematic diagram illustrating an embodiment of the present invention in which botnets may be detected using geo-spatial analysis.
  • the present invention relates to network monitoring and event management. More specifically it relates to processing network metadata obtained as a result of network monitoring activities and subsequent processing of the metadata, which may result in useful information being reported to an event management entity in a timely manner.
  • embodiments of the invention are applicable in contexts other than network metadata processing.
  • the system may receive NetFlow information from the network and output instructions to an OpenFlow Controller.
  • the method and system may be implemented using a NetFlow to Syslog Converter ("NF2SL”) - a software program which enables integrating NetFlow versions 1 through 8, NetFlow v9, jFlow, sflowd, sFlow, NetStream, IPFIX and similar (“NetFlow”) producers with any SIEM system capable of processing syslog.
  • NF2SL NetFlow to Syslog Converter
  • NetFlow NetFlow to Syslog Converter
  • SDN Software Defined Networking
  • a typical implementation of the SDN architecture puts the decision making process on a separate computing device such as a server and leaves packet forwarding to traditional network devices such as switches and routers.
  • communications between the control plane and the data forwarding plane are carried out by means of the OpenFlow protocol 100.
  • This protocol enables a central device, called OpenFlow Controller 101, to direct traffic through one or a plurality of OpenFlow compliant network devices 102 in its domain.
  • the OpenFlow Controller 101 may set up communications paths based on specific characteristics such as fewest number of hops, link bandwidth or latency.
  • the OpenFlow Controller 101 sets up communications paths using a flow table abstraction in which a flow is represented by a collection of packet fields against which each packet traversing a network device is matched.
  • a flow table abstraction in which a flow is represented by a collection of packet fields against which each packet traversing a network device is matched.
  • the OpenFlow Controller 101 makes its decisions based on the OSI Layer 2 (local network connectivity) and OSI Level 3 (routing) network level information.
  • OSI Layer 2 local network connectivity
  • OSI Level 3 routing
  • This deficiency of the OpenFlow Controller 101 could be alleviated by introducing an additional component which digests a higher level of information, such as OSI Layer 7 information (applications) and the users identity, according to a policy or a set of policies set forth by a System Administrator, and directs the OpenFlow Controller 101 how to make lower level network packets forwarding decisions taking into account such higher level information.
  • OSI Layer 7 information applications
  • OSI Layer 7 information applications
  • the OpenFlow Controller 101 could be alleviated by introducing an additional component which digests a higher level of information, such as OSI Layer 7 information (applications) and the users identity, according to a policy or a set of policies set forth by a System Administrator, and directs the OpenFlow Controller 101 how to make lower level network packets forwarding decisions taking into account such higher level information.
  • NFI Server 110 provides higher level information, including but not limited to the OSI Level 7 application-level data that enables the OpenFlow Controller 101 to make more intelligent decisions concerning how to utilize the network.
  • NFI Server 110 processes NetFlow information 111 generated by OpenFlow 100 compliant networking devices 102 and communicates consolidated flow data to the NFI OpenFlow Agent 113 implemented as an application capable of communicating with the OpenFlow Controller 101.
  • the communication between the NFI OpenFlow Agent 113 and the OpenFlow Controller 101 may be implemented by means of the OpenFlow "Northbound" API 114 which supports bidirectional communications between the NFI OpenFlow Agent 113 and the OpenFlow Controller 101.
  • NFI OpenFlow Agent 113 may communicate with a plurality of OpenFlow Controllers 101 and may receive flow related information from a plurality of NFI Servers 110. It is also appreciated that NFI Server 110 may send flow related
  • the NFI OpenFlow Agent 113 receives information about the flows, including but not limited to the OSI Level 7 application information and user identity information from the NFI Server 110 via a protected communications channel 112.
  • the NFI Server 110 receives OSI Level 7 application information in NetFlow messages generated by the network devices 102 and derives user information from the user- identity-aware NetFlow messages such as NetFlow Security Event Log (NSEL) or in the OSI Layer 2 extensions such as Cisco Secure Group Tags (SGT).
  • NSEL NetFlow Security Event Log
  • SGT Cisco Secure Group Tags
  • the OSI Level 7 application information may be supplied by means of a classification such as PANA-L7 accompanied by an application identifier or other similar application classification.
  • the communications channel 112 may be protected by standard cryptographic means such as the SSL/TLS or the DTLS protocol.
  • the NFI OpenFlow Agent 113 is able to retrieve information about the OSI Layer 2 (local network connectivity) and OSI Layer 3 (routing) from the OpenFlow Controller 101 by the means of the OpenFlow "Northbound" API 114. It is appreciated that the NFI OpenFlow Agent 113 may deduce the OSI Layer 2 (local network connectivity) and OSI Layer 3 (routing) information from the flow data received from the NFI Server 110 or by other means.
  • the NFI OpenFlow Agent 113 is able to map the OSI Level 7 application information and the user identity information received from the NFI Server 110 to the policy provided by the system administrator, determine if the state of the network comprised by the network devices 102 satisfies the policy, and instruct the OpenFlow Controller 101 to apply a corrective action if such is required.
  • Exemplary NFI OpenFlow Agent 113 policies could include enforcement of a certain network bandwidth allocated to an application for a certain user or a group as determined by a Cisco SGT associated with the network traffic; enforcement of an SLA for a subnet classified by an IP address prefix or a VLAN tag, and so on.
  • An exemplary policy could be expressed as a numeric threshold, in relative terms (e.g., "group A network bandwidth consumption should not exceed network bandwidth consumption of group B"), or in fuzzy terms (e.g., "if network traffic is low, network bandwidth allocated to group A may be increased”).
  • the policies could be expressed in many forms, for example and without any limitation, as an XML document, in a proprietary format, etc.
  • the policies could be based on the application type derived from the OSI Level 7 application information, user or group identity, user or group role, time of day, etc.
  • the NFI OpenFlow Agent 113 is capable of utilizing NetFlow information received from the NFI Server 110 to monitor the health of the network and report potential faults prior to their happening.
  • a conclusion about an impending network fault could be made by utilizing the NetFlow protocol for measuring average size of a packet traversing a network device interface.
  • a noticeable drop in the average packet size could indicate a higher level of the network packets fragmentation, which typically indicates faulty hardware.
  • the NFI Server 110 may notify the NFI OpenFlow Agent 113 about this event.
  • the NFI OpenFlow Agent 113 may instruct the OpenFlow Controller 101 to take a corrective action by rerouting the traffic around a problematic network device and/or notify the System Administrator about the problem.
  • the NFI Server 110 may forecast a network fault by comparing dispersion of the traffic rate by volume and processed packets against preset or dynamically computed thresholds. Comparison of dispersion of the flow reports arrival time to a computed or a preset threshold could be another NFI Server 110 network fault reporting criteria.
  • network fault threshold values could be computed by means of fuzzy-logic-based algorithms, statistical measurements and other methods and network faults may be predicted using linear prediction algorithms such as autoregressive model, moving average model, or other predictive analytics methods. It is also appreciated that the NFI OpenFlow Agent 113 may make its decisions based on information received from a plurality of NFI Servers 110.
  • a protocol used to control the data plane of the network devices 102 could be other than OpenFlow
  • the API used to communicate with the control plane could be other than the OpenFlow "Northbound" API 114
  • the NFI OpenFlow Agent 113 could be co-located with the control plane or be remote.
  • the NFI OpenFlow Agent 113 could utilize a local programmatic API or interact with the control plane using a network protocol.
  • An obvious benefit of integrating the application level information into the packet forwarding function is the simplicity in which the network administrator could express the network bandwidth utilization policies. This leads to a more optimal use of the existing network resources and increased customer satisfaction due to a better fulfillment of the existing SLA.
  • IaaS Infrastructure as a Service
  • OpenStack is a vendor independent cloud operating system designed to control large groups of computing resources, including servers, storage and networking devices, and manage those resources through a console called an OpenStack Dashboard 120.
  • the OpenStack system could be used by a service provider to manage its IaaS offering or by an organization to manage its own pool of computing resources.
  • OpenStack API 124 OpenStack Compute, OpenStack Object Storage
  • OpenStack Identity Service and OpenStack Image Store
  • the OpenStack API 124 enables cloud operators to provision cloud infrastructure, including virtual machine (VM) instances, storage and identity services, and manipulate Virtualized Devices 125 deployed in a Cloud 123.
  • the OpenStack system provides a number of tools, such as cURL, rest-client, nova, etc., for utilizing the OpenStack system services such as launching a Virtual Device 125, checking Virtual Device 125 status, shutting down a Virtual Device 125, and so on.
  • a robust OpenStack API 124 provides an opportunity to automate the OpenStack-based system provisioning and maintenance by utilizing the NetFlow information 111 reported by the Hardware or Virtual Network Devices 102. Furthermore, NetFlow 111 information reported by VM hypervisors provides a complete insight into the state of Virtualized Devices 125 by the means of the NFI Server 110.
  • the NFI Server 110 processes NetFlow information 111 generated by Hardware or Virtual Network Devices 102 and Virtualized Devices 125 and communicates consolidated flow data to the NFI OpenStack Agent 122 implemented as an application capable of communicating with the OpenStack controlled Virtualized Devices 125 deployed in the Cloud 123.
  • the communication between the NFI OpenStack Agent 122 and the OpenStack controlled Cloud 123 may be implemented by means of the OpenStack API 124 which supports bi-directional communications between the NFI OpenStack Agent 113 and the OpenStack controlled Cloud 123.
  • NFI Server 110 provides network flow information, including but not limited to the OSI Level 7 application-level data that enables the NFI OpenStack Agent 122 to make intelligent decisions how to utilize the Cloud 125 computing resources.
  • the NFI OpenStack Agent 122 receives information about the flows, including but not limited to the OSI Level 7 application information and user identity information from the NFI Server 110 via a protected communications channel 121.
  • the OSI Level 7 application information may be supplied by means of a classification such as PANA-L7 accompanied by an application identifier or other similar application classification.
  • the communications channel 121 may be protected by standard cryptographic means such as the SSL/TLS or the DTLS protocol.
  • the NFI Server 110 receives OSI Level 7 application information in NetFlow messages generated by the network devices 102 and derives user information from the user identity aware NetFlow messages such as NetFlow Security Event Log (NSEL) or in the OSI Layer 2 extensions such as Cisco Secure Group Tags (SGT).
  • NSEL NetFlow Security Event Log
  • SGT Cisco Secure Group Tags
  • System Administrator configures policies for
  • Virtualized Devices 125 provisioning and maintenance on the NFI OpenStack Agent 122 The policies could be expressed, without any limitation, as an XML document, in a proprietary format, etc. The policies could be based on the application type derived from the OSI Level 7 application information, user or group identity, user or group role, time of day, etc.
  • An exemplary policy configured by the System Administrator on the NFI OpenStack Agent 122 could be creating additional Virtualized Devices 125 when a demand for a particular application increases, provisioning additional resources to the existing Virtualized Devices 125, migration of existing Virtualized Devices 125 to more powerful hardware within the Cloud 123, shutting down idle Virtualized Devices 125, etc.
  • the NFI OpenStack Agent 122 is able to automate Cloud 123 management, thus reducing the cloud provider's or cloud owner's operational costs and improving utilization of the physical hardware resources.
  • OpenStack is an example of a cloud operating system and the method disclosed herein is applicable to any vendor specific or generic cloud operating system.
  • NFI for Virtualized Environment
  • NFI Server combined with NFI OpenFlow Agent and NFI OpenStack Agent becomes a linchpin of an integrated virtualized environment which includes an OpenFlow-based software defined network and an OpenStack driven cloud infrastructure.
  • Fig. 5 illustrates the NFI Server 110 application to an integrated setting which includes software defined networking and a cloud computing environment.
  • the NFI Server 110 processes NetFlow information 111 generated by Hardware or Virtual Network Devices 102 and Virtualized Devices 125 and communicates consolidated flow data to the NFI OpenStack Agent 122 implemented as an application capable of communicating with the OpenStack controlled Virtualized Devices 125 deployed in the Cloud 123.
  • the communication between the NFI OpenStack Agent 122 and the OpenStack controlled Cloud 123 may be implemented by means of the OpenStack API 124 which supports bi-directional communications between the NFI OpenStack Agent 113 and the OpenStack controlled Cloud 123.
  • NFI Server 110 processes NetFlow information 111 generated by OpenFlow compliant networking devices 102 and Virtualized Devices 125 and communicates consolidated flow data to the NFI OpenFlow Agent 113 implemented as an application capable of communicating with the OpenFlow Controller 101.
  • the communication between the NFI OpenFlow Agent 113 and the OpenFlow Controller 101 may be implemented by means of the OpenFlow "Northbound" API 114 which supports bi-directional communications between the NFI OpenFlow Agent 113 and the OpenFlow Controller 101.
  • the NFI Server 110 may interact with a plurality of Clouds 123 and a plurality of OpenFlow Controllers 101.
  • a protocol other than OpenFlow may be utilized and an API other than OpenStack may be employed for controlling virtualized computing resources.
  • Flow information data is notoriously voluminous: a single mid-range router like Cisco ASR1000 is capable of producing 400,000 NetFlow records per second which results in around 1.6TB of data per day. Due to a high rate and volume of data, many of the NFI policies are designed to consolidate and/or filter the data and report only a greatly reduced volume of essential information to a backend system, such as without limitation, a SIEM system.
  • NFI Network-to-Network Interface
  • the backend system may need more information about the conditions which preceded the event in question and conditions immediately after the event.
  • the backend system may be in a much better position to determine the scope and the consequences of the observed event.
  • configuration change event can signify an impersonation attack.
  • An embodiment of the NFI on-demand flow information mechanism disclosed in this invention enables the SIEM system to receive information required for correlating network information with other machine data a posteriori without the need of constantly processing all of what may be a huge flow of inbound network data.
  • NFI Server 110 receives NetFlow data 111 from one or a plurality of network devices.
  • NFI Server 110 processes NetFlow data 111 and reports Consolidated NetFlow data 142 to a SIEM System 140 in a format understood by the SIEM System 140.
  • NFI Server 110 propagates received NetFlow data 111 to the Short Term Storage 145 where the NetFlow data 111 is placed into the leftmost Time Window 144.
  • Short Term Storage 145 is a repository with a small access time, possibly in RAM, on SSD or some other fast and/or local storage device.
  • Short Term Storage 145 may be split into a configurable number of sections, e.g., Time Windows 144, each of which contains NetFlow data 111 information received over a configurable period At.
  • Short Term Storage 145 generally implements a sliding window schema in which after each period At the right-most Time Window 144 in an augmented NetFlow format 143 is forwarded to the Long Term Storage 146, the Short Term Storage 145 logically shifts and new left-most Time Window 144 is created for storing the incoming NetFlow data 111 information.
  • the Long Term Storage 146 generally has an access time and storage capacity that is greater or equal to the Short Term Storage 145 access time and storage capacity.
  • augmented NetFlow format 143 may be the same as the original NetFlow data 1 11 or may contain additional mark up information for use in the long term storage.
  • SIEM system 140 may execute a Set of Policies 150 which consume Consolidated NetFlow data 142 supplied by NFI Server 110 and, optionally, Other Machine Data 153. If in the process of execution of a policy from the Set of Policies 150, SIEM system 140 detects an Event 151 which took place at time T, SIEM system 140 can issue a Request 152 to the NFI Server 110 to provide additional NetFlow 111 data received by the NFI Server 110 during a time interval [T - 1, T + 1], where t is the interval half-width selected by the SIEM system 140.
  • the NFI Server 110 Upon receiving the SIEM system 140 Request 152, the NFI Server 110 determines location of the requested information in the storage based on the beginning time and the ending time of the requested time interval [T - 1, T + 1]. Assuming that at the time of Request 152 the Short Term Storage 145 contains NetFlow 111 data corresponding to the time interval [Tl, T2], T2 > Tl, and the requested time interval [T - 1, T + t] is within the Short Term Storage 145 time interval [Tl, T2], then the NFI Server 110 retrieves requested information from the Short Term Storage 145 and forwards 156 the retrieved information, optionally with additional processing, to the SIEM system 140.
  • the NFI Server 110 attempts to retrieve the requested information from the Long Term Storage 146 and if successful, upon optionally with additional processing, forwards the retrieved information in Response 156, to the SIEM system 140.
  • the NFI Server 110 retrieves first part of the requested information from the Short Term Storage 146 and the second part of the requested information from the Long Term Storage 146, concatenates the first retrieved part and the second retrieved part of information and forwards the concatenated information, optionally with additional processing, in Response 156 to the SIEM system 140.
  • the NFI Server 110 retrieves information for a truncated time range and notifies the SIEM system about the truncation in Response 156.
  • the NFI server 110 In the case in which the requested time interval [ T - 1, T + t ] is outside of the time range covered by the Short Term Storage 145 and the Long Term Storage 146, the NFI server 110 notifies the SIEM system about the error condition in Response 156.
  • the novel multi-tiered approach to storing NetFlow data disclosed herein provides a significant advantage when analyzing events which require immediate reporting or action as compared to the traditional single tiered NetFlow information storage used by prior NetFlow collectors. For the events which require immediate reporting or action, search for the requested information in the fast Short Term Storage 145 is significantly faster than in a slower Long Term Storage 146 which results in a better response time of the SIEM system 140.
  • the SIEM system 140 request 152 for additional information may include, besides specifying the time interval, other parameters such as the origin of the NetFlow record, specific flow information, such as, without limitation, a source or destination IP addresses, or a combination thereof. It is also appreciated that NetFlow information in the Short Term Storage 145 and the Long Term Storage 146 may be indexed by time and by zero, one or a plurality of keys based on the information pertinent to the NetFlow such as without limitation, source or destination IP addresses, source or destination OSI Layer 4 ports, and so on.
  • the Short Term Storage 145 and the Long Term Storage 146 may be operated by the NFI Server 110, by an instance of the NFI Server 110 other than the instance of the NFI Server 110 which originally processed NetFlow data 111, and/or by a process other than an NFI Server 110. It is also appreciated that the Short Term Storage 145 and the Long Term Storage 146 may be operated by different instances of the NFI Server 110 or by a process other than the NFI Server 110. Furthermore, the access time to the Short Term Storage 145 and the Long Term Storage 146 may be same and there may be a plurality of more than two storage tiers. It is also
  • the Long Term Storage 146 is an optional component and the information in the Short Term Storage 145 may be discarded when it ages past a configured life span.
  • a novel approach to associating network and other machine data disclosed herein enables detection of attacks which would be undetected if only the network or other machine data is taken into consideration.
  • a novel approach to the network information storage disclosed here enables provision of the network information on the "only when needed" basis without any preliminary processing.
  • Sophisticated malware agents engage complex evasion detection techniques when communicating with their masters. For example, an agent can contact the master at random time intervals, communicate with multiple masters by selecting next master based on information received during last communication session, obfuscate Command & Control channel traffic patterns, etc. [00108] Method
  • BIRCH BitTorrent - Balanced Iterative Reducing and Clustering using Hierarchies
  • dis and "az” are computed based on the source and destination IP addresses found in the flow record. Similarity function, "freq”, is frequency of communications to a particular geographic area. Applications are classified into groups each of which is associated with a category assigned to a monitored host ("standard applications").
  • Topology complexity increases configuration complexity and makes it more error prone.
  • tools which help a System Administrator to assess and validate configuration and security posture of the networks under his management. These tools use a variety of methods to determine vulnerabilities in the network. For example, penetration testing tools "attack" an organization's firewalls, configuration verification tools attempt to find loopholes in the authentication and authorization policies, IDS/IPS systems watch the traffic flowing in and out of an organization's network, and so on. These protective technologies were developed over a long period and are mature enough to stop known and sometimes even unpredicted threats.
  • a problem with the today's network defensive posture is its static nature: once configured, and possibly verified, network defenses are considered impregnable like the Maginot Line was before the World War II.
  • the protective measures are generally applied once, or at best, are assessed once in a while, leaving the organization without any quality assurance of the real security posture state in between the checks.
  • NetFlow is a technology which enables creation of the tools capable of providing dynamic quality control of the organization's networking infrastructure.
  • NFI technology disclosed in this invention, allows introducing arbitrary policies which could monitor network traffic throughout the organization and identify flow instances which were overseen by statically configured defenses.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

La présente invention concerne un procédé et un système améliorés de traitement de métadonnées de réseau. Les métadonnées de réseau peuvent être traitées par des modules logiciels exécutables instanciés dynamiquement qui prennent des décisions basées sur une politique à propos du caractère des métadonnées de réseau et de leur présentation aux utilisateurs des informations acheminées par les métadonnées de réseau. Les métadonnées de réseau peuvent être classées par type et chaque sous-catégorie d'un type peut être mappée par rapport à une définition par une valeur d'empreinte unique. La valeur d'empreinte peut être utilisée pour comparer les sous-catégories de métadonnées de réseau avec des politiques et des règles de transformation pertinentes. Pour les métadonnées de réseau basées sur des modèles comme NetFlow v9, un mode de réalisation de l'invention est capable de surveiller en continu le trafic réseau de modèles inconnus, de capturer des définitions de modèles, et d'informer les administrateurs au sujet des modèles pour lesquels aucune politique ni règle de conversion n'existent. Des modules de conversion peuvent convertir efficacement des types et/ou des sous-catégories sélectionnés de métadonnées de réseau en formats de métadonnées alternatifs.
PCT/US2014/010932 2013-01-10 2014-01-09 Procédé de streaming amélioré et système de traitement de métadonnées de réseau WO2014110293A1 (fr)

Priority Applications (5)

Application Number Priority Date Filing Date Title
CN201480012616.9A CN105051696A (zh) 2013-01-10 2014-01-09 用于处理网络元数据的改进的流式处理方法及系统
CA2897664A CA2897664A1 (fr) 2013-01-10 2014-01-09 Procede de streaming ameliore et systeme de traitement de metadonnees de reseau
KR1020157021506A KR20150105436A (ko) 2013-01-10 2014-01-09 향상된 스트리밍 방법과 네트워크 메타데이터를 처리하기 위한 시스템
JP2015552783A JP2016508353A (ja) 2013-01-10 2014-01-09 ネットワークメタデータを処理する改良されたストリーミング方法およびシステム
RU2015132628A RU2015132628A (ru) 2013-01-10 2014-01-09 Усовершенствованный потоковый способ и система обработки сетевых метаданных

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201361751243P 2013-01-10 2013-01-10
US61/751,243 2013-01-10
US13/830,924 2013-03-14
US13/830,924 US20140075557A1 (en) 2012-09-11 2013-03-14 Streaming Method and System for Processing Network Metadata

Publications (1)

Publication Number Publication Date
WO2014110293A1 true WO2014110293A1 (fr) 2014-07-17

Family

ID=51167380

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2014/010932 WO2014110293A1 (fr) 2013-01-10 2014-01-09 Procédé de streaming amélioré et système de traitement de métadonnées de réseau

Country Status (6)

Country Link
JP (1) JP2016508353A (fr)
KR (1) KR20150105436A (fr)
CN (1) CN105051696A (fr)
CA (1) CA2897664A1 (fr)
RU (1) RU2015132628A (fr)
WO (1) WO2014110293A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3300302A4 (fr) * 2015-06-29 2018-05-23 Huawei Technologies Co., Ltd. Procédé de mise en uvre d'application, et contrôleur de service
CN113507461A (zh) * 2021-07-01 2021-10-15 交通运输信息安全中心有限公司 基于大数据的网络监控系统及网络监控方法
US11425003B2 (en) 2017-08-03 2022-08-23 Drivenets Ltd. Network aware element and a method for using same

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10936966B2 (en) 2016-02-23 2021-03-02 At&T Intellectual Property I, L.P. Agent for learning and optimization execution
CN107665224B (zh) * 2016-07-29 2021-04-30 北京京东尚科信息技术有限公司 扫描hdfs冷数据的方法、系统和装置
US20180351806A1 (en) * 2017-05-31 2018-12-06 Cisco Technology, Inc. Intent specification checks for inconsistencies
CN107248959B (zh) * 2017-06-30 2020-07-24 联想(北京)有限公司 一种流量优化方法及装置
KR102045844B1 (ko) * 2018-04-18 2019-11-18 한국전자통신연구원 클라우드 시스템의 플로우 기반 트래픽 분석 방법 및 장치
CN111292523B (zh) * 2018-12-06 2023-04-07 中国信息通信科技集团有限公司 网络智能体系统
CN110417680A (zh) * 2019-08-16 2019-11-05 北京伏羲车联信息科技有限公司 车载网络流式数据优化方法及装置
JP7294764B2 (ja) * 2019-12-05 2023-06-20 日本電信電話株式会社 フォーマット変換装置及び方法並びにプログラム
RU2738337C1 (ru) * 2020-04-30 2020-12-11 Общество С Ограниченной Ответственностью "Группа Айби" Система и способ обнаружения интеллектуальных ботов и защиты от них
CN112256938B (zh) * 2020-12-23 2021-03-19 畅捷通信息技术股份有限公司 一种消息元数据的处理方法、装置、介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020083175A1 (en) * 2000-10-17 2002-06-27 Wanwall, Inc. (A Delaware Corporation) Methods and apparatus for protecting against overload conditions on nodes of a distributed network
US7633944B1 (en) * 2006-05-12 2009-12-15 Juniper Networks, Inc. Managing timeouts for dynamic flow capture and monitoring of packet flows
US20100071065A1 (en) * 2008-09-18 2010-03-18 Alcatel Lucent Infiltration of malware communications
US20110004876A1 (en) * 2009-07-01 2011-01-06 Riverbed Technology, Inc. Network Traffic Processing Pipeline for Virtual Machines in a Network Device

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9110976B2 (en) * 2010-10-15 2015-08-18 International Business Machines Corporation Supporting compliance in a cloud environment
CN101977146B (zh) * 2010-10-25 2013-04-17 成都飞鱼星科技开发有限公司 一种网络流量智能控制器及其实现方法
US8971196B2 (en) * 2011-03-08 2015-03-03 Riverbed Technology, Inc. Distributed network traffic data collection and storage

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020083175A1 (en) * 2000-10-17 2002-06-27 Wanwall, Inc. (A Delaware Corporation) Methods and apparatus for protecting against overload conditions on nodes of a distributed network
US7633944B1 (en) * 2006-05-12 2009-12-15 Juniper Networks, Inc. Managing timeouts for dynamic flow capture and monitoring of packet flows
US20100071065A1 (en) * 2008-09-18 2010-03-18 Alcatel Lucent Infiltration of malware communications
US20110004876A1 (en) * 2009-07-01 2011-01-06 Riverbed Technology, Inc. Network Traffic Processing Pipeline for Virtual Machines in a Network Device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
OPEN NETWORK FOUNDATION.: "Software-Defined Networking: The New Norm for Networks.", 13 April 2012 (2012-04-13), Retrieved from the Internet <URL:https:/www.opennetworking.org/images/stories/downloads/sdn-resourcis/white-papers/wp-sdn-newnorm.pdf> [retrieved on 20140520] *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3300302A4 (fr) * 2015-06-29 2018-05-23 Huawei Technologies Co., Ltd. Procédé de mise en uvre d'application, et contrôleur de service
JP2018521583A (ja) * 2015-06-29 2018-08-02 華為技術有限公司Ｈｕａｗｅｉ Ｔｅｃｈｎｏｌｏｇｉｅｓ Ｃｏ．，Ｌｔｄ． アプリケーション実装方法およびサービスコントローラ
EP3739847A1 (fr) * 2015-06-29 2020-11-18 Huawei Technologies Co. Ltd. Procédé de mise en uvre d'application et contrôleur de services
CN112073214A (zh) * 2015-06-29 2020-12-11 华为技术有限公司 一种实现应用的方法及业务控制器
CN112073215A (zh) * 2015-06-29 2020-12-11 华为技术有限公司 一种实现应用的方法及业务控制器
US11425003B2 (en) 2017-08-03 2022-08-23 Drivenets Ltd. Network aware element and a method for using same
CN113507461A (zh) * 2021-07-01 2021-10-15 交通运输信息安全中心有限公司 基于大数据的网络监控系统及网络监控方法
CN113507461B (zh) * 2021-07-01 2022-11-29 交通运输信息安全中心有限公司 基于大数据的网络监控系统及网络监控方法

Also Published As

Publication number Publication date
CA2897664A1 (fr) 2014-07-17
RU2015132628A (ru) 2017-02-15
KR20150105436A (ko) 2015-09-16
CN105051696A (zh) 2015-11-11
JP2016508353A (ja) 2016-03-17

Similar Documents

Publication Publication Date Title
US9860154B2 (en) Streaming method and system for processing network metadata
US10892964B2 (en) Systems and methods for monitoring digital user experience
US10728117B1 (en) Systems and methods for improving digital user experience
US10938686B2 (en) Systems and methods for analyzing digital user experience
US10079843B2 (en) Streaming method and system for processing network metadata
US11431550B2 (en) System and method for network incident remediation recommendations
WO2014110293A1 (fr) Procédé de streaming amélioré et système de traitement de métadonnées de réseau
Lin et al. A survey on network security-related data collection technologies
Fawcett et al. Tennison: A distributed SDN framework for scalable network security
EP3699766A1 (fr) Systèmes et procédés de surveillance, d&#39;analyse et d&#39;amélioration de l&#39;expérience numérique de l&#39;utilisateur
US10154053B2 (en) Method and apparatus for grouping features into bins with selected bin boundaries for use in anomaly detection
US20160359695A1 (en) Network behavior data collection and analytics for anomaly detection
US10355949B2 (en) Behavioral network intelligence system and method thereof
US8955091B2 (en) Systems and methods for integrating cloud services with information management systems
US11546266B2 (en) Correlating discarded network traffic with network policy events through augmented flow
Krishnan et al. OpenStackDP: a scalable network security framework for SDN-based OpenStack cloud infrastructure
US11343143B2 (en) Using a flow database to automatically configure network traffic visibility systems
US10296744B1 (en) Escalated inspection of traffic via SDN
Xu Network Behavior Analysis
KR20190055534A (ko) 심층 패킷 분석을 이용한 기계학습용 네트워크 데이터 생성 장치 및 그 동작방법
Ghosh et al. Managing high volume data for network attack detection using real-time flow filtering
Saravanan et al. An Investigation on Neuro-Fuzzy Based Alert Clustering for Statistical Anomaly of Attack Detection
ANALYTICS et al. PH. D. THESIS IN

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 201480012616.9

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14737995

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2897664

Country of ref document: CA

ENP Entry into the national phase

Ref document number: 2015552783

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 20157021506

Country of ref document: KR

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2015132628

Country of ref document: RU

Kind code of ref document: A

122 Ep: pct application non-entry in european phase

Ref document number: 14737995

Country of ref document: EP

Kind code of ref document: A1