US20160127180A1 - Streamlining configuration of protocol-based network data capture by remote capture agents - Google Patents

Streamlining configuration of protocol-based network data capture by remote capture agents Download PDF

Info

Publication number
US20160127180A1
US20160127180A1 US14/528,932 US201414528932A US2016127180A1 US 20160127180 A1 US20160127180 A1 US 20160127180A1 US 201414528932 A US201414528932 A US 201414528932A US 2016127180 A1 US2016127180 A1 US 2016127180A1
Authority
US
United States
Prior art keywords
event
user
set
data
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US14/528,932
Inventor
Vladimir A. Shcherbakov
Michael R. Dickey
Cary Glen Noel
Kishore R. Ramasayam
Mignon L. Belongie
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Splunk Inc
Original Assignee
Splunk Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Splunk Inc filed Critical Splunk Inc
Priority to US14/528,932 priority Critical patent/US20160127180A1/en
Assigned to SPLUNK INC. reassignment SPLUNK INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: NOEL, CARY GLEN, BELONGIE, MIGNON L., DICKEY, MICHAEL R., RAMASAYAM, KISHORE R., SCHERBAKOV, VLADIMIR A.
Publication of US20160127180A1 publication Critical patent/US20160127180A1/en
Application status is Abandoned legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance or administration or management of packet switching networks
    • H04L41/08Configuration management of network or network elements
    • H04L41/0803Configuration setting of network or network elements
    • H04L41/0813Changing of configuration
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/0482Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance interaction with lists of selectable items, e.g. menus
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04817Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance using icons
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object or an image, setting a parameter value or selecting a range
    • G06F3/04842Selection of a displayed object
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance or administration or management of packet switching networks
    • H04L41/22Arrangements for maintenance or administration or management of packet switching networks using GUI [Graphical User Interface]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing packet switching networks
    • H04L43/12Arrangements for monitoring or testing packet switching networks using dedicated network monitoring probes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance or administration or management of packet switching networks
    • H04L41/06Arrangements for maintenance or administration or management of packet switching networks involving management of faults or events or alarms
    • H04L41/0604Alarm or event filtering, e.g. for reduction of information
    • H04L41/0613Alarm or event filtering, e.g. for reduction of information based on the type or category of the network elements
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance or administration or management of packet switching networks
    • H04L41/06Arrangements for maintenance or administration or management of packet switching networks involving management of faults or events or alarms
    • H04L41/0604Alarm or event filtering, e.g. for reduction of information
    • H04L41/0622Alarm or event filtering, e.g. for reduction of information based on time
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance or administration or management of packet switching networks
    • H04L41/06Arrangements for maintenance or administration or management of packet switching networks involving management of faults or events or alarms
    • H04L41/0681Arrangements for maintenance or administration or management of packet switching networks involving management of faults or events or alarms involving configuration of triggering conditions
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance or administration or management of packet switching networks
    • H04L41/06Arrangements for maintenance or administration or management of packet switching networks involving management of faults or events or alarms
    • H04L41/0686Arrangements for maintenance or administration or management of packet switching networks involving management of faults or events or alarms involving notification enrichment
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing packet switching networks
    • H04L43/02Arrangements for monitoring or testing packet switching networks involving a reduction of monitoring data
    • H04L43/026Arrangements for monitoring or testing packet switching networks involving a reduction of monitoring data using flow generation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing packet switching networks
    • H04L43/04Processing of captured monitoring data
    • H04L43/045Processing of captured monitoring data for graphical visualization of monitoring data

Abstract

The disclosed embodiments provide a system that facilitates the processing of network data. During operation, the system provides a graphical user interface (GUI) for obtaining configuration information for configuring the generation of time-series event data from network packets captured by one or more remote capture agents. Next, the system provides, in the GUI, a first set of user-interface elements for including one or more event attributes in the time-series event data of an event stream associated with a protocol classification of the network packets. The system then includes the one or more event attributes specified through the first set of user-interface elements in the configuration information.

Description

    RELATED APPLICATION
  • The subject matter of this application is related to the subject matter in a co-pending non-provisional application by inventor Michael R. Dickey, entitled “Distributed Processing of Network Data Using Remote Capture Agents,” having Ser. No. 14/253,713, and filing date 15 Apr. 2014 (Attorney Docket No. SPLK14-1006).
  • The subject matter of this application is also related to the subject matter in a co-pending non-provisional application by inventors Vladimir A. Shcherbakov and Michael R. Dickey and filed on the same day as the instant application, entitled “Protocol-Based Capture of Network Data Using Remote Capture Agents,” having Ser. No. TO BE ASSIGNED, and filed 30 Oct. 2014 (Attorney Docket No. SPLK14-1021).
  • The subject matter of this application is also related to the subject matter in a co-pending non-provisional application by inventors Vijay Chauhan, Devendra M. Badhani, Luke K. Murphey and David Hazekamp and filed on the same day as the instant application, entitled “Capture Triggers for Capturing Network Data,” having Ser. No. TO BE ASSIGNED, and filed 30 Oct. 2014 (Attorney Docket No. SPLK14-1025).
  • BACKGROUND
  • 1. Field
  • The disclosed embodiments relate to techniques for processing network data. More specifically, the disclosed embodiments relate to techniques for streamlining the configuration of protocol-specific event streams for network data capture and processing.
  • 2. Related Art
  • Over the past decade, the age of virtualization has triggered a sea change in the world of network data capture. Almost every network capture product available today is a physical hardware appliance that customers have to purchase and configure. In addition, most network data capture technologies are built from scratch to serve a specific purpose and address the needs of a particular vertical market. For example, network capture systems may be customized to extract data for security and intrusion-detection purposes, collect network performance data, perform Quality of Service (QoS), redirect data, block network traffic, and/or perform other analysis or management of network traffic. Such targeted and/or fixed implementation and use of network capture technologies may preclude modification of the network capture technologies to address different and changing business needs.
  • Moreover, customers using conventional hardware-based network capture devices typically connect the devices to other hardware devices in a network. The connections may allow the network capture devices to access the network and monitor network traffic between two or more points in the network. Examples of such devices include a network Test Access Point (TAP) or Switched Port Analyzer (SPAN) port. After the network traffic is captured, cumbersome Extraction, Transform, and Load (“ETL”) processes may be performed to filter, transform, and/or aggregate data from the network traffic and enable the extraction of business value from the data.
  • However, customers are moving away from managing physical servers and data centers and toward public and private cloud computing environments that provide software, hardware, infrastructure, and/or platform resources as hosted services using computing, storage, and/or network devices at remote locations. For these customers, it is either impossible, or at best extremely challenging, to deploy physical network capture devices and infrastructure in the cloud computing environments.
  • Consequently, network data capture may be facilitated by mechanisms for streamlining the deployment and configuration of network capture technology at distributed and/or remote locations.
  • SUMMARY
  • The disclosed embodiments provide a system that facilitates the processing of network data. During operation, the system provides a graphical user interface (GUI) for obtaining configuration information for configuring the generation of time-series event data from network packets captured by one or more remote capture agents. Next, the system provides, in the GUI, a first set of user-interface elements for including one or more event attributes in the time-series event data of an event stream associated with a protocol classification of the network packets. The system then includes the one or more event attributes specified through the first set of user-interface elements in the configuration information.
  • In some embodiments, the system also provides the configuration information over a network to the one or more remote capture agents, wherein the configuration information is used to configure the generation of the time-series event data at the one or more remote capture agents during runtime of the one or more remote capture agents.
  • In some embodiments, the system also provides, in the GUI, a second set of user-interface elements for managing the event stream, and obtains the protocol classification for the event stream from the second set of user-interface elements.
  • In some embodiments, the GUI comprises a second set of user-interface elements for managing the event stream, and managing the event stream includes at least one of:
  • (i) cloning the event stream from an existing event stream;
  • (ii) deleting the event stream;
  • (iii) enabling the event stream; and
  • (iv) disabling the event stream.
  • In some embodiments, the system also provides, in the GUI, a second set of user-interface elements for filtering the network packets prior to generating the time-series event data from the network packets.
  • In some embodiments, the GUI includes a second set of user-interface elements for filtering the network packets, and filtering the network packets is associated with at least one of:
  • (i) a Boolean value;
  • (ii) a numeric comparison;
  • (iii) a definition;
  • (iv) a regular expression;
  • (v) an exact match;
  • (vi) a partial match; and
  • (vii) an ordering.
  • In some embodiments, the GUI includes a second set of user-interface elements for filtering the network packets, and the second set of user-interface elements is used to apply a logical disjunction or a logical conjunction to a set of filters for filtering the network packets.
  • In some embodiments, the GUI includes a second set of user-interface elements for filtering the network packets, and the second set of user-interface elements is used to match a filter to any or all elements of a multi-value event attribute in the event stream.
  • In some embodiments, the system also provides, in the GUI, a second set of user-interface elements for aggregating the one or more event attributes into aggregated event data that is included in the event stream.
  • In some embodiments, the GUI includes a second set of user-interface elements for aggregating the one or more event attributes into aggregated event data, and the second set of user-interface elements comprises a user-interface element for identifying an event attribute as a key attribute used to generate a key representing the aggregated event data, or an aggregation attribute to be aggregated prior to inclusion in the aggregated event data.
  • In some embodiments, the GUI includes a second set of user-interface elements for aggregating the one or more event attributes into aggregated event data, and the second set of user-interface elements includes a first user-interface element for identifying an event attribute as a key attribute used to generate a key representing the aggregated event data or an aggregation attribute to be aggregated prior to inclusion in the aggregated event data. The second set of user-interface elements also includes a second user-interface element for obtaining an aggregation interval over which the one or more event attributes are aggregated into the aggregated event data.
  • In some embodiments, the protocol classification includes at least one of a transport layer protocol, a session layer protocol, a presentation layer protocol, and an application layer protocol.
  • BRIEF DESCRIPTION OF THE FIGURES
  • FIG. 1 shows a schematic of a system in accordance with the disclosed embodiments.
  • FIG. 2A shows a remote capture agent in accordance with the disclosed embodiments.
  • FIG. 2B shows the protocol-based capture of network data using a remote capture agent in accordance with the disclosed embodiments.
  • FIG. 3 shows a configuration server in accordance with the disclosed embodiments.
  • FIG. 4A shows an exemplary screenshot in accordance with the disclosed embodiments.
  • FIG. 4B shows an exemplary screenshot in accordance with the disclosed embodiments.
  • FIG. 4C shows an exemplary screenshot in accordance with the disclosed embodiments.
  • FIG. 4D shows an exemplary screenshot in accordance with the disclosed embodiments.
  • FIG. 4E shows an exemplary screenshot in accordance with the disclosed embodiments.
  • FIG. 4F shows an exemplary screenshot in accordance with the disclosed embodiments.
  • FIG. 5A shows an exemplary screenshot in accordance with the disclosed embodiments.
  • FIG. 5B shows an exemplary screenshot in accordance with the disclosed embodiments.
  • FIG. 6 shows a flowchart illustrating the processing of network data in accordance with the disclosed embodiments.
  • FIG. 7 shows a flowchart illustrating the process of using configuration information associated with a protocol classification to build an event stream from a packet flow in accordance with the disclosed embodiments.
  • FIG. 8 shows a flowchart illustrating the process of facilitating the processing of network data in accordance with the disclosed embodiments.
  • FIG. 9 shows a flowchart illustrating the process of facilitating the processing of network data in accordance with the disclosed embodiments.
  • FIG. 10 shows a computer system in accordance with the disclosed embodiments.
  • In the figures, like reference numerals refer to the same figure elements.
  • DETAILED DESCRIPTION
  • The following description is presented to enable any person skilled in the art to make and use the embodiments, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Thus, the present invention is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.
  • The data structures and code described in this detailed description are typically stored on a computer-readable storage medium, which may be any device or medium that can store code and/or data for use by a computer system. The computer-readable storage medium includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs), DVDs (digital versatile discs or digital video discs), or other media capable of storing code and/or data now known or later developed.
  • The methods and processes described in the detailed description section can be embodied as code and/or data, which can be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and/or data stored on the computer-readable storage medium, the computer system performs the methods and processes embodied as data structures and code and stored within the computer-readable storage medium.
  • Furthermore, methods and processes described herein can be included in hardware modules or apparatus. These modules or apparatus may include, but are not limited to, an application-specific integrated circuit (ASIC) chip, a field-programmable gate array (FPGA), a dedicated or shared processor that executes a particular software module or a piece of code at a particular time, and/or other programmable-logic devices now known or later developed. When the hardware modules or apparatus are activated, they perform the methods and processes included within them.
  • The disclosed embodiments provide a method and system for facilitating the processing of network data. As shown in FIG. 1, the network data may be captured using a data-processing system 100 in a distributed network environment. In the illustrated embodiment, system 100 includes a set of configuration servers 120 in communication with a set of remote capture agents 151-153 over one or more networks 190.
  • Although system 100 only depicts three configuration servers 120 and three remote capture agents 151-153, any number of configuration servers 120 and/or remote capture agents 151-153 may be configured to operate and/or communicate with one another within the data-processing system. For example, a single physical and/or virtual server may perform the functions of configuration servers 120. Alternatively, multiple physical and/or virtual servers or network elements may be logically connected to provide the functionality of configuration servers 120. The configuration server(s) may direct the activity of multiple distributed remote capture agents 151-153 installed on various client computing devices across one or more networks. In turn, remote capture agents 151-153 may be used to capture network data from multiple remote network data sources.
  • Further, embodiments described herein can be configured to capture network data in a cloud-based environment, such as cloud 140 depicted in the illustrated embodiment, and to generate events such as timestamped records of network activity from the network data. Remote capture agents 151-153 may capture network data originating from numerous distributed network servers, whether they are physical hardware servers or virtual machines running in cloud 140. In cloud-based implementations, remote capture agents 151-153 will generally only have access to information that is communicated to and received from machines running in the cloud-based environment. This is because, in a cloud environment, there is generally no access to any of the physical network infrastructure, as cloud computing may utilize a “hosted services” delivery model where the physical network infrastructure is typically managed by a third party.
  • Embodiments further include the capability to separate the data capture technology into a standalone component that can be installed directly on client servers, which may be physical servers or virtual machines residing on a cloud-based network (e.g., cloud 140), and used to capture and generate events for all network traffic that is transmitted in and out of the client servers. This eliminates the need to deploy and connect physical hardware to network TAPS or SPAN ports, thus allowing users to configure and change their data capture configuration on-the-fly rather than in fixed formats.
  • In the illustrated embodiment, remote capture agents 152-153 are in communication with network servers 130 residing in cloud 140, and remote capture agent 151 is located in cloud 140. Cloud 140 may represent any number of public and private clouds, and is not limited to any particular cloud configuration. Network servers 130 residing in cloud 140 may be physical servers and/or virtual machines in cloud 140, and network traffic to and from network servers 130 may be monitored by remote capture agent 151 and/or other remote capture agents connected to network servers 130. Further, remote capture agents 152-153 may also run in cloud 140 on physical servers and/or virtual machines. Those skilled in the art will appreciate that any number of remote capture agents may be included inside or outside of cloud 140.
  • Remote capture agents 151-153 may analyze network packets received from the networks(s) to which remote capture agents 151-153 are connected to obtain network data from the network packets and generate a number of events from the network data. For example, each remote capture agent 151-153 may listen for network traffic on network interfaces available to the remote capture agent. Network packets transmitted to and/or from the network interfaces may be intercepted by the remote capture agent and analyzed, and relevant network data from the network packets may be used by the remote capture agent to create events related to the network data. Such events may be generated by aggregating network data from multiple network packets, or each event may be generated using the contents of only one network packet. A sequence of events from a remote capture agent may then be included in one or more event streams that are provided to other components of system 100.
  • Configuration servers 120, data storage servers 135, and/or other network components may receive event data (e.g., event streams) from remote capture agents 151-153 and further process the event data before the event data is stored by data storage servers 135. In the illustrated embodiment, configuration servers 120 may transmit event data to data storage servers 135 over a network 101 such as a local area network (LAN), wide area network (WAN), personal area network (PAN), virtual private network, intranet, mobile phone network (e.g., a cellular network), WiFi network, Ethernet network, and/or other type of network that enables communication among computing devices. The event data may be received over a network (e.g., network 101, network 190) at one or more event indexers (see FIG. 10) associated with data storage servers 135.
  • In addition, system 100 may include functionality to determine the types of network data collected and/or processed by each remote capture agent 151-153 to avoid data duplication at the indexers, data storage servers 135, and/or other components of system 100. For example, remote capture agents 152-153 may process network traffic from the same network. However, remote capture agent 152 may generate page view events from the network traffic, and remote capture agent 153 may generate request events (e.g., of HyperText Transfer Protocol (HTTP) requests and responses) from the network traffic.
  • In one or more embodiments, configuration servers 120 include configuration information that is used to configure the creation of events from network data on remote capture agents 151-153. In addition, such configuration may occur dynamically during event processing (e.g., at runtime). Conversely, because most conventional network capture technologies target specific end uses, they have been designed to operate in a fixed way and generally cannot be modified dynamically or easily to address different and changing business needs.
  • At least certain embodiments described herein are adapted to provide a distributed remote capture platform in which the times at which events are communicated to the configuration servers 120 and the fields to be included in the events are controlled by way of user-modifiable configuration rather than by “hard coding” fixed events with pre-determined fields for a given network capture mechanism. The remote configuration capability described herein also enables additional in-memory processing (e.g., filtering, transformation, normalization, aggregation, etc.) on events at the point of capture (e.g., remote capture agents 151-153) before the events are transmitted to other components of system 100.
  • Configuration information stored at each configuration server 120 may be created and/or updated manually at the configuration server and/or at a network element in communication with the configuration server. For example, a user may upload a configuration file containing configuration information for a remote capture agent to one or more configuration servers 120 for subsequent propagation to the remote capture agent. Alternatively, the user may use a GUI to provide the configuration information, as described in further detail below with respect to FIGS. 4A-4E. The configuration information may further be provided by one or more applications running on a separate server or network element, such as data storage servers 135.
  • Remote capture agents 151-153 may then use the configuration information to generate events from captured network packets. When changes in the configuration information at the configuration server are detected at the remote capture agents, logic in the remote capture agents may be automatically reconfigured in response. This means the remote capture agents may be configured dynamically to produce different events, transform the events, and/or communicate event streams to different components of system 100.
  • To detect changes in configuration information at configuration servers 120, remote capture agents 151-153 may poll configuration servers 120 at periodic intervals for updates to the configuration information. The updates may then be pulled from configuration servers 120 by remote capture agents 151-153. Conversely, updates to the configuration information may be pushed from configuration servers 120 to remote capture agents 151-153 at periodic intervals and/or when changes to the configuration information have been made.
  • In one embodiment, configuration servers 120 include a list of event streams generated by remote capture agents 151-153, as well as the configuration information used to generate the event streams at remote capture agents 151-153. The configuration information may include a unique identifier for each event stream, the types of events to be included in the event stream, one or more fields to be included in each event, and/or one or more filtering rules for filtering events to be included in the event stream. Using configuration information to dynamically modify network data capture by remote capture agents (e.g., remote capture agents 151-153) is described in a co-pending non-provisional application by inventor Michael Dickey, entitled “Distributed Processing of Network Data Using Remote Capture Agents,” having Ser. No. 14/253,713, and filing date 15 Apr. 2014 (Attorney Docket No. SPLK14-1006), which is incorporated herein by reference.
  • In one or more embodiments, system 100 includes functionality to perform protocol-based capture and analysis of network data using remote capture agents 151-153. First, remote capture agents 151-153 may be configured to generate event streams from packet flows captured at remote capture agents 151-153 based on protocol classifications for the packet flows, as described below with respect to FIGS. 2A-2B. Second, configuration servers 120 may include functionality to streamline the configuration of remote capture agents 151-153 in generating protocol-specific event streams, as described below with respect to FIGS. 3A-3B and 4A-4C. Third, configuration servers 120 and/or remote capture agents 151-153 may enable the use of capture triggers to capture additional network data based on the identification of potential security risks from previously generated event streams, as described below with respect to FIGS. 3A, 3C and 5A-5C.
  • FIG. 2A shows a remote capture agent 250 in accordance with the disclosed embodiments. In the illustrated embodiment, remote capture agent 250 is adapted to receive configuration information from one or more configuration servers 120 over network 101. Remote capture agent 250 may be installed at a customer's premises on one or more of the customer's computing resources. Remote capture agent 250 may also be installed in a remote computing environment such as a cloud computing system. For example, remote capture agent 250 may be installed on a physical server and/or in a virtual computing environment (e.g., virtual machine) that is distributed across one or more physical machines.
  • Remote capture agent 250 includes a communications component 203 configured to communicate with network elements on one or more networks (e.g., network 101) and send and receive network data (e.g., network packets) over the network(s). As depicted, communications component 203 may communicate with configuration servers 120 over network 101. Communications component 203 may also communicate with one or more sources of network data, such as network servers 130 of FIG. 1.
  • Network data received at communications component 203 may be captured by a capture component 205 coupled with communications component 203. Capture component 205 may capture some or all network data from communications component 203. For example, capture component 205 may capture network data based on the sources and/or destinations of the network data, the types of network data, the protocol associated with the network data, and/or other characteristics of the network data.
  • In addition, the network data may be captured based on configuration information stored in a configuration component 204 of remote capture agent 250. As mentioned above, the configuration information may be received from configuration servers 120 over network 101. The configuration information may then be used to dynamically configure or reconfigure remote capture agent 250 in real-time. For example, newly received configuration information in configuration component 204 may be used to configure the operation of remote capture agent 250 during processing of events from network data by remote capture agent 250.
  • To dynamically configure remote capture agent 250, configuration information received by configuration component 204 from configuration servers 120 may be provided to other components of remote capture agent 250. More specifically, remote capture agent 250 includes an events generator 207 that receives network data from network data capture component 205 and generates events from the network data based on configuration information from configuration component 204.
  • Using configuration information provided by configuration servers 120, remote capture agent 250 can be instructed to perform any number of event-based processing operations. For example, the configuration information may specify the generation of event streams associated with network (e.g., HTTP, Simple Mail Transfer Protocol (SMTP), Domain Name System (DNS)) transactions, business transactions, errors, alerts, clickstream events, and/or other types of events. The configuration information may also describe custom fields to be included in the events, such as values associated with specific clickstream terms. The configuration information may include additional parameters related to the generation of event data, such as an interval between consecutive events and/or the inclusion of transactions and/or errors matching a given event in event data for the event. Configuration information for configuring the generation of event streams from network data captured by remote capture agents is further described in the above-referenced application.
  • An events transformer 209 may further use the configuration information to transform some or all of the network data from capture component 205 and/or events from events generator 207 into one or more sets of transformed events. In one or more embodiments, transformations performed by events transformer 209 include aggregating, filtering, cleaning, and/or otherwise processing events from events generator 207. Configuration information for the transformations may thus include a number of parameters that specify the types of transformations to be performed, the types of data on which the transformations are to be performed, and/or the formatting of the transformed data.
  • A rules comparison engine 208 in remote capture agent 250 may receive events from event generator 207 and compare one or more fields from the events to a set of filtering rules in the configuration information to determine whether to include the events in an event stream. For example, the configuration information may specify packet-level, protocol-level, and/or application-level filtering of event data from event streams generated by remote capture agent 250.
  • Finally, a data enrichment component 211 may further transform event data to a different form or format based on the configuration information from configuration component 204. For example, data enrichment component 211 may use the configuration information to normalize the data so that multiple representations of the same value (e.g., timestamps, measurements, etc.) are converted into the same value in transformed event data.
  • Data can be transformed by data enrichment component 211 in any number of ways. For example, remote capture agent 250 may reside on a client server in Cupertino, Calif., where all the laptops associated with the client server have been registered with the hostname of the client server. Remote capture agent 250 may use the registration data to look up an Internet Protocol (IP) address in a look-up table (LUT) that is associated with one or more network elements of the client server's local network. Remote capture agent 250 may then resolve a user's IP address into the name of the user's laptop, thereby enabling inclusion of the user's laptop name in transformed event data associated with the IP address. The transformed event data may then be communicated to configuration servers 120 and/or a central transformation server residing in San Francisco for further processing, indexing, and/or storage.
  • As mentioned above, remote capture agent 250 may perform protocol-based generation of event streams from network data. As shown in FIG. 2B, configuration component 204 may obtain protocol-specific configuration information (e.g., protocol-specific configuration information A 212, protocol-specific configuration information B 214) from one or more configuration servers (e.g., configuration servers 120). For example, configuration information from the configuration server(s) may be transmitted over network 101 to communications component 203, which provides the configuration information to configuration component 204 for storage and/or further processing.
  • Protocol-specific configuration information from configuration component 204 may be used to configure the generation of event streams (e.g., event stream C 232, event stream D 234, event stream E 240, event stream F 242) based on protocol classifications of network packets (e.g., network packets C 216, network packets D 218) captured by capture component 205. For example, protocol-specific configuration information from configuration component 204 may specify the creation of event streams from the network packets based on the protocols used in the network packets, such as HTTP, DNS, SMTP, File Transfer Protocol (FTP), Server Message Block (SMB), Network File System (NFS), Internet Control Message Protocol (ICMP), email protocols, database protocols, and/or security protocols. Such event streams may include event attributes that are of interest to the respective protocols.
  • Before the event streams are generated from the network packets, capture component 205 may assemble the network packets into one or more packets flows (e.g., packet flow C 220, packet flow D 222). First, capture component 205 may identify the network packets in a given packet flow based on control information in the network packets. The packet flow may represent a communication path between a source and a destination (e.g., host, multicast group, broadcast domain, etc.) on the network. As a result, capture component 205 may identify network packets in the packet flow by examining network (e.g., IP) addresses, ports, sources, destinations, and/or transport protocols (e.g., Transmission Control Protocol (TCP), User Datagram Protocol (UDP), etc.) from the headers of the network packets.
  • Next, capture component 205 may assemble the packet flow from the network packets. For example, capture component 205 may assemble a TCP packet flow by rearranging out-of-order TCP packets. Conversely, capture component 205 may omit reordering of the network packets in the packet flow if the network packets use UDP and/or another protocol that does not provide for ordered packet transmission.
  • After the packet flow is assembled, capture component 205 and/or another component of remote capture agent 250 may detect encryption of the network packets in the packet flow by analyzing the byte signatures of the network packets' payloads. For example, the component may analyze the network packets' payloads for byte signatures that are indicative of Secure Sockets Layer (SSL) and/or Transport Layer Security (TLS) encryption. If the network packets are detected as encrypted, the component may decrypt the network packets. For example, the component may have access to private keys from an SSL server used by the network flow and perform decryption of the network packets to obtain plaintext payload data in the order in which the data was sent. Such access to private keys may be given to remote capture agent 250 by an administrator associated with the network flow, such as an administrator of the host from which the network packets are transmitted.
  • Events generator 207 may then obtain a protocol classification (e.g., protocol classification C 224, protocol classification D 226) for each packet flow identified, assembled, and/or decrypted by capture component 205. For example, events generator 207 may use a protocol-decoding mechanism to analyze the headers and/or payloads of the network packets in the packet flow and return protocol identifiers of one or more protocols used in the network packets. The protocol-decoding mechanism may additionally provide metadata related to the protocols, such as metadata related to traffic volume, application usage, application performance, user and/or host identifiers, content (e.g., media, files, etc.), and/or file metadata (e.g., video codecs and bit rates).
  • Once the protocol classification is obtained for a packet flow, events generator 207 may use protocol-specific configuration information associated with the protocol classification from configuration component 204 to build an event stream (e.g., event stream C 232, event stream D 234) from the packet flow. As mentioned above and in the above-referenced application, the event stream may include time-series event data generated from network packets in the packet flow. To create the event stream, events generator 207 may obtain one or more event attributes associated with the protocol classification from the configuration information. Next, event generator 207 may extract the event attribute(s) from the network packets in the first packet flow. Events generator 207 may then include the extracted event attribute(s) in the event stream.
  • For example, events generator 207 may obtain a protocol classification of DNS for a packet flow from capture component 205 and protocol-specific configuration information for generating event streams from DNS traffic from configuration component 204. The protocol-specific configuration information may specify the collection of event attributes such as the number of bytes transferred between the source and destination, network addresses and/or identifiers for the source and destination, DNS message type, DNS query type, return message, response time to a DNS request, DNS transaction identifier, and/or a transport layer protocol. In turn, events generator 207 may parse the protocol-specific configuration to identify the event attributes to be captured from the packet flow. Next, events generator 207 may extract the specified event attributes from the network packets in the packet flow and/or metadata received with the protocol classification of the packet flow and generate time-stamped event data from the extracted event attributes. Events generator 207 may then provide the time-stamped event data in an event stream to communications component 203 for transmission of the event stream over a network to one or more configuration servers, data storage servers, indexers, and/or other components for subsequent storage and processing of the event stream by the component(s).
  • As described above and in the above-referenced application, network data from capture component 205 and/or event data from events generator 207 may be transformed by events transformer 209 into transformed event data that is provided in lieu of or in addition to event data generated by events generator 207. For example, events transformer 209 may aggregate, filter, clean, and/or otherwise process event attributes from events generator 207 to produce one or more sets of transformed event attributes (e.g., transformed event attributes 1 236, transformed event attributes z 238). Events transformer 209 may then include the transformed event attributes into one or more additional event streams (e.g., event stream 1 240, event stream z 242) that may be transmitted over the network for subsequent storage and processing of the event stream(s) by other components on the network. Such transformation of event data at remote capture agent 250 may offload subsequent processing of the event data at configuration servers and/or other components on the network. Moreover, if the transformation reduces the size of the event data (e.g., by aggregating the event data), network traffic between remote capture agent 250 and the other components may be reduced, along with the storage requirements associated with storing the event data at the other components.
  • As with protocol-based generation of event data by events generator 207, events transformer 209 may use protocol-specific configuration information from configuration component 204 to transform network and/or event data from a given packet flow and/or event stream. For example, events transformer 209 may obtain protocol-specific configuration information for aggregating HTTP events and use the configuration information to generate aggregated HTTP events from HTTP events produced by events generator 207. The configuration information may include one or more key attributes used to generate a unique key representing an aggregated event from the configuration information. For example, key attributes for generating an aggregated HTTP event may include the source and destination IP addresses and ports in a set of HTTP events. A different unique key and aggregated HTTP event may thus be generated for each unique combination of source and destination IP addresses and ports in the HTTP events.
  • The configuration information may also specify one or more aggregation attributes to be aggregated prior to inclusion in the aggregated event. For example, aggregation attributes for generating an aggregated HTTP event from HTTP event data may include the number of bytes and packets sent in each direction between the source and destination. Data represented by the aggregation attributes may be included in the aggregated HTTP event by summing, averaging, and/or calculating a summary statistic from the number of bytes and packets sent in each direction between the source and destination. Aggregation of event data is described in further detail below with respect to FIG. 4C.
  • FIG. 3 shows a configuration server 320 in accordance with the disclosed embodiments. As shown in the illustrated embodiment, configuration server 320 is in communication with multiple remote capture agents 350 over network 190, and remote capture agents 350 are distributed throughout network 190 and cloud 140. Configuration server 320 includes a communications component 303 that receives events from remote capture agents 350 over networks 190 and/or 140. Communications component 303 may also communicate with one or more data storage servers, such as data storage servers 135 of FIG. 1.
  • Configuration server 320 also includes a configuration component 304 that stores configuration information for remote capture agents 350. As described above, the configuration information may specify the types of events to produce, data to be included in the events, and/or transformations to be applied to the data and/or events to produce transformed events. Some or all of the transformations may be specified in a set of filtering rules 321 that may be applied to event data at remote capture agents 350 to determine a subset of the event data to be included in one or more event streams that are sent to configuration server 320 and/or other components.
  • Configuration server 320 may also include a data processing component 311 that performs additional processing of the event streams based on configuration information from configuration component 304. As discussed in the above example with respect to FIG. 2, event data may be transformed at a remote capture agent (e.g., remote capture agent 250) during resolution of the user's IP address was into the name of the user's laptop. The transformed event data may be sent to configuration server 320 and/or a transformation server for additional processing and/or transformation, such as taking the host name from the transformed event data, using an additional LUT to obtain a user identifier (user ID) of the person to which the laptop is registered, and further transforming the event data by including the user ID in the event data before forwarding the event data to a third server (e.g., a transformation server) for another round of processing.
  • Configuration server 320 may also provide a GUI 325 that can be used to configure or reconfigure the information contained in configuration component 304. The operation of GUI 325 is discussed in further detail below with respect to FIGS. 4A-4E and 5A-5C.
  • Finally, configuration server 320 may provide a risk-identification mechanism 307 for identifying a security risk from time-series event data generated by remote capture agents 350, as well as a capture trigger 309 for generating additional time-series event data based on the security risk. For example, risk-identification mechanism 307 may allow a user to view and/or search for events that may represent security risks through GUI 325. Risk-identification mechanism 307 and/or GUI 325 may also allow the user to set and/or activate capture trigger 309 based on the events shown and/or found through risk-identification mechanism 307 and/or GUI 325.
  • In particular, risk-identification mechanism 307 and/or GUI 325 may allow the user to manually activate capture trigger 309 after discovering a potential security risk. In turn, the activated capture trigger 309 may modify configuration information in configuration component 304 that is propagated to remote capture agents 350 to trigger the capture of additional network data by remote capture agents 350.
  • Alternatively, risk-identification mechanism 307 may allow the user to create a search and/or recurring search for time-series event data that may match a security risk. If the search and/or recurring search finds time-series event data that matches the security risk, capture trigger 309 may automatically be activated to enable the generation of additional time-series event data, such as event data containing one or more attributes associated with one or more protocols that facilitate analysis of the security risk. Such automatic activation of capture trigger 309 may allow the additional event data to be generated immediately after a notable event is detected, thus averting the loss of captured network data that results from enabling additional network data capture only after a potential security risk is manually identified (e.g., by an analyst). Triggering the generation of additional time-series event data from network packets on remote agents based on potential security risks is described in further detail below with respect to FIGS. 5A-5C.
  • FIG. 4A shows an exemplary screenshot in accordance with the disclosed embodiments. More specifically, FIG. 4A shows a screenshot of a GUI, such as GUI 325 of FIG. 3. As described above, the GUI may be used to obtain configuration information that is used to configure the generation of event streams containing time-series event data at one or more remote capture agents distributed across a network.
  • As shown in FIG. 4A, the GUI includes a table with a set of columns 402-408 containing high-level information related to event streams that may be created using the configuration information. Each row of the table may represent an event stream, and rows of the table may be sorted by column 402.
  • Column 402 shows an alphabetized list of names of the event streams, and column 404 provides descriptions of the event streams. For example, columns 402-404 may include names and descriptions of event streams generated from HTTP, Dynamic Host Configuration Protocol (DHCP), DNS, FTP, email protocols, database protocols, NFS, Secure Message Block (SMB), security protocols, Session Initiation Protocol (SIP), TCP, and/or UDP network traffic. Columns 402-404 may thus indicate that event streams may be generated based on transport layer protocols, session layer protocols, presentation layer protocols, and/or application layer protocols.
  • A user may select a name of an event stream under column 402 to access and/or update configuration information for configuring the generation of the event stream. For example, the user may select “DemoHTTP” in column 402 to navigate to a screen of the GUI that allows the user to specify event attributes, filters, and/or aggregation information related to creating the “DemoHTTP” event stream, as discussed in further detail below with respect to FIGS. 4B-4E.
  • Column 406 specifies whether each event stream is enabled or disabled. For example, column 406 may indicate that the “AggregateHTTP,” “DemoHTTP,” “dns,” “ftp,” “mysql-query,” “sip,” “tcp,” and “udp” event streams are enabled. If an event stream is enabled, time-series event data may be included in the event stream based on the configuration information for the event stream.
  • Column 408 specifies whether each event stream is cloned from an existing event stream. For example, column 408 may indicate that the “AggregateHTTP” and “DemoHTTP” event streams have been cloned (e.g., copied) from other event streams, while the remaining event streams may be predefined with default event attributes.
  • The GUI also includes a user-interface element 410 (e.g., “Clone Stream”). A user may select user-interface element 410 to create a new event stream as a copy of an event stream listed in the GUI. After user-interface element 410 is selected, an overlay may be displayed that allows the user to specify a name for the new event stream, a description of the new event stream, and an existing event stream from which the new event stream is to be cloned. The new event stream may then be created with the same event attributes and/or configuration options as the existing event stream, and the user may use the GUI to customize the new event stream as a variant of the existing event stream (e.g., by adding or removing event attributes, filters, and/or aggregation information).
  • FIG. 4B shows an exemplary screenshot in accordance with the disclosed embodiments. More specifically, FIG. 4B shows a screenshot of the GUI of FIG. 4A after the user has selected “DemoHTTP” from column 402. In response to the selection, the GUI displays configuration information and/or configuration options for the “DemoHTTP” event stream.
  • Like the GUI of FIG. 4A, the GUI of FIG. 4B may include a table. Each row in the table may represent an event attribute that is eligible for inclusion in the event stream. For example, an event attribute may be included in the table if the event attribute can be obtained from network packets that include the protocol of the event stream. Columns 412-420 of the table may allow the user to use the event attributes to generate time-series event data that is included the event stream. First, column 412 includes a series of checkboxes that allows the user to include individual event attributes in the event stream or exclude the event attributes from the event stream. If a checkbox is checked, the corresponding event attribute is added to the event stream, and the row representing the event attribute is shown with other included event attributes in an alphabetized list at the top of the table. If a checkbox is not checked, the corresponding event attribute is omitted from the event stream, and the row representing the event attribute is shown with other excluded event attributes in an alphabetized list following the list of included event attributes. Those skilled in the art will appreciate that the GUI may utilize other sortings and/or rankings of event attributes in columns 412-420.
  • Columns 414-418 may provide information related to the event attributes. Column 414 may show the names of the event attributes, column 416 may provide a description of each event attribute, and column 418 may provide a term representing the event attribute. In other words, columns 414-418 may allow the user to identify the event attributes and decide whether the event attributes should be included in the event stream.
  • Column 420 may include a series of links labeled “Add.” The user may select one of the links to access a portion of the GUI that allows the user to set a filter for the corresponding event attribute. The filter may then be used in the generation of the event stream from network data. Creation of filters for generating event streams from network packets is described in further detail below with respect to FIGS. 4D-4E.
  • The GUI of FIG. 4B also includes information 422 related to the event stream. For example, information 422 may include the name (e.g., “DemoHTTP”) of the event stream, the protocol classification and/or type (e.g., “http.event”) of the event stream, and the number of filters (e.g., “0 filters configured”) set for the event stream. Information 422 may also include a checkbox 436 that identifies if the event stream contains aggregated event data. If checkbox 436 is checked, the GUI may be updated with options associated with configuring the generation of an aggregated event stream, as described below with respect to FIG. 4C.
  • Finally, the GUI of FIG. 4B includes a set of user-interface elements 424-434 for managing the event stream. First, the user may select user-interface element 424 (e.g., “Enabled”) to enable generation of the event stream from network data and user-interface element 426 (e.g., “Disabled”) to disable the generation of the event stream from the network data.
  • Next, the user may select user-interface element 428 (e.g., “Clone”) to clone the event stream and user-interface element 430 (e.g., “Delete”) to delete the event stream. If the user selects user-interface element 428, the GUI may obtain a name and description for the cloned event stream from the user. Next, the GUI may copy the content of columns 412-420, including configuration options (e.g., checkboxes in column 412 and filters added using links in column 420) that have been changed but not yet saved by the user, to a new screen for configuring the generation of the cloned event stream.
  • If the user selects user-interface element 430, the GUI may remove the event stream from the table in FIG. 4A. In turn, a representation of the event stream may be removed from the configuration information to stop the generation of time-series event data in the event stream by one or more remote capture agents.
  • The user may select user-interface element 432 (e.g., “Cancel”) to discharge changes to the configuration information made in the current screen of the GUI. Conversely, the user may select user-interface 434 (e.g., “Save”) to propagate the changes to the configuration information, and in turn, update the generation of event data from network packets captured by the remote capture agents based on the changes.
  • FIG. 4C shows an exemplary screenshot in accordance with the disclosed embodiments. In particular, FIG. 4C shows a screenshot of the GUI of FIG. 4B after checkbox 436 has been checked. Because checkbox 436 is checked, the GUI includes a number of user-interface elements for configuring the generation of an aggregated event stream. The aggregated event stream may include aggregated event data, which in turn may be generated by aggregating and/or extracting event attributes from one or more network packets in a packet flow. For example, an HTTP event may be generated from one to several HTTP packets representing an HTTP request/response pair. Event attributes from multiple HTTP events may then be aggregated into a single aggregated HTTP event to reduce the amount of event data generated from the network data without losing important attributes of the event data.
  • As shown in FIG. 4C, a new column 438 is added to the table. Each row in column 438 may include a pair of user-interface elements (e.g., buttons) that allow the user to identify the corresponding event attribute as a key attribute or an aggregation attribute. One or more key attributes may be used to generate a unique key representing each aggregated event, and one or more aggregation attributes may be aggregated prior to inclusion in the aggregated event. Some event attributes (e.g., “dest_ip,” “src_ip,” “uri_path”) may only be used as key attributes because the event attributes are not numeric in nature. On the other hand, event attributes that may be summed (e.g., “dest_port,” “status,” “bytes,” “bytes_in,” “bytes_out,” “time_taken”) may have numeric values.
  • Event attributes identified as key attributes in column 438 may be sorted at the top of the table, followed by event attributes identified as aggregation attributes. Event attributes that are not included in the event stream (e.g., event attributes with unchecked checkboxes in column 412) may be shown below the aggregation attributes in the table. Alternatively, event attributes may be displayed in the table according to other sortings and/or rankings.
  • While sums are the only type of aggregation shown in the GUI of FIG. 4C, other types of aggregation may also be used to generate aggregated event data. For example, aggregated event streams may be created using minimums, maximums, averages, standard deviations, and/or other summary statistics of event attributes.
  • The GUI of FIG. 4C also includes a user-interface element 440 (e.g., a text box) for obtaining an aggregation interval over which event attributes are to be aggregated into a single aggregated event. The aggregation interval may be increased to increase the amount of aggregation in the aggregated event stream and reduced to decrease the amount of aggregation in the aggregated event stream.
  • For example, column 438 may indicate that the “dest_ip,” “dest_port,” “src_ip,” “status,” and “uri_path” event attributes are specified as key attributes and the “bytes,” “bytes_in,” “bytes_out,” and “time_taken” event attributes are specified as aggregation attributes. Similarly, an aggregation interval of 60 seconds may be obtained from user-interface element 440. As a result, the aggregated event stream may include aggregated events generated from event data over a 60-second interval. After each 60-second interval has passed, a separate aggregated event with a unique key may be generated for each unique combination of “dest_ip,” “dest_port,” “src_ip,” “status,” and “uri_path” key attributes encountered during the interval. Values of “bytes,” “bytes_in,” “bytes_out,” and “time_taken” for events within the interval that match the unique combination of key attributes may also be summed and/or otherwise aggregated into the aggregated event. Aggregated events generated from the configuration options may then be shown in the same GUI, as described in further detail below with respect to FIG. 4F.
  • Such configuration of event streams and/or aggregated event streams may allow network data to be captured at different levels of granularity and/or for different purposes. For example, an aggregated event stream may include all possible event attributes for the event stream to enable overall monitoring of network traffic. On the other hand, one or more unaggregated event streams may be created to capture specific types of network data at higher granularities than the aggregated event stream. In addition, multiple event streams may be created from the same packet flow and/or event data to provide multiple “views” of the packet flow and/or event data.
  • FIG. 4D shows an exemplary screenshot in accordance with the disclosed embodiments. More specifically, FIG. 4D shows a screenshot of the GUI of FIGS. 4B-4C after an “Add” link in column 420 is selected. For example, the GUI of FIG. 4D may be shown as an overlay on the screens of FIGS. 4B-4C to enable the addition of filters to configuration information for the event stream(s) and/or aggregated event stream(s) shown on the screens.
  • As with the screenshots of FIGS. 4A-4C, the GUI of FIG. 4D includes information and/or user-interface elements organized into a table. Rows of the table may represent filters for an event stream and/or aggregated event stream, and columns 442-450 of the table may facilitate identification and/or configuration of the filters.
  • First, column 442 may provide a list of terms representing event attributes to which the filters are to be applied. For example, column 422 may specify an “http.status” term representing the “status” event attribute and an “http.uri-stem” term representing the “uri_path” event attribute.
  • Column 444 may be used to provide a comparison associated with each filter. For example, a user may select a cell under column 444 to access a drop-down menu of possible comparisons for the corresponding filter. As shown in FIG. 4D, the second cell of column 444 is selected to reveal a drop-down menu of comparisons for a string-based event attribute (e.g., “uri_path”). Within the drop-down menu, “Regular Expression” is selected, while other options for the comparison may include “False,” “True,” “Is defined,” “Is not defined,” “Not Regular Expression,” “Exactly matches,” “Does not exactly match,” “Contains,” “Does not contain,” “Starts with,” “Does not start with,” “Ends with,” “Does not end with,” “Ordered before,” “Not ordered before,” “Ordered after,” and “Not ordered after.” As a result, a number of comparisons may be made with string-based event attributes during filtering of network data by the string-based event attributes.
  • Column 446 may allow the user to specify a value against which the comparison in column 444 is made. Cells in column 446 may be text-editable fields and/or other user-interface elements that accept user input. For example, the second cell of column 446 may include a value of “admin” that is entered by the user. Consequently, the values in the second cells of columns 444-446 may be used to generate a filter that determines if the “uri_path” event attribute from network data matches a regular expression of “admin.” If the network data matches the regular expression, the network data may be used to generate event data, which may subsequently be used to generate aggregated event data. If the network data does not match the regular expression, generation of event data from the network data may be omitted.
  • Column 448 may include a set of checkboxes with a “Match All” header. The user may check a checkbox in column 448 to require each value in a multi-value event attribute to match the filter. For example, the user may check a checkbox in column 448 for a filter that is applied to a checksum event attribute to ensure that each of multiple checksums in a given network packet and/or event satisfies the comparison in the filter.
  • Column 450 may allow the user to delete filters from the configuration information. For example, the user may select a user-interface (e.g., an icon) in a cell of column 450 to remove the corresponding filter from the configuration information.
  • The GUI also includes a set of user-interface elements 452-454 for determining the applicability of individual filters or all filters to the network data. For example, the user may select user-interface element 452 (e.g., “All”) to apply the filters so that only data that matches all filters in the table is used to generate events. Conversely, the user may select user-interface element 454 (e.g., “Any”) to apply the filters so that data matching any of the filters in the data is used to generate events. In other words, user-interface element 452 may be selected to apply a logical conjunction to the filters, while user-interface element 454 may be selected to apply a logical disjunction to the filters.
  • FIG. 4E shows an exemplary screenshot in accordance with the disclosed embodiments. As with the screenshot of FIG. 4D, FIG. 4E shows a GUI for adding and/or managing filters for generating event data at one or more remote capture components.
  • Within the GUI of FIG. 4E, the first cell of column 444 is selected. In turn, a drop-down menu of possible comparisons is shown for the corresponding filter. Because the filter relates to a numeric event attribute (e.g., an HTTP status code), comparisons in column 444 may be numeric in nature. For example, the “Greater than” comparison is selected, while other possible comparisons may include “False,” “True,” “Is defined,” “Is not defined,” “Equals,” “Does not equal,” “Less than,” “Greater than or equal to,” and “Less than or equal to.” The differences in comparisons shown in FIG. 4E and FIG. 4D may ensure that comparisons that are meaningful and/or relevant to the types of event attributes specified in the filters are used with the filters.
  • FIG. 4F shows an exemplary screenshot in accordance with the disclosed embodiments. More specifically, FIG. 4F shows a screenshot of a GUI, such as GUI 325 of FIG. 3. The GUI of FIG. 4F may provide information related to aggregated events, such as aggregated events generated using the GUI of FIG. 4C.
  • As shown in FIG. 4F, a first column 456 contains a timestamp of an aggregated event, and a second column 458 shows the aggregated event. Within column 458, the aggregated event includes a number of event attributes. Some of the event attributes (e.g., “dest_ip,” “dest_port,” “src_ip,” “status,” “uri_path”) are key attributes that are used to uniquely identify the aggregated event, and other event attributes (e.g., “dest_port,” “status,” “bytes,” “bytes_in,” “bytes_out,” “time_taken”) may be numerically summed before the event attributes are included in the aggregated event.
  • FIG. 5A shows an exemplary screenshot in accordance with the disclosed embodiments. More specifically, FIG. 5A shows a screenshot of a GUI, such as GUI 325 of FIG. 3. The GUI may be used with a risk-identification mechanism and/or a capture trigger, such as risk-identification mechanism 307 and capture trigger 309 of FIG. 3.
  • The GUI of FIG. 5A may include a portion 502 that represents the risk-identification mechanism. For example, portion 502 may display a dashboard of time-series event data that represents security risks. The dashboard includes a number of potential security risks, such as “HTTP Errors,” “DNS Errors,” “Cloud Email,” “NFS Activity,” and “Threat List Activity.” Events that match one of the listed potential security risks may be represented as bars within a time interval represented by the horizontal dimension of the dashboard. For example, a security risk 506 may be shown as a series of bars clustered around an interval of time under “DNS Errors” in portion 502.
  • On the other hand, the dashboard may lack data for other potential security risks because the data volume associated with capturing network data across all protocols and/or security risks may be too large to effectively store and/or consume. As a result, portion 502 may indicate that no data is available (e.g., “Search returned no results”) for the “HTTP Errors,” “Cloud Email,” “NFS Activity,” and “Threat List Activity” security risks.
  • The GUI may also include a portion 504 that represents a capture trigger for generating additional time-series event data based on identified security risks from portion 502. For example, portion 504 may include a checkbox that allows a user to activate the capture trigger upon identifying security risk 506 in portion 502. Portion 504 may also include a first drop-down menu that allows the user to specify one or more protocols (e.g., “HTTP,” “DNS,” “All Email,” “NFS/SMB,” “All Protocols”) of additional time-series event data to be captured with the capture trigger. Portion 504 may additionally include a second drop-down menu that allows the user to specify a period (e.g., “4 Hours”) over which the additional time-series event data is to be captured after the capture trigger is activated.
  • After the capture trigger is activated, configuration information on one or more remote capture agents used to generate the time-series event data may be updated to include the additional protocol(s) specified in portion 504. For example, configuration information for configuring the generation of additional event streams from the specified protocol(s) may be propagated to the remote capture agents, and the remote capture agents may use the configuration to create the event streams from network data and/or event data at the remote capture agents. The configuration information may include default event attributes for the protocol(s) and/or event attributes that may be of interest to the security assessment of network packet flows. For example, the configuration information may specify the generation of event data related to other security risks, such as the security risks shown in the dashboard. Once the event data is generated and/or indexed, the event data may be shown in the dashboard to facilitate verification, monitoring, and/or analysis of the security risk. After the pre-specified period obtained from portion 504 has passed, the configuration information on the remote capture agents may be updated to disable the generation of the additional event streams and reduce the volume of network data captured by the remote capture agents.
  • As with the user interfaces of FIGS. 4A-4E, the user may add one or more filters that are applied during the generation of the additional time-series event data. For example, the user may use the user interfaces of FIGS. 4D-4E to add a filter for network and/or event data that exactly matches the IP address (e.g., 10.160.26.206) from which the security risk was detected. As a result, the additional time-series data may be generated only from network data containing the same source IP address. The user may also use the user interfaces of FIGS. 4A-4C to customize the collection of additional time-series event data by protocol and/or event attributes.
  • FIG. 5B shows an exemplary screenshot in accordance with the disclosed embodiments. In particular, FIG. 5B shows a screenshot of a GUI, such as GUI 325 of FIG. 3. Like the GUI of FIG. 5A, the GUI of FIG. 5B includes a first portion 506 representing a risk-identification mechanism and a second portion 508 representing a capture trigger.
  • Portion 506 may allow a user to create a recurring search for time-series event data that matches a security risk. For example, portion 506 may include user-interface elements for obtaining a domain, application context, description, search terms, time range (e.g., start and end times), and/or frequency (e.g., daily, hourly, every five minutes, etc.) for the recurring search. The user may use the user-interface elements of portion 506 to specify a recurring search for an excessive number of failed login attempts in captured network and/or event data, which may represent brute force access behavior that constitutes a security risk.
  • Portion 508 may allow the user to provide the capture trigger, which is automatically activated if the recurring search finds time-series event data that matches the security risk. As with portion 504 of FIG. 5A, portion 508 may allow the user to set the capture trigger, specify one or more protocols to be captured with the capture trigger, and/or a pre-specified period over which network data using the protocol(s) is to be captured.
  • After the user has finished defining the recurring search and capture trigger, the user may select a user-interface 510 (e.g., “Save”) to save the recurring search and capture trigger. The capture trigger may then be activated without additional input from the user once an iteration of the recurring search identifies the security risk. Conversely, the user may select a user-interface element 512 (e.g., “Cancel”) to exit the screen of FIG. 5B without creating the recurring search and/or capture trigger.
  • FIG. 6 shows a flowchart illustrating the processing of network data in accordance with the disclosed embodiments. In one or more embodiments, one or more of the steps may be omitted, repeated, and/or performed in a different order. Accordingly, the specific arrangement of steps shown in FIG. 6 should not be construed as limiting the scope of the embodiments.
  • Initially, configuration information is obtained at a remote capture agent from a configuration server over a network (operation 602). The remote capture agent may be located on a separate network from that of the configuration server. For example, the remote capture agent may be installed on a physical and/or virtual machine on a remote network and/or cloud. As discussed above, the remote capture agent and other remote capture agents may be used to capture network data from a set of remote networks in a distributed manner.
  • Next, the configuration information is used to configure the generation of event data from network packets captured by the remote capture agent during the runtime of the remote capture agent (operation 604). For example, the configuration information may be used to configure the remote capture agent to identify certain types of network packets, extract network data from the network packets, and/or include the network data in the event data.
  • The remote capture agent may identify network packets in a packet flow based on control information in the network packets (operation 608). For example, network packets between a source and destination may be identified based on source and/or destination network addresses, source and/or destination ports, and/or transport layer protocols in the headers of the network packets.
  • The remote capture agent may also assemble the packet flow from the network packets (operation 608) and/or decrypt the network packets upon detecting encryption of the network packets (operation 610). For example, the remote capture agent may rearrange out-of-order TCP packets into a TCP stream. The remote capture agent may also analyze the byte signatures of the network packets' payloads to identify encryption of the network packets and use an available private key to decrypt the network packets.
  • After the packet flow is identified, assembled and/or decrypted, the remote capture agent may obtain a protocol classification for the packet flow (operation 612). For example, the remote capture agent may provide network packets in the packet flow to a protocol-decoding mechanism and receive one or more protocol identifiers representing the protocols used by the network packets from the protocol-decoding mechanism.
  • Next, the remote capture agent may use configuration information associated with the protocol classification to build an event stream from the packet flow (operation 614), as described in further detail below with respect to FIG. 7. The remote capture agent may then transmit the event stream over a network for subsequent storage and processing of the event stream by one or more components on the network (operation 616). For example, the remote capture agent may transmit the event stream to one or more data storage servers, configuration servers, and/or indexers on the network.
  • An update to the configuration information may be received (operation 616). For example, the remote capture agent may receive an update to the configuration information after the configuration information is modified at a configuration server. If an update to the configuration information is received, the update is used to reconfigure the generation of time-series event data at the remote capture agent during runtime of the remote capture agent (operation 620). For example, the remote capture agent may be use the updated configuration information to generate one or more new event streams, discontinue the generation of one or more existing event streams, and/or modify the generation of one or more existing event streams.
  • The remote capture agent may continue to be used (operation 622) to capture network data. If the remote capture agent is to be used, packet flows captured by the remote capture agent are identified (operation 606), and network packets in the packet flows are assembled into the packet flows and/or decrypted (operations 608=610). Protocol classifications for the packet flows are also obtained and used, along with configuration information associated with the protocol classifications, to build event streams from the packet flows (operations 612-614). The event streams are then transmitted over the network (Operation 616), and any updates to the configuration information are used to reconfigure the operation of the remote capture agent (operations 618-620) during generation of the event streams. Capture of network data by the remote capture agent may continue until the remote capture agent is no longer used to generate event data from network data.
  • FIG. 7 shows a flowchart illustrating the process of using configuration information associated with a protocol classification to build an event stream from a packet flow in accordance with the disclosed embodiments. In one or more embodiments, one or more of the steps may be omitted, repeated, and/or performed in a different order. Accordingly, the specific arrangement of steps shown in FIG. 7 should not be construed as limiting the scope of the embodiments.
  • First, one or more event attributes associated with the protocol classification are obtained from the configuration information (operation 702). For example, the event attribute(s) may be obtained from a portion of the configuration information that specifies the generation of an event stream from network data matching the protocol classification.
  • Next, the event attribute(s) are extracted from network packets in the packet flow (operation 704). For example, the event attribute(s) may be used to generate event data from the network packets. The configuration information may optionally be used to transform the extracted event attribute(s) (operation 706). For example, the configuration information may be used to aggregate the event data into aggregated event data that reduces the volume of event data generated while retaining the important aspects of the event data.
  • Finally, the extracted and/or transformed event attributes are included in the event stream (operation 708). For example, the event stream may be include a series of events and/or aggregated events that contain event attributes that are relevant to the protocol classification of the network packets represented by the events.
  • FIG. 8 shows a flowchart illustrating the process of facilitating the processing of network data in accordance with the disclosed embodiments. In one or more embodiments, one or more of the steps may be omitted, repeated, and/or performed in a different order. Accordingly, the specific arrangement of steps shown in FIG. 8 should not be construed as limiting the scope of the embodiments.
  • First, a GUI for obtaining configuration information for configuring the generation of time-series event data from network packets captured by one or more remote agents is provided (operation 802). The GUI may include a number of user-interface elements for streamlining the creation and/or update of the configuration information. First, the GUI may provide a set of user-interface elements for including one or more event attributes in the time-series event data of an event stream associated with a protocol classification of the network packets (operation 804). For example, the GUI may include a set of checkboxes that enable the selection of individual event attributes for inclusion in the time-series event data.
  • Second, the GUI may provide a set of user-interface elements for managing the event stream (operation 806) and/or obtaining the protocol classification for the event stream. For example, the GUI may include one or more user-interface elements for cloning the event stream from an existing event stream, which imparts the protocol classification of the existing event stream on the cloned event stream. The GUI may also include user-interface elements for deleting the event stream, enabling the event stream, and/or disabling the event stream.
  • Third, the GUI may provide a set of user-interface elements for filtering the network packets (operation 808) prior to generating the time-series event data from the network packets. Each filter may identify an event attribute, a comparison to be performed on the event attribute, and/or a value to which the event attribute is to be compared. For example, the filter may match the event attribute to a Boolean value (e.g., true or false), perform a numeric comparison (e.g., equals, greater, less than, greater than or equal to, less than or equal to), and/or verify the definition of (e.g., the existence of) the event attribute in network data. The filter may also compare the event attribute to a regular expression, perform an exact match of the event attribute to the value, perform a partial match of the event attribute to the value, and/or determine the event attribute's position in an ordering.
  • Fourth, the GUI may provide a set of user-interface elements for aggregating the event attribute(s) into aggregated event data that is included in the event stream (operation 810). For example, the GUI may provide user-interface elements for identifying event attributes as key attributes used to generate a key representing the aggregated event data and/or aggregation attributes to be aggregated prior to inclusion in the aggregated event data. The GUI may also include one or more user-interface elements for obtaining an aggregation interval over which the one or more event attributes are aggregated into the aggregated event data.
  • Finally, the event attribute(s), protocol classification, filtering information, and/or aggregation information obtained from the GUI are included in the configuration information (operation 812). The configuration information may then be used to configure the protocol-based capture, filtering, and/or aggregation of network data at the remote capture agent(s).
  • FIG. 9 shows a flowchart illustrating the process of facilitating the processing of network data in accordance with the disclosed embodiments. In one or more embodiments, one or more of the steps may be omitted, repeated, and/or performed in a different order. Accordingly, the specific arrangement of steps shown in FIG. 9 should not be construed as limiting the scope of the embodiments.
  • Initially, a risk-identification mechanism for identifying a security risk from time-series event data generated from network packets captured by one or more remote capture agents distributed across a network is provided (operation 902). The risk-identification mechanism may include a GUI that displays an event of interest related to the security risk. For example, the GUI may show potential security risks in a dashboard and/or other visualization of the time-series event data. Alternatively, the risk-identification mechanism may include a search and/or recurring search for a subset of the time-series event data matching the security risk. For example, the risk-identification mechanism may include a search mechanism that allows a user to search for threats, attacks, errors, and/or other notable events in the time-series event data.
  • Next, a capture trigger for generation additional time-series data from the network packets on the remote capture agent(s) based on the security risk is provided (operation 904). The capture trigger may be received through one or more user-interface elements of a GUI, such as the same GUI used to provide the risk-identification mechanism. For example, the capture trigger may be activated in a portion of the GUI that is above, below, and/or next to a dashboard that displays security risks to the user. Alternatively, the capture trigger may be linked to a recurring search for time-series event data that matches a security risk. As a result, the capture trigger may automatically be activated once time-series event data matching the security risk is found.
  • After the capture trigger is activated, the capture trigger is used to configure the generation of the additional time-series event data from the network packets (operation 906). For example, activation of the capture trigger may result in the updating of configuration information for the remote capture agent(s), which causes the remote capture agent(s) to generate additional event streams containing event attributes associated with protocols that facilitate analysis of the security risk.
  • Finally, generation of the additional time-series event data is disabled after a pre-specified period has passed (operation 908). For example, generation of the additional time-series event data may be set to expire a number of hours or days after the capture trigger is activated. The expiry may be set by the user and/or based on a default expiration for security-based capture of additional network data from network packets.
  • FIG. 10 shows a computer system 1000 in accordance with the disclosed embodiments. Computer system 1000 includes a processor 1002, memory 1004, storage 1006, and/or other components found in electronic computing devices. Processor 1002 may support parallel processing and/or multi-threaded operation with other processors in computer system 1000. Computer system 1000 may also include input/output (I/O) devices such as a keyboard 1008, a mouse 1010, and a display 1012.
  • Computer system 1000 may include functionality to execute various components of the disclosed embodiments. In particular, computer system 1000 may include an operating system (not shown) that coordinates the use of hardware and software resources on computer system 1000, as well as one or more applications that perform specialized tasks for the user. To perform tasks for the user, applications may obtain the use of hardware resources on computer system 1000 from the operating system, as well as interact with the user through a hardware and/or software framework provided by the operating system.
  • In one or more embodiments, computer system 1000 provides a system for facilitating the processing of network data. The system may include a remote capture agent. The remote capture agent may obtain a first protocol classification for a first packet flow captured at the remote capture agent and use configuration information associated with the first protocol classification to build a first event stream from the first packet flow at the remote capture agent. The remote capture agent may also transmit the first event stream over a network for subsequent storage and processing of the first event stream by one or more components on the network. The remote capture agent may further obtain a second protocol classification for a second packet flow captured at the remote capture agent, use configuration information associated with the second protocol classification to build a second event stream from the second packet flow at the remote capture agent, and transmit the second event stream over the network.
  • The system may also provide a configuration server that provides the configuration information to the remote capture agent. The configuration server may provide a GUI for obtaining configuration for configuring the generation of time-series event data from network packets captured by the remote capture agent. The GUI may include user-interface elements for including one or more event attributes in the time-series event data of an event stream associated with a protocol classification of the network packets. The GUI may also include user-interface elements for managing the event stream, filtering the network packets, and/or aggregating the event attributes into aggregated event data.
  • The system may additionally provide a risk-identification mechanism for identifying a security risk from the time-series event data generated by the remote capture agent. Finally, the system may provide a capture trigger for generating additional time-series event data from the network packets on the remote capture agent based on the security risk. The additional time-series data may include one or more event attributes for facilitating analysis of the security risk, such as an event attribute associated with a protocol that facilitates analysis of the security risk.
  • The foregoing descriptions of various embodiments have been presented only for purposes of illustration and description. They are not intended to be exhaustive or to limit the present invention to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the present invention.

Claims (24)

What is claimed is:
1. A method for facilitating the processing of network data, comprising:
providing, on a computer system, a graphical user interface (GUI) for obtaining configuration information for configuring the generation of time-series event data from network packets captured by one or more remote capture agents;
providing, in the GUI, a first set of user-interface elements for including one or more event attributes in the time-series event data of an event stream associated with a protocol classification of the network packets; and
including the one or more event attributes specified through the first set of user-interface elements in the configuration information.
2. The method of claim 1, further comprising:
providing the configuration information over a network to the one or more remote capture agents, wherein the configuration information is used to configure the generation of the time-series event data at the one or more remote capture agents during runtime of the one or more remote capture agents.
3. The method of claim 1, further comprising:
providing, in the GUI, a second set of user-interface elements for managing the event stream; and
obtaining the protocol classification for the event stream from the second set of user-interface elements.
4. The method of claim 1,
wherein the GUI comprises a second set of user-interface elements for managing the event stream, and
wherein managing the event stream comprises at least one of:
cloning the event stream from an existing event stream;
deleting the event stream;
enabling the event stream; and
disabling the event stream.
5. The method of claim 1, further comprising:
providing, in the GUI, a second set of user-interface elements for filtering the network packets prior to generating the time-series event data from the network packets.
6. The method of claim 1,
wherein the GUI comprises a second set of user-interface elements for filtering the network packets, and
wherein filtering the network packets is associated with at least one of:
a Boolean value;
a numeric comparison;
a definition;
a regular expression;
an exact match;
a partial match; and
an ordering.
7. The method of claim 1,
wherein the GUI comprises a second set of user-interface elements for filtering the network packets, and
wherein the second set of user-interface elements is used to apply a logical disjunction or a logical conjunction to a set of filters for filtering the network packets.
8. The method of claim 1,
wherein the GUI comprises a second set of user-interface elements for filtering the network packets, and
wherein the second set of user-interface elements is used to match a filter to any or all elements of a multi-value event attribute in the event stream.
9. The method of claim 1, further comprising:
providing, in the GUI, a second set of user-interface elements for aggregating the one or more event attributes into aggregated event data that is included in the event stream.
10. The method of claim 1,
wherein the GUI comprises a second set of user-interface elements for aggregating the one or more event attributes into aggregated event data, and
wherein the second set of user-interface elements comprises a user-interface element for identifying an event attribute as:
a key attribute used to generate a key representing the aggregated event data; or
an aggregation attribute to be aggregated prior to inclusion in the aggregated event data.
11. The method of claim 1,
wherein the GUI comprises a second set of user-interface elements for aggregating the one or more event attributes into aggregated event data, and
wherein the second set of user-interface elements comprises:
a first user-interface element for identifying an event attribute as:
a key attribute used to generate a key representing the aggregated event data; or
an aggregation attribute to be aggregated prior to inclusion in the aggregated event data; and
a second user-interface element for obtaining an aggregation interval over which the one or more event attributes are aggregated into the aggregated event data.
12. The method of claim 1, wherein the protocol classification comprises at least one of:
a transport layer protocol;
a session layer protocol;
a presentation layer protocol; and
an application layer protocol.
13. An apparatus, comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
provide a graphical user interface (GUI) for obtaining configuration information for configuring the generation of time-series event data from network packets captured by one or more remote capture agents;
provide, in the GUI, a first set of user-interface elements for including one or more event attributes in the time-series event data of an event stream associated with a protocol classification of the network packets; and
include the one or more event attributes specified through the first set of user-interface elements in the configuration information.
14. The apparatus of claim 13, wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
provide the configuration information over a network to the one or more remote capture agents, wherein the configuration information is used to configure the generation of the time-series event data at the one or more remote capture agents during runtime of the one or more remote capture agents.
15. The apparatus of claim 13, wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
provide, in the GUI, a second set of user-interface elements for managing the event stream; and
obtaining the protocol classification for the event stream from the second set of user-interface elements.
16. The apparatus of claim 13, wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
provide, in the GUI, a second set of user-interface elements for filtering the network packets prior to generating the time-series event data from the network packets.
17. The apparatus of claim 13,
wherein the GUI comprises a second set of user-interface elements for filtering the network packets, and
wherein filtering the network packets is associated with at least one of:
a Boolean value;
a numeric comparison;
a definition;
a regular expression;
an exact match;
a partial match; and
an ordering.
18. The apparatus of claim 13, wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
provide, in the GUI, a second set of user-interface elements for aggregating the one or more event attributes into aggregated event data that is included in the event stream.
19. The apparatus of claim 13,
wherein the GUI comprises a second set of user-interface elements for aggregating the one or more event attributes into aggregated event data, and
wherein the second set of user-interface elements comprises:
a first user-interface element for identifying an event attribute as:
a key attribute used to generate a key representing the aggregated event data; or
an aggregation attribute to be aggregated prior to inclusion in the aggregated event data; and
a second user-interface element for obtaining an aggregation interval over which the one or more event attributes are aggregated into the aggregated event data.
20. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for facilitating the processing of network data, the method comprising:
providing, on a computer system, a graphical user interface (GUI) for obtaining configuration information for configuring the generation of time-series event data from network packets captured by one or more remote capture agents;
providing, in the GUI, a first set of user-interface elements for including one or more event attributes in the time-series event data of an event stream associated with a protocol classification of the network packets; and
including the one or more event attributes specified through the first set of user-interface elements in the configuration information.
21. The non-transitory computer-readable storage medium of claim 20, the method further comprising:
providing the configuration information over a network to the one or more remote capture agents, wherein the configuration information is used to configure the generation of the time-series event data at the one or more remote capture agents during runtime of the one or more remote capture agents.
22. The non-transitory computer-readable storage medium of claim 20, the method further comprising:
providing, in the GUI, a second set of user-interface elements for managing the event stream; and
obtaining the protocol classification for the event stream from the second set of user-interface elements.
23. The non-transitory computer-readable storage medium of claim 20, the method further comprising:
providing, in the GUI, a second set of user-interface elements for filtering the network packets prior to generating the time-series event data from the network packets.
24. The non-transitory computer-readable storage medium of claim 20,
wherein the GUI comprises a second set of user-interface elements for aggregating the one or more event attributes into aggregated event data, and
wherein the second set of user-interface elements comprises:
a first user-interface element for identifying an event attribute as:
a key attribute used to generate a key representing the aggregated event data; or
an aggregation attribute to be aggregated prior to inclusion in the aggregated event data; and
a second user-interface element for obtaining an aggregation interval over which the one or more event attributes are aggregated into the aggregated event data.
US14/528,932 2014-10-30 2014-10-30 Streamlining configuration of protocol-based network data capture by remote capture agents Abandoned US20160127180A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US14/528,932 US20160127180A1 (en) 2014-10-30 2014-10-30 Streamlining configuration of protocol-based network data capture by remote capture agents

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US14/528,932 US20160127180A1 (en) 2014-10-30 2014-10-30 Streamlining configuration of protocol-based network data capture by remote capture agents

Publications (1)

Publication Number Publication Date
US20160127180A1 true US20160127180A1 (en) 2016-05-05

Family

ID=55853906

Family Applications (1)

Application Number Title Priority Date Filing Date
US14/528,932 Abandoned US20160127180A1 (en) 2014-10-30 2014-10-30 Streamlining configuration of protocol-based network data capture by remote capture agents

Country Status (1)

Country Link
US (1) US20160127180A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160283307A1 (en) * 2014-07-28 2016-09-29 Hitachi, Ltd. Monitoring system, monitoring device, and test device

Citations (55)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5436618A (en) * 1992-09-17 1995-07-25 U.S. Philips Corporation Multi-apparatus consumer electronics system
US5892903A (en) * 1996-09-12 1999-04-06 Internet Security Systems, Inc. Method and apparatus for detecting and identifying security vulnerabilities in an open network computer communication system
US5920711A (en) * 1995-06-02 1999-07-06 Synopsys, Inc. System for frame-based protocol, graphical capture, synthesis, analysis, and simulation
US5983270A (en) * 1997-03-11 1999-11-09 Sequel Technology Corporation Method and apparatus for managing internetwork and intranetwork activity
US20030061506A1 (en) * 2001-04-05 2003-03-27 Geoffrey Cooper System and method for security policy
US6542861B1 (en) * 1999-03-31 2003-04-01 International Business Machines Corporation Simulation environment cache model apparatus and method therefor
US20030120619A1 (en) * 2001-05-23 2003-06-26 Osborn Mark David Optimizing storage and retrieval of monitoring data
US6594634B1 (en) * 1998-09-14 2003-07-15 Medtronic Physio-Control Corp. Method and apparatus for reporting emergency incidents
US20040015579A1 (en) * 2001-06-14 2004-01-22 Geoffrey Cooper Method and apparatus for enterprise management
US20040042470A1 (en) * 2000-06-16 2004-03-04 Geoffrey Cooper Method and apparatus for rate limiting
US20040109453A1 (en) * 2002-12-10 2004-06-10 Jarryl Wirth Graphical system and method for editing multi-layer data packets
US20050060402A1 (en) * 2002-09-10 2005-03-17 Randy Oyadomari Propagation of signals between devices for triggering capture of network data
US20050131876A1 (en) * 2003-12-10 2005-06-16 Ahuja Ratinder Paul S. Graphical user interface for capture system
US6988141B1 (en) * 2000-05-17 2006-01-17 Ricoh Company, Ltd. Method and system of remote diagnostic, control and information collection using a dynamic linked library of multiple formats and multiple protocols with restriction on protocol
US20060077895A1 (en) * 2004-06-04 2006-04-13 Cary Wright Protocol emulator
US7080399B1 (en) * 1999-01-22 2006-07-18 Matsushita Electric Industrial Co., Ltd. Network control system, and controller, target and consumer for use in the network control system
US20060256735A1 (en) * 2005-05-13 2006-11-16 Hewlett Packard Company Intellectual Property Administration Method and apparatus for centrally configuring network devices
US20070011309A1 (en) * 2003-12-03 2007-01-11 Network Intelligence Corporation Network event capture and retention system
US20070050846A1 (en) * 2005-08-30 2007-03-01 Fortinet, Inc. Logging method, system, and device with analytical capabilities for the network traffic
US20070067450A1 (en) * 2005-08-19 2007-03-22 Malloy Patrick J Managing captured network traffic data
US20070076312A1 (en) * 2005-08-11 2007-04-05 Wi-Tronix, Llc Universal event/data recorder system
US20070083644A1 (en) * 2005-10-12 2007-04-12 Microsoft Corporation Capturing, displaying, and re-creating network conversations and state information
US20070121872A1 (en) * 2005-09-09 2007-05-31 Infineon Technologies Ag Apparatus and method for controlling a telecommunications conference
US7228348B1 (en) * 2002-08-13 2007-06-05 Finisar Corporation System and method for triggering communications data capture
US20070156916A1 (en) * 2005-12-30 2007-07-05 Senactive It-Dienstleistungs Gmbh Method of correlating events in data packet streams
US7246162B2 (en) * 2000-12-06 2007-07-17 Intelliden System and method for configuring a network device
US7260846B2 (en) * 2002-07-30 2007-08-21 Steelcloud, Inc. Intrusion detection system
US20070260932A1 (en) * 2006-04-11 2007-11-08 Ryan Prichard Event log management system
US20080082679A1 (en) * 2006-09-29 2008-04-03 Sap Ag Method and system for management protocol-based data streaming
US20080259910A1 (en) * 2004-07-13 2008-10-23 International Business Machines Corporation Dynamic Media Content For Collaborators With VOIP Support For Client Communications
US20090271504A1 (en) * 2003-06-09 2009-10-29 Andrew Francis Ginter Techniques for agent configuration
US20090319247A1 (en) * 2008-06-18 2009-12-24 Eads Na Defense Security And Systems Solutions Inc Systems and Methods for A Simulated Network Environment and Operation Thereof
US7644438B1 (en) * 2004-10-27 2010-01-05 Arcsight, Inc. Security event aggregation at software agent
US7660892B2 (en) * 2005-01-24 2010-02-09 Daintree Networks, Pty. Ltd. Network analysis system and method
US7814218B1 (en) * 2002-10-17 2010-10-12 Astute Networks, Inc. Multi-protocol and multi-format stateful processing
US20100318836A1 (en) * 2009-06-11 2010-12-16 Microsoft Corporation Monitoring and healing a computing system
US20110026521A1 (en) * 2009-07-31 2011-02-03 Gamage Nimal K K Apparatus and methods for forwarding data packets captured from a network
US7886054B1 (en) * 2000-10-11 2011-02-08 Siddhartha Nag Graphical user interface (GUI) for administering a network implementing media aggregation
US20110246134A1 (en) * 2010-04-05 2011-10-06 Tektronix, Inc. Real Time Statistical Triggers on Data Streams
US20120106354A1 (en) * 2009-07-31 2012-05-03 Anue Systems, Inc. Superset packet forwarding for overlapping filters and related systems and methods
US20130179855A1 (en) * 2012-01-09 2013-07-11 John Elliott Methods and apparatuses to display interface protocol activity
US20130246925A1 (en) * 2009-03-25 2013-09-19 Mcafee, Inc. System and method for managing data and policies
US8583772B2 (en) * 2008-08-14 2013-11-12 International Business Machines Corporation Dynamically configurable session agent
US20140046645A1 (en) * 2009-05-04 2014-02-13 Camber Defense Security And Systems Solutions, Inc. Systems and methods for network monitoring and analysis of a simulated network
US20140173512A1 (en) * 2012-12-19 2014-06-19 Teledyne Lecroy, Inc. Field Selection Graphical User Interface
US20140201375A1 (en) * 2013-01-11 2014-07-17 Anuta Networks, Inc. Method, apparatus and system pertaining to cloud computing
US20140237292A1 (en) * 2013-02-21 2014-08-21 Advantest Corporation Gui implementations on central controller computer system for supporting protocol independent device testing
US20140280737A1 (en) * 2013-03-14 2014-09-18 Cisco Technology, Inc. Method for streaming packet captures from network access devices to a cloud server over http
US20140351415A1 (en) * 2013-05-24 2014-11-27 PacketSled Inc. Selective packet capture
US20150062113A1 (en) * 2009-11-09 2015-03-05 International Business Machines Corporation Activity triggered photography in metaverse applications
US20150125807A1 (en) * 2012-04-06 2015-05-07 Aseptico Inc. Method for cutting or abrading with a tool, and related drivers and systems
US20150156170A1 (en) * 2013-12-03 2015-06-04 Alcatel-Lucent Usa Inc. Security Event Routing In a Distributed Hash Table
US9110101B2 (en) * 2012-02-17 2015-08-18 Vencore Labs, Inc. Method and system for packet acquisition, analysis and intrusion detection in field area networks
US20150319058A1 (en) * 2012-12-07 2015-11-05 Telefonaktiebolaget L M Ericsson (Publ) Selective event reporting in a mobile telecommunications network
US20160155314A1 (en) * 2013-07-10 2016-06-02 Seal Innovation, Inc. Water Safety Monitoring Systems and Related Methods

Patent Citations (63)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5436618A (en) * 1992-09-17 1995-07-25 U.S. Philips Corporation Multi-apparatus consumer electronics system
US5920711A (en) * 1995-06-02 1999-07-06 Synopsys, Inc. System for frame-based protocol, graphical capture, synthesis, analysis, and simulation
US5892903A (en) * 1996-09-12 1999-04-06 Internet Security Systems, Inc. Method and apparatus for detecting and identifying security vulnerabilities in an open network computer communication system
US5983270A (en) * 1997-03-11 1999-11-09 Sequel Technology Corporation Method and apparatus for managing internetwork and intranetwork activity
US6594634B1 (en) * 1998-09-14 2003-07-15 Medtronic Physio-Control Corp. Method and apparatus for reporting emergency incidents
US7080399B1 (en) * 1999-01-22 2006-07-18 Matsushita Electric Industrial Co., Ltd. Network control system, and controller, target and consumer for use in the network control system
US6542861B1 (en) * 1999-03-31 2003-04-01 International Business Machines Corporation Simulation environment cache model apparatus and method therefor
US6988141B1 (en) * 2000-05-17 2006-01-17 Ricoh Company, Ltd. Method and system of remote diagnostic, control and information collection using a dynamic linked library of multiple formats and multiple protocols with restriction on protocol
US20040042470A1 (en) * 2000-06-16 2004-03-04 Geoffrey Cooper Method and apparatus for rate limiting
US7886054B1 (en) * 2000-10-11 2011-02-08 Siddhartha Nag Graphical user interface (GUI) for administering a network implementing media aggregation
US7246162B2 (en) * 2000-12-06 2007-07-17 Intelliden System and method for configuring a network device
US7650396B2 (en) * 2000-12-06 2010-01-19 Intelliden, Inc. System and method for defining a policy enabled network
US20030061506A1 (en) * 2001-04-05 2003-03-27 Geoffrey Cooper System and method for security policy
US20030120619A1 (en) * 2001-05-23 2003-06-26 Osborn Mark David Optimizing storage and retrieval of monitoring data
US20040015579A1 (en) * 2001-06-14 2004-01-22 Geoffrey Cooper Method and apparatus for enterprise management
US7260846B2 (en) * 2002-07-30 2007-08-21 Steelcloud, Inc. Intrusion detection system
US7228348B1 (en) * 2002-08-13 2007-06-05 Finisar Corporation System and method for triggering communications data capture
US8266271B2 (en) * 2002-09-10 2012-09-11 Jds Uniphase Corporation Propagation of signals between devices for triggering capture of network data
US20050060402A1 (en) * 2002-09-10 2005-03-17 Randy Oyadomari Propagation of signals between devices for triggering capture of network data
US7814218B1 (en) * 2002-10-17 2010-10-12 Astute Networks, Inc. Multi-protocol and multi-format stateful processing
US20040109453A1 (en) * 2002-12-10 2004-06-10 Jarryl Wirth Graphical system and method for editing multi-layer data packets
US20090271504A1 (en) * 2003-06-09 2009-10-29 Andrew Francis Ginter Techniques for agent configuration
US20070011309A1 (en) * 2003-12-03 2007-01-11 Network Intelligence Corporation Network event capture and retention system
US20050131876A1 (en) * 2003-12-10 2005-06-16 Ahuja Ratinder Paul S. Graphical user interface for capture system
US20060077895A1 (en) * 2004-06-04 2006-04-13 Cary Wright Protocol emulator
US20080259910A1 (en) * 2004-07-13 2008-10-23 International Business Machines Corporation Dynamic Media Content For Collaborators With VOIP Support For Client Communications
US7644438B1 (en) * 2004-10-27 2010-01-05 Arcsight, Inc. Security event aggregation at software agent
US7660892B2 (en) * 2005-01-24 2010-02-09 Daintree Networks, Pty. Ltd. Network analysis system and method
US20060256735A1 (en) * 2005-05-13 2006-11-16 Hewlett Packard Company Intellectual Property Administration Method and apparatus for centrally configuring network devices
US20070076312A1 (en) * 2005-08-11 2007-04-05 Wi-Tronix, Llc Universal event/data recorder system
US7953425B2 (en) * 2005-08-11 2011-05-31 Wi-Tronix, Llc Universal event/data recorder system
US20070067450A1 (en) * 2005-08-19 2007-03-22 Malloy Patrick J Managing captured network traffic data
US8601122B2 (en) * 2005-08-19 2013-12-03 Riverbed Technology, Inc. Managing captured network traffic data
US8140665B2 (en) * 2005-08-19 2012-03-20 Opnet Technologies, Inc. Managing captured network traffic data
US20070050846A1 (en) * 2005-08-30 2007-03-01 Fortinet, Inc. Logging method, system, and device with analytical capabilities for the network traffic
US20070121872A1 (en) * 2005-09-09 2007-05-31 Infineon Technologies Ag Apparatus and method for controlling a telecommunications conference
US7899444B2 (en) * 2005-09-09 2011-03-01 Infineon Technologies Ag Apparatus and method for controlling a telecommunications conference
US20070083644A1 (en) * 2005-10-12 2007-04-12 Microsoft Corporation Capturing, displaying, and re-creating network conversations and state information
US20070156916A1 (en) * 2005-12-30 2007-07-05 Senactive It-Dienstleistungs Gmbh Method of correlating events in data packet streams
US20070260932A1 (en) * 2006-04-11 2007-11-08 Ryan Prichard Event log management system
US20080082679A1 (en) * 2006-09-29 2008-04-03 Sap Ag Method and system for management protocol-based data streaming
US20090319247A1 (en) * 2008-06-18 2009-12-24 Eads Na Defense Security And Systems Solutions Inc Systems and Methods for A Simulated Network Environment and Operation Thereof
US8583772B2 (en) * 2008-08-14 2013-11-12 International Business Machines Corporation Dynamically configurable session agent
US20130246925A1 (en) * 2009-03-25 2013-09-19 Mcafee, Inc. System and method for managing data and policies
US20140046645A1 (en) * 2009-05-04 2014-02-13 Camber Defense Security And Systems Solutions, Inc. Systems and methods for network monitoring and analysis of a simulated network
US20100318836A1 (en) * 2009-06-11 2010-12-16 Microsoft Corporation Monitoring and healing a computing system
US8842548B2 (en) * 2009-07-31 2014-09-23 Anue Systems, Inc. Superset packet forwarding for overlapping filters and related systems and methods
US20110026521A1 (en) * 2009-07-31 2011-02-03 Gamage Nimal K K Apparatus and methods for forwarding data packets captured from a network
US20120106354A1 (en) * 2009-07-31 2012-05-03 Anue Systems, Inc. Superset packet forwarding for overlapping filters and related systems and methods
US20150062113A1 (en) * 2009-11-09 2015-03-05 International Business Machines Corporation Activity triggered photography in metaverse applications
US20110246134A1 (en) * 2010-04-05 2011-10-06 Tektronix, Inc. Real Time Statistical Triggers on Data Streams
US20130179855A1 (en) * 2012-01-09 2013-07-11 John Elliott Methods and apparatuses to display interface protocol activity
US9110101B2 (en) * 2012-02-17 2015-08-18 Vencore Labs, Inc. Method and system for packet acquisition, analysis and intrusion detection in field area networks
US20150125807A1 (en) * 2012-04-06 2015-05-07 Aseptico Inc. Method for cutting or abrading with a tool, and related drivers and systems
US20150319058A1 (en) * 2012-12-07 2015-11-05 Telefonaktiebolaget L M Ericsson (Publ) Selective event reporting in a mobile telecommunications network
US20140173512A1 (en) * 2012-12-19 2014-06-19 Teledyne Lecroy, Inc. Field Selection Graphical User Interface
US20140201375A1 (en) * 2013-01-11 2014-07-17 Anuta Networks, Inc. Method, apparatus and system pertaining to cloud computing
US20140237292A1 (en) * 2013-02-21 2014-08-21 Advantest Corporation Gui implementations on central controller computer system for supporting protocol independent device testing
US20140280737A1 (en) * 2013-03-14 2014-09-18 Cisco Technology, Inc. Method for streaming packet captures from network access devices to a cloud server over http
US9043439B2 (en) * 2013-03-14 2015-05-26 Cisco Technology, Inc. Method for streaming packet captures from network access devices to a cloud server over HTTP
US20140351415A1 (en) * 2013-05-24 2014-11-27 PacketSled Inc. Selective packet capture
US20160155314A1 (en) * 2013-07-10 2016-06-02 Seal Innovation, Inc. Water Safety Monitoring Systems and Related Methods
US20150156170A1 (en) * 2013-12-03 2015-06-04 Alcatel-Lucent Usa Inc. Security Event Routing In a Distributed Hash Table

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Wireshark User’s Guide for Wireshark 1.9 (10/18/2012 http://wayback.archive.org/web/20121018193345/http://www.wireshark.org/download/docs/user-guide-us.pdf; dated 10/18/2012; last accessed 11/29/2016) *
Zabbix Network Monitoring Essentials Tutorial Video (https://www.youtube.com/watch?v=NLt_qR6yKWM&list=PLYoq8isOGa_N7mT7t4k2I9lCFdg2O64wj&index=17; pub date: 10/22/2013) *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160283307A1 (en) * 2014-07-28 2016-09-29 Hitachi, Ltd. Monitoring system, monitoring device, and test device

Similar Documents

Publication Publication Date Title
Moustafa et al. UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set)
US10193901B2 (en) Interface providing an interactive timeline for evaluating instances of potential network compromise
US8010689B2 (en) Locational tagging in a capture system
US6549208B2 (en) Information security analysis system
US7047423B1 (en) Information security analysis system
US8069210B2 (en) Graph based bot-user detection
US20160359697A1 (en) Mdl-based clustering for application dependency mapping
US20110296005A1 (en) Method and system for monitoring control signal traffic over a computer network
US10142353B2 (en) System for monitoring and managing datacenters
US8996728B2 (en) Obfuscating network traffic from previously collected network traffic
US9130826B2 (en) System and related method for network monitoring and control based on applications
US8549650B2 (en) System and method for three-dimensional visualization of vulnerability and asset data
US10122575B2 (en) Log collection, structuring and processing
Maier et al. Enriching network security analysis with time travel
US8248227B2 (en) Method and apparatus for suppressing duplicate alarms
US20110314148A1 (en) Log collection, structuring and processing
EP1999890B1 (en) Automated network congestion and trouble locator and corrector
Kandula et al. What's going on?: learning communication rules in edge networks
US9225601B2 (en) Network-wide verification of invariants
US20120246303A1 (en) Log collection, structuring and processing
US9003023B2 (en) Systems and methods for interactive analytics of internet traffic
US20140075557A1 (en) Streaming Method and System for Processing Network Metadata
US9813447B2 (en) Device and related method for establishing network policy based on applications
JP2015502060A (en) Streaming method and system for processing network metadata
Jiang et al. Lightweight application classification for network management

Legal Events

Date Code Title Description
AS Assignment

Owner name: SPLUNK INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SCHERBAKOV, VLADIMIR A.;DICKEY, MICHAEL R.;NOEL, CARY GLEN;AND OTHERS;SIGNING DATES FROM 20141028 TO 20141029;REEL/FRAME:034219/0707

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION