CN117579523A

CN117579523A - Distributed event high-speed acquisition and analysis system

Info

Publication number: CN117579523A
Application number: CN202311529853.XA
Authority: CN
Inventors: 陶源; 李末岩; 刘楠; 游志勇
Original assignee: Third Research Institute of the Ministry of Public Security
Current assignee: Third Research Institute of the Ministry of Public Security
Priority date: 2023-11-15
Filing date: 2023-11-15
Publication date: 2024-02-20

Abstract

The invention discloses a distributed event high-speed acquisition analysis system and a distributed event high-speed acquisition analysis method, wherein the distributed event high-speed acquisition analysis system comprises a distributed event acquisition probe and an analysis platform; the distributed event acquisition probes are distributed and deployed, the working state is adjusted according to an analysis platform management strategy, and events sent by equipment are received; the analysis platform can monitor the running state of the event probes, can issue a management strategy to the distributed event acquisition probes, adjust the working state of the event acquisition probes, and simultaneously receive and analyze the event probes in different dimensions according to different event acquisition probes. The distributed event acquisition scheme provided by the invention supports data acquisition and processing under complex environment and huge data volume, can adaptively and dynamically adjust the event acquisition probes at proper positions in a management domain, improves the event receiving and forwarding efficiency, reduces load imbalance and improves the transmission efficiency.

Description

Distributed event high-speed acquisition and analysis system

Technical Field

The invention relates to a network data technology, in particular to data acquisition and processing.

Background

Log acquisition and traffic acquisition are two common event acquisition schemes. Log collection is typically used to capture detailed log information generated by an application, system, or device, which logs events such as operations, errors, warnings, and the like. The logs may be stored and analyzed centrally in text files or sent to a central log server via a specific protocol. Traffic collection is the acquisition of data by monitoring network traffic, which captures detailed information of network packets, including source IP, destination IP, ports, protocols, etc. Both acquisition schemes play an important role in different application scenarios, but they also each face some problems in acquisition efficiency and reliability.

In the log acquisition scheme, acquisition efficiency is affected by a number of factors. First, the frequency and number of logs generated may increase dramatically, especially in high load environments. This can lead to performance problems in terms of storage and transmission, requiring the use of efficient storage and transmission mechanisms to handle large amounts of log data. In addition, too many logs may complicate the analysis process, reducing real-time. In terms of reliability, log collection may face the problem of log loss, especially in the event of system failure or high load, where critical event information may be lost. Furthermore, the time stamps of the logs may not guarantee an accurate order, which may affect the chronological analysis of the events. Proper log storage and retention policies are also necessary, which may otherwise lead to data loss or compliance issues.

For traffic collection schemes, collection efficiency is limited by hardware and network equipment. Capturing a large number of network packets requires high throughput hardware while requiring efficient storage and processing power to handle the vast data streams. Appropriate filtering and aggregation mechanisms are also necessary to reduce the amount of data and improve efficiency. In terms of reliability, traffic collection may face data loss problems, especially in high load situations, where data packet loss may occur, resulting in event loss or intrusion detection holes. Furthermore, capturing network traffic may involve sensitive information and thus requires careful handling to comply with privacy regulations. Note also that traffic collection may place a burden on the network itself, possibly leading to network congestion problems, requiring careful planning and configuration of the traffic capture device.

Disclosure of Invention

Aiming at the problems of the existing event acquisition scheme in acquisition efficiency and reliability, an efficient event acquisition scheme is needed.

Therefore, the invention aims to provide a distributed event high-speed acquisition analysis system based on a distributed event acquisition probe so as to realize efficient acquisition of events.

In order to achieve the above purpose, the distributed event high-speed acquisition and analysis system provided by the invention mainly comprises a distributed event acquisition probe and an analysis platform;

the distributed event acquisition probes are used for carrying out event acquisition by placing proper number of event acquisition probes at proper positions in a management domain according to a self-adaptive algorithm, and an acquisition scheme is formed;

the analysis platform can monitor the running state of the event probes, can issue a management strategy to the distributed event acquisition probes, adjust the working state of the event acquisition probes, and simultaneously receive and analyze the event probes in different dimensions according to different event acquisition probes.

In some embodiments of the present invention, the distributed event acquisition probes are distributed on different network nodes, monitor network changes based on an adaptive algorithm, acquire network state information in real time, and transmit the network state information to an analysis platform; the distributed event acquisition probes can receive a management strategy formed by the analysis platform according to the real-time network state information, and automatically adjust the working states of the corresponding event acquisition probes according to the management strategy so as to realize dynamic adjustment of the distributed states of the distributed event acquisition probes.

In some embodiments of the present invention, the distributed event collection probe can automatically select the most suitable transmission protocol according to the real-time network environment characteristics based on the transmission protocol selection policy issued by the analysis platform.

In some embodiments of the present invention, the analysis platform is configured to set a code for each event acquisition probe according to the status information reported by the distributed event acquisition probes, and accordingly determine the working status of the distributed event acquisition probes according to a genetic algorithm, and simultaneously issue a relevant management policy to the corresponding event acquisition probes.

In some embodiments of the invention, the distributed event acquisition probe deploys event acquisition probes based on encoded genetic algorithms under link coverage and message coverage constraints.

In some embodiments of the present invention, corresponding data receiving interfaces are established for different types of event acquisition probes in the analysis platform, and corresponding data analysis modules are established for different data dimensions.

In some embodiments of the present invention, the event collection probes in the distributed event collection probes can form a registration state in the analysis platform, and can actively report own ready information to the analysis platform.

The analysis platform monitors the running state of each event acquisition probe according to the registration information of the event acquisition probe in the platform and the actively reported ready information, and simultaneously generates a corresponding management strategy and sends the management strategy to the corresponding event acquisition probe. The event acquisition probe enters a working state according to the received management strategy, acquires the event and forwards the event to the analysis platform. And the analysis platform collects probes according to different events and receives and analyzes the probes in different dimensionalities.

In some embodiments of the present invention, the analysis platform is capable of screening out an optimal working group of event acquisition probes in the distributed event acquisition probes based on a genetic algorithm according to the running state of each event acquisition probe.

In some embodiments of the present invention, the analysis platform sets a gray code for each event acquisition probe according to the equipment status information and the load condition reported by the event acquisition probe, adopts playback-free random selection as a selection operator, performs an arithmetic crossover operation on the selected individuals, and finally performs gaussian approximate variation to obtain the individuals of the optimal population.

In some embodiments of the invention, the analysis platform adopts a heartbeat strategy based on time difference, and simultaneously adopts bidirectional alternate communication detection operation, so that the highest requirement of transmission efficiency can be met, and the operation state of the event probe can be monitored in real time in a practical manner to the greatest extent.

In some embodiments of the invention, the system enables fast support for added device types based on XSD files.

In some embodiments of the present invention, a rule engine is further configured in the system, and the rule engine interacts with the distributed event collection probe data, so that the events collected and forwarded by the distributed event collection probe can be filtered and screened.

In some embodiments of the present invention, in the collecting method, for the collected event forwarded by the event collecting probe, first, a rule engine is used to perform selection filtering to generate relevant alarm information, and the alarm information and the screened event are sent to an analysis platform.

The distributed event acquisition scheme provided by the invention supports data acquisition and processing under complex environment and huge data volume, can adaptively and dynamically adjust the event acquisition probes at proper positions in a management domain, improves the event receiving and forwarding efficiency, reduces load imbalance and improves the transmission efficiency.

The distributed event acquisition scheme provided by the invention supports multiple transmission mechanisms and can effectively make up the defect of a single transmission mechanism; meanwhile, each event acquisition probe adopts various mechanisms to receive logs sent by equipment, and uniformly sends the logs to a rule engine for screening and filtering, so that the accuracy and reliability requirements on the logs are increased.

The distributed event acquisition scheme provided by the invention supports event acquisition through the distributed probe, the event type comprises log and flow information, and various log acquisition modes are supported.

The event acquisition probe in the distributed event acquisition scheme provided by the invention supports various transmission protocols, and does not need to specify which transmission mechanism is required to be adopted by equipment, so that the operability of operation and maintenance personnel is improved, and the implementation efficiency is improved.

The distributed event acquisition scheme provided by the invention can automatically adjust the distribution of the local event acquisition probes according to the change of the network, so that the analysis platform reaches an approximately optimal state; meanwhile, the event acquisition probes can select an optimal transmission protocol according to the characteristics of the network environment of the user, and the analysis platform receives and analyzes different dimensionalities according to different event acquisition probes.

The distributed event acquisition scheme provided by the invention supports event acquisition through the distributed probe, and the event type comprises log and flow information. The supported log acquisition mode at least comprises a system log, an SNMP Trap, a log file, a JDBC, a Java database connection, a system plug-in, an Ftp, a file transfer protocol and the like, and supports flow data acquisition including NetFlow, sFlow, netStream and the like; and meanwhile, the unified log format is adopted for storage, so that the high-speed log acquisition capability is realized.

The types of the data source equipment or the system supported by the distributed event acquisition scheme provided by the invention cover the network equipment, the security equipment, the server, the database, the middleware, the application system and the like of the main stream, and the types of the equipment directly supported are not less than 100.

The distributed event acquisition scheme provided by the invention can realize the support for the newly added equipment type by only loading the configuration file without modifying codes.

The distributed event acquisition scheme provided by the invention can dynamically control the event acquisition probes, and dynamically increase and decrease the number of the event acquisition probes according to actual needs.

Drawings

The invention is further described below with reference to the drawings and the detailed description.

FIG. 1 is a flow chart of a process for receiving a system log and a Netflow log in an example of the invention;

FIG. 2 is a flowchart of a Windows system log collection in an example of the present invention;

FIG. 3 is a flow chart of event source log collection performed in an example of the invention;

FIG. 4 is a flowchart of a database table log collection performed in accordance with an embodiment of the present invention.

Detailed Description

The invention is further described with reference to the following detailed drawings in order to make the technical means, the creation characteristics, the achievement of the purpose and the effect of the implementation of the invention easy to understand.

The event types involved in the scheme of the invention comprise logs and flow information.

The scheme of the invention is based on the distributed event acquisition probe to acquire the event, thereby realizing data acquisition and processing under the conditions of complex environment and huge data volume; on the basis, a distributed log acquisition analysis scheme is formed by further combining a remote dictionary service, a multi-line Cheng Wanglao library based on a message queue, a search data analysis Engine (ES) and the like.

Accordingly, the invention discloses a distributed event high-speed acquisition and analysis system which is used for carrying out distributed event high-speed acquisition and analysis based on a distributed event acquisition probe.

The distributed event high-speed acquisition and analysis system is mainly formed by mutually matching a distributed event acquisition probe and an analysis platform.

The distributed event acquisition probe in the system is used for realizing data acquisition and processing under the conditions of complex environment and huge data volume;

specifically, the distributed event acquisition probes are configured to place a proper number of event acquisition probes at proper positions in the management domain according to the adaptive algorithm for event acquisition; and on the basis, a distributed log acquisition scheme can be formed by further combining a remote dictionary service, a multi-line Cheng Wanglao library based on a message queue, a search data analysis Engine (ES) and the like.

In cooperation with the system, the analysis platform in the system interacts with the front-end distributed event acquisition probe data, can monitor the running states of all distributed event acquisition probes, can issue a management strategy to the distributed event acquisition probes, adjusts the working states of the event acquisition probes, and simultaneously receives and analyzes different dimensionalities according to different event acquisition probes.

Accordingly, the analysis platform manages the event acquisition probes by configuring the corresponding event acquisition probe unified manager.

The implementation scheme and corresponding technical characteristics of the distributed event high-speed acquisition and analysis system (hereinafter referred to as a system) provided by the scheme of the invention are specifically described below.

The distributed event acquisition probes in the system are configured to cooperate with the analysis platform to automatically adjust the distribution of the local event acquisition probes according to the change of the network, so that the analysis platform reaches an approximately optimal state.

According to the system scheme, firstly, a plurality of event acquisition probes are deployed in a network and distributed on different network nodes, each event acquisition probe is configured to monitor network changes by adopting an adaptive algorithm, acquire network state information such as flow, delay and topological structure in real time, and transmit the network state information to an analysis platform.

And secondly, the distributed event acquisition probes can receive a management strategy formed by the analysis platform according to the real-time network state information, and automatically adjust the working states of the corresponding event acquisition probes according to the management strategy so as to realize the dynamic adjustment of the distributed event acquisition probes, for example, the distributed event acquisition probes can be realized by dynamically adjusting the number, the positions, the acquisition ranges and the like of the event acquisition probes in the corresponding working states. As a supplementary explanation, the dynamic adjustment process needs to consider the load balance of the network, so as to avoid overload or acquisition overlapping of a single probe;

finally, the analysis platform continuously optimizes the self-adaptive algorithm by monitoring and analyzing the event data in the network so as to adapt to the event acquisition requirements under different network environments, and realize the automatic adjustment of the distributed event acquisition probes, so that the analysis platform reaches an approximate optimal state.

In some embodiments of the system, the distributed event acquisition probe is further configured to cooperate with the analysis platform to enable selection of an optimal transmission protocol according to characteristics of a user network environment.

Accordingly, the system scheme firstly acquires user network state information such as flow, delay and topology structure in real time based on a plurality of event acquisition probes distributed on different network nodes, and transmits the information to an analysis platform.

Secondly, the analysis platform analyzes the characteristics of the network environment of the user, including factors such as network topology, bandwidth, delay and the like, so as to determine the selection standard of the optimal transmission protocol;

and finally, the analysis platform establishes a transmission protocol selection strategy according to the analysis result and sends the transmission protocol selection strategy to a corresponding event acquisition probe, and the event acquisition probe can automatically select the most suitable transmission protocol according to the real-time network environment characteristics based on the transmission protocol selection strategy to establish data transmission with the analysis platform. By way of example, the transport protocol herein may be TCP, UDP or other custom protocols.

In some embodiments of the system solution, when the distributed event acquisition probes are deployed across domains, the distributed event acquisition probes can place a proper number of event acquisition probes at proper positions in the management domain according to the adaptive algorithm.

In the specific implementation, based on an event acquisition probe manager configured in an analysis platform, the event acquisition probe manager is configured to set codes for each event acquisition probe according to state information reported by the event acquisition probes, so that which event acquisition probes are selected to work according to a genetic algorithm, and related strategies are issued.

On the basis, the distributed event acquisition probes perform event acquisition probe deployment based on the encoded genetic algorithm under the constraint conditions of link coverage and message coverage.

Specifically, the network topology and event message coverage are coded and represented, and then optimized searching is performed by using a genetic algorithm, so that the optimal probe deployment scheme is formed. By way of example, a new deployment scenario is generated by genetic algorithms through crossover, mutation, etc., and is evaluated and selected according to link coverage and message coverage constraints to gradually optimize probe deployment.

In the iterative process of the genetic algorithm, the deployment scheme is continuously updated and optimized until the optimal solution meeting the constraint condition is reached. The method can realize intelligent deployment of the distributed event acquisition probes under the condition of considering the constraint of link coverage and message coverage, so as to improve the event data acquisition efficiency and coverage to the greatest extent.

By way of example, when a plurality of sets of event acquisition probes are deployed in a cross-domain manner according to the actual network environment of a user, the above scheme is combined to be deployed in a complex area of the network environment, and the network environment is simpler, so that the distributed event acquisition probes are deployed simply, and the effect of maximizing the acquisition efficiency by minimizing the number of event acquisition probes can be achieved.

As a further illustration, each of the distributed event collection probes, after deployment, will be data-connected with a corresponding collection object, such as a proxy or original equipment, for real-time event collection, such as log and traffic information collection.

Specifically, each event acquisition probe in the distributed event acquisition probes performs event acquisition on an agent or original equipment based on a corresponding passive monitoring port, namely the acquisition agent or the original equipment sends an original log to the passive monitoring port of the event acquisition probe, and the event acquisition probe acquires system logs and flow data through the passive monitoring port of a data acquisition layer.

As a further illustration, all event collection probes in the distributed event collection probes work independently, process received events in parallel and forward to an analysis platform.

In some embodiments of the system scheme, the analysis platform is matched with the distributed event acquisition probes at the front end, so that the receiving and analyzing of different dimensionalities are realized according to different event acquisition probes.

In concrete implementation, corresponding data receiving interfaces are established in the analysis platform aiming at different types of event acquisition probes so as to support various data formats and transmission protocols; meanwhile, corresponding data analysis modules are constructed for different data dimensions in the analysis platform, and the analysis platform specifically comprises a log data analysis module and a flow data analysis module, so that the collected log data and the flow data can be analyzed in parallel, the speed of data analysis processing is ensured, and the diversified data processing requirements can be met.

When the analysis platform configured in this way is matched with the distributed event acquisition probe at the front end, in the data receiving process, a proper data receiving interface is adapted according to the type and the data characteristics of the event acquisition probe, and the received data is preprocessed, including operations such as de-duplication, data format conversion and the like; then, according to different dimensionalities of the preprocessed data, a corresponding analysis module (a log data analysis module or a flow data analysis module) is called to analyze the data, and the data is converted into a format which can be analyzed and stored; and finally, storing the parsed data into a big data engine ES, and providing corresponding query and analysis interfaces to support multi-dimensional data visualization presentation.

In some embodiments of the system scheme, a registration management mode is preferably adopted between the analysis platform and the distributed event acquisition probes in the system scheme to perform real-time and dynamic management on the distributed event acquisition probes deployed in different areas. That is, the analysis platform receives the event sent by the event acquisition probe which completes registration in the analysis platform, and can only issue a management policy to the event acquisition probe which completes registration in the analysis platform to manage the event acquisition probe.

All event acquisition probes in a specific distributed event acquisition probe are registered in an analysis platform before working, and a registration state can be formed in the analysis platform; after registration is completed, each event acquisition probe actively reports own ready information, such as an IP address, an MAC address, performance indexes such as CPU utilization rate, memory utilization rate and the like, to an analysis platform, so that the analysis platform can know the equipment state of each event acquisition probe so as to screen out an optimal population, and each event acquisition probe is ready to receive a strategy issued by the analysis platform in real time and enters a working state.

For example, when the event collection probe completes registration in the analysis platform, the event collection probe preferably reports its own device unique ID to the analysis platform, and the device can be monitored after reporting.

On the basis, the analysis platform in the system can monitor the running state of each event acquisition probe according to the registration information of the event acquisition probe in the platform and the actively reported ready information, and can also form a corresponding management strategy according to the running state of each event acquisition probe, and send the management strategy to the corresponding event acquisition probe to control and adjust the working state of the corresponding event acquisition probe.

Further, after the event acquisition probe is registered in the analysis platform, the analysis platform can perform centralized control on the registered probes, including start and stop, and key parameter configuration.

By further describing, the analysis platform in the system monitors the working states of the event acquisition probes, can determine the running state of each event acquisition probe in real time according to the registration information of each event acquisition probe in the platform and the ready information actively reported according to the distributed event acquisition probes deployed in different areas, then screens out the optimal event acquisition probes in the distributed event acquisition probes based on the coded genetic algorithm by combining with the acquisition task to form an optimal event acquisition probe working group, and accordingly forms a corresponding management strategy and sends the optimal event acquisition probes to the screened event acquisition probes to control the screened event acquisition probes to enter the corresponding working states, so that the effect of maximizing the acquisition efficiency by minimizing the number of the event acquisition probes is achieved. According to the method, the number of the event acquisition probes can be dynamically increased or decreased and the distribution area and range of the event acquisition probes can be adjusted according to actual conditions and demand changes.

As a preferred scheme, when the analysis platform in the system screens distributed event acquisition probes deployed in different areas, an event acquisition probe unified manager deployed in the analysis platform sets a Gray code for each event acquisition probe according to equipment state information and load conditions reported by the event acquisition probes, and accordingly, a playback-free random selection is adopted as a selection operator, a crossing operation is performed on selected individuals in an arithmetic crossing mode, and finally, gaussian approximate variation is performed to obtain optimal population individuals. Such an optimization process enables individuals in the entire population to achieve optimal performance in the current environment. In addition, the unified event acquisition probe manager also records individuals in the optimal population, and issues the individuals to the corresponding event acquisition probes through strategies so as to adjust the working states of the event acquisition probes, thereby improving the efficiency and performance of the system.

In some embodiments of the system scheme, the analysis platform adopts a heartbeat strategy based on time difference to control the data communication state between the analysis platform and the distributed acquisition probes or other network devices.

The heartbeat strategy of the time difference is specifically that the analysis platform records the current time (recorded as a recvedTime) after receiving a heartbeat, meanwhile, judges the time of the timer to arrive, calculates the time (T) of how long the heartbeat is not received, namely the current time-recvedTime (the time recorded above), and if T is larger than a certain set value, the Client is considered to be overtime, the connection is disconnected.

The analysis platform adopts a heartbeat strategy based on time difference to control the data communication state between the distributed acquisition probes or other network equipment, and the specific implementation process is as follows:

the distributed acquisition probe or other network equipment periodically transmits heartbeat signals to the analysis platform to indicate the normal operation of the distributed acquisition probe or other network equipment;

after the analysis platform receives the heartbeat signal, recording the current time as a receiving time (recvedTime);

after the timing of the analysis platform timer arrives, calculating the time difference (T) between the current time and the receiving time, namely T=current time-recvedTime;

if the time difference T is greater than a certain set value, the distributed acquisition probe or other network equipment is considered to be overtime, and a fault or disconnection can occur;

the analysis platform can generate an alarm or perform corresponding fault treatment according to the timeout condition, such as attempting to reconnect the device, notifying a system administrator, etc.;

through the cooperation, the analysis platform can monitor the running state of the distributed event acquisition probe in real time, discover faults in time and process the faults, so that the stability and the reliability of the system are improved.

Furthermore, the analysis platform adopts bidirectional alternate communication detection operation at the same time, namely, both communication parties can send information, but both communication parties cannot send and receive information at the same time. The bidirectional alternate communication mode realizes that one party transmits and the other party receives, and the other party receives after a period of time. The specific implementation process is as follows:

The distributed acquisition probes periodically send the health state and performance index data of the system, such as CPU utilization rate, memory occupancy rate, network flow rate and the like, to the analysis platform;

after receiving the data sent by the acquisition probes, the analysis platform performs real-time analysis and processing to generate corresponding reports or alarms;

meanwhile, the analysis platform also sends a detection request to the distributed acquisition probes, and the acquisition probes are required to return specific performance index data or perform specific detection operation;

after receiving the detection request of the analysis platform, the acquisition probe executes corresponding operation and returns the result to the analysis platform.

The bidirectional alternate communication detection operation mode can enable the analysis platform to monitor the health state and performance index of the system in real time, and meanwhile, monitoring and management of the acquisition probes can be achieved by sending detection requests.

Therefore, the analysis platform can meet the requirement of highest transmission efficiency, and the operation state of the event probe is monitored in real time in a practical mode to the greatest extent.

In some embodiments of the system scheme, the system can rapidly realize support for the newly added device type through loading the configuration file through the analysis platform.

As a further illustration, the present system solution preferably enables fast support for newly added device types based on XSD files. Specifically, the system scheme defines index parameters to be transmitted by equipment of different equipment types through an XSD file, so that when the equipment type is newly added, only index items of the corresponding equipment type are registered in the XSD file; based on the method, the system performs dynamic loading analysis on the XML configuration file of the newly added equipment according to the XSD file through the OTA upgrading mode.

Accordingly, the implementation process of the system scheme for rapidly supporting the newly added equipment type based on the XSD file is as follows:

firstly, updating an XSD file by an analysis platform according to the characteristics of a new equipment type, and defining index parameters which need to be transmitted by the new equipment type, wherein the parameters can comprise equipment states, performance indexes, log information and the like;

and secondly, the distributed event acquisition probe can dynamically load and analyze the XML configuration file of the newly added equipment according to the updated XSD file through an OTA upgrading technology. When the new equipment is accessed into the system, the distributed acquisition probe can dynamically analyze the configuration file of the new equipment according to the rule defined by the XSD file and start to acquire index parameters of the new equipment;

And finally, the analysis platform performs corresponding data processing and analysis according to the index parameters of the new equipment types so as to meet the service requirements, thus realizing the rapid support of the newly added equipment types, and improving the flexibility and the expandability of the system.

In some embodiments of the system scheme, a corresponding rule engine is additionally arranged in the system to be used for filtering event data acquired by the distributed event acquisition probe; and receiving and analyzing the events collected and forwarded by the different event collecting probes in different dimensionalities by arranging corresponding search data analysis Engines (ES).

Specifically, the rule engine operates independently of or in the analysis platform, and interacts with the distributed event collection probe data to filter and screen events collected and forwarded by the distributed event collection probe.

On the basis, the analysis platform of the system and a rule engine and search data analysis Engine (ES) preferably adopt the following synergistic scheme to realize cooperation with a distributed event acquisition probe:

the analysis platform issues a strategy to the screened event acquisition probe, receives a system log and a flow log (namely an event) sent by the event acquisition probe by the acquisition probe, and transmits the system log and the flow log (namely the event) to the rule engine through a multi-line Cheng Wanglao library based on a message queue; at that time, the rule engine performs selection filtering through rules to generate relevant alarm information, and sends the alarm information, the screened system log and the flow log (i.e. event) to the search data analysis Engine (ES).

In some embodiments of the system scheme, the above configuration scheme forms a distributed event high-speed acquisition system, and meanwhile, the distributed event high-speed acquisition is completed through the following synergistic effect among a distributed event acquisition probe, an analysis platform and a rule engine in the distributed event high-speed acquisition system:

firstly, deploying a plurality of event acquisition probes in a cross-domain manner, wherein the event acquisition probes are deployed in a region with a complex network environment, the event acquisition probes are deployed in a key manner, and the event acquisition probes are deployed in a simple region with a simpler network environment, and all event acquisition probes are simultaneously transmitted to an analysis platform to form a registration state in the analysis platform; meanwhile, the registration is completed in the analysis platform, and the ready information of the analysis platform is actively reported to the analysis platform.

At that time, in the operation process, the analysis platform monitors the operation state of each event acquisition probe according to the registration information of the event acquisition probe in the platform and the actively reported ready information, and simultaneously sets a thunder code for each event acquisition probe by combining with the corresponding load condition, so that the required event acquisition probe is selected according to a genetic algorithm to form an optimal event acquisition probe working population, and simultaneously generates a corresponding management strategy and transmits the management strategy to the event acquisition probes in the working population.

Further, the event collection probes in the working population receive the management policies below the analysis platform, adjust the corresponding working states (e.g. enter the working states), and receive the log data and the flow data (i.e. the events) sent by the equipment according to the management policies below the analysis platform. Meanwhile, all event acquisition probes work independently, received events are processed in parallel and forwarded to a rule engine of an analysis platform for further filtering and screening, relevant alarm information is generated, the alarm information and the screened events are sent to a search data analysis Engine (ES) in the analysis platform, different dimensionalities of receiving and analyzing are carried out according to different event acquisition probes, and in addition, data can be visually presented according to service requirements.

Further, when the distributed event high-speed acquisition system is operated, the distribution of the local event acquisition probes can be automatically adjusted according to the change of the network, so that the analysis platform reaches an approximately optimal state; meanwhile, the event acquisition probe can select the optimal transmission protocol according to the characteristics of the network environment of the user.

The present invention is further illustrated below in conjunction with specific embodiments. It is to be understood that these examples are illustrative of the present invention and are not intended to limit the scope of the present invention.

Example 1:

the distributed event high-speed acquisition system provided by the invention is used for realizing acquisition of the log of the receiving system and the Netflow log.

In the example, aiming at corresponding equipment and Linux and Unix systems, logs of specified types and levels are recorded through configuration, and a server for receiving the logs is specified; accordingly, after the device configures the sending address, the device sends the log to the server receiving the log through the system log protocol. In the embodiment, the server address for receiving the log is set as the address of the event acquisition probe of the audit and pursuit platform during deployment, so that the acquired data is acquired and processed by the data receiving engine and then transferred to the file management engine for file storage, and the event acquisition probe for forwarding the log to the audit and pursuit platform is realized.

Accordingly, the types of device forwarding logs that the present example can receive include system logs and Netflow logs. The corresponding log acquisition detection is formed by matching a log receiver with a log analyzer, wherein the log receiver is written in a C language and is mainly responsible for receiving and forwarding the log; the log analyzer is written in java language, matched with the log receiver and used for analyzing and normalizing the forwarding log of the log receiver.

Referring to fig. 1, a flow chart of the processing of the system log and Netflow log received by the present example is shown.

As can be seen from the figure, the processing flow of the receiving system log and the Netflow log in this example is as follows:

1. the device sends system log and Netflow log data to a log collector in a log collection probe.

2. The log collector sends the collected logs to a log analyzer in the log collection probe.

3. The log analyzer normalizes the format of the log data according to the log type, the grade and the log analysis rule, sends the log data in the unified format to a rule filter of a rule engine, and forwards the log data to an ES of a big data platform through the rule filter.

When the system log and the Netflow log are collected, the log sending grade and the IP address of the collector are pre-configured; meanwhile, the log analysis should be pre-configured with analysis rules according to the log type.

Example 2:

the embodiment is based on the distributed event high-speed acquisition system provided by the invention to acquire Windows host logs through the system plug-in service.

The Windows system does not support log forwarding, and to obtain log information of Windows, a plug-in is installed on the Windows system to forward or open a plug-in service of the Windows system, and the log information is obtained by calling a corresponding interface of the plug-in service of the Windows system. The event acquisition probe supports acquiring Windows log through a system plug-in service mode of a Windows system, and when the event acquisition probe is to be configured, the Windows system plug-in service is required to be started and necessary access rights are provided for the acquisition probe.

Collecting Windows system logs, and installing Sensor plug-ins; the plug-in acquires a log through a system plug-in service interface and forwards the log to a log collector.

Referring to FIG. 2, a flow chart for Windows system log collection in this example is shown. As can be seen from the figure, the process of collecting the Windows system log in this example is as follows:

1. downloading Windows log to collect Sensor plug-in and installing;

2. after the Sensor plug-in is successfully installed, clicking to start the Sensor;

3. filling in a system name, an IP address, an administrator user name and a password to be monitored in a Sensor system, setting a state to be effective, and supporting batch operation; the address of the log collector and the use protocol are added in the server of the Sensor.

4. And starting a Sensor log forwarding function in a server of the Sensor.

5. The Windows system sends the log to a log collector of the log collection probe through a Sensor plug-in.

When the Windows system log acquisition is carried out, a Sensor plug-in needs to be installed in the embodiment; and meanwhile, when the Sensor plug-in is installed, the Sensor is added into a trust list of the antivirus software according to the prompt.

Example 3:

the example realizes reading of the local log file of the event source based on the distributed event high-speed acquisition system.

In this example, log forwarding is not supported for some devices or systems, but rather, the log is stored in a local log file, and some policy may be set to use the file in a rotation. The plug-in is required to be installed in the equipment where the event source is located, the plug-in polls the log file at regular time, judges the newly added log through a time stamp or an event ID (depending on the event source equipment or the system implementation mode), reads the newly added log and forwards the newly added log to the event acquisition probe through a system log protocol.

In this case, when the local log file of the event source is read, the Sensor plug-in needs to be installed in the present example; the plug-in forwards the log file of the event source to the log collector by configuring the event source information.

The event source monitoring supports event source catalog monitoring and event source log file monitoring; during directory monitoring, the file type, namely the file suffix name, needs to be specified; in file monitoring, a file to be monitored needs to be selected, and selection of a plurality of files is supported.

Accordingly, the specific implementation flow of the present example for reading the event source local log file is as follows (see fig. 3):

1. the user downloads a log acquisition Sensor plug-in the system and installs the Sensor plug-in;

2. after the Sensor plug-in is successfully installed, starting the Sensor;

3. Filling in a file name to be monitored in a Sensor file monitoring configuration module, filling in a monitoring directory, a suffix name and an event processor in directory monitoring, selecting a file (optional), a content separator and an event processor to be monitored in file monitoring, and setting the state to be effective; the address of the log collector and the use protocol are added in the server of the Sensor.

4. And starting a Sensor log forwarding function in a server of the Sensor.

5. The Sensor plug-in sends the acquired event source log to a log collector of the log collection probe.

When the local log file of the event source is read, a Sensor plug-in needs to be installed in the embodiment; and meanwhile, when the Sensor plug-in is installed, the Sensor is added into a trust list of the antivirus software according to the prompt.

Example 4:

the example realizes access to the event source log database through JDBC connection based on the distributed event high-speed acquisition system provided by the invention.

For some devices or systems that store logs in database tables on local or designated servers, the event acquisition probes in this example support connecting to a remote database through JDBC, accessing the corresponding database table to obtain log information. The account number of the event acquisition probe for accessing the database and the access right to the log data table are provided during deployment.

In the embodiment, a Sensor plug-in is required to be installed in the process of collecting database logs; the plug-in is connected with a remote database through JDBC, periodically queries a data table according to preset SQL sentences, and forwards the query result to a log collector as a log. The database types here support MySql, SQL Server, oracle, SQLite.

Accordingly, the concrete implementation flow of accessing the event source log database through the JDBC connection in this example is as follows (see fig. 4):

2. after the Sensor plug-in is successfully installed, starting the Sensor;

3. filling in a Sensor data volume module with a name to be monitored, a database type, a database driver, a database name, a user name, a password, an IP address, a port number, a monitoring table SQL (e.g. a plurality of tables are separated by: "), a collection frequency (seconds) and setting a state to be effective; the address of the log collector and the use protocol are added in the server of the Sensor.

4. And starting a Sensor log forwarding function in a server of the Sensor.

5. The Sensor plug-in sends the collected database table log to a log collector of the log collection probe.

When the event source log database is accessed through JDBC connection, the Sensor plug-in needs to be installed in the embodiment; and meanwhile, when the Sensor plug-in is installed, the Sensor is added into a trust list of the antivirus software according to the prompt.

According to the distributed event high-speed acquisition system scheme provided by the invention, the data acquisition and processing under the conditions of complex environment and huge data volume are supported, the access log of Nginx can be acquired, analyzed and displayed in real time, the challenges to the log acquisition and analysis system caused by the scale sharp increase of log data are fundamentally solved, and meanwhile, effective information can be timely extracted from massive logs to provide information support for enterprise security aiming at the attack activity of Internet security.

The above method of the present invention, or specific system units, or parts thereof, are pure software structures, and can be distributed on physical media, such as hard disks, optical discs, or any electronic devices (such as smart phones, computer readable storage media), when the machine loads the program codes and executes (such as smart phones loads and executes), the machine becomes a device for implementing the present invention. The methods and apparatus of the present invention may also be embodied in the form of program code that is transmitted over some transmission medium, such as over electrical wiring, optical fiber, or any other transmission medium, when the program code is received and loaded into and executed by a machine, such as a smart phone, the machine thereby providing an apparatus for practicing the methods.

The foregoing has shown and described the basic principles, principal features and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, and that the above embodiments and descriptions are merely illustrative of the principles of the present invention, and various changes and modifications may be made without departing from the spirit and scope of the invention, which is defined in the appended claims. The scope of the invention is defined by the appended claims and equivalents thereof.

Claims

1. The distributed event high-speed acquisition and analysis system is characterized by comprising a distributed event acquisition probe and an analysis platform;

2. The distributed event high-speed acquisition and analysis system according to claim 1, wherein the distributed event acquisition probes are distributed on different network nodes, monitor network changes based on an adaptive algorithm, acquire network state information in real time, and transmit the network state information to an analysis platform; the distributed event acquisition probes can receive a management strategy formed by the analysis platform according to the real-time network state information, and automatically adjust the working states of the corresponding event acquisition probes according to the management strategy so as to realize dynamic adjustment of the distributed states of the distributed event acquisition probes.

3. The distributed event high-speed acquisition and analysis system according to claim 1, wherein the distributed event acquisition probe is capable of automatically selecting the most appropriate transmission protocol according to the real-time network environment characteristics based on the transmission protocol selection policy issued by the analysis platform.

4. The distributed event high-speed acquisition and analysis system according to claim 1, wherein the analysis platform is configured to set a code for each event acquisition probe according to status information reported by the distributed event acquisition probes, and determine an operating status of the distributed event acquisition probes according to a genetic algorithm according to the code, and issue a related management policy to the corresponding event acquisition probes.

5. The distributed event high-speed acquisition analysis system of claim 4 wherein the distributed event acquisition probes are configured based on encoded genetic algorithms under link coverage and message coverage constraints.

6. The distributed event high-speed acquisition and analysis system according to claim 1, wherein corresponding data receiving interfaces are established for different types of event acquisition probes in the analysis platform, and corresponding data analysis modules are established for different data dimensions.

7. The distributed event high-speed acquisition and analysis system according to claim 1, wherein an event acquisition probe in the distributed event acquisition probes can form a registration state in an analysis platform and can actively report own ready information to the analysis platform;

the analysis platform is configured to monitor the running state of each event acquisition probe according to the registration information of the event acquisition probe in the platform and the actively reported ready information, and simultaneously generate a corresponding management strategy and issue the management strategy to the corresponding event acquisition probe;

the event acquisition probe enters a working state according to the received management strategy, acquires the event and forwards the event to the analysis platform; and the analysis platform collects probes according to different events and receives and analyzes the probes in different dimensionalities.

8. The distributed event high-speed acquisition analysis system of claim 1, wherein the analysis platform is capable of screening out an optimal event acquisition probe workgroup in the distributed event acquisition probes based on a genetic algorithm according to an operational state of each event acquisition probe.

9. The distributed event high-speed acquisition and analysis system according to claim 1, wherein equipment state information and load conditions reported by event acquisition probes in the analysis platform are set for each event acquisition probe, gray code codes are set for each event acquisition probe, playback-free random selection is adopted as a selection operator, selected individuals are subjected to cross operation in an arithmetic cross mode, and finally gaussian approximate variation is carried out, so that an optimal population of individuals is obtained.

10. The distributed event high-speed acquisition and analysis system according to claim 1, wherein the analysis platform adopts a heartbeat strategy based on time difference, and simultaneously adopts bidirectional alternate communication detection operation, so that the highest requirement of transmission efficiency can be met, and the operation state of an event probe can be monitored in a maximized real-time manner.

11. The distributed event high-speed acquisition and analysis system according to claim 1, wherein the system enables fast support of newly added device types based on XSD files.

12. The distributed event high-speed acquisition and analysis system according to claim 1, wherein a rule engine is further configured in the system, and the rule engine interacts with the distributed event acquisition probe data and can filter and screen the events acquired and forwarded by the distributed event acquisition probe.

13. The system of claim 12, wherein the collected events forwarded by the event collection probe in the collection method are first selected and filtered by a rule engine to generate relevant alarm information, and the alarm information and the filtered events are sent to an analysis platform.