CN113507461B

CN113507461B - Network monitoring system and network monitoring method based on big data

Info

Publication number: CN113507461B
Application number: CN202110750202.8A
Authority: CN
Inventors: 戴明; 杜渐; 李飞; 刘天宇
Original assignee: Traffic And Transportation Information Security Center Co ltd
Current assignee: Traffic And Transportation Information Security Center Co ltd
Priority date: 2021-07-01
Filing date: 2021-07-01
Publication date: 2022-11-29
Anticipated expiration: 2041-07-01
Also published as: CN113507461A

Abstract

The present disclosure describes a big data based network monitoring system. In the network monitoring system, an analyzer reads network flow data from a collector based on a shared memory mode and analyzes the network flow data to obtain protocol data; receiving and managing protocol data by a message cluster server deployed in a distributed mode to provide the protocol data for a processing module; the first processing module detects the protocol data to obtain analysis data, asset information and geographic information related to the analysis data and stores the analysis data by using the first storage module, and the second processing module processes the protocol data to obtain compressed protocol data and stores the compressed protocol data by using the second storage module; the monitoring module is used for outputting the analysis data obtained by the first processing module and asset information and geographic information related to the analysis data; and the flow playback module is used for outputting the compressed protocol data obtained by the second processing module. Therefore, the packet loss rate, the missing report rate and the storage cost can be reduced.

Description

Network monitoring system and network monitoring method based on big data

Technical Field

The present disclosure generally relates to a network monitoring system and a network monitoring method based on big data.

Background

With the popularization of computer technology, network technology and communication technology, informatization has become a powerful guarantee for various organizations to realize stable development and improve competitiveness. In the information-based construction, the network security is particularly important, and if the network is attacked, the assets of an organization can be damaged, and further, serious loss can be caused. Therefore, how to improve the defense capability against network attacks and discover the network attack behavior in time has become a research direction of various organizations.

At present, network attack behaviors are often identified by using network security tools such as firewalls, vulnerability scanning tools or intrusion detection systems based on network traffic data. The intrusion detection system analyzes the acquired network traffic data and presents the network traffic data to the network security responsible person in a visual mode, so that the network security responsible person can monitor the network traffic data and discover network attack behaviors in time. However, the above-mentioned network attack behavior identification method is often difficult to adapt to the increasingly massive network traffic data, and is prone to situations such as packet loss and missing report, and may not support playback of the historical network traffic data.

Disclosure of Invention

The present disclosure has been made in view of the above circumstances, and an object thereof is to provide a network monitoring system and a network monitoring method based on big data, which can reduce a packet loss rate, a missing report rate, and a storage cost.

Therefore, a first aspect of the present disclosure provides a network monitoring system based on big data, which is a network monitoring system for performing network monitoring based on network traffic data, and includes a collector, an analyzer, a message cluster server based on a message queue, a first processing module, a second processing module, a first storage module, a second storage module, a monitoring module, a traffic playback module, and an asset management module; the collector is configured to obtain the network flow data from a network device and copy the network flow data to a kernel buffer area; the parser is configured to read the network traffic data from the kernel buffer based on a shared memory manner and parse the network traffic data to obtain protocol data, create a protocol message including the protocol data and send the protocol message to the message cluster server; the message cluster server is configured to be deployed in a distributed manner, and is configured to receive the protocol message, store the protocol message, and provide the protocol data in the protocol message for the first processing module and the second processing module; the first processing module is configured to read protocol data in the protocol message from the message cluster server, detect the protocol data through an alarm engine for acquiring alarm information, a blacklist engine for acquiring a blacklist tag based on a blacklist, and a whitelist engine for acquiring a whitelist tag based on a whitelist to acquire analysis data including the protocol data and detection information, and store the analysis data in the first storage module after associating asset information and geographic information with the analysis data, wherein if the detection information exists, the detection information includes at least one of the alarm information, the blacklist tag, and the whitelist tag; the second processing module is configured to read protocol data in the protocol message from the message cluster server, sequentially perform serialization and compression processing on the protocol data to acquire compressed protocol data, and store the compressed protocol data in the second storage module; the first storage module is configured to store analysis data within a first preset time and asset information and geographic information related to the analysis data; the second storage module is configured to store compressed protocol data within a second preset time, wherein the compressed protocol data is stored in corresponding storage spaces based on the receiving time of the compressed protocol data, each storage space is used for storing the compressed protocol data within a corresponding time range, and the first preset time is shorter than the second preset time; the monitoring module is configured to acquire and output the analysis data and asset information and geographic information associated with the analysis data from the first storage module; the flow playback module is configured to acquire compressed protocol data within a preset time range from the second storage module, decompress and deserialize the compressed protocol data to acquire the protocol data, and analyze the protocol data by using the first processing module to acquire and output the analysis data, and asset information and geographic information associated with the analysis data; the asset management module is configured to manage the asset information. Under the condition, the requirements of real-time query and historical query can be met, the storage cost is low, in addition, the protocol data are managed on the basis of the message cluster server, the coupling performance can be reduced, the distributed deployment can be supported, the processing capacity of the big data is further improved, the packet loss rate is reduced, in addition, the protocol data which are subjected to false report or missed report can be identified on the basis of the alarm information, the blacklist tags and the white list tags, and the missed report rate can be reduced.

In addition, in the network monitoring system according to the first aspect of the present disclosure, optionally, the first storage module configuration and/or the second storage module configuration are deployed in a distributed manner. Thereby, storage of mass data can be supported.

In addition, in the network monitoring system according to the first aspect of the present disclosure, optionally, the asset information includes an asset name, an asset address, and asset principal information, and the geographic information includes latitude and longitude information and a geographic name. In this case, when the assets are attacked, corresponding asset responsible persons can be contacted to process the assets in time. This reduces the risk of damage to the asset and enables the source of the attack to be intuitively obtained.

Further, in the network monitoring system according to the first aspect of the present disclosure, optionally, the blacklist engine is configured to manage the blacklist and mark the protocol data based on the blacklist to obtain the blacklist tag; the whitelist engine is configured to manage the whitelist and tag the protocol data based on the whitelist to obtain the whitelist tag. In this case, it is possible to manage the white list and the black list and mark the protocol data with the white list and the black list. Therefore, whether false alarm or false alarm exists or not can be identified subsequently based on the white list and the black list.

In addition, in the network monitoring system according to the first aspect of the present disclosure, optionally, the alarm engine includes a rule engine having a monitoring rule, an intelligence engine having intelligence information, and an attack source engine for detecting an attack source; the rule engine is configured to manage the monitoring rule and perform rule matching on the protocol data by using the monitoring rule to acquire first alarm information; the intelligence engine is configured to manage the intelligence information and utilize the intelligence information to carry out intelligence matching on the protocol data so as to obtain second alarm information; the attack source engine is configured to manage the attack source and match a source address of the protocol data with the attack source to acquire third alarm information; the alarm information includes the first alarm information, the second alarm information, and the third alarm information. In this case, the protocol data can be detected more comprehensively. Thus, the rate of missing reports can be reduced.

In addition, in the network monitoring system related to the first aspect of the present disclosure, optionally, the first alarm information, the second alarm information, and the third alarm information have corresponding scores, which are obtained based on the monitoring rule, the intelligence information, and the score of the attack source, respectively; the network monitoring system further comprises a scoring module, wherein the scoring module is configured to obtain scores in the alarm information of the analysis data and collect the scores to obtain a total score, in the collecting, the score of the first alarm information with the highest score is obtained as a first score, the score of the second alarm information with the highest score is obtained as a second score, the score of the third alarm information is used as a third score, and the total score is obtained based on the first score, the second score and the third score. In this case, the degree of criticality corresponding to the analysis data can be quantified. Therefore, the network attack behavior with high criticality can be conveniently and quickly identified.

In addition, in the network monitoring system related to the first aspect of the present disclosure, optionally, the monitoring module is further configured to determine that there is protocol data that is falsely reported and highlight analysis data corresponding to the detection information to identify the protocol data that is falsely reported if there are alarm information and a white list label in the detection information, and determine that there is protocol data that is not falsely reported and highlight analysis data corresponding to the detection information to identify the protocol data that is falsely reported if there is no alarm information and a black list label in the detection information. Thus, the protocol data which is falsely reported can be identified based on the white list, and the protocol data which is missed can be identified based on the black list.

In addition, in the network monitoring system related to the first aspect of the present disclosure, optionally, the network monitoring system further includes a dynamic loading mechanism, where the dynamic loading mechanism is configured to create a rule change message including change information when rule information changes, send the rule change message to the message cluster server, and a monitor corresponding to each piece of rule information monitors the message cluster server to find the rule change message, and further reads the rule change message from the message cluster server and reloads the rule information based on the rule change message, so as to detect the protocol data by using the reloaded rule information to obtain the detection information, where the rule information includes the monitoring rule, the white list, the black list, and the attack source. In this case, the rule information can be reloaded based on the message cluster server. This can reduce the coupling.

In addition, in the network monitoring system according to the first aspect of the present disclosure, optionally, the monitoring rule includes a first monitoring rule and a second monitoring rule, the first monitoring rule is a monitoring rule set before the rule engine is started, and the first monitoring rule is automatically loaded when the rule engine is started; the second monitoring rule is a monitoring rule which is customized by a user through a visual interface after the rule engine is started, and the second monitoring rule is loaded through the dynamic loading mechanism. Under the condition, the monitoring rules are classified and managed according to the objects for managing the monitoring rules, so that the requirement of network monitoring can be better met. In addition, the method also supports dynamic loading of user-defined monitoring rules and can adapt to the change of network conditions. Therefore, the false alarm rate and the missing report rate can be reduced, and the monitoring rule can be managed conveniently.

A second aspect of the present disclosure provides a network monitoring method based on big data, which is a network monitoring method for performing network monitoring based on network traffic data, and includes: acquiring the network flow data from network equipment, and copying the network flow data to a kernel buffer area; reading the network flow data from the kernel buffer area based on a shared memory mode, analyzing the network flow data to obtain protocol data, creating a protocol message comprising the protocol data, and sending the protocol message to a message cluster server which is deployed in a distributed mode and is based on a message queue for management; reading protocol data in the protocol message from the message cluster server, detecting the protocol data through an alarm engine for acquiring alarm information, a blacklist engine for acquiring a blacklist tag based on a blacklist and a whitelist engine for acquiring a whitelist tag based on a whitelist to acquire analysis data including the protocol data and detection information, and storing the analysis data in a first storage module after associating the analysis data with asset information and geographic information, wherein the first storage module is used for storing the analysis data in a first preset time and the asset information and geographic information associated with the analysis data, and if the detection information exists, the detection information includes at least one of the alarm information, the blacklist tag and the whitelist tag; reading protocol data in the protocol message from the message cluster server, sequentially serializing and compressing the protocol data to obtain compressed protocol data, and storing the compressed protocol data in a second storage module, wherein the second storage module is used for storing the compressed protocol data within a second preset time, the compressed protocol data are stored in corresponding storage spaces based on the receiving time of the compressed protocol data, each storage space is used for storing the compressed protocol data within a corresponding time range, and the first preset time is shorter than the second preset time; acquiring the analysis data and asset information and geographic information associated with the analysis data from the first storage module and outputting the analysis data and the asset information and the geographic information; and acquiring compressed protocol data within a preset time range from the second storage module, decompressing and deserializing the compressed protocol data to acquire the protocol data, detecting the protocol data to acquire the analysis data, and outputting the analysis data associated with asset information and geographic information. In addition, the protocol data are managed based on the message cluster server, the coupling can be reduced, distributed deployment can be supported, the processing capacity of big data is improved, the packet loss rate is reduced, in addition, protocol data which are reported in error or are not reported can be identified based on alarm information, black list labels and white list labels, and the report missing rate can be reduced.

According to the disclosure, a network monitoring system and a network monitoring method based on big data are provided, which can reduce packet loss rate, missing report rate and storage cost.

Drawings

The disclosure will now be explained in further detail by way of example only with reference to the accompanying drawings, in which:

fig. 1 is a diagram illustrating an application scenario of a big data based network monitoring system according to an example of the present disclosure.

Fig. 2 is a block diagram illustrating a big data based network monitoring system according to an example of the present disclosure.

Fig. 3 is a block diagram illustrating another example of a big data based network monitoring system to which examples of the present disclosure relate.

Fig. 4 is a block diagram illustrating another example of a big data based network monitoring system to which examples of the present disclosure relate.

Fig. 5 is a flow chart illustrating a big data based network monitoring method according to an example of the present disclosure.

Detailed Description

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It is noted that the terms "comprises" and "comprising," and any variations thereof, in this disclosure, such that a process, method, system, article, or apparatus that comprises or has a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include or have other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. All methods described in this disclosure can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context.

The network monitoring system and the network monitoring method based on the big data can reduce the packet loss rate, the missing report rate and the storage cost. The big data based network monitoring system related to the present disclosure may also be sometimes referred to as a network monitoring system, a monitoring system, an attack detection system, an attack recognition system, an intrusion detection system, or the like. In addition, the network monitoring method according to the present disclosure may also be referred to as a network monitoring method, a monitoring method, an attack detection method, an attack identification method, or an intrusion detection method. The present disclosure is described in detail below with reference to the attached drawings. In addition, the application scenarios described in the examples of the present disclosure are for more clearly illustrating the technical solutions of the present disclosure, and do not constitute a limitation on the technical solutions provided by the present disclosure.

Fig. 1 is a schematic diagram illustrating an application scenario of a big data based network monitoring system 100 according to an example of the present disclosure. As shown in fig. 1, the network monitoring system 100 may obtain network traffic data passing through the traffic aggregation device 21 to identify network attack behavior by monitoring the network traffic data. The traffic aggregation device 21 may aggregate network traffic data for a plurality of traffic devices 22 within an organization. In some examples, end devices 23 within an organization may connect to the internet 24 through the traffic device 22 and the traffic aggregation device 21 and interact with other devices connected to the internet 24 to generate network traffic data.

In some examples, traffic aggregation device 21 may be used to aggregate network traffic data within an organization. In some examples, the traffic aggregation device 21 may be used for devices connected to the internet 24. For example, the traffic aggregation device 21 may be a router. In some examples, network traffic data passing through the traffic aggregation device 21 may be replicated into the network monitoring system 100 by way of traffic mirroring. I.e., network traffic data may be obtained in a bypass manner. In this case, the network traffic data of the traffic aggregation device 21 can be copied to the network monitoring system 100 in a traffic mirroring manner without affecting the original network, so that the network traffic data can be analyzed and monitored. In some examples, a network device, such as a network card, of network monitoring system 100 may receive the replicated network traffic data.

In some examples, traffic mirroring may be implemented by means including, but not limited to, port mirroring (Port Monitoring) and Optical Splitter (Optical Splitter) traffic collection. In some examples, the traffic device 22 may be a device within an organization for regional communications, such as local area network communications. For example, the traffic device 22 may be a switch. In this case, the terminal device 23 within the local area network is able to interact through the traffic device 22. In some examples, a terminal device 23 within the local area network can connect to the internet 24 through the traffic device 22 and the traffic aggregation device 21. However, examples of the present disclosure are not limited thereto, and in other examples, the terminal device 23 may be directly connected to the internet 24 through the traffic aggregation device 21.

In some examples, end device 23 may be a device within an organization capable of generating network traffic data. As an example of the terminal device 23, as shown in fig. 1, the terminal device 23 may include, but is not limited to, an application server 23a, a printer 23b, a personal computer 23c, a database server 23d, and the like. In some examples, an organization may include, but is not limited to, an enterprise unit, a business unit, a social group, and the like.

In some examples, network monitoring system 100 may obtain network traffic data that passes through traffic device 22. In this case, network traffic data generated based on the interaction of the traffic device 22 can be monitored. In some examples, network monitoring system 100 may obtain network traffic data between traffic devices 22. In this case, network traffic data between the traffic devices 22 can be monitored.

In some examples, the network traffic data may be data traffic generated on a network by a device capable of connecting to the network. In this case, by analyzing the network traffic data, the use condition of the network traffic can be known, the network attack behavior can be identified, and the network security can be improved.

In this embodiment, the network monitoring system 100 may perform network monitoring based on the network traffic data. Hereinafter, the network monitoring system 100 according to the present disclosure will be described in detail with reference to the drawings. Fig. 2 is a block diagram illustrating a big data based network monitoring system 100 according to an example of the present disclosure.

As shown in fig. 2, in some examples, network monitoring system 100 may include a collector 110. The collector 110 may collect network traffic data.

In some examples, collector 110 may be configured to obtain network traffic data from a network device, such as a network card. In some examples, the network traffic data obtained from the network device may be a duplicate of the network traffic data passing through the traffic aggregation device 21. In some examples, a network device, such as a network card, may be provided on the device for receiving the replicated network traffic data.

In some examples, the device for receiving the replicated network traffic data may be a server. In some examples, a server may include one or more processors and one or more memories. Wherein the processor may include a central processing unit, a graphics processing unit, and any other electronic components capable of processing data, capable of executing computer program instructions. The memory may be used to store computer program instructions.

In some examples, collector 110 may copy network traffic data obtained from a network device to a kernel buffer. The kernel buffer may be an allocated block of buffer in kernel space. In this case, the network traffic data may be subsequently read from the kernel buffer for parsing based on a shared memory manner, thereby reducing copying and system calls. Therefore, the acquisition efficiency of the network traffic data can be improved, and the packet loss rate can be reduced.

As shown in fig. 2, in some examples, network monitoring system 100 may include a parser 111. The parser 111 may be configured to parse the network traffic data to obtain protocol data.

In some examples, the parser 111 may be configured to read network traffic data from a kernel buffer. In this case, copying and system calls can be reduced. Therefore, the acquisition efficiency of the network traffic data can be improved, and the packet loss rate can be reduced. In some examples, network traffic data may be read from a kernel buffer based on a shared memory approach. In particular, the parser 111 may map a user space corresponding to the parser 111 to a kernel buffer to enable the parser 111 to read network traffic data directly from the kernel buffer. In some examples, memory may be shared through mmap functions. That is, the user space corresponding to the parser 111 may be mapped to the kernel buffer by the mmap function. Examples of the disclosure are not limited thereto and in other examples, network traffic data may also be read from user space. In other examples, the network traffic data may be data stored on a storage medium, such as a hard disk.

In some examples, the protocol data may include fields of different protocols. As an example, the fields of the different protocols may comprise, for example, a source address and a destination address. In some examples, the protocol data may have a receive time. In some examples, the receive time may be the time at which the collector 110 obtained the network traffic data.

In some examples, protocols that the parser 111 may parse include, but are not limited to, internet Protocol Version 4 (Internet Protocol Version 4, ipv 4), internet Protocol Version 6 (Internet Protocol Version 6, ipv 6), transmission Control Protocol (Transmission Control Protocol, TCP), user Datagram Protocol (UDP), stream Control Transmission Protocol (SCTP), internet Control information Protocol Version 4 (Internet Control Message Protocol Version 4, ICMPv 4), internet Control information Protocol Version six (Internet Control Message Protocol Version 4, icmp 6), generic Routing Encapsulation Protocol (GRE), ethernet Protocol (Ethernet), point-to-Point Protocol (Point Protocol Point), PPP), point-to-Point Protocol Over Ethernet (PPPoE), raw Datagram socket (Raw), secure socket Protocol (SLL), virtual Local Area Network (Virtual Local Area Network, VLAN), QINQ, multi-Protocol Label Switching (MPLS), traffic mirror Protocol (ERSPAN), virtual eXtensible Local Area Network (Virtual eXtensible Local Area Network, VXLAN), hyperText Transfer Protocol (HTTP), secure socket Protocol (SSL), secure Transport Layer Protocol (Transport Layer, TLS), microsoft's software program level Network Transfer Protocol (SMB, server), DCPC, simple Transport Protocol (MAC Protocol), SMTP), file Transfer Protocol (FTP), secure Shell Protocol (SSH), domain Name System (DNS), modbus, ENIP/CIP, distributed Network Protocol 3 (Distributed Network Protocol 3, dnp 3), network File System (NFS), network Time Protocol (NTP), dynamic Host Configuration Protocol (DHCP), simple File Transfer Protocol (TFTP), KRB5, internet Key Exchange Protocol Version 2 (Internet Exchange Version 2, ikev 2), session Initiation Protocol (SIP), simple Network Management Protocol (Management Network Key, SNMP), remote Display Protocol (RDP), and the like.

In some examples, parser 111 may be configured to create a protocol message and send the protocol message to message cluster server 112. The protocol message may include protocol data. In some examples, the protocol message may be sent to message cluster server 112.

As shown in fig. 2, in some examples, network monitoring system 100 may include a message cluster server 112. Message cluster server 112 may receive and store protocol messages that include protocol data. That is, message cluster server 112 may manage the protocol messages.

In some examples, message cluster server 112 may be a cluster that manages protocol messages based on message queues. In particular, message cluster server 112 may receive protocol messages, store protocol messages, and provide protocol messages through a message queue. In this case, the coupling can be reduced by uniformly managing the protocol messages by the message cluster server 112. In some examples, message cluster server 112 may provide processing module 113 (described later) with protocol data in a protocol message.

In some examples, message cluster server 112 may run one or more distributed systems. That is, the message cluster servers 112 may be deployed in a distributed manner. Thus, the network traffic data with large data volume can be processed. In some examples, message cluster server 112 may manage protocol messages through a producer, consumer schema. For example, the parser 111 may be a producer of the protocol message and the module or program using the protocol message may be a consumer of the protocol message. In some examples, the processing module 113 may be a consumer of the protocol message. In some examples, message clustering server 112 may be implemented based on kafka clusters.

Additionally, in some examples, message cluster server 112 may also receive protocol messages that include log information (e.g., a log of devices within an organization). In this case, the network traffic data can be monitored in conjunction with the log information. Therefore, the accuracy of identifying the network attack behavior can be improved.

As shown in fig. 2, in some examples, network monitoring system 100 may include a processing module 113. The processing module 113 may process the protocol data in the protocol message to obtain analytics data, asset information and geographic information associated with the analytics data (which may be referred to simply as associated data), and/or compress the protocol data.

In some examples, the processing module 113 may include a first processing module 113a (see fig. 2). The first processing module 113a may be configured to detect protocol data to obtain analysis data. In some examples, the first processing module 113a may be configured to read protocol data in a protocol message from the message cluster server 112. That is, the message cluster server 112 may provide the first processing module 113a with protocol data in a protocol message.

Fig. 3 is a block diagram illustrating another example of a big data based network monitoring system 100 to which examples of the present disclosure relate. As shown in fig. 3, in some examples, network monitoring system 100 may include at least one of an alert engine 116, a blacklist engine 117, and a whitelist engine 118.

In some examples, the first processing module 113a may detect the protocol data through the alert engine 116, the blacklist engine 117, and the whitelist engine 118 to obtain the analysis data.

In some examples, the analysis data may include protocol data and detection information. In some examples, the detection information may be obtained based on the alert engine 116, the blacklist engine 117, and the whitelist engine 118. In this case, the alarm information may include first alarm information, second alarm information, and third alarm information. In some examples, if the detection information is present (i.e., the detection information is not null), the detection information may include at least one of alarm information, a blacklist tag, and a whitelist tag. In some examples, the detection information may also include a total score and/or an alert level. The total score and/or the alert level may be obtained by a scoring module 121 (described later).

In some examples, the first processing module 113a may detect the protocol data through the alert engine 116 to obtain the alert information. That is, the alert engine 116 may be used to obtain alert information. In some examples, the alert engine 116 may include a rules engine, intelligence engine, and attack sources engine. In some examples, the rules engine may have monitoring rules. In some examples, the intelligence engine may have intelligence information. In some examples, an attack source engine may be used to detect the source of the attack. In this case, the protocol data can be detected more comprehensively. Thus, the rate of missing reports can be reduced.

As described above, in some examples, the alert engine 116 may include a rules engine. In some examples, the rules engine may be configured to manage monitoring rules. In some examples, the rules engine may be configured to utilize the monitoring rules to rule match the protocol data to obtain the first alarm information.

In some examples, managing the monitoring rules may include adding monitoring rules, modifying monitoring rules, and deleting monitoring rules. In some examples, the monitoring rule may be used to match values of fields in the protocol data to obtain the first alert information. Thereby, protocol data in compliance with the monitoring rule can be obtained. In some examples, the monitoring rule may have a score. In this case, the score of the first alarm information corresponding to the monitoring rule can be obtained based on the score of the monitoring rule, and thus the total score and/or the alarm level of the protocol data can be obtained.

In some examples, the monitoring rules may include a first monitoring rule and a second monitoring rule. In some examples, the first monitoring rule may be a predefined monitoring rule (which may also be referred to as a built-in monitoring rule). In some examples, the first monitoring rule may be a monitoring rule set before the rule engine is started. In some examples, the first monitoring rule may be automatically loaded upon startup of the rules engine. In some examples, the first monitoring rule may be managed by a developer. In some examples, the second monitoring rule may be a monitoring rule (which may also be referred to as a user-defined rule) that is customized by a user, such as a network security principal, through a visualization interface, such as a browser interface, after the rule engine is started. In some examples, the second monitoring rule may be loaded via a dynamic loading mechanism 119 (described later). Under the condition, the monitoring rules are classified and managed according to the objects for managing the monitoring rules, so that the requirement of network monitoring can be better met. In addition, the method also supports dynamic loading of user-defined monitoring rules and can adapt to the change of network conditions. Therefore, the false alarm rate and the false missing rate can be reduced, and the monitoring rule can be managed conveniently.

In some examples, the first monitoring rule may include a protocol type, a rule number, a data flow direction, an alarm content, a rule content, a score, and a match field. In some examples, the first monitoring rule may further include an alarm identification, an identification of whether case differentiation is performed when the rule content matches, a rule version, a rule creation time, and a rule update time. This enables the monitoring rule to be arranged more comprehensively.

In addition, the protocol type may be a different protocol (i.e., a protocol that the parser 111 may parse). For example, the protocol type may be HTTP, FTP, PPP, or the like as described above. The rule number may uniquely identify a monitoring rule. In addition, the data flow direction may indicate the flow direction, the source address and the destination address of the network traffic data corresponding to the monitoring rule. In some examples, the flow direction of the network traffic data may include both extranet to intranet and intranet to extranet flow directions. In some examples, the source Address and the destination Address may include an Internet Protocol Address (IP Address) and a port, respectively. In addition, the alarm content may be description information of the monitoring rule. As an example of the alert content, the alert content may be "suspected attempted sql injection behavior", for example. In some examples, the first alert information may include alert content. In addition, the rule content may be used to determine whether there is data that conforms to the format corresponding to the rule content in the values of the fields in the protocol data. In some examples, the rule content may be a regular expression or a fixed string of characters. In addition, the match field may be a field in the protocol data. As an example of the matching field, for example, the matching field in the protocol data corresponding to the Http protocol may be Http _ uri. In addition, the alarm flag may identify whether the monitoring rule would generate an alarm.

In some examples, the second monitoring rule may include a rule name, a rule number, a rule alarm level, a match range, a match field, and a match value. This makes it possible to easily arrange the monitoring rule. Unless otherwise specified, the same configuration, e.g., rule number, in the second monitoring rule as in the first monitoring rule applies to the second monitoring rule, with reference to the associated description in the first monitoring rule.

In some examples, the regular alert levels may include critical, high-risk, medium-risk, and low-risk. Therefore, the criticality corresponding to the second monitoring rule can be obtained more intuitively. In some examples, the rule alert level may correspond to a score. That is, the rule alarm level and the score can be mutually converted. For example, if the score range is 0 to 12 points, 10 to 12 points may be assigned to critical, 7 to 9 points may be assigned to high-risk, 4 to 6 points may be assigned to medium-risk, and 1 to 3 points may be assigned to low-risk. In some examples, the regular alarm level may correspond to a median of the corresponding range when the regular alarm level is converted to a score. For example, a crisis may correspond to a median in the range of 10 to 12 points, i.e. a crisis may correspond to 11 points. Thus, the rule alarm level and the score can be mutually converted. In some examples, the matching range of the second monitoring rule may include a full network, an internal network, and an external network. The matching range is that the whole network can match all source addresses. The matching range is the address of the intranet which can be matched with the source address. The matching range is that the external network can match the source address with the address of the external network. In some examples, the matching value of the second monitoring rule may be a fixed string. Therefore, the user can conveniently configure the second monitoring rule.

As described above, in some examples, the alert engine 116 may include a intelligence engine. In some examples, the intelligence engine may be configured to manage intelligence information. In some examples, the intelligence engine may be configured to utilize the intelligence information to intelligence match the protocol data to obtain the second alarm information. In some examples, managing the intelligence information may include adding, modifying, and deleting the intelligence information. In some examples, the intelligence information may include at least one of a malicious internet protocol address, a malicious domain name, a mine pool address, and a trojan file. In this case, more favorable evidence can be provided for the network attack behavior. In some examples, the intelligence information may have a score. In this case, the score of the second alarm information corresponding to the informative information is obtained based on the score of the informative information, and the total score and/or the alarm level of the protocol data can be obtained.

As described above, in some examples, the alert engine 116 may include an attack sources engine. In some examples, the attack sources engine may be configured to manage the attack sources. In some examples, the attack source engine may be configured to match a source address of the protocol data with the attack source to obtain the third alarm information. In some examples, the attack sources may include attack internet protocol addresses and attack domain names. In some examples, managing attack sources may include adding attack sources, modifying attack sources, and deleting attack sources. In some examples, the attack source may have a score. In this case, the score of the third alarm information corresponding to the attack source is obtained based on the score of the attack source, and then the total score and/or the alarm level of the protocol data can be obtained.

In some examples, the first processing module 113a may detect the protocol data by the blacklist engine 117 to obtain the blacklist label. In some examples, blacklist engine 117 may obtain blacklist tags based on a blacklist. In some examples, blacklist engine 117 may be configured to manage blacklists. In some examples, the blacklist engine 117 can be configured to tag the protocol data based on a blacklist to obtain a blacklist label. In this case, it is possible to manage the blacklist and to tag the protocol data with the blacklist. Therefore, whether the report is missed or not can be identified based on the blacklist subsequently.

In some examples, if there is no alarm information in the detection information in the analysis data and there is a blacklist tag (i.e., there is protocol data that is identified as a blacklist but does not generate alarm information), it may be determined that there is protocol data that is not reported. This enables identification of protocol data that has been missed. In some examples, the protocol data for the false negative may be configured by the alert engine 116 with corresponding monitoring rules, intelligence information, or attack sources to identify the false negative protocol data. In this case, the blacklist engine 117 can identify whether there is a false negative, and adjust the alarm engine 116 accordingly. Thus, the rate of missing reports can be reduced.

In some examples, the blacklist may include a match field and a match value. The match field may be a field in the protocol data. In some examples, the match value may be a fixed string. Specifically, if the field in the protocol data is consistent with the matching field of the blacklist, and the value of the field in the protocol data has the matching value of the blacklist, the protocol data may be marked to obtain the blacklist tag.

In some examples, the first processing module 113a may detect the protocol data by the whitelist engine 118 to obtain the whitelist tag. In some examples, the whitelist engine 118 may obtain the whitelist label based on the whitelist. In some examples, the whitelist engine 118 may be configured to manage whitelists. In some examples, the whitelist engine 118 may be configured to tag the protocol data based on a whitelist to obtain whitelist tags. In this case, the white list can be managed and the protocol data can be tagged with the white list. Therefore, whether false alarm exists or not can be identified based on the white list subsequently.

In some examples, if the alarm information and the white list tag (i.e., the protocol data that is identified as the white list but which generates the alarm information) are present in the detection information, it may be determined that there is protocol data that is being false-reported. This enables identification of protocol data that has been misinformed. In some examples, the protocol data for the false positive may be configured with monitoring rules, intelligence information, or attack sources by alert engine 116 to identify the protocol data that is false positive. In this case, the white list engine 118 can identify whether there is a false positive and adjust the alarm engine 116 accordingly. This can reduce the false alarm rate.

In some examples, the white list may include a match field and a match value. The match field may be a field in the protocol data. In some examples, the match value may be a fixed string. Specifically, if the field in the protocol data is consistent with the matching field of the white list and the value of the field in the protocol data has the matching value of the white list, the protocol data may be marked to obtain the white list tag

In some examples, the first processing module 113a may be further configured to associate the analytics data with asset information and geographic information. Thus, the attacked asset and the attack source can be intuitively obtained.

In some examples, the first processing module 113a may be configured to associate the analytics data with asset information. In some examples, the asset information may include an asset name, an asset address, and asset principal information. In this case, when the asset is attacked, the corresponding asset principal can be contacted in time for processing. Thereby, the risk of damage to the assets can be reduced. In some examples, the source address and/or the destination address in the analytics data (i.e., the source address and/or the destination address in the protocol data) may be associated by an asset address to associate the analytics data with the asset information. In some examples, the first processing module 113a may be configured to associate the analysis data with geographic information. In some examples, the geographic information may include latitude and longitude information and a geographic name. Thus, the attack source can be intuitively obtained. In some examples, the geographic information may be obtained by analyzing a source address and/or a destination address in the data (i.e., a source address and/or a destination address in the protocol data) to associate the analyzed data with the geographic information. For example, the geographic information may be obtained by an internet protocol address or a domain name in the source address.

In some examples, the first processing module 113a may be further configured to store the analysis data in association with the asset information and the geographic information. In some examples, the first processing module 113a may be further configured to store the analysis data in association with asset information and geographic information to a first storage module 114a (described later).

In some examples, the processing module 113 may include a second processing module 113b (see fig. 2). The second processing module 113b may be configured to sequentially perform serialization and compression processing on the protocol data to obtain compressed protocol data. In this case, the size of the protocol data can be reduced, and the storage space can be reduced in the subsequent storage. This can reduce the storage cost.

In some examples, protocol data may be serialized using the avro middleware. Thereby, the protocol data in binary form can be obtained. In some examples, the serialized protocol data may be compressed using a snappy algorithm. This can improve the compression efficiency.

In some examples, the second processing module 113b may be configured to read protocol data in a protocol message from the message cluster server 112. That is, the message cluster server 112 may provide the second processing module 113b with protocol data in a protocol message. In some examples, the second processing module 113b may also be configured to store compression protocol data. In some examples, the second processing module 113b may also be configured to store the compression protocol data in a second storage module 114b (described later). This can reduce the storage cost.

As shown in fig. 2, in some examples, network monitoring system 100 may include a storage module 114. The storage module 114 may store data obtained by the processing module 113. In some examples, the storage module 114 may store the analysis data and/or the association information obtained by the first processing module 113 a. In some examples, the storage module 114 may store the compression protocol data obtained by the second processing module 113 b.

In some examples, the storage module 114 may include a first storage module 114a (see fig. 2). The first storage module 114a may be configured to store the analysis data and the associated information for a first preset time. In some examples, the first preset time may be less than or equal to half a year. For example, the first preset time may be 1 month, 2 months, 3 months, 4 months, 5 months, or half a year, etc. In some examples, the first storage module 114a may be configured to delete the analysis data that exceeds a first preset time. In this case, the analysis data in a short time range and the asset information and the geographic information associated with the analysis data are stored, and the storage cost can be reduced under the condition of satisfying the real-time data query. However, examples of the present disclosure are not limited thereto, and the first preset time may be set according to actual situations such as a budget of a storage cost.

In some examples, the first storage module 114a may be deployed in a distributed manner. Thereby, storage of mass data can be supported. In some examples, the first storage module 114a may support full-text retrieval. Therefore, real-time query of the analysis data can be facilitated. In some examples, the first storage module 114a may store the analysis data and the associated information based on an elasticsearch. In this case, full-text retrieval can be supported. Therefore, the analysis data can be conveniently inquired in real time.

In some examples, the storage module 114 may include a second storage module 114b (see fig. 2). The second storage module 114b may be configured to store the compressed protocol data for a second preset time. In some examples, the second preset time may be greater than or equal to 1 year. For example, the second preset time may be 1 year, 2 years, 3 years, 5 years, or the like. In this case, compressed protocol data in a longer time range is stored, and it is possible to support query of historical protocol data with a low requirement for real-time performance while reducing the storage cost as much as possible. However, examples of the present disclosure are not limited thereto, and the second preset time may be set according to actual situations such as budget of storage cost.

In some examples, in the second storage module 114b, the compressed protocol data may be stored in a corresponding storage space, e.g., a folder, based on a reception time of the compressed protocol data (which may also be referred to as a reception time of the protocol data), and each storage space may be used to store the compressed protocol data within a corresponding time range. That is, the compressed protocol data may be stored to the corresponding storage space by the reception time. Under the condition, the corresponding storage space can be positioned according to the query time, and then the compressed protocol data in the time range corresponding to the query time can be obtained and flow playback is carried out.

In some examples, the time range corresponding to the storage space may be set on an hourly basis. For example, the compressed protocol data may be stored in the storage space corresponding to a preset period, for example, one hour, two hours, or three hours. As an example, if the reception time of the compressed protocol data is 2021-06-01. Under the condition, the corresponding storage space can be positioned according to the query time, and then the corresponding compression protocol data can be obtained for flow playback.

In some examples, the first preset time may be less than the second preset time. In this case, the corresponding storage time is set according to the requirement of using the protocol data, so that under the condition of meeting the real-time query, the data with longer time can be stored by using less storage space to support the query of the historical protocol data. This can reduce the storage cost.

In some examples, the second storage module 114b may be deployed in a distributed manner. This enables mass data storage. In some examples, the second storage module 114b may be a distributed file system. In this case, the bearer can be carried with less configured hardware. This can reduce the storage cost. In some examples, the second storage module 114b may store the compression protocol data based on a Hadoop Distributed File System (HDFS). In this case, storage of massive amounts of compression protocol data can be supported.

As shown in fig. 2, in some examples, network monitoring system 100 may include a display module 115. The display module 115 may display the analysis data and associated information, and/or perform traffic playback on the compressed protocol data.

In some examples, the display module 115 may include a monitoring module 115a (see fig. 2). The monitoring module 115a may be configured to obtain and output the analysis data and the associated information. Therefore, network monitoring can be carried out based on more comprehensive information so as to identify the network attack behaviors. In some examples, the monitoring module 115a may be configured to obtain the analysis data and the associated information from the storage module 114 and output. In some examples, the monitoring module 115a may be configured to obtain and output the analysis data and the associated information from the first storage module 114 a. In this case, the analysis data and the asset information and the geographic information associated with the analysis data can be queried in real time and quickly within a short time frame. Thereby, the network attack behavior can be recognized quickly. Examples of the disclosure are not limited thereto, and in other examples, the monitoring module 115a may be configured to obtain and output the analysis data.

As described above, the detection information may include at least one of alarm information, a blacklist tag and a whitelist tag. In some examples, the monitoring module 115a is further configured to highlight protocol data that is misinformed and/or protocol data that is false-declared. Specifically, if there are alarm information and a white list tag in the detection information in the analysis data (that is, there is protocol data that is identified as a white list but generates alarm information), it may be determined that there is protocol data that is misreported and highlight the analysis data corresponding to the detection information to identify the protocol data that is misreported. If the detection information in the analysis data does not have the alarm information and the blacklist tag exists (that is, the protocol data which is identified as the blacklist but does not generate the alarm information exists), the existence of the protocol data which is not reported can be judged, and the analysis data corresponding to the detection information is highlighted to identify the protocol data which is not reported. Thus, the protocol data which is falsely reported can be identified based on the white list, and the protocol data which is missed can be identified based on the black list. In some examples, protocol data that is being misinformed or protocol data that is being misreported may be highlighted with a different color or a different icon.

In some examples, the display module 115 may include a traffic playback module 115b (see fig. 2). The traffic playback module 115b may be configured to obtain compressed protocol data and detect to obtain and output analysis data and/or associated information. This enables the playback of the traffic of the compressed protocol data.

In some examples, in the traffic playback module 115b, the compressed protocol data may be decompressed and deserialized to obtain protocol data, which is then detected to obtain analytics data and/or associated information. In some examples, the compression protocol data may be compressed using a snappy algorithm to obtain decompressed compression protocol data. In some examples, the decompressed compression protocol data may be deserialized using the avro middleware.

In some examples, in the traffic playback module 115b, the protocol data may be detected by the first processing module 113a to obtain the analysis data and/or the associated information. In some examples, the compression protocol data may be obtained from the second storage module 114 b. In this case, the query of the historical protocol data can be supported with a lower storage cost. Specifically, the compressed protocol data within a preset time range may be acquired from the second storage module 114b, decompressed and deserialized to acquire the protocol data, and associated information may be performed on the protocol data by the first processing module 113a to acquire analysis data and the associated information and output the analysis data and the associated information. In some examples, the preset time range may include a start time and an end time. Thereby, the compressed protocol data from the start time to the end time can be inquired.

In some examples, network monitoring system 100 may also include a dynamic loading mechanism 119 (see fig. 3). The dynamic loading mechanism 119 may be configured to monitor for changes in the rule information and reload the rule information. In some examples, the rule information may include at least one of a monitoring rule, a white list, a black list, and a source of the attack. Thus, a variety of rule information can be dynamically loaded.

Specifically, in the dynamic loading mechanism 119, when the rule information changes, a rule change message including the change information may be created, the rule change message may be sent to the message cluster server 112, and a listener corresponding to each rule information listens to the message cluster server 112 to find the rule change message, and then the rule change message may be read from the message cluster server 112 and the rule information may be reloaded based on the rule change message, so as to detect the protocol data using the reloaded rule information to obtain the detection information. In this case, the rule information can be reloaded based on the message cluster server 112. This can reduce the coupling property.

In some examples, the monitor corresponding to each rule information may be a monitor set in each engine, such as the rule engine, the blacklist engine 117, the whitelist engine 118, and the attack source engine. For example, a listener provided in the rule engine may read the rule change message from the message cluster server 112 and reload the monitoring rule, e.g., the second monitoring rule, based on the rule change message, so as to detect the protocol data using the reloaded monitoring rule to obtain the first alarm information. Other engines are similar and will not be described in detail herein.

Fig. 4 is a block diagram illustrating another example of a big data based network monitoring system 100 to which examples of the present disclosure relate. As shown in fig. 4, in some examples, network monitoring system 100 may include an asset management module 120. The asset management module 120 may be configured to manage asset information. In some examples, managing asset information may include adding asset information, modifying asset information, and deleting asset information. In some examples, assets may be resources that have value to a organization of organization and are objects that are protected by security policies. For example, an asset may be an information system or a hardware device within an organization. In some examples, assets may be classified into types of data, software, hardware, services, and personnel, etc., according to their representation.

As shown in fig. 4, in some examples, network monitoring system 100 may include a scoring module 121. The scoring module 121 may be configured to obtain scores in the alarm information of the analysis data and aggregate the scores to obtain a total score. The first warning information, the second warning information, and the third warning information may have scores, respectively. In some examples, the total score may be obtained based on a score of at least one of the first warning information, the second warning information, and the third warning information.

In some examples, in the aggregating, a score of the first alarm information with the highest score may be obtained as the first score, a score of the second alarm information with the highest score may be obtained as the second score, and a score of the third alarm information may be obtained as the third score. In some examples, the total score may be obtained based on the first score, the second score, and the third score. In this case, the degree of criticality corresponding to the analysis data can be quantified. Therefore, the network attack behavior with high criticality can be conveniently and quickly identified. In some examples, the first score, the second score, and the third score may be weighted and summed to obtain a total score. In some examples, the first score, the second score, and the third score may be summed to obtain a total score.

In some examples, the scoring module 121 is further configured to obtain an alarm level corresponding to the analysis data based on the total score. This makes it possible to intuitively obtain the degree of criticality corresponding to the analysis data. In some examples, the alert levels may include critical, high-risk, medium-risk, and low-risk. The corresponding relationship between the alarm level and the total score may refer to the relevant description of the interconversion between the alarm level and the score in the rule in the second monitoring rule.

Hereinafter, the big data based network monitoring method of the present disclosure is described in detail with reference to fig. 5. The network monitoring method is applied to the network monitoring system 100. Unless otherwise indicated, the description related to the network monitoring system 100 applies to the network monitoring method. Fig. 5 is a flow chart illustrating a big data based network monitoring method according to an example of the present disclosure.

In some examples, as shown in fig. 5, a network monitoring method may include collecting network traffic data (step S110). In some examples, in step S110, network traffic data may be obtained from a network device. In some examples, the network traffic data may be copied to a kernel buffer. The detailed description can refer to the relevant description of collector 110.

In some examples, as shown in fig. 5, the network monitoring method may include parsing the network traffic data to obtain protocol data (step S120). In some examples, in step S120, the network traffic data may be read from the kernel buffer based on a shared memory manner and parsed to obtain the protocol data. The detailed description can be referred to the related description of the parser 111.

In some examples, as shown in fig. 5, a network monitoring method may include sending a protocol message including protocol data to a message queue-based message cluster server 112 for management (step S130). In some examples, message cluster servers 112 may be deployed in a distributed manner. In some examples, a protocol message including protocol data may be created and sent to message cluster server 112 for management. The detailed description may refer to the related description of the parser 111 and the message cluster server 112.

In some examples, as shown in fig. 5, the network monitoring method may include detecting agreement data to obtain analysis data, and asset information and geographic information associated with the analysis data (step S141). In some examples, in step S141, protocol data in the protocol message may be read from message cluster server 112. In some examples, the protocol data may be detected by the alert engine 116, the blacklist engine 117, and the whitelist engine 118 to obtain the analysis data. In some examples, the analysis data may include protocol data and detection information. In some examples, the detection information, if present, may include at least one of alarm information, blacklist tags, and whitelist tags. In some examples, the analytics data may be associated with asset information and geographic information to obtain asset information and geographic information associated with the analytics data. The detailed description can be referred to the alarm engine 116, the blacklist engine 117, the whitelist engine 118 and the relevant description of the first processing module 113 a.

In some examples, as shown in fig. 5, the network monitoring method may include storing the analytics data, and the asset information and geographic information associated with the analytics data, to the first storage module 114a (step S142). In some examples, the first storage module 114a may be used to store the analytics data for a first preset time, and asset information and geographic information associated with the analytics data. The detailed description may refer to the related description of the first storage module 114 a.

In some examples, as shown in fig. 5, the network monitoring method may include reading and outputting the analysis data and the asset information and the geographic information associated with the analysis data from the first storage module 114a (step S143). The detailed description may refer to the related description of the monitoring module 115 a.

In some examples, as shown in fig. 5, the network monitoring method may include sequentially serializing and compressing the protocol data to obtain compressed protocol data (step S151). In some examples, in step S151, protocol data in the protocol message may be read from message cluster server 112. The detailed description may refer to the related description of the second processing module 113 b.

In some examples, as shown in fig. 5, the network monitoring method may include storing the compressed protocol data to the second storage module 114b (step S152). In some examples, the second storage module 114b may be configured to store the compressed protocol data for a second predetermined time. In some examples, the compressed protocol data may be stored in the respective storage space based on a time of receipt of the compressed protocol data. In some examples, each storage space may be used to store compressed protocol data over a corresponding time range. In some examples, the first preset time may be less than the second preset time. The detailed description may refer to the related description of the second storage module 114 b.

In some examples, as shown in fig. 5, the network monitoring method may include reading the compressed protocol data from the second storage module 114b and detecting to obtain the analysis data and outputting the analysis data in association with the asset information and the geographic information (step S153). In some examples, the compression protocol data within a preset time range may be retrieved from the second storage module 114 b. In some examples, the compressed protocol data may be decompressed and deserialized to obtain and detect the protocol data. The detailed description may refer to the related description of the traffic playback module 115 b.

The network monitoring system 100 and the network monitoring method disclosed by the present disclosure acquire and analyze network traffic data from a network device based on a shared memory manner to acquire protocol data, manage protocol messages including the protocol data through a message cluster server 112 deployed in a distributed manner based on a message queue, detect the protocol data in the message cluster server 112 to acquire and store analysis data capable of supporting real-time query and asset information and geographic information associated with the analysis data, and serialize and compress the protocol data to acquire and store compressed protocol data capable of supporting historical query. Under the condition, the requirements of real-time query and historical query can be met, the storage cost is low, in addition, the protocol data are managed on the basis of the message cluster server 112, the coupling performance can be reduced, the distributed deployment is supported, the processing capacity of the big data is further improved, the packet loss rate is reduced, in addition, the protocol data which are subjected to false report or missed report can be identified on the basis of the alarm information, the black list labels and the white list labels, and the missed report rate can be reduced.

While the invention has been described in detail in connection with the drawings and the embodiments, it is to be understood that the above description is not intended to limit the invention in any way. Those skilled in the art can make modifications and variations as necessary without departing from the true spirit and scope of the invention, and such modifications and variations are intended to be within the scope of the invention.

Claims

1. A network monitoring system based on big data is characterized in that the network monitoring system is used for carrying out network monitoring based on network flow data and comprises a collector, a resolver, a message cluster server based on a message queue, a first processing module, a second processing module, a first storage module, a second storage module, a monitoring module, a flow playback module and an asset management module; the collector is configured to obtain the network traffic data from a network device and copy the network traffic data to a kernel buffer area; the parser is configured to read the network traffic data from the kernel buffer based on a shared memory manner and parse the network traffic data to obtain protocol data, create a protocol message including the protocol data and send the protocol message to the message cluster server; the message cluster server is configured to be deployed in a distributed manner, and is configured to receive the protocol message, store the protocol message, and provide the protocol data in the protocol message for the first processing module and the second processing module; the first processing module is configured to read protocol data in the protocol message from the message cluster server, detect the protocol data through an alarm engine for acquiring alarm information, a blacklist engine for acquiring a blacklist tag based on a blacklist, and a whitelist engine for acquiring a whitelist tag based on a whitelist to acquire analysis data including the protocol data and detection information, and store the analysis data in the first storage module after associating asset information and geographic information with the analysis data, wherein if the detection information exists, the detection information includes at least one of the alarm information, the blacklist tag, and the whitelist tag; the second processing module is configured to read protocol data in the protocol message from the message cluster server, sequentially perform serialization and compression processing on the protocol data to acquire compressed protocol data, and store the compressed protocol data in the second storage module; the first storage module is configured to store analysis data within a first preset time and asset information and geographic information related to the analysis data; the second storage module is configured to store compressed protocol data within a second preset time, wherein the compressed protocol data is stored in corresponding storage spaces based on the receiving time of the compressed protocol data, each storage space is used for storing the compressed protocol data within a corresponding time range, and the first preset time is shorter than the second preset time; the monitoring module is configured to acquire and output the analysis data and asset information and geographic information associated with the analysis data from the first storage module; the flow playback module is configured to acquire compressed protocol data within a preset time range from the second storage module, decompress and deserialize the compressed protocol data to acquire the protocol data, and analyze the protocol data by using the first processing module to acquire and output the analysis data, and asset information and geographic information associated with the analysis data; the asset management module is configured to manage the asset information.

2. The network monitoring system of claim 1, wherein:

the first storage module configuration and/or the second storage module configuration is deployed in a distributed manner.

3. The network monitoring system of claim 1, wherein:

the asset information comprises an asset name, an asset address and asset responsible person information, and the geographic information comprises longitude and latitude information and a geographic name.

4. The network monitoring system of claim 1, wherein:

the blacklist engine is configured to manage the blacklist and to mark the protocol data based on the blacklist to obtain the blacklist label; the whitelist engine is configured to manage the whitelist and tag the protocol data based on the whitelist to obtain the whitelist tag.

5. The network monitoring system of claim 1, wherein:

the alarm engine comprises a rule engine with monitoring rules, an intelligence engine with intelligence information and an attack source engine for detecting attack sources; the rule engine is configured to manage the monitoring rule and perform rule matching on the protocol data by using the monitoring rule to acquire first alarm information; the intelligence engine is configured to manage the intelligence information and utilize the intelligence information to perform intelligence matching on the protocol data to obtain second alarm information; the attack source engine is configured to manage the attack source and match a source address of the protocol data with the attack source to obtain third alarm information; the alarm information includes the first alarm information, the second alarm information, and the third alarm information.

6. The network monitoring system of claim 5, wherein:

the first alarm information, the second alarm information and the third alarm information have corresponding scores, and the scores are obtained based on the monitoring rule, the intelligence information and the scores of the attack sources respectively; the network monitoring system further comprises a scoring module, wherein the scoring module is configured to acquire scores in the alarm information of the analysis data and collect the scores to acquire a total score, in the collecting, the score of the first alarm information with the highest score is acquired as a first score, the score of the second alarm information with the highest score is acquired as a second score, the score of the third alarm information is acquired as a third score, and the total score is acquired based on the first score, the second score and the third score.

7. The network monitoring system of claim 1, wherein:

the monitoring module is further configured to determine that there is protocol data that is falsely reported and highlight analysis data corresponding to the detection information to identify the protocol data that is falsely reported if there are alarm information and a white list tag in the detection information, and determine that there is protocol data that is not falsely reported and highlight analysis data corresponding to the detection information to identify the protocol data that is falsely reported if there is no alarm information and a black list tag in the detection information.

8. The network monitoring system of claim 5, wherein:

the network monitoring system further comprises a dynamic loading mechanism, wherein the dynamic loading mechanism is configured to create a rule change message including change information when rule information changes, send the rule change message to the message cluster server, monitor the message cluster server by a monitor corresponding to each rule information to find the rule change message, further read the rule change message from the message cluster server and reload the rule information based on the rule change message, so as to detect the protocol data by using the reloaded rule information to obtain the detection information, and the rule information includes the monitoring rule, the white list, the black list and the attack source.

9. The network monitoring system of claim 8, wherein:

the monitoring rules comprise a first monitoring rule and a second monitoring rule, the first monitoring rule is a monitoring rule set before the rule engine is started, and the first monitoring rule is automatically loaded when the rule engine is started; the second monitoring rule is a monitoring rule which is customized by a user through a visual interface after the rule engine is started, and the second monitoring rule is loaded through the dynamic loading mechanism.

10. A network monitoring method based on big data is characterized in that the network monitoring method based on network flow data carries out network monitoring, and comprises the following steps: acquiring the network flow data from network equipment, and copying the network flow data to a kernel buffer area; reading the network flow data from the kernel buffer area based on a shared memory mode, analyzing the network flow data to obtain protocol data, creating a protocol message comprising the protocol data, and sending the protocol message to a message cluster server which is deployed in a distributed mode and is based on a message queue for management; reading protocol data in the protocol message from the message cluster server, detecting the protocol data through an alarm engine for acquiring alarm information, a blacklist engine for acquiring a blacklist tag based on a blacklist, and a whitelist engine for acquiring a whitelist tag based on a whitelist to acquire analysis data including the protocol data and detection information, associating the analysis data with asset information and geographic information, and storing the analysis data into a first storage module, wherein the first storage module is used for storing the analysis data within a first preset time and the asset information and geographic information associated with the analysis data, and if the detection information exists, the detection information includes at least one of the alarm information, the blacklist tag and the whitelist tag; reading protocol data in the protocol message from the message cluster server, sequentially serializing and compressing the protocol data to obtain compressed protocol data, and storing the compressed protocol data in a second storage module, wherein the second storage module is used for storing the compressed protocol data within a second preset time, the compressed protocol data are stored in corresponding storage spaces based on the receiving time of the compressed protocol data, each storage space is used for storing the compressed protocol data within a corresponding time range, and the first preset time is shorter than the second preset time; acquiring the analysis data and asset information and geographic information associated with the analysis data from the first storage module and outputting the analysis data and the asset information and the geographic information; and acquiring compressed protocol data within a preset time range from the second storage module, decompressing and deserializing the compressed protocol data to acquire the protocol data, detecting the protocol data to acquire the analysis data, and outputting the analysis data associated with asset information and geographic information.