CN116800777A - Message processing method and device, electronic equipment and storage medium - Google Patents

Abstract

The message processing method executed by the network backtracking analysis system provided by the embodiment of the disclosure comprises the following steps: receiving a message; uploading the received message to a distributed storage system in a stream type additional mode, wherein the message is stored in the distributed storage system in the stream type additional mode; determining whether abnormal messages exist in the received messages; when the abnormal message exists, the metadata of the abnormal message is transmitted to a preset data processing platform, wherein the metadata are used for carrying out abnormal analysis processing on the preset data processing platform. The network backtracking analysis system can receive the message and analyze whether the message is abnormal or not, the received message is uploaded to the distributed storage system in a stream type additional mode, the distributed storage system can store large-flow network data, metadata of the abnormal message is transmitted to the preset data processing platform, the preset data processing platform can conduct abnormal analysis processing through the metadata of the abnormal message, and analysis and backtracking of the large-flow network data are achieved.

Description

Message processing method, device, electronic equipment and storage medium

Technical field

The present disclosure relates to but is not limited to the field of data processing, and in particular, to a message processing method, device, electronic equipment and storage medium.

Background technique

In network security scenarios, various ransomware viruses and network attacks appear more and more frequently. In response to network security challenges, traditional intrusion prevention systems (IPS, Intrusion Prevention System) and intrusion detection systems (IDS, intrusion detection systems) ) When a threat is detected, the device cannot provide a single message information to analyze, determine, and process network threats in a timely manner. In the current complex cloud-network integration environment, network data traffic needs to be completely analyzed, stored, and traced back. .

For enterprises, especially financial and Internet enterprises, network security is very important, and backtracking and analysis of network traffic is also imperative. However, enterprise-level network data traffic is very large, and all large-traffic network data traffic must be completely analyzed. , storage and traceback are very difficult. Existing network traceback analysis systems often use disk arrays in the system to store, analyze and traceback. Due to the problems of disk arrays with small storage traffic and read and write limitations, the existing network traceback analysis systems have been It cannot support analysis, storage and backtracking of large traffic data.

Contents of the invention

In view of this, embodiments of the present disclosure disclose a message processing method, device, electronic device, and storage medium.

According to a first aspect of an embodiment of the present disclosure, a message processing method is provided, which is executed by a network traceback analysis system. The method includes:

receive messages;

Upload the received message to the distributed storage system in a streaming append mode, where the message is stored by the distributed storage system in a streaming append mode;

Determine whether there is an abnormal message in the received message;

When the abnormal message exists, the metadata of the abnormal message is transmitted to a preset data processing platform, where the metadata is used by the preset data processing platform to perform exception analysis and processing.

In one embodiment, determining whether there is an abnormal message in the received message includes: determining whether the received message contains a field matching an abnormal field; when the received message contains a When the message field matches at least one exception field, it is determined that the message containing the message field matching the exception field is the abnormal message.

In one embodiment, the metadata of the abnormal message includes at least one of the following: source port; destination port; source Internet Protocol IP address; destination Internet Protocol IP address; object name; timestamp included in the abnormal message; threat Information, the threat information at least includes: the abnormal field contained in the abnormal message.

In one embodiment, uploading the received message to the distributed storage system in a streaming append mode includes: uploading the received message to the distributed storage system in a streaming append mode based on sending a post request; , wherein the uploading of received messages to the distributed storage system in a streaming append mode based on the post request includes: storing the received messages in a predetermined message pool; and storing the received messages in the message pool. The message is uploaded to the distributed storage system by sending a post request; before the end of a session, in a streaming append, the message with the same session ID in the message pool is uploaded to the distributed storage system.

In one embodiment, the post request includes: a bucket name, where the bucket name is named by the Internet Protocol IP address of the server that the network traceback analysis system receives the message; wherein the bucket name A name, used to identify the storage bucket in the distributed storage system where the message is stored; a predetermined object name, where the predetermined object name is based on the server that received the message in the network traceback analysis system. The predetermined object name obtained by the universal unique identification code UUID and the session identification ID of the received message is determined; the predetermined object name is used to determine the file in which the distributed storage system stores the message.

In a second aspect, embodiments of the present disclosure provide a message processing method, which is executed by a distributed storage system. The method includes:

Receive messages uploaded by the network traceback analysis system in streaming append mode;

Distributed storage of messages uploaded in a streaming append mode; wherein the messages are used for tracing abnormal messages.

In a third aspect, embodiments of the present disclosure provide a message processing method, which is executed by a data processing platform. The method includes:

Receive metadata of abnormal messages sent by the network backtracking analysis system;

A predetermined download link is generated according to the metadata of the abnormal message, where the download link is used to obtain the abnormal message stored in a streaming append mode in a distributed storage system.

In one embodiment, the method further includes: determining threat information based on metadata of the abnormal message; and outputting an alert based on the threat information.

In a fourth aspect, embodiments of the present disclosure provide an electronic device, the electronic device including: a processor and a memory for storing a computer program capable of running on the processor; wherein when the processor runs the computer program , perform the steps of the method described in one or more of the foregoing technical solutions.

In a fifth aspect, embodiments of the present disclosure provide a computer-readable storage medium that stores computer-executable instructions; after the computer-executable instructions are executed by a processor, one or more of the foregoing technical solutions can be implemented described method.

According to the message processing method provided by the embodiments of the present disclosure, the network traceback analysis system can receive messages and determine abnormal messages, and upload the received messages to the distributed storage system in a streaming append manner. Compared with directly uploading the received messages, Messages are stored in disks and uploaded to the distributed storage system to increase the number and efficiency of saving messages. By uploading the metadata of abnormal messages to the data processing platform, compared with directly uploading abnormal messages, exceptions are The metadata of messages is more concise and clear, improving the efficiency of abnormal data transmission.

The distributed storage system can distribute and store uploaded messages in a streaming manner. Compared with disks, the distributed storage system does not need to be frequently erased and can dynamically allocate and store large-traffic data to multiple objects through distributed storage. In the storage device, messages are quickly and completely saved, read and written, thereby realizing the storage and traceback of large-traffic network data. The data processing platform can perform exception analysis and processing by receiving the metadata of abnormal messages sent by the network traceback analysis system. Compared with abnormal analysis and processing through manual or network traceback analysis systems, the preset data processing platform can perform analysis and processing more quickly and accurately. Analysis and processing of multiple abnormal messages generated by large traffic; by generating a predetermined download link for the metadata of the abnormal message, and calling the message through the download link, compared to directly searching for the call message in the distributed storage system , the download link can specifically call messages related to abnormal messages, and can also be shared through the download link, allowing multiple users and terminals to analyze and view the messages.

In this way, the network traceback analysis system, distributed storage system and data processing platform can solve the problem of the traceback analysis system performing full flow storage and complete flow traceback of large-traffic network data, and improve the analysis and processing of large-traffic network data. Storage and backtracking efficiency.

Description of the drawings

Figure 1 is a schematic flowchart 1 of a packet processing method provided by an embodiment of the present disclosure.

Figure 2 is a schematic flowchart 2 of a message processing method provided by an embodiment of the present disclosure.

Figure 3 is a schematic flowchart 3 of a packet processing method provided by an embodiment of the present disclosure.

Figure 4 is a schematic flowchart 4 of a message processing method provided by an embodiment of the present disclosure.

Figure 5 is a schematic structural diagram of a message processing system provided by an embodiment of the present disclosure.

Figure 6 is a schematic flowchart of a message processing method provided by an embodiment of the present disclosure.

FIG. 7 is a schematic diagram 1 of a packet processing device provided by an embodiment of the present disclosure.

Figure 8 is a second schematic diagram of a packet processing device provided by an embodiment of the present disclosure.

Figure 9 is a schematic diagram three of a packet processing device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present disclosure. Those of ordinary skill in the art will not make any All other embodiments obtained under the premise of creative work belong to the scope of protection of this disclosure.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or a different subset of all possible embodiments, and Can be combined with each other without conflict.

In the following description, the terms "first\second\third" are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understandable that "first\second\third" is used in Where permitted, the specific order or sequence may be interchanged so that the disclosed embodiments described herein can be practiced in other sequences than illustrated or described herein.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of the disclosure only and is not intended to limit the disclosure.

In order to better understand the embodiments of the present disclosure, some scenario examples are described below:

In the storage and traceback of network traffic, the mechanical hard disk write/read (I/O, input/output) volume can be 230MB/s. When the size of network data traffic reaches 30Gbps, 18 disks need to be written simultaneously to write The traffic is completely preserved; although the solid-state disk has high read and write performance, it has a limit on the number of erasing and writing, and cannot save and trace back multiple times; common industrial computers are also limited by the transmission rate of the PCIE bus and cannot save large flows in a timely manner.

In addition, before writing to the disk, the packets need to be hashed according to the session to ensure that the packets are read and written successfully on the disk. Frequent reading and writing of the disk during the process of network traffic backtracking will also bring problems to the disk. More I/O losses reduce the efficiency of network traffic traceback analysis. Therefore, traditional disk array methods cannot support large-traffic network backtracking analysis.

As shown in Figure 1, an embodiment of the present disclosure provides a message processing method, which is executed by a network traceback analysis system. The method includes:

Step S101: Receive message;

Step S102: Upload the received message to the distributed storage system in a streaming append mode, where the message is stored by the distributed storage system in a streaming append mode;

Step S103: Determine whether there is an abnormal message in the received message;

Step S104: When the abnormal message exists, transmit the metadata of the abnormal message to the preset data processing platform, where the metadata is used by the preset data processing platform to perform exception analysis and processing.

In one embodiment, the network backtracking analysis system may include: a network backtracking analysis system composed of a Moloch server. Among them, Moloch is an open source system that can be used to capture large-scale traffic data packets and save them to a hard disk, as well as browse, retrieve and analyze traffic data packets.

In one embodiment, the network traceback analysis system may include a Data Plane Development Kit (DPDK), wherein the DPDK may integrate function libraries and drivers for fast processing of data packets to improve packet data processing. performance and throughput. In this way, the network backtracking analysis system can process large-traffic network data in real time.

In one embodiment, step S101 may include receiving traffic data packets interconnected with the server network through multiple servers in the network backtracking analysis system. Wherein, the multiple servers may correspondingly receive traffic packets from different network segments or different network connection devices. In this way, when an abnormal message occurs, the message associated with the abnormal message can be searched based on the server that received the abnormal message. Wherein, the associated packets may include packets from devices connected to the abnormal packet network, in the same network segment as the abnormal packet, and/or from devices connected to the same network as the abnormal packet.

In one embodiment, the streaming appending method may be a streaming data appending operation method based on streaming caching technology. The step S102 may include continuously uploading the streaming message data to the distributed storage system, and additionally uploading the received message data to the back of the last associated uploaded message data. In this way, uploading message data through streaming append can store associated message data together. Compared with creating a new cache and index for each message data, through streaming append, the associated message data can be stored together. Message data is cached, searched and read together, improving the efficiency of processing large traffic data.

In one embodiment, the abnormal messages may include messages that may include threat information such as viruses, Trojans, or network attacks.

In one embodiment, determining whether the abnormal message exists may include determining whether the abnormal message exists based on abnormal behavior or feature detection. Wherein, the abnormal behavior or characteristics can be determined based on threat information. For example, a flooding attack may include a large number of Synchronize Sequence Numbers (SYN) and/or Acknowledge character (ACK) attack messages. The abnormal behavior can be set to receive SYN and/or within a predetermined time. Or the number of ACK messages exceeds the predetermined threshold range.

In one embodiment, the metadata (metadata) may be data used to describe data, and the metadata of the abnormal message may include key attribute information of the abnormal message. In this way, the metadata of the abnormal message is transmitted to the preset data processing platform. Compared with transmitting all abnormal messages to the preset data processing platform, transmitting the metadata of the abnormal message can speed up the transmission and improve the efficiency. The speed of processing cyber threat data.

As shown in Figure 2, determining whether there is an abnormal message in the received message includes:

Step S201: Determine whether the received message contains a field matching the abnormal field;

Step S202: When the message field contained in the received message matches at least one exception field, determine that the message containing the message field matching the exception field is the abnormal message.

In one embodiment, the abnormal fields may include abnormal fields determined based on existing threat information and abnormal fields that are different from normal fields.

For example, the exception field may include: an abnormal number of message bytes, an abnormal message flag bit, an abnormal message Internet Protocol (IP, Internet Protocol) address, and/or an abnormal message connection rate, etc.

Wherein, the abnormal number of message bytes may include that the number of message bytes is less than the minimum number of bytes of the message or that the number of message bytes exceeds the maximum number of bytes of the message, such as Internet Control Message Protocol (ICMP). The maximum number of bytes in a ) message is 65535. When the number of bytes in an ICMP message is greater than 65535, the ICMP message is an abnormal message and may correspond to the network threat of ping of death.

Wherein, the abnormal message flag bit may be to set an unconventional message flag bit. For example, when the network threat is a TCP message flag bit attack, the abnormal flag bit may include: Transmission Control Protocol (TCP, Transmission Control Protocol) packet The flag bits of the file are all set to 1 or all set to 0, the flag bit SYN and the end flag bit (FIN, finish) are both 1 and/or the flag bit SYN and the reset flag bit (RST, reset) are both 1, etc. .

The abnormal IP address of the message may include abnormality of the source IP address and/or destination IP address of the message. For example, the source IP address of the message is the IP address of the attacking host or the destination IP address is the same as the source IP address. The abnormal packet connection rate may include that the packet connection rate exceeds the connection rate threshold, which may be used to prevent the server from being unable to respond normally due to the server receiving a large number of attack connection packets in a short period of time.

In one embodiment, the step S201 may include: determining a filtering rule according to a predetermined exception field, searching and filtering the received message according to the filtering rule, and determining the received message according to the search and filtered result. Whether the packet contains fields that match the exception fields.

In one embodiment, the filtering rules may include determining filtering rules based on at least one exception field. For example, the filtering rules may include filtering packets whose source IP address is 127.0.0.1 and/or the number of message bytes is greater than 65535, and the packets obtained by filtering according to the filtering rules are the abnormal packets. In this way, determining the filtering rule based on at least one abnormal field can more accurately determine the type of network threat that the abnormal message may correspond to, allowing the data processing platform to more accurately determine network threats, and improve the response speed to network threats.

In one embodiment, the method may further include: determining whether all received messages within a predetermined time period contain fields that match an abnormal field. When the received message contains a message field that matches at least one exception, When the fields match, it is determined that the message containing the message field matching the abnormal field is the abnormal message. In this way, the network backtracking analysis system determines the abnormal message based on the messages received within the predetermined time period. Compared with determining the abnormal message based on one received message, it can determine the abnormal message based on multiple received messages, which is more accurate. Comprehensively identify abnormal messages.

In one embodiment, the metadata of the abnormal message includes at least one of the following: source port, destination port, source Internet Protocol IP address, destination Internet Protocol IP address, object name, timestamp included in the abnormal message, and/or Or threat information, the threat information at least includes: the abnormal field contained in the abnormal message.

Exemplarily, the exception message may include an exception message sent by the server that sends the exception message to the service exception that receives the exception message, and the source port and source Internet Protocol (IP, Internet Protocol) address may include the server sending the exception message. The port and IP address of the server of the message. The destination port and destination IP address may include the port and IP address of the server that receives the abnormal message.

In one embodiment, the timestamp contained in the abnormal message can be used to determine the time contained in the abnormal message. The timestamp contained in the abnormal message may include: an original timestamp, used to determine the time of sending the abnormal message; and/or a receiving timestamp, used to determine the time of receiving the abnormal message, etc.

In one embodiment, the threat information can be used to determine network threats. Sending the threat information in the metadata of the abnormal message to the data processing platform can enable the data processing platform to quickly determine the type of network threat based on the threat information and issue an alarm, compared to directly sending the abnormal message, thereby improving Improve network security response efficiency.

By way of example, the format of the metadata may be:

{"objName":"3F2504E0-4F89-11D3-9A0C-0305E82C3301","srcPort":60409,"dstPort":53,"sessionId":"191217-aQuFA_-5t5RLFrsjujNaVgwU","IP":"10.230.1.117" ,"timestamp":1576566105002,"srcIp":"10.230.1.151","dstIp":"10.88.7.10","firstConnectCountPackets":0,"node":"localhost","packetPos":[-145912325,500317654 ],"dns":{"ip":["107.163.177.82"],"hostName":["www.j2kdoan.top"],"status":["NOERROR"],"ttl":[600] ,"qt":["A"],"qc":["IN"],"rrType":[1]},"protocol":["udp","dns"],"suricata":{" flowIdCnt":1,"flowId":["155404755143330"],"actionCnt":1,"action":["allowed"],"signatureCnt":1,"signature":["ET DNS Query to a*. top domain–Likely Hostile"],"categoryCnt":1,"category":["PotentiallyBadTraffic"],"gidCnt":1,"gid":[1],"signatureIdCnt":1,"signatureId":[2023883 ],"severityCnt":1,"severity":[0]}};

The metadata includes: object name (objName), source port (srcPort), destination port (dstPort), session identifier (sessionId), network traceback analysis system server IP that receives the message, and source network protocol IP address. (srcIp), destination IP address (dstIp), and threat information including signature and category.

In one embodiment, uploading the received message to the distributed storage system in a streaming append mode includes:

Uploading the received message to the distributed storage system in a streaming append mode based on sending a post request; wherein, uploading the received message to the distributed storage system in a streaming append mode based on the post request includes:

Store received messages in a predetermined message pool;

Upload the received messages in the predetermined message pool to the distributed storage system by sending a post request;

Before a session ends, in a streaming append, the packets with the same session ID in the packet pool are uploaded to the distributed storage system.

In one embodiment, the post request may include the target request address, the requested protocol and/or the message data to be sent, etc. Wherein, the target request address may include: a target server address and a target interface address of the target server, and the target interface address may include a path where the target server stores files. The requested protocol uses Hyper Text Transfer Protocol (HTTP).

In one embodiment, the post request may also include the length of the uploaded message in the same session, where the length of the uploaded message may be recorded according to the storage thread calling the Software Development Kit (SDK) interface. , the length of the uploaded message can be used to store object files in the distributed storage system.

In one embodiment, the post request may be used to send the received message data to the distributed storage system, and the target request address may be the IP address of the distributed storage system.

For example, the syntax of the post request may include:

POST/10-230-1-117/objName? append&position=Position HTTP/1.1

Host:127.0.0.1:8000

User-Agent:python-requests/2.14.2

Accept-Encoding:gzip,deflate

Accept:*/*

Connection:keep-alive

date:Tue,20Jul 2021 02:36:36GMT

Authorization:AWS chx:p27eQoZ/PxY6Nbuud3SZq/r9ZMk＝

In one embodiment, the message pool pool may be used to save and manage the received messages.

In one embodiment, uploading the received packets in the predetermined packet pool to the distributed storage system by sending a post request may include uploading the packets received in the cache buffer in the packet pool by sending a post request. Upload to distributed storage system.

In one embodiment, the session may be a continuous calling process from the client to the server, including multiple requests and responses. For example, a session may include a user opening a browser and performing multiple operations until closing the browser. A series of operations and the process of generating multiple requests and responses between the corresponding user client and the server. Request messages in a session and The response messages all have the same session ID.

In one embodiment, before a session ends, the packets with the same session ID in the packet pool are determined, and the packets with the same session ID are uploaded to the back of the same session packet in the distributed storage system through streaming append. .

In this way, instead of directly uploading all messages to the distributed storage system, the messages of the same session are uploaded to the back of the same session messages in the distributed storage system through streaming append. The messages of the same session can be The messages are stored together in order, and all messages in the same session can be called according to the session ID, which improves the storage and calling efficiency of the distributed storage system.

In one embodiment, the method may include: before the end of a session, setting the target request address in the post request of the message with the same session ID in the message pool to the same target request address, based on the same target request The address streams additionally uploaded packets with the same session ID in the packet pool to the distributed storage system.

In one embodiment, the post request includes:

Bucket name, wherein the bucket name is named by the Internet Protocol IP address of the server that the network traceback analysis system receives the message; wherein the bucket name is used to identify the storage in the distributed storage system The storage bucket of the message.

In one embodiment, the network traceback analysis system may include multiple servers, and the bucket name may correspond to the IP address of one of the servers that receives the message. In this way, by identifying the bucket storing the message in the distributed storage system through the bucket name, the message can be classified and saved according to the server that receives the message. Classifying and saving the message received by the same server can improve the understanding of all messages. Describe the efficiency of packet traceback and analysis.

In one embodiment, the target interface address in the target server in the post request may include a bucket name.

In one embodiment, the post request also includes:

Predetermined object name, wherein the predetermined object name is a predetermined object name obtained based on the universal unique identification code UUID of the server that received the message in the network backtracking analysis system and the session identification ID of the received message. Determined; the predetermined object name is used to determine the file in which the distributed storage system stores the message.

In one embodiment, the predetermined object name may include a universal unique identification code (UUID) of the server that received the message in the network backtracking analysis system and a session identification ID of the received message through a hash. ) operation to obtain the predetermined object name. In this way, messages of the same session can have the same scheduled object name.

In one embodiment, the object file determined according to the predetermined object name may be in a bucket determined according to the bucket name, and one bucket may include multiple object files determined by the predetermined object name.

In one embodiment, the target interface address in the target server in the post request may include a bucket name and a predetermined object name. For messages of the same session received by the same server, the target receiving address in the post request can be the same scheduled object name in the same bucket name.

As shown in Figure 3, an embodiment of the present disclosure provides a message processing method, which is executed by a distributed storage system. The method includes:

Step S301: Receive the message uploaded by the network traceback analysis system in the streaming append mode;

Step S302: Distributed storage of messages uploaded in a streaming append mode; wherein the messages are used for tracing abnormal messages.

In one embodiment, the distributed storage can dispersely store data on multiple independent storage devices and configure the dispersed storage resources into a virtual storage device. According to the distributed storage, large-traffic message data can be stored in a decentralized manner. Compared with storing a large amount of message data in the network traceback analysis system, the analysis and storage of traffic are decoupled, and the network traceback analysis system does not need to save a large number of reports. Text data reduces the operational burden and improves the efficiency of analysis. The distributed storage system can store large-traffic packet data and improves storage efficiency. The combination of the two improves the overall processing efficiency of large-traffic data.

In one embodiment, the distributed storage system may include: a distributed object storage cluster ceph, etc. Wherein, the ceph storage cluster may include multiple ceph servers, and the ceph servers may include multiple large-capacity storage devices.

In one embodiment, the storage threads in the distributed object storage cluster can adopt technologies such as multi-threaded massive concurrency and/or remote direct data access (RDMA), so that it can be used for a large number of customers. The terminal simultaneously performs operations such as storing, reading, modifying and/or deleting message data, which improves the efficiency of saving and tracing large-traffic message data.

In one embodiment, the distributed object storage cluster can also perform disaster recovery backup and/or scheduled deletion of the message data, wherein the scheduled deletion can periodically delete expired data by configuring an aging policy. , improving the efficiency of distributed object storage cluster storage, and the disaster recovery backup improves the security and stability of message data storage.

In one embodiment, the messages uploaded by the distributed storage in a streaming append mode include:

The uploaded message is stored in the object storage device OSD through the load balancing server.

In one embodiment, the load balancing server may be used to dynamically store uploaded messages into different object-based storage devices (OSDs) according to load balancing policies. The load balancing server can dynamically distribute and store uploaded messages, thereby improving the efficiency of message storage.

In one embodiment, the load balancing server may include: Linux virtual server (LVS, LinuxVirtual Server), etc.

In one embodiment, the object storage device OSD may be a ceph OSD in a ceph storage cluster. The OSD may be used to store the uploaded message in the form of an object on the physical disk of each node in the cluster.

In one embodiment, the packets uploaded by the network backtracking analysis system in a streaming append mode include:

The network traceback analysis system receives messages uploaded in streaming append mode based on sending post requests.

In one embodiment, the distributed storage system parses the target request address, requested protocol and sent message data in the post request to obtain the message uploaded by the network traceback analysis system in a streaming append mode.

In one embodiment, the post request also includes at least one of the following:

Bucket name, used by the distributed system to determine the storage bucket in which the message is stored; the bucket name is named by the Internet Protocol IP address of the server that the network traceback analysis system receives the message;

Predetermined object name, the predetermined object name is used by the distributed storage system to determine the file to store the message; wherein the predetermined object name is based on the message received in the network traceback analysis system. The predetermined object name is determined by the universal unique identification code UUID of the server and the session identification ID of the received message.

In one embodiment, the distributed storage system may store uploaded messages in a predetermined object file in a predetermined bucket, which may include storing a message in a predetermined object file in the bucket, and storing the same message in a predetermined object file in the bucket. Subsequent messages of the session are appended to the same scheduled object file in the same bucket until the session ends. In this way, the contents of the same session are stored in the same scheduled object file in the same bucket, and the messages of the same session can be viewed and downloaded based on the bucket name and scheduled object name, which facilitates viewing when an exception occurs and improves the efficiency of network traffic traceback. efficiency.

As shown in Figure 4, an embodiment of the present disclosure provides a message processing method, which is executed by a data processing platform. The method includes:

Step S401: Receive metadata of the abnormal message sent by the network backtracking analysis system;

Step S402: Generate a predetermined download link according to the metadata of the abnormal message, where the download link is used to call the predetermined message in the distributed storage system.

In one embodiment, the data processing platform may include a big data analysis and processing platform, etc.

In one embodiment, the data processing platform can call a predetermined message from the distributed storage system based on the bucket name and object name in the metadata of the exception message. The predetermined message may include a message received by a corresponding server in the network traceback analysis system that receives the message corresponding to the bucket name and a message in the session corresponding to the object name.

In one embodiment, generating a predetermined download link based on the metadata of the exception message may include generating a predetermined URL download link based on the predetermined object name and the bucket name in the metadata of the exception message. , the user can call the message with the same session ID of the abnormal message in the distributed storage system according to the download link. In this way, according to the download link, multiple users or terminals can view, download, analyze and/or share scheduled messages including abnormal messages, and can support functions such as multi-terminal linkage analysis and/or off-site analysis. Improved the flexibility and convenience of traffic traceback.

In one embodiment, the method further includes obtaining a get request based on the metadata of the exception message; and obtaining the exception message based on sending the get request to the distributed storage system. In this way, the abnormal messages or messages related to the abnormal messages that can be obtained by the data processing platform are further analyzed and processed.

For example, the format of the get request may include:

GET/10-230-1-117/objName HTTP/1.1

Host:127.0.0.1:8000

User-Agent:python-requests/2.14.2

Accept-Encoding:gzip,deflate

Accept:*/*

Connection:keep-alive

date:Tue,20Jul 2021 06:59:46GMT

Authorization:AWS chx:ELUe+eMFxxK/58KnEIfZoPGPNfk＝

In one embodiment, the method further includes:

Determine threat information based on the metadata of the abnormal message;

An alert is output based on the threat information.

In one embodiment, the method further includes: outputting and displaying the threat information and alerts in a web form.

In one embodiment, determining the threat information based on the metadata of the abnormal message may include using a big data analysis and processing platform to perform data operations based on the metadata of the abnormal message to determine the threat information, wherein the data operation may include Query data, statistical data and/or data analysis, etc., wherein the data operations can be performed through machine learning algorithms.

In one embodiment, determining the threat information may include determining whether there is a network attack or virus, determining the type and level of the network threat information, and/or determining the source of the network threat and determining a processing method corresponding to the network threat.

In one embodiment, outputting an alert according to the threat information may include outputting alerts of different levels according to the type and level of the threat information.

Embodiments of the present disclosure provide a message processing system. The message processing system includes:

A network traceback analysis system, configured to: receive messages; upload the received messages to a distributed storage system in a streaming append mode, wherein the messages are stored by the distributed storage system in a streaming append mode ; Determine whether there is an abnormal message in the received message; when the abnormal message exists, transmit the metadata of the abnormal message to the preset data processing platform, where the metadata is used for all The above-mentioned preset data processing platform performs abnormal analysis and processing;

A distributed storage system, used for: receiving messages uploaded by the network traceback analysis system in a streaming append mode; distributed storage of messages uploaded by a streaming append mode; wherein the messages are used for traceability of abnormal messages ;

A data processing platform, configured to: receive metadata of abnormal messages sent by the network traceback analysis system; generate a predetermined download link based on the metadata of the abnormal message, wherein the download link is used to obtain in the distributed storage system The abnormal message stored in streaming append mode.

In one embodiment, the network traceback analysis system is also used to: determine whether the received message contains a field that matches an abnormal field; when the received message contains a message field that matches at least one abnormal field When , it is determined that the message containing the message field matching the abnormal field is the abnormal message.

In one embodiment, the metadata of the abnormal message includes at least one of the following: source port; destination port; source Internet Protocol IP address; destination Internet Protocol IP address; object name; timestamp included in the abnormal message; threat Information, the threat information at least includes: the abnormal field contained in the abnormal message.

In one embodiment, the network traceback analysis system is also configured to: upload the received message to the distributed storage system in a streaming append mode based on the post request; wherein, the received message based on the post request is uploaded to the distributed storage system in a streaming append mode. The streaming append method is uploaded to the distributed storage system, including: storing the received messages in a predetermined message pool; uploading the received messages in the predetermined message pool to the distributed storage system by sending a post request ; Before the end of a session, in a streaming append, the packets with the same session ID in the packet pool are uploaded to the distributed storage system.

In one embodiment, the post request includes: a bucket name, where the bucket name is named by the Internet Protocol IP address of the server that the network traceback analysis system receives the message; wherein the bucket name A name, used to identify the storage bucket in the distributed storage system where the message is stored; a predetermined object name, where the predetermined object name is based on the server that received the message in the network traceback analysis system. The predetermined object name obtained by the universal unique identification code UUID and the session identification ID of the received message is determined; the predetermined object name is used to determine the file in which the distributed storage system stores the message.

In one embodiment, the distributed storage system is further configured to: store the uploaded message into an object storage device OSD through a load balancing server.

In one embodiment, the distributed storage system is further configured to: receive messages uploaded by the network traceback analysis system in a streaming append mode based on sending a post request.

In one embodiment, the data processing platform is further configured to: determine threat information based on metadata of the abnormal message; and output an alert based on the threat information.

As shown in Figure 5, in one embodiment, the network backtracking analysis system in a message processing system can be composed of a moloch server, and the distributed storage system can be composed of a ceph cluster. The ceph cluster includes an LVS server. and ceph server, the data processing platform may be composed of a big data analysis and processing platform.

In one embodiment, as shown in Figure 6, the process of packet processing through the packet processing system may include:

Receive messages through the network traceback analysis system;

Upload the received message to the distributed storage system in a streaming append mode through the network traceback analysis system;

Determine abnormal messages through the network traceback analysis system, and transmit the metadata of the abnormal messages to the preset data processing platform;

The distributed storage system receives the message uploaded by the network traceback analysis system in a streaming append mode, and stores the uploaded message in the object storage device OSD according to the load balancing server in the distributed storage system;

Receive metadata of abnormal messages sent by the network backtracking analysis system through the data processing platform;

Generate a predetermined download link based on the metadata of the abnormal message through the data processing platform, where the download link is used to call the predetermined message in the distributed storage system;

Use the data processing platform to determine threat information based on the metadata of the abnormal message and output an alert;

Users view threat information and alerts on the data processing platform, and download scheduled messages from the distributed storage system through scheduled download links.

As shown in Figure 7, an embodiment of the present disclosure provides a message processing device. The device includes:

Receiving module, used to receive messages;

An upload module, configured to upload the received message to the distributed storage system in a streaming append mode, wherein the message is stored by the distributed storage system in a streaming append mode;

A determination module, used to determine whether there is an abnormal message in the received message;

A transmission module, used to transmit the metadata of the abnormal message to a preset data processing platform when the abnormal message exists, wherein the metadata is used for the preset data processing platform to perform abnormal analysis. deal with.

In one embodiment, the determination module is further configured to: determine whether the received message contains a field that matches an abnormal field; when the message field contained in the received message matches at least one abnormal field , determining that the message containing the message field matching the abnormal field is the abnormal message.

In one embodiment, the metadata of the abnormal message in the transmission module includes at least one of the following: source port; destination port; source Internet Protocol IP address; destination Internet Protocol IP address; object name; the exception message contains Timestamp; threat information, the threat information at least includes: the abnormal field contained in the abnormal message.

In one embodiment, the upload module is also configured to: upload the received message to the distributed storage system in a streaming append mode based on sending a post request; the upload module also includes: a storage module, used to The received messages are stored in a predetermined message pool; the upload module also includes: a sending module, used to upload the received messages in the message pool to the distributed storage system by sending a post request; the upload The module is also used to upload messages with the same session ID in the message pool to the distributed storage system in a streaming append before the end of a session.

In one embodiment, the post request includes: a bucket name, where the bucket name is named by the Internet Protocol IP address of the server that the network traceback analysis system receives the message; wherein the bucket name A name, used to identify the storage bucket in the distributed storage system where the message is stored; a predetermined object name, where the predetermined object name is based on the server that received the message in the network traceback analysis system. The predetermined object name obtained by the universal unique identification code UUID and the session identification ID of the received message is determined; the predetermined object name is used to determine the file in which the distributed storage system stores the message.

As shown in Figure 8, an embodiment of the present disclosure provides a message processing device. The device includes:

The receiving module is used to receive messages uploaded by the network traceback analysis system in the streaming append mode;

The distributed storage module is used for distributed storage of messages uploaded in a streaming append mode; wherein the messages are used for tracing abnormal messages.

As shown in Figure 9, an embodiment of the present disclosure provides a message processing device. The device includes:

The receiving module is used to receive metadata of abnormal messages sent by the network backtracking analysis system;

A generation module, configured to: generate a predetermined download link according to the metadata of the abnormal message, wherein the download link is used to obtain the abnormal message stored in a streaming append mode in a distributed storage system.

In one embodiment, the device further includes: a determination module, configured to determine threat information based on the metadata of the abnormal message; and an output module, configured to output an alert based on the threat information.

It should be noted that those skilled in the art can understand that the methods provided in the embodiments of the present disclosure can be executed alone or together with some methods in the embodiments of the present disclosure or some methods in related technologies.

An embodiment of the present disclosure also provides an electronic device. The electronic device includes: a processor and a memory for storing a computer program that can be run on the processor. When the processor runs the computer program, it executes one or more of the foregoing The steps of the method described in the technical solution.

Embodiments of the present disclosure also provide a computer-readable storage medium that stores computer-executable instructions. After the computer-executable instructions are executed by the processor, the method described in one or more of the foregoing technical solutions can be implemented. .

The computer storage medium provided in this embodiment may be a non-transient storage medium. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components may be combined, or can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be electrical, mechanical, or other forms. of.

The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units; Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure can be all integrated into one processing module, or each unit can be separately used as a unit, or two or more units can be integrated into one unit; the above-mentioned integration The unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

In some cases, if any two of the above technical features do not conflict, they can be combined into a new method and technical solution.

In some cases, if any two of the above technical features do not conflict, they can be combined into a new equipment technical solution.

Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, It includes the steps of the above method embodiment; and the aforementioned storage media includes: mobile storage devices, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disks or optical disks, etc. A medium on which program code can be stored.

The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure. should be covered by the protection scope of this disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims (10)

1. A message processing method, characterized in that it is executed by a network traceback analysis system, and the method includes: receive messages; Upload the received message to the distributed storage system in a streaming append mode, where the message is stored by the distributed storage system in a streaming append mode; Determine whether there is an abnormal message in the received message; When the abnormal message exists, the metadata of the abnormal message is transmitted to the preset data processing platform, and the metadata is used by the preset data processing platform to perform exception analysis and processing. 2. The method according to claim 1, wherein the determining whether there is an abnormal message in the received message includes: Determine whether the received message contains a field matching the abnormal field; When the message field contained in the received message matches at least one exception field, it is determined that the message containing the message field matching the exception field is the abnormal message. 3. The method according to claim 1 or 2, characterized in that the metadata of the abnormal message includes at least one of the following: source port; destination port; Source Internet Protocol IP address; Destination Internet Protocol IP address; object name; The timestamp included in the exception message; Threat information, the threat information at least includes: the abnormal field contained in the abnormal message. 4. The method according to claim 1 or 2, characterized in that uploading the received message to the distributed storage system in a streaming append mode includes: Upload the received message to the distributed storage system in a streaming append method based on the post request; Among them, the received messages are uploaded to the distributed storage system in a streaming append mode based on the post request, including: Store received messages in a predetermined message pool; Upload the received messages in the predetermined message pool to the distributed storage system by sending a post request; Before a session ends, in a streaming append, the packets with the same session ID in the packet pool are uploaded to the distributed storage system. 5. The method according to claim 4, characterized in that the post request includes: Bucket name, where the bucket name is named by the Internet Protocol IP address of the server that the network traceback analysis system receives the message; where the bucket name is used to identify the storage in the distributed storage system The storage bucket of the message; Predetermined object name, wherein the predetermined object name is a predetermined object name obtained based on the universal unique identification code UUID of the server that received the message in the network traceback analysis system and the session identification ID of the received message. Determined; the predetermined object name is used to determine the file in which the distributed storage system stores the message. 6. A message processing method, characterized in that it is executed by a distributed storage system, and the method includes: Receive messages uploaded by the network traceback analysis system in streaming append mode; Distributed storage of messages uploaded in a streaming append mode; wherein the messages are used for tracing abnormal messages. 7. A message processing method, characterized in that it is executed by a data processing platform, and the method includes: Receive metadata of abnormal messages sent by the network backtracking analysis system; A predetermined download link is generated according to the metadata of the abnormal message, where the download link is used to obtain the abnormal message stored in a streaming append mode in a distributed storage system. 8. The method according to claim 7, characterized in that the method further comprises: Determine threat information based on the metadata of the abnormal message; An alert is output based on the threat information. 9. An electronic device, characterized in that the electronic device includes: a processor and a memory for storing a computer program capable of running on the processor, wherein when the processor runs the computer program, claim 1 is executed Go to the steps of the message processing method described in any one of 8. 10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-executable instructions; after the computer-executable instructions are executed by a processor, the computer-readable storage medium can realize any one of claims 1 to 8 The message processing method described in the item.