CN116800777A - Message processing method and device, electronic equipment and storage medium - Google Patents
Message processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116800777A CN116800777A CN202210822715.XA CN202210822715A CN116800777A CN 116800777 A CN116800777 A CN 116800777A CN 202210822715 A CN202210822715 A CN 202210822715A CN 116800777 A CN116800777 A CN 116800777A
- Authority
- CN
- China
- Prior art keywords
- message
- abnormal
- distributed storage
- storage system
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 230000002159 abnormal effect Effects 0.000 claims abstract description 136
- 238000012545 processing Methods 0.000 claims abstract description 80
- 238000004458 analytical method Methods 0.000 claims abstract description 79
- 238000000034 method Methods 0.000 claims description 54
- 238000004590 computer program Methods 0.000 claims description 5
- 238000001914 filtration Methods 0.000 description 9
- 230000005540 biological transmission Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000004044 response Effects 0.000 description 5
- 238000007405 data analysis Methods 0.000 description 4
- 230000018109 developmental process Effects 0.000 description 4
- 206010000117 Abnormal behaviour Diseases 0.000 description 3
- 241000509958 Moloch Species 0.000 description 3
- 241000700605 Viruses Species 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 3
- 238000012217 deletion Methods 0.000 description 3
- 230000037430 deletion Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000013475 authorization Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000002265 prevention Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 101100368149 Mus musculus Sync gene Proteins 0.000 description 1
- 241001501944 Suricata Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000032683 aging Effects 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005206 flow analysis Methods 0.000 description 1
- ZXQYGBMAQZUVMI-GCMPRSNUSA-N gamma-cyhalothrin Chemical compound CC1(C)[C@@H](\C=C(/Cl)C(F)(F)F)[C@H]1C(=O)O[C@H](C#N)C1=CC=CC(OC=2C=CC=CC=2)=C1 ZXQYGBMAQZUVMI-GCMPRSNUSA-N 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer And Data Communications (AREA)
Abstract
The message processing method executed by the network backtracking analysis system provided by the embodiment of the disclosure comprises the following steps: receiving a message; uploading the received message to a distributed storage system in a stream type additional mode, wherein the message is stored in the distributed storage system in the stream type additional mode; determining whether abnormal messages exist in the received messages; when the abnormal message exists, the metadata of the abnormal message is transmitted to a preset data processing platform, wherein the metadata are used for carrying out abnormal analysis processing on the preset data processing platform. The network backtracking analysis system can receive the message and analyze whether the message is abnormal or not, the received message is uploaded to the distributed storage system in a stream type additional mode, the distributed storage system can store large-flow network data, metadata of the abnormal message is transmitted to the preset data processing platform, the preset data processing platform can conduct abnormal analysis processing through the metadata of the abnormal message, and analysis and backtracking of the large-flow network data are achieved.
Description
Technical Field
The present disclosure relates to the field of data processing, and in particular, to a method, an apparatus, an electronic device, and a storage medium for processing a message.
Background
In a network security scenario, various lux viruses and network attacks are more and more frequent, under the condition of coping with network security challenges, traditional intrusion prevention systems (IPS, intrusion Prevention System) and intrusion detection system (IDS, intrusion detection system) devices provide single message information when detecting a threat, which cannot be well analyzed and determined to be a network threat and timely processed, and network data traffic needs to be completely analyzed, stored and traced back under the current complex cloud network integrated environment.
In enterprises, especially finance and internet enterprises, network security is very important, backtracking and analysis of network traffic are imperative, but enterprise-level network data traffic is very large, and it is very difficult to completely analyze, store and backtrack all large-traffic network data traffic, and the existing network backtracking analysis system often stores, analyzes and backtracks through a disk array in the system, and the existing network backtracking analysis system cannot support analysis, storage, backtracking and the like of large-traffic data due to the problems of small storage traffic, read-write limitation and the like of the disk array.
Disclosure of Invention
In view of this, an embodiment of the disclosure discloses a method, a device, an electronic device and a storage medium for processing a message.
According to a first aspect of embodiments of the present disclosure, there is provided a method for processing a message, which is performed by a network backtracking analysis system, the method including:
receiving a message;
uploading the received message to a distributed storage system in a stream type additional mode, wherein the message is stored in the distributed storage system in the stream type additional mode;
determining whether abnormal messages exist in the received messages;
when the abnormal message exists, metadata of the abnormal message is transmitted to a preset data processing platform, wherein the metadata are used for carrying out abnormal analysis processing on the preset data processing platform.
In one embodiment, the determining whether the received message has an abnormal message includes: determining whether the received message has a field matched with the abnormal field; when the message field contained in the received message is matched with at least one abnormal field, determining the message containing the message field matched with the abnormal field as the abnormal message.
In one embodiment, the metadata of the exception message includes at least one of: a source port; a destination port; a source internet protocol IP address; a destination internet protocol IP address; an object name; a timestamp contained in the exception message; threat information comprising at least: and an exception field contained in the exception message.
In one embodiment, the uploading the received message to the distributed storage system in a stream append mode includes: uploading the received message to a distributed storage system in a stream type additional mode based on a post request; the uploading the received message to the distributed storage system in a stream append mode based on the post request comprises the following steps: storing the received message in a preset message pool; uploading the received message in the message pool to a distributed storage system by sending a post request; before one session ends, in one stream type additional, the messages with the same session identification in the message pool are uploaded to the distributed storage system.
In one embodiment, the post request includes: a bucket name, wherein the bucket name is named by an Internet Protocol (IP) address of a server of the network backtracking analysis system receiving the message; the bucket name is used for identifying a storage bucket for storing the message in the distributed storage system; the preset object name is determined according to a universal unique identifier UUID of a server receiving the message in the network backtracking analysis system and the preset object name obtained by the received session identifier ID of the message; and the preset object name is used for determining the file of the message stored in the distributed storage system.
In a second aspect, an embodiment of the present disclosure provides a method for processing a packet, which is performed by a distributed storage system, where the method includes:
receiving a message uploaded by a network backtracking analysis system in a stream type additional mode;
the distributed storage is used for storing the messages uploaded in a stream type additional mode; the message is used for tracing abnormal messages.
In a third aspect, an embodiment of the present disclosure provides a method for processing a packet, performed by a data processing platform, where the method includes:
receiving metadata of abnormal messages sent by a network backtracking analysis system;
and generating a preset download link according to the metadata of the abnormal message, wherein the download link is used for acquiring the abnormal message stored in a stream additional mode in a distributed storage system.
In one embodiment, the method further comprises: determining threat information according to the metadata of the abnormal message; and outputting an alarm according to the threat information.
In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor; wherein the processor, when running the computer program, performs the steps of the method of one or more of the foregoing technical solutions.
In a fifth aspect, embodiments of the present disclosure provide a computer-readable storage medium having stored thereon computer-executable instructions; the computer-executable instructions, when executed by the processor, are capable of performing the method of one or more of the foregoing aspects.
According to the message processing method provided by the embodiment of the disclosure, the network backtracking analysis system can receive the message and determine the abnormal message, the received message is uploaded to the distributed storage system in a stream type additional mode, compared with the method that the received message is directly stored in a disk, the quantity and efficiency of the stored message can be increased by uploading the message to the distributed storage system, the metadata of the abnormal message is uploaded to the data processing platform, compared with the method that the abnormal message is directly uploaded, the metadata of the abnormal message is more concise and clear, and the efficiency of abnormal data transmission is improved.
Compared with a disk, the distributed storage system does not need to perform frequent erasing, can dynamically allocate and store large-flow data into a plurality of object storage devices through the distributed storage, and rapidly and completely store and read and write the message, thereby realizing the storage and backtracking of the large-flow network data. The data processing platform can perform exception analysis processing by receiving metadata of the exception messages sent by the network backtracking analysis system, and compared with the exception analysis processing by manual or network backtracking analysis system, the data processing platform can perform analysis processing on a plurality of exception messages generated by large flow more rapidly and accurately by presetting the data processing platform; and generating a preset download link for the metadata of the abnormal message, calling the message through the download link, wherein compared with the process of directly searching the call message in a distributed storage system, the download link can pointedly call the message related to the abnormal message and can also share the message through the download link, so that a plurality of users and terminals can analyze and search the message.
Therefore, the problem that the backtracking analysis system stores the high-flow network data in a full flow way and backtracks the high-flow network data in a complete flow way can be solved through the network backtracking analysis system, the distributed storage system and the data processing platform, and the efficiency of analyzing, storing and backtracking the high-flow network data is improved.
Drawings
Fig. 1 is a schematic flow chart of a message processing method according to an embodiment of the disclosure.
Fig. 2 is a flow chart diagram of a message processing method according to an embodiment of the disclosure.
Fig. 3 is a flowchart illustrating a message processing method according to an embodiment of the present disclosure.
Fig. 4 is a flow chart diagram of a message processing method according to an embodiment of the disclosure.
Fig. 5 is a schematic structural diagram of a message processing system according to an embodiment of the present disclosure.
Fig. 6 is a flow chart of a message processing method according to an embodiment of the disclosure.
Fig. 7 is a schematic diagram of a message processing apparatus according to an embodiment of the disclosure.
Fig. 8 is a schematic diagram two of a message processing apparatus according to an embodiment of the disclosure.
Fig. 9 is a schematic diagram III of a message processing apparatus according to an embodiment of the disclosure.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present disclosure more apparent, the present disclosure will be further described in detail with reference to the accompanying drawings, and the described embodiments should not be construed as limiting the present disclosure, and all other embodiments obtained by those skilled in the art without making inventive efforts are within the scope of protection of the present disclosure.
In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with one another without conflict.
In the following description, the terms "first", "second", "third" and the like are merely used to distinguish similar objects and do not represent a particular ordering of the objects, it being understood that the "first", "second", "third" may be interchanged with a particular order or sequence, as permitted, to enable embodiments of the disclosure described herein to be practiced otherwise than as illustrated or described herein.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of the present disclosure only and is not intended to be limiting of the present disclosure.
For a better understanding of the disclosed embodiments, the following is a description of some example embodiments:
in the preservation and backtracking of network traffic, the input/output (I/O) amount of a mechanical hard disk can be 230MB/s, and when the size of the network data traffic reaches 30Gbps, 18 disks are needed to be written at the same time to completely preserve the traffic; although the solid state disk has high read-write performance, the solid state disk has the limit of erasing times and can not be stored and traced back for many times; common industrial personal computers are also limited by the transmission rate of the PCIE bus, and cannot save large traffic in time.
In addition, before the disk is written, hash operation is performed on the message according to the session, so that the success of reading and writing of the message on the disk can be ensured, and frequent reading and writing of the disk in the network traffic backtracking process can bring more I/O loss to the disk, so that the efficiency of network traffic backtracking analysis is reduced. Therefore, the conventional disk array method cannot support the network backtracking analysis of large traffic.
As shown in fig. 1, an embodiment of the present disclosure provides a method for processing a message, which is executed by a network backtracking analysis system, and includes:
step S101: receiving a message;
step S102: uploading the received message to a distributed storage system in a stream type additional mode, wherein the message is stored in the distributed storage system in the stream type additional mode;
Step S103: determining whether abnormal messages exist in the received messages;
step S104: when the abnormal message exists, metadata of the abnormal message is transmitted to a preset data processing platform, wherein the metadata are used for carrying out abnormal analysis processing on the preset data processing platform.
In one embodiment, the network backtracking analysis system may include: and a network backtracking analysis system formed by the Moroc moloch server. The moloch is an open source system which can be used for capturing the flow data messages on a large scale, storing the flow data messages on a hard disk and browsing, searching and analyzing the flow data messages.
In one embodiment, the network traceback analysis system may include a data plane development suite (DPDK, data Plane Development Kit), where the DPDK may aggregate a library of functions and a driver for fast processing of data packets, improving performance and throughput of packet data processing. Thus, the network backtracking analysis system can process large-flow network data in real time.
In one embodiment, the step S101 may include receiving, by a plurality of servers in the network backtracking analysis system, a traffic data packet interconnected with a network of servers. The servers can correspondingly receive traffic messages of different network segments or different network connection devices. In this way, when an abnormal message occurs, the message associated with the abnormal message can be searched according to the server receiving the abnormal message. Wherein, the associated message may include a message connected to the abnormal message network, a message connected to the same network segment as the abnormal message, and/or a message connected to the same network device as the abnormal message.
In one embodiment, the streaming additional mode may be an additional operation mode of streaming data based on a streaming caching technology. The step S102 may include continuously uploading the streaming message data to the distributed storage system, and adding the received message data to the last associated uploaded message data. In this way, the associated message data can be stored together by uploading the message data in a stream type additional mode, and compared with the method of creating a new cache and an index for each message data, the method of stream type additional can cache, search and read the associated message data together, thereby improving the processing efficiency of large-flow data.
In one embodiment, the abnormal message may include a message that may include threat information such as a virus Trojan or a network attack.
In one embodiment, determining whether the abnormal message exists may include determining whether the abnormal message exists based on abnormal behavior or feature detection. Wherein the abnormal behavior or characteristic may be determined from threat information. For example, a number of sync sequence numbers (SYN, synchronize Sequence Numbers) and/or acknowledgement characters (ACK, acknowledge character) attack messages may be included in the flooding attack, and the abnormal behavior may be set such that the number of times that a SYN and/or ACK message is received within a predetermined time exceeds a predetermined threshold range.
In one embodiment, the metadata (metadata) may be data for describing data, and the metadata of the abnormal message may include key attribute information of the abnormal message. In this way, the metadata of the abnormal message is transmitted to the preset data processing platform, and compared with the case that all the abnormal messages are transmitted to the preset data processing platform, the metadata of the abnormal message can be transmitted at a higher speed, so that the processing speed of the network threat data is improved.
As shown in fig. 2, the determining whether an abnormal message exists in the received message includes:
step S201: determining whether the received message has a field matched with the abnormal field;
step S202: when the message field contained in the received message is matched with at least one abnormal field, determining the message containing the message field matched with the abnormal field as the abnormal message.
In one embodiment, the exception fields may include an exception field determined from existing threat information and an exception field different from a normal field.
Illustratively, the exception field may include: abnormal number of message bytes, abnormal message flag bits, abnormal message internet protocol (IP, internet Protocol) address and/or abnormal message connection rate, etc.
The abnormal number of bytes may include that the number of bytes is smaller than the minimum number of bytes of the message or the number of bytes of the message exceeds the maximum number of bytes of the message, for example, the maximum number of bytes of the message is 65535 in the control message protocol (ICMP, internet Control Message Protocol), and when the number of bytes of the ICMP message is greater than 65535, the ICMP message is an abnormal message and may correspond to a network threat of death transmission (Ping of death).
The message flag bit abnormality may be to set an irregular message flag bit, for example, in a case that the network threat is a TCP message flag bit attack, the flag bit abnormality may include: the flag bit of the transmission control protocol (TCP, transmission Control Protocol) message is set to 1 or 0, the flag bit SYN and the end flag bit (FIN, finish) are 1 at the same time and/or the flag bit SYN and the reset flag bit (RST, reset) are 1 at the same time, etc.
The message IP address exception may include an exception of a source IP address and/or a destination IP address of the message, e.g., the source IP address of the message is an IP address of the attacking host or the destination IP address is the same as the source IP address. The abnormal message connection rate may include that the connection rate of the message exceeds a connection rate threshold, which may be used to prevent the server from failing to respond normally due to receiving a large number of attack connection messages in a short time.
In one embodiment, the step S201 may include: and determining a filtering rule according to a preset abnormal field, searching, filtering and matching the received message according to the filtering rule, and determining whether the received message has a field matched with the abnormal field according to the result of searching and filtering.
In one embodiment, the filtering rules may include determining filtering rules based on at least one exception field. For example, the filtering rule may include screening a message with a source IP address of 127.0.0.1 and/or a number of bytes greater than 65535, where the obtained message is screened as the abnormal message according to the filtering rule. Therefore, the type of the network threat possibly corresponding to the abnormal message can be judged more accurately by determining the filtering rule according to at least one abnormal field, the network threat can be determined more accurately by the data processing platform, and the response speed to the network threat is improved.
In one embodiment, the method may further comprise: determining whether all received messages have fields matched with abnormal fields in a preset time period, and determining that the message containing the message fields matched with the abnormal fields is the abnormal message when the message fields contained in the received messages are matched with at least one abnormal field. In this way, the network backtracking analysis system determines the abnormal message according to the message received in the predetermined time period, and can determine the abnormal message more accurately and comprehensively by a plurality of received messages than determining the abnormal message according to one received message.
In one embodiment, the metadata of the exception message includes at least one of: a source port, a destination port, a source Internet Protocol (IP) address, a destination IP address, an object name, a timestamp included in an abnormal message, and/or threat information, where the threat information at least includes: and an exception field contained in the exception message.
For example, the exception message may include an exception message that is sent by a server that sent the exception message to a service exception that received the exception message, the source port and source internet protocol (IP, internet Protocol) address may include a port and an IP address of the server that sent the exception message, and the destination port and destination IP address may include a port and an IP address of the server that received the exception message.
In one embodiment, the timestamp included in the exception message may be used to determine the time included in the exception message. The timestamp included in the abnormal message may include: an original timestamp for determining the time of sending the abnormal message; and/or a receive timestamp for determining when the abnormal message was received, etc.
In one embodiment, the threat information may be used to determine a cyber threat. Threat information in metadata of the abnormal message is sent to a data processing platform, and compared with the case of directly sending the abnormal message, the data processing platform can quickly determine the type of the network threat according to the threat information and alarm, and response efficiency of network security is improved.
Illustratively, the metadata may be in the format of:
{"objName":"3F2504E0-4F89-11D3-9A0C-0305E82C3301","srcPort":60409,"dstPort":53,"sessionId":"191217-aQuFA_-5t5RLFrsjujNaVgwU","IP":"10.230.1.117","timestamp":1576566105002,"srcIp":"10.230.1.151","dstIp":"10.88.7.10","firstConnectCountPackets":0,"node":"localhost","packetPos":[-145912325,500317654],"dns":{"ip":["107.163.177.82"],"hostName":["www.j2kdoan.top"],"status":["NOERROR"],"ttl":[600],"qt":["A"],"qc":["IN"],"rrType":[1]},"protocol":["udp","dns"],"suricata":{"flowIdCnt":1,"flowId":["155404755143330"],"actionCnt":1,"action":["allowed"],"signatureCnt":1,"signature":["ET DNS Query to a*.top domain–Likely Hostile"],"categoryCnt":1,"category":["PotentiallyBadTraffic"],"gidCnt":1,"gid":[1],"signatureIdCnt":1,"signatureId":[2023883],"severityCnt":1,"severity":[0]}}；
wherein the metadata includes: object name (obj name), source port (srcPort), destination port (dstPort), session identification (session id), network backtracking analysis system server IP receiving the message, source network protocol IP address (srip), destination IP address (dstpip), threat information including signature (signature) and category (category), etc.
In one embodiment, the uploading the received message to the distributed storage system in a stream append mode includes:
uploading the received message to a distributed storage system in a stream type additional mode based on a post request; the uploading the received message to the distributed storage system in a stream type additional mode based on the post request comprises the following steps:
storing the received message in a preset message pool;
uploading the received message in the preset message pool to a distributed storage system by sending a post request;
before one session ends, in one stream type additional, the messages with the same session identification in the message pool are uploaded to the distributed storage system.
In one embodiment, the post request may include a target request address, a protocol of the request, and/or message data to be sent, etc. Wherein the target request address may include: a target server address and a target interface address of the target server, the target interface address may include a path for the target server to deposit a file. The protocol of the request uses the hypertext transfer protocol (HTTP, hyper Text Transfer Protocol).
In one embodiment, the post request may further include a length of an uploaded message in the same session, where the length of the uploaded message may be recorded according to a storage thread call software development kit (SDK, software Development Kit) interface, and the length of the uploaded message may be used for a distributed storage system to store an object file.
In one embodiment, the post request may be used to send the received message data to the distributed storage system, and the target request address may be an IP address of the distributed storage system.
Illustratively, the syntax of the post request may include:
POST/10-230-1-117/objNameappend&position＝Position HTTP/1.1
Host:127.0.0.1:8000
User-Agent:python-requests/2.14.2
Accept-Encoding:gzip,deflate
Accept:*/*
Connection:keep-alive
date:Tue,20Jul 2021 02:36:36GMT
Authorization:AWS chx:p27eQoZ/PxY6Nbuud3SZq/r9ZMk＝
in one embodiment, the message pool can be used to store and manage the received messages.
In one embodiment, the uploading the received message in the predetermined message pool to the distributed storage system by sending a post request may include uploading the received message in the buffer in the message pool to the distributed storage system by sending a post request.
In one embodiment, the session may be a continuous call procedure from the client to the server, including multiple requests and responses. For example, a session may include a process that a user opens a browser to perform a plurality of operations until the browser is closed, and a process that generates a plurality of requests and responses between a corresponding user client and a corresponding server, where a request message and a response message in a session have the same session identifier.
In one embodiment, before a session ends, a message with the same session identifier in the message pool is determined, and the message with the same session identifier is uploaded to the back of the message with the same session in the distributed storage system in a stream additional mode.
In this way, compared with the method that all messages are directly uploaded to the distributed storage system, the messages of the same session are uploaded to the back of the messages of the same session in the distributed storage system in a stream type additional mode, the messages of the same session can be stored together in sequence, all the messages in the same session can be called according to the session identification, and the storage and calling efficiency of the distributed storage system is improved.
In one embodiment, the method may include: before one session is ended, setting a target request address in a post request of a message with the same session identifier in a message pool as the same target request address, and additionally uploading the message stream with the same session identifier in the message pool to a distributed storage system based on the same target request address.
In one embodiment, the post request includes:
a bucket name, wherein the bucket name is named by an Internet Protocol (IP) address of a server of the network backtracking analysis system receiving the message; and the bucket name is used for identifying a storage bucket for storing the message in the distributed storage system.
In one embodiment, the network backtracking analysis system may include a plurality of servers, and the bucket name may correspond to an ip address of one of the servers that receives the packet. Therefore, the storage barrel for storing the messages in the distributed storage system can be identified through the barrel names, the messages can be classified and stored according to the server for receiving the messages, and the efficiency of backtracking and analysis of the messages can be improved through the classified and stored of the messages received by the same server.
In one embodiment, the target interface address in the target server in the post request may include a bucket name.
In one embodiment, the post request further comprises:
the preset object name is determined according to a universal unique identifier UUID of a server receiving the message in the network backtracking analysis system and the preset object name obtained by the received session identifier ID of the message; and the preset object name is used for determining the file of the message stored in the distributed storage system.
In one embodiment, the predetermined object name may include a predetermined object name obtained by a hash (hash) operation according to a UUID of a universal unique identifier of a server receiving the message in the network backtracking analysis system and a session ID of the received message. In this way, messages of the same session may have the same predetermined object name.
In one embodiment, the object files determined according to the predetermined object names may be in a bucket determined according to the bucket names, and a bucket may include a plurality of object files determined according to the predetermined object names.
In one embodiment, the target interface address in the target server in the post request may include a bucket name and a predetermined object name. The target receiving address of the message of the same session received by the same server in the post request may be the same predetermined object name in the same bucket name.
As shown in fig. 3, an embodiment of the present disclosure provides a method for processing a message, which is performed by a distributed storage system, and includes:
step S301: receiving a message uploaded by a network backtracking analysis system in a stream type additional mode;
step S302: the distributed storage is used for storing the messages uploaded in a stream type additional mode; the message is used for tracing abnormal messages.
In one embodiment, the distributed storage may store data in a decentralized manner on multiple independent storage devices and form the decentralized storage resources into a virtual storage device. According to the distributed storage, large-flow message data can be stored in a scattered manner, compared with the case that a large amount of message data is stored in a network backtracking analysis system, flow analysis and storage are decoupled, the network backtracking analysis system does not need to store a large amount of message data, so that the operation burden is reduced, the analysis efficiency is improved, the distributed storage system can store the large-flow message data, the storage efficiency is improved, and the overall processing efficiency of the large-flow data is improved by combining the large-flow message data with the network backtracking analysis system.
In one embodiment, the distributed storage system may include: distributed object storage clusters ceph, and the like. Wherein the ceph storage cluster may comprise a plurality of ceph servers, which may comprise a plurality of mass storage devices.
In one embodiment, the storage threads in the distributed object storage cluster may use technologies such as multi-thread massive concurrency and/or remote direct data access (RDMA, remote Direct Memory Access), so that operations such as disk-drop storage, reading, modification and/or deletion can be performed on the message data by a large number of clients at the same time, and efficiency of storing and backtracking of the large-flow message data is improved.
In one embodiment, the distributed object storage cluster may further perform disaster recovery backup and/or timing deletion on the message data, where the timing deletion may delete expired data by configuring an aging policy, thereby improving storage efficiency of the distributed object storage cluster, and the disaster recovery backup improves safety and stability of message data storage.
In one embodiment, the distributed storage of the message uploaded in the streaming append manner includes:
And storing the uploaded message into an Object Storage Device (OSD) through a load balancing server.
In one embodiment, the load balancing server may be configured to dynamically store the uploaded message to a different Object storage device (OSD, object-based Storage Device) according to a load balancing policy. The load balancing server can dynamically allocate and store the uploaded messages, and the efficiency of storing the messages is improved.
In one embodiment, the load balancing server may include: linux virtual servers (LVS, linux Virtual Server), etc.
In one embodiment, the object storage device OSD may be a ceph OSD in a ceph storage cluster. The OSD may be configured to store the uploaded packet in an object form on a physical disk of each node of the cluster.
In one embodiment, the message uploaded by the receiving network backtracking analysis system in a stream append mode includes:
the receiving network backtracking analysis system is based on the message uploaded in a stream type additional mode by sending a post request.
In one embodiment, the distributed storage system analyzes the target request address, the request protocol and the sent message data in the post request to obtain the message uploaded by the network backtracking analysis system in a stream additional mode.
In one embodiment, the post request further comprises at least one of:
the bucket name is used for determining a storage bucket for storing the message by the distributed system; the barrel names are named by the Internet Protocol (IP) addresses of servers receiving the messages by the network backtracking analysis system;
a predetermined object name, wherein the predetermined object name is used for determining a file for storing the message by the distributed storage system; the preset object name is determined according to a universal unique identifier UUID of a server receiving the message in the network backtracking analysis system and a preset object name obtained by a received session identifier ID of the message.
In one embodiment, the distributed storage system may store the uploaded message in a predetermined object file of a predetermined bucket, and may include storing a message in the predetermined object file in the bucket, and appending a subsequent message of the same session to the same predetermined object file in the same bucket until the session ends. Therefore, the content of the same session is stored in the same preset object file in the same storage bucket, and the message of the same session can be checked and downloaded according to the bucket name and the preset object name, so that the content can be checked conveniently when an abnormality occurs, and the efficiency of network traffic backtracking is improved.
As shown in fig. 4, an embodiment of the present disclosure provides a method for processing a message, which is executed by a data processing platform, and includes:
step S401: receiving metadata of abnormal messages sent by a network backtracking analysis system;
step S402: and generating a preset download link according to the metadata of the abnormal message, wherein the download link is used for calling the preset message in the distributed storage system.
In one embodiment, the data processing platform may include a big data analysis processing platform or the like.
In one embodiment, the data processing platform may call a predetermined message from the distributed storage system according to a bucket name and an object name in metadata of the exception message. The predetermined message may include a message received by a corresponding server in a network backtracking analysis system that receives the message and corresponds to a bucket name, and a message in a session that corresponds to an object name.
In one embodiment, the generating the predetermined download link according to the metadata of the abnormal message may include generating a predetermined URL download link according to the predetermined object name and the bucket name in the metadata of the abnormal message, and the user may call the message with the same session identifier of the abnormal message in the distributed storage system according to the download link. Therefore, according to the download link, a plurality of users or terminals can check, download, analyze and/or share the preset message comprising the abnormal message, and the like, and the functions of multi-terminal linkage analysis and/or remote analysis and the like can be supported, so that the flexibility and convenience of flow backtracking are improved.
In one embodiment, the method further comprises obtaining a get request from metadata of the exception message; and acquiring an abnormal message according to the get request sent to the distributed storage system. In this way, the data processing platform can further analyze and process the obtained abnormal message or the message related to the abnormal message.
Illustratively, the format of the get request may include:
GET/10-230-1-117/objName HTTP/1.1
Host:127.0.0.1:8000
User-Agent:python-requests/2.14.2
Accept-Encoding:gzip,deflate
Accept:*/*
Connection:keep-alive
date:Tue,20Jul 2021 06:59:46GMT
Authorization:AWS chx:ELUe+eMFxxK/58KnEIfZoPGPNfk＝
in one embodiment, the method further comprises:
determining threat information according to the metadata of the abnormal message;
and outputting an alarm according to the threat information.
In one embodiment, the method further comprises: and outputting and displaying the threat information and the alarm in a web form.
In one embodiment, the determining threat information according to the metadata of the abnormal message may include determining threat information by performing, by a big data analysis processing platform, a data operation according to the metadata of the abnormal message, where the data operation may include querying data, statistical data, and/or data analysis, and the like, and where the data operation may be operated by a machine learning algorithm.
In one embodiment, the determining threat information may include determining whether a cyber attack or virus is present, determining a type and level of cyber threat information, and/or determining a source of cyber threat and determining a processing method corresponding to the cyber threat.
In one embodiment, the outputting the alert according to the threat information may include outputting different levels of the alert according to types and levels of threat information.
The embodiment of the disclosure provides a message processing system, which comprises:
the network backtracking analysis system is used for: receiving a message; uploading the received message to a distributed storage system in a stream type additional mode, wherein the message is stored in the distributed storage system in the stream type additional mode; determining whether abnormal messages exist in the received messages; when the abnormal message exists, transmitting metadata of the abnormal message to a preset data processing platform, wherein the metadata are used for carrying out abnormal analysis processing on the preset data processing platform;
a distributed storage system for: receiving a message uploaded by a network backtracking analysis system in a stream type additional mode; the distributed storage is used for storing the messages uploaded in a stream type additional mode; the message is used for tracing abnormal messages;
a data processing platform for: receiving metadata of abnormal messages sent by a network backtracking analysis system; and generating a preset download link according to the metadata of the abnormal message, wherein the download link is used for acquiring the abnormal message stored in a stream additional mode in a distributed storage system.
In one embodiment, the network backtracking analysis system is further configured to determine whether a field matching an abnormal field exists in the received packet; when the message field contained in the received message is matched with at least one abnormal field, determining the message containing the message field matched with the abnormal field as the abnormal message.
In one embodiment, the metadata of the exception message includes at least one of: a source port; a destination port; a source internet protocol IP address; a destination internet protocol IP address; an object name; a timestamp contained in the exception message; threat information comprising at least: and an exception field contained in the exception message.
In one embodiment, the network backtracking analysis system is further configured to: uploading the received message to a distributed storage system in a stream type additional mode based on a post request; the uploading the received message to the distributed storage system in a stream type additional mode based on the post request comprises the following steps: storing the received message in a preset message pool; uploading the received message in the preset message pool to a distributed storage system by sending a post request; before one session ends, in one stream type additional, the messages with the same session identification in the message pool are uploaded to the distributed storage system.
In one embodiment, the post request includes: a bucket name, wherein the bucket name is named by an Internet Protocol (IP) address of a server of the network backtracking analysis system receiving the message; the bucket name is used for identifying a storage bucket for storing the message in the distributed storage system; the preset object name is determined according to a universal unique identifier UUID of a server receiving the message in the network backtracking analysis system and the preset object name obtained by the received session identifier ID of the message; and the preset object name is used for determining the file of the message stored in the distributed storage system.
In one embodiment, the distributed storage system is further configured to: and storing the uploaded message into an Object Storage Device (OSD) through a load balancing server.
In one embodiment, the distributed storage system is further configured to: the receiving network backtracking analysis system is based on the message uploaded in a stream type additional mode by sending a post request.
In one embodiment, the data processing platform is further configured to: determining threat information according to the metadata of the abnormal message; and outputting an alarm according to the threat information.
As shown in fig. 5, in one embodiment, the network backtracking analysis system in a packet processing system may be composed of moloch servers, the distributed storage system may be composed of ceph clusters, the ceph clusters include LVS servers and ceph servers, and the data processing platform may be composed of big data analysis processing platforms.
In one embodiment, as shown in fig. 6, the flow of the message processing performed by the message processing system may include:
receiving a message through a network backtracking analysis system;
uploading the received message to a distributed storage system in a stream type additional mode through a network backtracking analysis system;
determining an abnormal message through a network backtracking analysis system, and transmitting metadata of the abnormal message to a preset data processing platform;
the distributed storage system receives a message uploaded by the network backtracking analysis system in a stream type additional mode, and stores the uploaded message into an Object Storage Device (OSD) according to a load balancing server in the distributed storage system;
the metadata of the abnormal message sent by the network backtracking analysis system is received through the data processing platform;
generating a preset download link according to the metadata of the abnormal message through a data processing platform, wherein the download link is used for calling the preset message in a distributed storage system;
Determining threat information and outputting an alarm according to the metadata of the abnormal message through a data processing platform;
the user views the threat information and the alarm at the data processing platform and downloads the predetermined message from the distributed storage system via the predetermined download link.
As shown in fig. 7, an embodiment of the present disclosure provides a message processing apparatus, including:
the receiving module is used for receiving the message;
the uploading module is used for uploading the received message to a distributed storage system in a stream type additional mode, wherein the message is stored in the distributed storage system in the stream type additional mode;
the determining module is used for determining whether abnormal messages exist in the received messages;
and the transmission module is used for transmitting the metadata of the abnormal message to a preset data processing platform when the abnormal message exists, wherein the metadata is used for carrying out abnormal analysis processing on the preset data processing platform.
In one embodiment, the determining module is further configured to: determining whether the received message has a field matched with the abnormal field; when the message field contained in the received message is matched with at least one abnormal field, determining the message containing the message field matched with the abnormal field as the abnormal message.
In one embodiment, the metadata of the abnormal message in the transmission module includes at least one of the following: a source port; a destination port; a source internet protocol IP address; a destination internet protocol IP address; an object name; a timestamp contained in the exception message; threat information comprising at least: and an exception field contained in the exception message.
In one embodiment, the uploading module is further configured to: uploading the received message to a distributed storage system in a stream type additional mode based on a post request; the uploading module further comprises: the storage module is used for storing the received message in a preset message pool; the uploading module further comprises: the sending module is used for uploading the received message in the message pool to the distributed storage system through sending a post request; the uploading module is further configured to: before one session ends, in one stream type additional, the messages with the same session identification in the message pool are uploaded to the distributed storage system.
In one embodiment, the post request includes: a bucket name, wherein the bucket name is named by an Internet Protocol (IP) address of a server of the network backtracking analysis system receiving the message; the bucket name is used for identifying a storage bucket for storing the message in the distributed storage system; the preset object name is determined according to a universal unique identifier UUID of a server receiving the message in the network backtracking analysis system and the preset object name obtained by the received session identifier ID of the message; and the preset object name is used for determining the file of the message stored in the distributed storage system.
As shown in fig. 8, an embodiment of the present disclosure provides a message processing apparatus, where the apparatus includes:
the receiving module is used for receiving the message uploaded by the network backtracking analysis system in a stream type additional mode;
the distributed storage module is used for distributed storage of the messages uploaded in the stream type additional mode; the message is used for tracing abnormal messages.
As shown in fig. 9, an embodiment of the present disclosure provides a message processing apparatus, including:
the receiving module is used for receiving the metadata of the abnormal message sent by the network backtracking analysis system;
a generation module for: and generating a preset download link according to the metadata of the abnormal message, wherein the download link is used for acquiring the abnormal message stored in a stream additional mode in a distributed storage system.
In one embodiment, the apparatus further comprises: the determining module is used for determining threat information according to the metadata of the abnormal message; and the output module is used for outputting an alarm according to the threat information.
It should be noted that, as those skilled in the art may understand, the methods provided in the embodiments of the present disclosure may be performed alone or together with some methods in the embodiments of the present disclosure or some methods in the related art.
The embodiment of the disclosure also provides an electronic device, which includes: a processor and a memory for storing a computer program capable of running on the processor, which when run performs the steps of the method of one or more of the preceding claims.
Embodiments of the present disclosure also provide a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, enable implementation of the method according to one or more of the foregoing technical solutions.
The computer storage medium provided in this embodiment may be a non-transitory storage medium. In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above described device embodiments are only illustrative, e.g. the division of the units is only one logical function division, and there may be other divisions in practice, such as: multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or units, whether electrically, mechanically, or otherwise.
The units described as separate units may or may not be physically separate, and units displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present disclosure may be integrated in one processing module, or each unit may be separately used as one unit, or two or more units may be integrated in one unit; the integrated units may be implemented in hardware or in hardware plus software functional units.
In some cases, the two technical features do not conflict, and a new method technical scheme can be combined.
In some cases, the above two technical features may be combined into a new device technical scheme without any conflict.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware associated with program instructions, where the foregoing program may be stored in a computer readable storage medium, and when executed, the program performs steps including the above method embodiments; and the aforementioned storage medium includes: a mobile storage device, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk or an optical disk, or the like, which can store program codes.
The foregoing is merely specific embodiments of the disclosure, but the protection scope of the disclosure is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the disclosure, and it is intended to cover the scope of the disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (10)
1. A method of message processing performed by a network backtracking analysis system, the method comprising:
receiving a message;
uploading the received message to a distributed storage system in a stream type additional mode, wherein the message is stored in the distributed storage system in the stream type additional mode;
determining whether abnormal messages exist in the received messages;
when the abnormal message exists, metadata of the abnormal message is transmitted to a preset data processing platform, wherein the metadata are used for carrying out abnormal analysis processing on the preset data processing platform.
2. The method of claim 1, wherein said determining whether an abnormal message exists in said received message comprises:
determining whether the received message has a field matched with the abnormal field;
When the message field contained in the received message is matched with at least one abnormal field, determining the message containing the message field matched with the abnormal field as the abnormal message.
3. The method according to claim 1 or 2, wherein the meta data of the exception message comprises at least one of:
a source port;
a destination port;
a source internet protocol IP address;
a destination internet protocol IP address;
an object name;
a timestamp contained in the exception message;
threat information comprising at least: and an exception field contained in the exception message.
4. The method according to claim 1 or 2, wherein said uploading said received message to a distributed storage system in a streaming append manner comprises:
uploading the received message to a distributed storage system in a stream type additional mode based on a post request;
the uploading the received message to the distributed storage system in a stream type additional mode based on the post request comprises the following steps:
storing the received message in a preset message pool;
uploading the received message in the preset message pool to a distributed storage system by sending a post request;
Before one session ends, in one stream type additional, the messages with the same session identification in the message pool are uploaded to the distributed storage system.
5. The method of claim 4, wherein the post request comprises:
a bucket name, wherein the bucket name is named by an Internet Protocol (IP) address of a server of the network backtracking analysis system receiving the message; the bucket name is used for identifying a storage bucket for storing the message in the distributed storage system;
the preset object name is determined according to a universal unique identifier UUID of a server receiving the message in the network backtracking analysis system and the preset object name obtained by the received session identifier ID of the message; and the preset object name is used for determining the file of the message stored in the distributed storage system.
6. A method of processing a message, performed by a distributed storage system, the method comprising:
receiving a message uploaded by a network backtracking analysis system in a stream type additional mode;
the distributed storage is used for storing the messages uploaded in a stream type additional mode; the message is used for tracing abnormal messages.
7. A method of processing a message, the method being performed by a data processing platform, the method comprising:
receiving metadata of abnormal messages sent by a network backtracking analysis system;
and generating a preset download link according to the metadata of the abnormal message, wherein the download link is used for acquiring the abnormal message stored in a stream additional mode in a distributed storage system.
8. The method of claim 7, wherein the method further comprises:
determining threat information according to the metadata of the abnormal message;
and outputting an alarm according to the threat information.
9. An electronic device, the electronic device comprising: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor performs the steps of the message processing method of any of claims 1 to 8 when the computer program is run.
10. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions; the computer executable instructions, when executed by a processor, are capable of implementing the message processing method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210822715.XA CN116800777A (en) | 2022-07-12 | 2022-07-12 | Message processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210822715.XA CN116800777A (en) | 2022-07-12 | 2022-07-12 | Message processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116800777A true CN116800777A (en) | 2023-09-22 |
Family
ID=88035146
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210822715.XA Pending CN116800777A (en) | 2022-07-12 | 2022-07-12 | Message processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116800777A (en) |
-
2022
- 2022-07-12 CN CN202210822715.XA patent/CN116800777A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11863587B2 (en) | Webshell detection method and apparatus | |
| US10666680B2 (en) | Service overload attack protection based on selective packet transmission | |
| US9917850B2 (en) | Deterministic reproduction of client/server computer state or output sent to one or more client computers | |
| US8844034B2 (en) | Method and apparatus for detecting and defending against CC attack | |
| US20170085583A1 (en) | Detecting malicious http redirections using user browsing activity trees | |
| US9817969B2 (en) | Device for detecting cyber attack based on event analysis and method thereof | |
| US10218733B1 (en) | System and method for detecting a malicious activity in a computing environment | |
| CN114145004A (en) | System and method for using DNS messages to selectively collect computer forensics data | |
| US8713674B1 (en) | Systems and methods for excluding undesirable network transactions | |
| US10122722B2 (en) | Resource classification using resource requests | |
| JP7388613B2 (en) | Packet processing method and apparatus, device, and computer readable storage medium | |
| CN111914126A (en) | Processing method, equipment and storage medium for indexed network security big data | |
| CN105577670A (en) | Warning system of database-hit attack | |
| US20240106730A1 (en) | Network Forensic System and Method | |
| CN112839054A (en) | Network attack detection method, device, equipment and medium | |
| KR102125966B1 (en) | System for collecting traffic and feature of TOR network using private network and virtual machine | |
| US10757118B2 (en) | Method of aiding the detection of infection of a terminal by malware | |
| KR20030039732A (en) | Attacker traceback method by using edge router's log information in the internet | |
| CN116800777A (en) | Message processing method and device, electronic equipment and storage medium | |
| CN113329035B (en) | Method and device for detecting attack domain name, electronic equipment and storage medium | |
| US10992702B2 (en) | Detecting malware on SPDY connections | |
| TW201928746A (en) | Method and apparatus for detecting malware | |
| CN114567472A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN112637171A (en) | Data traffic processing method, device, equipment, system and storage medium | |
| RU2777348C1 (en) | Computing apparatus and method for identifying compromised apparatuses based on dns tunnelling detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |