CN112311717A - Network data recovery method and device, storage medium and computer equipment - Google Patents
Network data recovery method and device, storage medium and computer equipment Download PDFInfo
- Publication number
- CN112311717A CN112311717A CN201910672616.6A CN201910672616A CN112311717A CN 112311717 A CN112311717 A CN 112311717A CN 201910672616 A CN201910672616 A CN 201910672616A CN 112311717 A CN112311717 A CN 112311717A
- Authority
- CN
- China
- Prior art keywords
- data
- message
- network
- thread
- network data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
Abstract
The application relates to a network data recovery method, a device, a computer readable storage medium and a computer device, wherein the method comprises the following steps: when a network data processing request is acquired, a virtual port to be monitored carried by the network processing request is determined, a message capturing thread and a data recovery thread are synchronously called, a data message received and sent through the virtual port is captured in real time through the message capturing thread, the data message is written into a buffer area, the data message of the buffer area is read through the data recovery thread, and data recovery processing is carried out on the data message, so that recovered network data are obtained. The recovery processing efficiency of the network data is improved, and the recovered network data is obtained quickly.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for recovering network data, a computer-readable storage medium, and a computer device.
Background
With the wide application of the internet, the security risk of the internet becomes more and more obvious, for example, network hackers spread various malicious codes or malicious software by using the internet, and the security is improved by relying on a server and a personal computer, which is difficult to meet the security requirement. Therefore, it is necessary to monitor the data stream transmitted in the network, analyze the network status and the user behavior, assist the network administrator to find out the security problem and provide appropriate defense measures to ensure the normal operation of the network.
In the conventional technology, a monitoring manner for network data is generally to query the sniffed binary network data by using a packet capture tool, such as wireshark, sniffer, etc., and to show the protocol type and data format of the data. Among them, Wireshark (ethernet) is a network packet analysis software. The function of the network packet analysis software is to capture the network packets and display the most detailed network packet data as possible. Wireshark uses WinPCAP (Windows packet capture, which is a free and public network access system under a Windows platform and provides access network bottom capability for win 32) as an interface to directly exchange data messages with a network card. Sniffer, also called Sniffer or packet capture software, is a network analysis method based on the principle of passive interception, and by using this technical method, the state of the network, the data flow situation and the information transmitted on the network can be monitored.
However, the packet capturing tool can only analyze a binary network data structure, and a network administrator cannot visually perceive specific contents of the network data according to the binary data, and needs to additionally process the binary data obtained by the packet capturing tool to recover the data. For complex data, context analysis of network data is required, and then data recombination is performed, so that the recovery processing process of the network data is complex, the data recovery efficiency is low, and the recovered network data cannot be obtained quickly.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a network data recovery method, apparatus, computer-readable storage medium, and computer device for solving the technical problem of inefficient network data recovery.
A method of network data recovery, comprising:
when a network data processing request is acquired, determining a virtual port to be monitored carried by the network processing request, and synchronously calling a message capturing thread and a data recovery thread;
capturing data messages sent and received through the virtual port in real time through the message capturing thread, and writing the data messages into a buffer area;
and reading the data message of the buffer area through the data recovery thread, and performing data recovery processing on the data message to obtain recovered network data.
A network data recovery apparatus, the apparatus comprising:
the system comprises a synchronization module, a message acquisition module and a data recovery module, wherein the synchronization module is used for determining a virtual port to be monitored carried by a network processing request when the network data processing request is acquired, and synchronously calling a message acquisition thread and a data recovery thread;
the flow capturing module is used for capturing the data message which is transmitted and received through the virtual port in real time through the message capturing thread and writing the data message into a buffer zone;
and the flow recovery module is used for reading the data message of the buffer area through the data recovery thread and performing data recovery processing on the data message to obtain recovered network data.
A computer-readable storage medium, storing a computer program which, when executed by a processor, causes the processor to perform the steps of the method of:
when a network data processing request is acquired, determining a virtual port to be monitored carried by the network processing request, and synchronously calling a message capturing thread and a data recovery thread;
capturing data messages sent and received through the virtual port in real time through the message capturing thread, and writing the data messages into a buffer area;
and reading the data message of the buffer area through the data recovery thread, and performing data recovery processing on the data message to obtain recovered network data.
A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method of:
when a network data processing request is acquired, determining a virtual port to be monitored carried by the network processing request, and synchronously calling a message capturing thread and a data recovery thread;
capturing data messages sent and received through the virtual port in real time through the message capturing thread, and writing the data messages into a buffer area;
and reading the data message of the buffer area through the data recovery thread, and performing data recovery processing on the data message to obtain recovered network data.
According to the network data recovery method, the device, the computer readable storage medium and the computer equipment, on one hand, the data message received and sent by the terminal to be monitored is captured through the monitoring virtual port, the real-time data message can be monitored, and the network data recovery processing is carried out on the data message without manual analysis processing, so that a network administrator can visually read the transmitted network data, on the other hand, the message capturing thread and the data recovery thread are synchronously executed by additionally arranging the buffer area, the recovery processing efficiency of the network data is improved, and the recovered network data is quickly obtained.
Drawings
FIG. 1 is a diagram of an exemplary network data recovery method;
FIG. 2 is a flow diagram illustrating a method for network data recovery in one embodiment;
FIG. 3 is a flow diagram illustrating the determination of a virtual port to be monitored according to an embodiment;
FIG. 4 is a schematic diagram of an input interface for configuration data in one embodiment;
FIG. 5 is a diagram illustrating the structure of distributed threads in one embodiment;
FIG. 6 is a flow diagram that illustrates the writing of a data packet into a buffer in one embodiment;
FIG. 7 is a flow diagram illustrating network data recovery in one embodiment;
FIG. 8 is a diagram illustrating an interface for data message capture results, according to an embodiment;
FIG. 9 is a schematic diagram of an interface for a data message capture result in another embodiment;
FIG. 10 is a schematic diagram of an input interface for configuration data in another embodiment;
FIG. 11 is a diagram illustrating the results of restoring network data stored to a database in one embodiment;
FIG. 12 is a diagram illustrating the results of restoring network data stored to a database in another embodiment;
FIG. 13 is a flowchart illustrating a method for recovering network data in another embodiment;
FIG. 14 is a block diagram showing the construction of a network data recovery apparatus according to an embodiment;
fig. 15 is a block diagram showing the construction of a network data recovery apparatus according to another embodiment;
FIG. 16 is a block diagram showing a configuration of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In one embodiment, the application environment of the network data recovery method is shown in fig. 1. The application environment relates to the terminal 102, the network data administration platform 104 and the terminal interaction object 106, and the terminal interaction object 106 may be a server or other terminal. The terminal 102 is connected to the terminal interaction object 106 through a network to access data, and the terminal interaction object 106 responds to an access request sent by the terminal 102 and sends data corresponding to the access request to the terminal 102 through a data message. In practical application, a network hacker may perform an aggressive or sensitive data traffic operation by using data interaction between a terminal and a server, and need to perform data recovery on network data in order to obtain information transmitted by the data traffic and provide a proper defense capability according to the transmitted information to improve information security. The network data supervision platform 104 monitors the virtual port of the terminal 102, captures the data message transmitted and received by the virtual port, recovers the data of the captured data message, and recombines the data into information such as image-text, video, web page and the like which can be visually identified by people, so as to visually check and analyze the recovered network data, thereby finding out a safety problem and providing a proper defense measure to ensure the normal operation of the network. In one embodiment, the network data administration platform 104 includes a terminal providing a human-computer interaction interface, which may be the terminal 102 or another terminal different from the terminal 102 and the terminal interaction object 106.
The terminal 102 may be a local computer, and may specifically be a desktop terminal or a mobile terminal, and the mobile terminal may specifically be at least one of a mobile phone, a tablet computer, a notebook computer, and the like. The terminal interaction object 106 may be a server, and the server may be implemented by a stand-alone server or a server cluster composed of a plurality of servers.
In one embodiment, a method of network data recovery is provided. The embodiment is mainly illustrated by applying the method to the network data monitoring platform 104 in fig. 1. As shown in fig. 2, the network data recovery method specifically includes steps S202 to S206.
S202, when a network data processing request is acquired, determining a virtual port to be monitored carried by the network processing request, and synchronously calling a message capturing thread and a data recovery thread.
The network data processing request refers to a processing request generated based on the configuration data when the trigger condition is satisfied. The trigger condition includes that the user triggers a set 'run' function button by clicking, and can also be based on a timer, the current time meets the requirement of the timer, and the like. The 'operation' function button comprises a function button arranged on a User Interface (UI for short) based on a Qt (cross-platform C + + graphical User Interface application development framework) Interface, or a function button configured in a Web page, wherein the Web page is suitable for monitoring and recovering network data on a machine in a mode of installing an agent (representing software for processing query and returning result of an application program), and providing conditions of query monitoring and recovery of a Web interaction inlet.
In an embodiment, referring to fig. 3, when the network data processing request is acquired, determining the virtual port to be monitored carried by the network processing request includes steps S302 to S304.
And S302, when the network data processing request is acquired, reading the port selection parameters in the configuration data carried by the network data processing request.
S304, determining the virtual port corresponding to the port selection parameter in the terminal to be detected as the virtual port to be monitored.
In an embodiment, the port selection parameter is obtained by reading parameter information in a filter expression, the filter expression may be set according to a domain name to be accessed, taking www.qq.com as an example, since an application layer Protocol adopted in www.qq.com is an HTTP ((HyperText Transfer Protocol) Protocol, and a corresponding terminal virtual port is an 80 port, the filter expression may be set to capture only data with a port number of 80, that is, an HTTP Protocol.
In one embodiment, the configuration data includes configuration data entered through a UI interface or configuration data in an Agent profile. The Agent is a software and hardware system with the characteristics of autonomy, sociality, reflexibility and motility, monitors and recovers network data by installing the Agent, and provides a Web interaction entrance for inquiring, monitoring and recovering conditions. Taking the example of determining the configuration data by inputting data through the UI interface, please refer to fig. 4, the configuration data includes specific parameter values of the configuration parameters such as the network card, the data recovery processing time interval, the storage mode, and the like, in addition to the port selection parameter. The user can input parameter values of various configuration parameters through a UI interface or a Web page, wherein the various data are provided with default data. When the network card corresponds to default data, the application program automatically selects an available network card, the default setting of the data recovery processing time interval can be 1 second, and the storage mode comprises any one of non-storage, storage to a file and storage to a database. The user generates a network data processing request by triggering a 'run' button of a UI (user interface), and the network data supervision platform determines a virtual port to be monitored carried by the network processing request by responding to the network data supervision platform and synchronously calls a message capturing thread and a data recovery thread.
The port is an outlet for communication between the device and the outside, and the virtual port refers to a port inside a computer or inside a switch router. Such as 80 ports, 21 ports, 23 ports, etc. in a computer. On the Internet, each host transmits and receives data packets through a network Protocol, such as a TCP (Transmission Control Protocol)/IP (Internet Protocol Address) Protocol, and each data packet performs routing in the Internet according to an IP Address of its destination host, and smoothly transmits the data packet to the destination host. The local operating system will assign protocol ports to those processes that have a need, each protocol port being identified by a positive integer, such as: 80, 139, 445, etc. When the destination host receives the data packet, the data is sent to the corresponding port according to the destination port number of the header of the packet, and the process corresponding to the port will take the data and wait for the next group of data to arrive. The virtual port to be monitored refers to a target port which needs to perform data message capture processing. When the configuration data is set with a filter expression, the virtual ports to be monitored correspond to the virtual ports in the filter expression, and when the configuration data is not set with the filter expression, the virtual ports to be monitored are all the virtual ports.
The message capture refers to the process of acquiring the data message transmitted and received by the virtual port. In one embodiment, network packet capture is performed using a TcpDump open source library, which completely intercepts packets transmitted in the network for analysis. TcpDump is one of the powerful network data collection and analysis tools in Linux, supports filtering for network layers, protocols, hosts, networks or ports, and provides logic statements such as and, or, not and the like to help you to remove useless information, TcpDump supports quite a number of different parameters, such as specifying a network port to which TcpDump monitors using an-i parameter, which is very useful when a computer has a plurality of network ports, specifying the number of packets to be monitored using a-c parameter, writing the monitored packets into a file to be saved using a-w parameter, and the like.
The network data recovery is a process of analyzing and data recombining the data message to recover an original data stream carried by the data message. The network data is generally transmitted by directly using a corresponding port of a standard protocol through protocol packaging, the data obtained through the message capturing thread is generally secondary system data, and the network data carried in the data message is recovered into visual and visible pictures and texts, videos, webpages and the like through data recovery processing. After capturing the traffic data, the network data recovery thread needs to be executed after capturing the message data, and if the serial sequence execution is used, the network data recovery thread cannot be executed during capturing the message data, so that the execution efficiency is affected. The message capturing thread and the data recovery thread are synchronously called, so that the message capturing thread and the data recovery thread are executed simultaneously, and the processing efficiency is improved.
S204, capturing the data message transmitted and received through the virtual port in real time through the message capturing thread, and writing the data message into a buffer zone.
The buffer area is used for storing the captured message data and providing the captured message data for other modules needing to read flow, such as a data recovery thread, to read the message data. In design, the problem of execution efficiency is considered, and because synchronous read-write of network data is an efficiency bottleneck, a parallel execution mode is introduced. After the buffer area is introduced, the captured message data is serialized and stored, when other threads read, the provided reading interface is called to take out the message data, and the bottom layer uses a synchronization mode of copying the message data, namely, a plurality of copies of the data are written in at one time, and only one copy is taken in one reading. By introducing the buffer area, the execution of the message capturing thread and the data recovery thread is independent of the execution condition of the other party, the message capturing thread and the data recovery thread can be synchronously executed, the recovery processing efficiency of the network data is improved, and the recovered network data is quickly obtained.
In the transmission process of network data, a data stream of network data with a large data volume is usually divided into a plurality of subdata and transmitted in the form of a plurality of data messages, and when the network data is restored, the data messages corresponding to all the subdata of the data stream must be read to restore the data stream to obtain a complete data stream. In order to realize convenient and effective data message reading, in the process of writing the data messages into the buffer area, the data messages are serialized, so that the serial numbers of the data messages belonging to the same data flow are continuous, and the data messages belonging to the same data flow are written into the buffer area according to the serial numbers to store message data.
S206, reading the data message of the buffer area through the data recovery thread, and performing data recovery processing on the data message to obtain recovered network data.
The data recovery processing comprises reading, analyzing and recombining the data message. Reading the data messages comprises reading the data messages belonging to the same data flow according to the serial number of the message data. The analysis of the data message comprises the steps of sequentially analyzing the data message according to message protocols of all layers, and thus obtaining network data loaded by an application layer. And the data message recombination comprises the step of recombining the data message according to the application layer load data obtained by analysis and the serial number of the data message to obtain the complete data information corresponding to the data stream.
According to the network data recovery method, on one hand, the data messages received and sent by the terminal to be monitored are captured through the monitoring virtual port, real-time data messages can be monitored, network data recovery processing is carried out on the data messages, manual analysis processing is not needed, a network administrator can read transmitted network data visually, on the other hand, the message capturing thread and the data recovery thread are executed synchronously through adding the buffer area, the recovery processing efficiency of the network data is improved, and recovered network data are obtained quickly.
In one embodiment, at least one of the packet capture thread and the data recovery thread is a distributed thread.
Distributed means that a processing task is broken up into many small parts, which are allocated to multiple threads for processing. Thereby saving the whole processing time and greatly improving the processing efficiency. In one embodiment, the packet capturing thread is a distributed architecture in which a plurality of threads execute simultaneously, and each packet capturing thread is configured to capture packet data received and sent by each virtual port, and store a data packet capturing result of each thread in a buffer. In another embodiment, the data recovery threads are a distributed architecture in which multiple threads execute simultaneously, and each data recovery thread reads a data packet from the buffer and performs data recovery processing respectively. Referring to fig. 5, in one embodiment, a plurality of packet capturing threads and a plurality of data recovery threads are included, the plurality of packet capturing threads write captured data into a buffer, and the data recovery threads read data from the buffer, where the number of threads of the packet capturing threads and the number of data recovery threads may be matched according to a rate, for example, when the packet capturing speed is fast, more data recovery threads are correspondingly used for performing data recovery processing synchronously.
In one embodiment, referring to FIG. 6, writing the data packet to the buffer includes steps S602-S604.
S602, identifying the session ending identifier carried by the data message, determining the data messages belonging to the same session, and performing serialization processing on the data messages of the same session to determine the serial number of each data message.
S604, writing the data message into a buffer zone according to the sequence number.
When data flow is transmitted in a data message form, each data message is associated with a session sequence number and a recombination state, the recombination state comprises an independent message or a recombination message, the independent message refers to that network data needing to be transmitted is transmitted through a single data message, and the complete network data can be obtained after the data message is analyzed. The reassembly packets refer to network data to be transmitted, which is split and transmitted through a plurality of data packets, and after the data packets are analyzed, the load data of each packet needs to be reassembled to obtain complete network data. And performing data classification on the data messages through the session serial numbers of the sessions to determine the data messages belonging to the same session, specifically, the numerical values of the session serial numbers of the data messages of the same session are adjacent, and the state control bit message carried in the last data message of the same session is Finish. Based on the identification of the last data message of the same session, the sequence of each data message in the session is determined, the data messages of the same session are serialized, the serial number of each data message is determined, and the data messages are written into a buffer zone according to the serial numbers.
In one embodiment, referring to fig. 7, performing data recovery processing on the data packet to obtain a recovered network data stream includes steps S702 to S704.
S702, according to the protocol format of the data message, decapsulating the data message to obtain the load data of the data message.
S704, the load data of the data messages with continuous serial numbers are recombined to obtain a network data stream, and the recovered network data is the network data stream.
Analyzing data messages, which relates to a link layer interface, a network layer interface, a transport layer interface and an application layer interface, and expanding a specific network protocol analysis class for each interface, for example, the link layer interface is embodied as a Media Access Control (MAC) protocol analysis class; the network forming interface is embodied as an IP protocol analysis class; the transport layer interface is embodied as a TCP (transmission control Protocol) and UDP (User Datagram Protocol) Protocol analysis class; the application Layer interface is embodied as HTTP, HTTPs (Hyper Text Transfer Protocol over Secure Socket Layer or Hyper Text Transfer Protocol Secure) Protocol parsing class. In the analysis process, firstly, the network protocol corresponding to the link layer interface is analyzed, and the source MAC address, the destination MAC address and the network protocol type field are determined according to the head of the MAC message. The data with the MAC header removed is the IP message, and the header of the IP message comprises a source IP address and a destination IP address. Then, the header of the IP message is removed to obtain the source port and the destination port of the TCP message or the UDP message, and finally, the application layer protocol is analyzed to obtain the load data.
In one embodiment, when the data message is captured, the relevant information of the data message is displayed on a UI interface or a Web interface, and the data capture result can also be displayed. The website www.qq.com is visited for traffic capture and the results are shown in FIG. 8, starting from left to right, with fields representing the time, data type, data behavior, and data size at which the data was captured, respectively. Taking the 10 th data as an example, at the time point of 2017/5/22Mon 18:14:12, response data of the data type HTTP protocol is captured, the data is sent from the port of the host 80 with IP address 58.205.217.1 and MAC address 00:0C:29:3E:23: F4 to the port of the host 43466 (local computer) with IP address 192.168.142.129 and MAC address 00:50:56: E0:82:6B, and the data size is 60 bytes. If a default filter expression is used, i.e., the filter expression field is empty, all data traffic will be captured. If a default filter expression is used, i.e., the filter expression field is empty, all data traffic will be captured. Referring to fig. 9, it can be seen that there is non-HTTP protocol type data, the 751 st piece of network data is UDP protocol type, and the data size is 205 bytes from host 53 port with IP address 192.168.142.2 and MAC address 00:0C:29:3E:23: F4 to host 62074 port (local computer) with IP address 192.168.142.129 and MAC address 00:50:56: E0:82: 6B. Since 192.168.142.2IP address is gateway IP, if UDP is used to communicate on 53 port, it can be determined that this data is a response to domain name resolution.
In one embodiment, after obtaining the recovered network data, the method further includes: and reading the storage address and the storage protocol in the configuration data, and storing the recovered network data to the storage address according to the storage protocol.
The storage address refers to a position where the recovered network data are to be stored, the storage address comprises a specified database or a specified file, and the recovered network data comprise pictures and texts, videos, webpages and the like. Taking the storage address as the file storage as an example, as shown in fig. 10, the file root path is set as the ZRestore folder in the application program execution directory, the restored network data can be viewed through the terminal, and the network data is classified into the port, IP, and receiving/sending behaviors, and the result is displayed as shown in fig. 11 and fig. 12. In addition, html, pictures, videos, json and the like data can be viewed through local previews, and files can be decompressed locally.
Fig. 13 is a flowchart illustrating a network data recovery method according to an embodiment. It should be understood that, although the steps in the flowchart of fig. 13 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 13 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, a flow chart of a network data recovery method is provided, which includes steps S1302 to S1322.
S1302, a network data processing request generated by triggering UI interface input or Web interface is acquired.
The network data processing request refers to a processing request generated based on the configuration data when the trigger condition is satisfied. The trigger condition includes that the user triggers a set 'run' function button by clicking, and can also be based on a timer, the current time meets the requirement of the timer, and the like. The "run" function button includes a function button set on the Qt-based UI interface or a function button configured in the Web page.
S1304, reads the port selection parameter in the configuration data carried by the network data processing request.
The port selection parameter is obtained by reading parameter information in a filter expression, which is set according to a domain name to be accessed.
And S1306, determining that the virtual port corresponding to the port selection parameter in the terminal to be detected is the virtual port to be monitored.
The virtual port to be monitored refers to a target port which needs to perform data message capture processing. When the configuration data is set with a filter expression, the virtual ports to be monitored correspond to the virtual ports in the filter expression, and when the configuration data is not set with the filter expression, the virtual ports to be monitored are all the virtual ports.
S1308, the message capturing thread and the data recovery thread are synchronously called. At least one of the packet capture thread and the data recovery thread is a distributed thread.
The message capturing thread and the data recovery thread are synchronously called, so that the message capturing thread and the data recovery thread are executed simultaneously, and the processing efficiency is improved. The distributed thread is a processing mode of parallel execution of a plurality of threads, can meet the requirement that a plurality of data processing tasks are carried out simultaneously, and further improves the processing efficiency.
S1310, capturing the data message transmitted and received through the virtual port in real time through the message capturing thread.
Message capture is the process of acquiring the data message transmitted and received by the virtual port.
S1312, identifying the session ending identifier carried by the data message, determining the data messages belonging to the same session, performing serialization processing on the data messages of the same session, and determining the serial number of each data message.
In the transmission process of network data, a data stream of network data with a large data volume is usually divided into a plurality of subdata and transmitted in the form of a plurality of data messages, and when the network data is restored, the data messages corresponding to all the subdata of the data stream must be read to restore the data stream to obtain a complete data stream. In order to realize convenient and effective data message reading, the data messages are serialized in the process of writing the data messages into the buffer area, so that the serial numbers of the data messages belonging to the same data stream are continuous, and the message data can be conveniently stored.
And S1314, writing the data message into a buffer zone according to the sequence number.
The buffer area is used for storing the captured message data and providing the captured message data for other modules needing to read flow, such as a data recovery thread, to read the message data. In design, the problem of execution efficiency is considered, and because synchronous read-write of network data is an efficiency bottleneck, a parallel execution mode is introduced. After the buffer area is introduced, the captured message data is serialized and stored, when other threads read, the provided reading interface is called to take out the message data, and the bottom layer uses a synchronization mode of copying the message data, namely, a plurality of copies of the data are written in at one time, and only one copy is taken in one reading. By introducing the buffer area, the execution of the message capturing thread and the data recovery thread is independent of the execution condition of the other party, the message capturing thread and the data recovery thread can be synchronously executed, the recovery processing efficiency of the network data is improved, and the recovered network data is quickly obtained.
S1316, reading the data message of the buffer area through the data recovery thread.
The data recovery processing comprises reading, analyzing and recombining the data message. Reading the data messages comprises reading the data messages belonging to the same data flow according to the serial number of the message data. The analysis of the data message comprises the steps of sequentially analyzing the data message according to message protocols of all layers, and thus obtaining network data loaded by an application layer. And the data message recombination comprises the step of recombining the data message according to the application layer load data obtained by analysis and the serial number of the data message to obtain the complete data information corresponding to the data stream.
S1318, decapsulating the data packet according to the protocol format of the data packet, to obtain load data of the data packet.
The analysis of the data message relates to a link layer interface, a network layer interface, a transport layer interface and an application layer interface, and specific network protocol analysis classes are expanded for each interface, for example, the link layer interface is embodied as an MAC protocol analysis class; the network forming interface is embodied as an IP protocol analysis class; the transport layer interface is embodied as TCP and UDP protocol analysis; the application layer interface is embodied as HTTP and HTTPS protocol analysis classes. In the analysis process, firstly, the network protocol corresponding to the link layer interface is analyzed, and the source MAC address, the destination MAC address and the network protocol type field are determined according to the head of the MAC message. The data with the MAC header removed is the IP message, and the header of the IP message comprises a source IP address and a destination IP address. Then, the header of the IP message is removed to obtain the source port and the destination port of the TCP message or the UDP message, and finally, the application layer protocol is analyzed to obtain the load data.
S1320, the load data of the data packets with consecutive serial numbers is reassembled to obtain a network data stream, and the recovered network data is the network data stream.
And carrying out data recombination on the load data of the data messages in the same session according to the sequence of the sequence numbers so as to obtain complete network data and realize data recovery.
S1322, reading the storage address and the storage protocol in the configuration data, and storing the recovered network data to the storage address according to the storage protocol, wherein the storage address comprises a designated database or a designated file.
The storage address refers to a position where the recovered network data is to be stored, the storage address comprises a designated database or a designated file, and the recovered network data comprises pictures and texts, videos, webpages and the like. When the storage address is selected as file storage, the file root path is set as a ZRestore folder in an application program execution directory, recovered network data including port, IP and receiving/sending behavior classification can be viewed through the terminal, in addition, data such as html, pictures, videos, json and the like can be viewed through local preview, and files can be decompressed locally.
In one embodiment, as shown in fig. 14, there is provided a network data recovery apparatus 1400, the apparatus comprising:
the synchronization module 1402 is configured to determine a virtual port to be monitored carried by a network processing request when the network data processing request is acquired, and synchronously invoke a packet capture thread and a data recovery thread.
The traffic capturing module 1404 is configured to capture, in real time, a data packet sent and received through the virtual port by using a packet capturing thread, and write the data packet into a buffer.
The traffic recovery module 1406 is configured to read the data packet in the buffer through the data recovery thread, and perform data recovery processing on the data packet to obtain recovered network data.
In an embodiment, the synchronization module 1402 is further configured to, when the network data processing request is obtained, read a port selection parameter in configuration data carried in the network data processing request, and determine that a virtual port corresponding to the port selection parameter in the terminal to be detected is a virtual port to be monitored.
In an embodiment, the traffic capturing module 1404 is further configured to identify a session end identifier carried in the data packet, determine data packets belonging to the same session, perform serialization processing on the data packets of the same session, determine a serial number of each data packet, and write the data packets into the buffer according to the serial numbers.
In an embodiment, the traffic recovery module 1406 is further configured to decapsulate the data packet according to a protocol format of the data packet, to obtain load data of the data packet, and reassemble the load data of the data packet with consecutive sequence numbers to obtain a network data stream, where the recovered network data is the network data stream.
In one embodiment, the network data recovery apparatus 1400 further includes a storage module, configured to read a storage address and a storage protocol in the configuration data, and store the recovered network data to the storage address according to the storage protocol.
In one embodiment, the network data recovery apparatus is shown in fig. 15 and specifically includes a user interface module, a parallelism module, a traffic capture module, a traffic recovery module, a buffer module, a storage module, and a protocol rules module. All modules cooperate with each other to recover network data and realize supervision on the network data, wherein the functions of all modules are summarized as follows:
a user interface module: the main function is to realize User interaction, which is a module which is only contacted between a User and the platform and only has a GUI (Graphical User Interface) program, and a Qt is used for developing a Graphical Interface.
A parallel module: the main function is to call the message capturing thread of the flow capturing module and the data recovery module of the flow recovery module to execute them in parallel, so as to improve the resource utilization and task execution efficiency of the software system. The parallel module improves the time utilization rate, and after the data message corresponding to the network flow is captured, the capturing does not need to be continued until the execution of the recovery task and the monitoring task is finished, so that more and more complete network flows can be captured conveniently.
A flow capture module: the main function is to read the parameters of the capturing options according to the configuration parameters provided by the user interface module, capture the data messages of the network flow, perform serialized representation and store the data messages in the buffer area for other modules to use. The flow capturing module is called by the parallel module, and simultaneously, the captured data message is stored in the buffer module.
A flow recovery module: the main function is to pay attention to the upper layer protocol of network flow according to the known network protocol message format, support connection-oriented message recombination and html, jpg, MP4 and other types of data recovery, and identify the compression type and classify the encrypted data and the plaintext data.
A storage module: the main function is to provide data storage, including file storage and database storage. A plurality of modules in the software system need to store data, so that one storage module is abstracted to better support software modularization, the development cost is reduced by reusing the storage module, and the additional overhead brought by function synchronization is reduced.
A protocol rule module: the method mainly provides the functions of analyzing and analyzing the network protocol, decapsulates the message and extracts the specific information of the protocol through the known message format of the network protocol. The module abstracts the protocol to a link layer interface, a network layer interface, a transport layer interface and an application layer interface, and expands a concrete network protocol analysis class for each interface, for example, the link layer interface is embodied to an MAC protocol analysis class; the network forming interface is embodied as an IP protocol analysis class; the transport layer interface is embodied as TCP and UDP protocol analysis; the application layer interface is embodied as HTTP and HTTPS protocol analysis classes. Protocol parsing and functional extension are provided more flexibly in a polymorphic manner through common implementation of inheritance to an abstraction layer.
A buffer module: the main function is to store the data captured by the traffic and provide it to other modules that need to read the traffic. Considering that synchronous read-write of network data is an efficiency bottleneck, after a buffer module is introduced, captured data is serialized and stored, a provided read interface is called to fetch the data when other modules read, and a bottom layer uses a synchronous mode of data copying, namely, a plurality of data are written at one time, and only one data is fetched after one-time reading.
FIG. 16 is a diagram illustrating an internal structure of a computer device in one embodiment. The computer device may be specifically the network data administration platform 104 in fig. 1. As shown in fig. 16, the computer apparatus includes a processor, a memory, a network interface, an input device, and a display screen connected through a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program that, when executed by the processor, causes the processor to implement the network data recovery method. The internal memory may also have stored therein a computer program that, when executed by the processor, causes the processor to perform a network data recovery method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 16 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, the network data recovery apparatus provided in the present application may be implemented in the form of a computer program, and the computer program may be run on a computer device as shown in fig. 16. The memory of the computer device may store therein various program modules constituting the network data recovery apparatus, such as the synchronization module, the traffic capture module, and the traffic recovery module shown in fig. 14. The computer program constituted by the respective program modules causes the processor to execute the steps in the network data recovery method of the respective embodiments of the present application described in the present specification.
For example, the computer device shown in fig. 16 may execute, by using a synchronization module in the network data recovery apparatus shown in fig. 14, determining a virtual port to be monitored carried by a network processing request when the network data processing request is acquired, and synchronously invoking a packet capture thread and a data recovery thread. The computer equipment can capture the data message transmitted and received through the virtual port in real time through the message capture thread by the flow capture module, and write the data message into the buffer zone. The computer equipment can execute the data message read from the buffer area through the data recovery thread through the flow recovery module, and perform data recovery processing on the data message to obtain recovered network data.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the network data recovery method described above. Here, the steps of the network data recovery method may be steps in the network data recovery methods of the above embodiments.
In one embodiment, a computer readable storage medium is provided, storing a computer program that, when executed by a processor, causes the processor to perform the steps of the above-described network data recovery method. Here, the steps of the network data recovery method may be steps in the network data recovery methods of the above embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, and the program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (11)
1. A method of network data recovery, comprising:
when a network data processing request is acquired, determining a virtual port to be monitored carried by the network processing request, and synchronously calling a message capturing thread and a data recovery thread;
capturing data messages sent and received through the virtual port in real time through the message capturing thread, and writing the data messages into a buffer area;
and reading the data message of the buffer area through the data recovery thread, and performing data recovery processing on the data message to obtain recovered network data.
2. The method of claim 1, wherein at least one of the packet capture thread and the data recovery thread is a distributed thread.
3. The method according to claim 1, wherein the determining, when the network data processing request is obtained, the virtual port to be monitored carried by the network processing request comprises:
when a network data processing request is acquired, reading a port selection parameter in configuration data carried by the network data processing request;
and determining a virtual port corresponding to the port selection parameter in the terminal to be detected as the virtual port to be monitored.
4. The method of claim 3, wherein the configuration data comprises configuration data entered via a UI interface or configuration data in a configuration file.
5. The method of claim 1, wherein writing the data packet to the buffer comprises:
identifying a session ending identifier carried by the data message, determining the data messages belonging to the same session, performing serialization processing on the data messages of the same session, and determining the serial number of each data message;
and writing the data message into a buffer zone according to the serial number.
6. The method of claim 5, wherein performing data recovery processing on the data packet to obtain a recovered network data stream comprises:
according to the protocol format of the data message, decapsulating the data message to obtain load data of the data message;
and recombining the load data of the data messages with continuous serial numbers to obtain a network data stream, wherein the recovered network data is the network data stream.
7. The method of claim 1, wherein obtaining the recovered network data further comprises:
and reading a storage address and a storage protocol in the configuration data, and storing the recovered network data to the storage address according to the storage protocol.
8. The method of claim 7, wherein the storage address comprises a specified database or a specified file.
9. An apparatus for network data recovery, the apparatus comprising:
the system comprises a synchronization module, a message acquisition module and a data recovery module, wherein the synchronization module is used for determining a virtual port to be monitored carried by a network processing request when the network data processing request is acquired, and synchronously calling a message acquisition thread and a data recovery thread;
the flow capturing module is used for capturing the data message which is transmitted and received through the virtual port in real time through the message capturing thread and writing the data message into a buffer zone;
and the flow recovery module is used for reading the data message of the buffer area through the data recovery thread and performing data recovery processing on the data message to obtain recovered network data.
10. A computer-readable storage medium, storing a computer program which, when executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 8.
11. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910672616.6A CN112311717B (en) | 2019-07-24 | 2019-07-24 | Network data recovery method and device, storage medium and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910672616.6A CN112311717B (en) | 2019-07-24 | 2019-07-24 | Network data recovery method and device, storage medium and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112311717A true CN112311717A (en) | 2021-02-02 |
| CN112311717B CN112311717B (en) | 2022-08-23 |
Family
ID=74329194
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910672616.6A Active CN112311717B (en) | 2019-07-24 | 2019-07-24 | Network data recovery method and device, storage medium and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112311717B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101621587A (en) * | 2008-06-30 | 2010-01-06 | 成都市华为赛门铁克科技有限公司 | Method, device and system for network monitoring |
| CN101883081A (en) * | 2009-05-05 | 2010-11-10 | 昆明智讯达科技开发有限公司 | Method for carrying out video stream transmission filtering based on content of network data packet |
| CN102333007A (en) * | 2011-09-28 | 2012-01-25 | 重庆大学 | On-line Web service quality monitoring system and method |
| CN103067218A (en) * | 2012-12-14 | 2013-04-24 | 华中科技大学 | High speed network data package content analysis device |
| US9170848B1 (en) * | 2010-07-27 | 2015-10-27 | Google Inc. | Parallel processing of data |
| CN105337797A (en) * | 2015-10-15 | 2016-02-17 | 哈尔滨工业大学 | Data capturing method of network protocol of complex electronic information system |
| CN107666486A (en) * | 2017-09-27 | 2018-02-06 | 清华大学 | A kind of network data flow restoration methods and system based on message protocol feature |
-
2019
- 2019-07-24 CN CN201910672616.6A patent/CN112311717B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101621587A (en) * | 2008-06-30 | 2010-01-06 | 成都市华为赛门铁克科技有限公司 | Method, device and system for network monitoring |
| CN101883081A (en) * | 2009-05-05 | 2010-11-10 | 昆明智讯达科技开发有限公司 | Method for carrying out video stream transmission filtering based on content of network data packet |
| US9170848B1 (en) * | 2010-07-27 | 2015-10-27 | Google Inc. | Parallel processing of data |
| CN102333007A (en) * | 2011-09-28 | 2012-01-25 | 重庆大学 | On-line Web service quality monitoring system and method |
| CN103067218A (en) * | 2012-12-14 | 2013-04-24 | 华中科技大学 | High speed network data package content analysis device |
| CN105337797A (en) * | 2015-10-15 | 2016-02-17 | 哈尔滨工业大学 | Data capturing method of network protocol of complex electronic information system |
| CN107666486A (en) * | 2017-09-27 | 2018-02-06 | 清华大学 | A kind of network data flow restoration methods and system based on message protocol feature |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112311717B (en) | 2022-08-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11855967B2 (en) | Method for identifying application information in network traffic, and apparatus | |
| CN109756501B (en) | High-privacy network proxy method and system based on HTTP (hyper text transport protocol) | |
| CN112019575B (en) | Data packet processing method and device, computer equipment and storage medium | |
| CN108363662A (en) | A kind of applied program testing method, storage medium and terminal device | |
| CN105573733B (en) | Method, web front-end and the system that browser is communicated with web front-end | |
| WO2014206089A1 (en) | Terminal mirroring synchronization method, device, terminal and server | |
| US10775751B2 (en) | Automatic generation of regular expression based on log line data | |
| CN106941418B (en) | SSL VPN configuration information synchronization method and device | |
| CN113472798B (en) | Method, device, equipment and medium for backtracking and analyzing network data packet | |
| WO2019043804A1 (en) | Log analysis device, log analysis method, and computer-readable recording medium | |
| CN112235300B (en) | Cloud virtual network vulnerability detection method, system, device and electronic equipment | |
| CN112311717B (en) | Network data recovery method and device, storage medium and computer equipment | |
| CN110581792B (en) | Message transmission method and device | |
| Velea et al. | Feature extraction and visualization for network pcapng traces | |
| CN112822204A (en) | NAT detection method, device, equipment and medium | |
| US9794274B2 (en) | Information processing apparatus, information processing method, and computer readable medium | |
| CN109792617B (en) | Application resiliency system for applications deployed on a platform and method thereof | |
| CN114553546A (en) | Message capturing method and device based on network application | |
| CN114301802A (en) | Confidential evaluation detection method and device and electronic equipment | |
| CN113836464A (en) | Page data processing method and device, computer equipment and storage medium | |
| CN112929357A (en) | Virtual machine data analysis method, device, equipment and storage medium | |
| EP4319094A1 (en) | Control method and apparatus, and computing device | |
| CN107306264B (en) | Network security monitoring method and apparatus | |
| Boonyopakorn | Applying Data Analytics to Findings of User Behaviour Usage in Network Systems | |
| Vandenberghe | Network traffic exploration application: A tool to assess, visualize, and analyze network security events |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |