CN115473948A - Data packet analysis method and device, computer equipment and storage medium - Google Patents

Abstract

The invention provides a data packet analysis method, a data packet analysis device, computer equipment and a storage medium, wherein the method comprises the following steps: under the condition that the signal connection with a first port of the core outlet network equipment is established, acquiring a first data packet through the first port; based on a preset target function, filing the first data packet to obtain a second data packet in a target format; unpacking the second data packet based on a preset unpacking program to obtain structured data; carrying out consumption processing on the structured data to obtain first structured data, and storing the first structured data to a target library; performing aggregation analysis processing on the first structural data in the target library to obtain a data analysis result; and executing the target operation based on the IP related information and/or the operation related information. By the method and the device, the problem of high operation and maintenance difficulty caused by low data packet monitoring precision is solved, and the effect of reducing the operation and maintenance difficulty is achieved.

Description

Data packet analysis method and device, computer equipment and storage medium

Technical Field

The present application relates to the field of data transmission technologies, and in particular, to a method and an apparatus for analyzing a data packet, a computer device, and a storage medium.

Background

With the popularization of cloud computing technology, more and more enterprises begin to use cloud hosts, a long time elapses from the beginning of questioning the security and stability of the cloud hosts to the later approval, with the popularization of the use of the cloud hosts, the development of the IT internet and the electronic commerce, the number of users of each cloud manufacturer is larger and larger, and the complexity of services, DDOS attacks and abnormal network traffic all bring great challenges to data centers.

For example, the conditions of network delay, network traffic, abnormal packet loss, TCP reset network connection, TCP retransmission, and the like between two IP terminals cannot be effectively monitored, so that the existing IT infrastructure cannot be effectively linked according to the abnormal network traffic, and the operation and maintenance difficulty is greatly increased.

In view of the above problems, no better solution is available at present.

Disclosure of Invention

The embodiment of the invention provides a data packet analysis method, a data packet analysis device, computer equipment and a storage medium, which at least solve the problem of high operation and maintenance difficulty caused by the incapability of effectively monitoring an IP terminal in the related technology.

According to an embodiment of the present invention, there is provided a packet analysis method including:

under the condition that signal connection with a first port of core outlet network equipment is established, acquiring a first data packet through the first port, wherein the first data packet is obtained by processing an initial data packet which enters and exits an external data center through the core outlet network equipment;

based on a preset objective function, filing the first data packet to obtain a second data packet in an objective format;

unpacking the second data packet based on a preset unpacking program to obtain structured data;

performing consumption processing on the structured data to obtain first structured data, and storing the first structured data to a target library;

performing aggregation analysis processing on the first structure data in the target library to obtain a data analysis result, wherein the data analysis result comprises IP (Internet protocol) associated information and/or operation associated information;

and executing target operation based on the IP related information and/or the operation related information, wherein the target operation at least comprises event visualization processing and/or abnormal event linkage processing.

In an optional embodiment, the method further comprises:

after the unpacking the second data packet based on the preset unpacking program to obtain the structured data, the method further includes: transmitting the structured data to a target message queue in multiple threads based on timestamps contained in the structured data;

the consuming the structured data to obtain the first structured data comprises: sequentially acquiring the structural data from the target message queue based on the timestamp; and processing the structured data to obtain the first structural data.

In an optional embodiment, the processing the structured data to obtain the first structured data includes:

acquiring IP attribution information contained in the structured data;

and performing identification processing on the structured data based on the IP attribution information to obtain the first structured data.

In an optional embodiment, before the obtaining the first data packet through the first port, the method further comprises:

under the condition that the second port receives an initial data packet, copying the initial data packet through a target program to obtain a first data packet, and transmitting the first data packet to the first port, wherein the second port is in communication connection with the external data center, and the initial data packet comprises data packets entering and exiting the external data center.

In an optional embodiment, the obtaining, through the first port, the first data packet in the case that it is determined that the signal connection is established with the first port of the core egress network device includes:

under the condition that signal connection with the first port is determined, network card scanning operation is carried out on a target network card through a first program so as to determine network card information of the target network card, wherein the target network card is connected with the first port;

the first program obtains the first data packet from the first port according to the network card information and sends the first data packet to the first port

And the second program sends a starting instruction to instruct the second program to carry out filing processing on the first data packet based on a preset target function so as to obtain a second data packet in a target format.

In an optional embodiment, based on the IP association information and/or the operation association information, performing the target operation includes at least one of:

based on the IP associated information and/or the operation associated information, executing information visualization operation;

determining the network quality between the areas where the target objects are located based on the IP associated information and/or the operation associated information;

sending a configuration instruction to a target server based on the IP associated information and/or the operation associated information to indicate the target server to generate a first list;

and under the condition that the IP associated information and/or the operation associated information are determined to be in an abnormal state, sending a work order operation order to a target object to indicate the target object to generate a target work order and send the target work order.

According to another embodiment of the present invention, there is provided a packet device including:

the data capturing module is used for acquiring a first data packet through a first port of core outlet network equipment under the condition that the signal connection with the first port of the core outlet network equipment is established, wherein the first data packet is obtained by processing an initial data packet which enters and exits an external data center through the core outlet network equipment;

the data archiving module is used for archiving the first data packet based on a preset target function to obtain a second data packet in a target format;

the data unpacking module is used for unpacking the second data packet based on a preset unpacking program to obtain structured data;

the data consumption module is used for performing consumption processing on the structured data to obtain first structure data and storing the first structure data to a target library;

the aggregation analysis module is used for performing aggregation analysis processing on the first structure data in the target library to obtain a data analysis result, wherein the data analysis result comprises IP (Internet protocol) associated information and/or operation associated information;

and the target operation module is used for executing target operation based on the IP related information and/or the operation related information, wherein the target operation at least comprises event visualization processing and/or abnormal event linkage processing.

According to yet another embodiment of the invention, there is also provided a storage medium, the computer program when executed, implementing the steps of the method of any of the preceding claims.

There is also provided, in accordance with yet another embodiment of the present invention, computer apparatus including a memory and a processor, the processor coupled to the memory, wherein at least one program instruction or code is stored in the memory, and the at least one program instruction or code is loaded and executed by the processor to cause the computer apparatus to implement the aforementioned packet analysis method.

According to the invention, because the second data packet is from the external data center and the second data packet is subjected to unpacking and consumption aggregation analysis, the actual condition of the IP terminal accessing the external data center can be determined according to the data analysis result, so that whether the object accessing the external data is abnormal can be accurately determined, and then the target operation is executed according to the abnormal condition, therefore, the problem of high operation and maintenance difficulty caused by the fact that the IP terminal cannot be effectively monitored can be solved, and the effect of improving the operation and maintenance efficiency can be achieved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention and do not constitute a limitation of the invention. In the drawings:

FIG. 1 is a prior art system architecture diagram provided by an embodiment of the present application;

FIG. 2 is a diagram of a system architecture of the present application provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of an application environment provided by an embodiment of the present application;

FIG. 4 is a flow diagram illustrating a method for packet analysis, according to one embodiment;

FIG. 5 is a flow diagram illustrating data packet capture in one embodiment;

FIG. 6 is a block diagram showing a schematic configuration of a packet analyzing apparatus according to an embodiment;

FIG. 7 is a block diagram showing a schematic configuration of a computer device according to an embodiment.

Detailed Description

The embodiment of the application provides a data packet analysis method, which can effectively monitor a data packet and solve the problem of high operation and maintenance difficulty caused by the fact that an IP terminal cannot be effectively monitored.

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

For convenience of understanding the data packet analysis system architecture provided in the embodiments of the present application, the data packet analysis system architecture in the prior art is first described in the present application, and the following description is specifically provided.

Please refer to fig. 1, which is a diagram illustrating a prior art packet analysis architecture according to an embodiment of the present application.

In the prior art, an external data center transmits data to a core egress network device, and then the data is transmitted to a user side or an operator through the core egress network device, and the core egress network device is usually configured as an intermediate device such as a gateway and a router.

Referring to fig. 2 and fig. 3, an architecture diagram of a packet analysis system according to an embodiment of the present invention includes an operator (corresponding to the client 202 in fig. 2), a core egress network device (corresponding to the network in fig. 2), an external data center (corresponding to the server 204 in fig. 2), and an external server (corresponding to the network in fig. 2) connected to a target port of the location 2 of the core egress network device, where the external server forms another branch of the core egress network device.

It should be noted that, in the practical application process, the packet analysis system may further include more devices. For example, the system may further include a user device, and when the packet analysis system needs to be used, the user device may send or receive related data information forwarded from the core egress network device to the core egress network device by connecting to the core egress network device, or replace an operator port with a user device port, thereby implementing data interaction between the user device port and the external data center.

Based on the above-mentioned data packet analysis system architecture shown in fig. 2, the method in the embodiment of the present application is described in detail below.

The packet analysis method provided by the present application may be executed entirely on the server side, entirely on the client side, or may be executed by both the server 204 and the client 202. The packet analysis method can be applied to the application environment shown in fig. 2 when the server and the client perform together. Wherein the client 202 and the server 204 communicate over a network.

The server 204 may obtain the operation or operation instruction forwarded by the terminal 202 through the core egress network device, and the server 204 may execute the operation or operation instruction from the user or operator of the terminal 202.

The client 202 executes receiving user instructions, and performs recognition and classification on the user instructions.

The client 202 may be, but not limited to, various user devices, such as a computer, a notebook computer, a smart phone, a tablet computer, and a portable smart device, and the server 204 may be implemented by an independent server or a server cluster formed by a plurality of servers.

Example 1

Referring to fig. 4, a schematic flow chart of a data packet analysis method according to an embodiment of the present application is shown, where the data packet analysis method includes:

step S401, under the condition that the signal connection with a first port of a core exit network device is established, acquiring a first data packet through the first port, wherein the first data packet is obtained by processing an initial data packet transmitted by an external data center through the core exit network device;

in this embodiment, the branch is established at the core egress network device, and the analysis of the data packet is completed at the branch in order to avoid occupying the computing resource of the core egress network device, so that the core egress network device has enough computing resource to perform data communication with other devices (such as an external data center, an operator, or a user side, etc.), and compared with the prior art, the timeliness of data communication between the core egress network device and other devices can be ensured, and further extension of the function can be realized at the same time, that is, the function of the whole system architecture is further extended by adding other functional devices at the branch, so that the application range can be extended; meanwhile, the first data packet is filed through a preset objective function, so that the identifiers of the data packets with different formats are unified, and the data packets can be conveniently identified and processed by the subsequent process.

For example, as shown in fig. 3, since all network packets entering and exiting the data center need to be forwarded through the network device at the core egress, a server may be connected to the first port at location 2 (where the server runs the aforementioned packet capture program capable of obtaining the first packet according to the packet capture instruction).

It can be thought that the number of the first ports may be at least 2, and thus the number of the corresponding external physical servers may also be at least two, so that a plurality of ports may process different types of data packets captured according to different requirements at different time intervals, different sizes, and the like, and similarly, the number of the servers corresponding to the same port may also be at least two, so as to improve the archiving efficiency of the first data packet.

Step S402, based on a preset objective function, filing the first data packet to obtain a second data packet in an objective format;

in this embodiment, the preset target function may be (but is not limited to) a function of libpcap, or another function capable of implementing a format unification archiving function, and similarly, the target format may be (but is not limited to) a file format with suffix of pcap; and the instruction feedback is carried out after the first port receives the first data packet, so that the first port can accurately receive the first data packet, the feedback of wrong instructions is avoided, and the continuity and the accuracy of subsequent operation are ensured.

For example, as shown in fig. 3, after the external server is connected to the network device at the core egress, the first packet is archived as a second packet suffixed by the pcap file through a function of libpcap, and the archived second packet is transmitted to the message queue to wait for further subsequent processing.

Step S403, unpacking the second data packet based on a preset unpacking program to obtain structured data;

in this embodiment, the structured data is obtained to obtain information such as an IP included in a target Protocol layer, and determine, through subsequent processing, whether the IP is an abnormal IP or not, where the target Protocol layer includes (but is not limited to) a data link layer, a network layer, a transport layer, and an application layer, and the structured data includes (but is not limited to) an ARP (Address Resolution Protocol) of a data link layer Protocol, RARP (Reverse Address Resolution Protocol), information of a Reverse Address translation Protocol, and the like, an IP Protocol (Internet Protocol) of a network layer Protocol, an Internet interconnection Protocol (Internet interconnection Protocol), an ICMP Protocol (Internet Control Message Protocol, internet network Control Message Protocol), and the like, a TCP (Transmission Control Protocol) of a transport layer Protocol, a UDP (User data Protocol, user Datagram Protocol), a Text Protocol (HTTP), a Text Transfer Protocol (Text Transfer Protocol), and a security System (Secure Domain Name System), and a security System (DNS) that may be modified according to a request of a real Secure environment.

It is conceivable that the unpacking program in this application may (but is not limited to) analyze a packet protocol of each layer based on four layers of TCP/IP (i.e., a data link layer, a network layer, a transport layer, and an application layer), and when an actual requirement and an operating environment change, the unpacking program may also be a packet protocol analyzed for a certain layer of protocol or other protocols, which is not described herein again.

Step S404, performing consumption processing on the structured data to obtain first structure data, and storing the first structure data to a target library;

in the embodiment, the consumption processing is performed on the structured data to determine the attribution condition of the IP in the structured data, so that different processing strategies are determined according to different IP attributions. Wherein, the consumption processing of the protocol information refers to further processing of the protocol, such as tag adding, data screening and classification, confidence calculation, and the like; the first structure data includes (but is not limited to) the structured data after adding a home label to the structured data, for example, determining whether the structured data is an internal IP or an external IP, whether the structured data is a white list or a black list IP, for example, if the recognition result is the internal IP, determining the home attribute thereof, for example, the structured data belongs to the internal IP of a service system, a data center, etc., the specific identification method thereof may be identified by a field such as a region, etc., and if the recognition result is the external IP, determining the attribute of the country, the city, the province, the city, the operator, the user principal, etc., thereof.

The purpose of storing the target protocol information in the target database is to make the obtained target protocol information persistent, so that the relevant target protocol information can be directly called when other operations are subsequently executed.

Wherein the target database may be (but is not limited to) clickhouse, electronics research, mongodb, etc., in the present application, clickhouse is preferred because clickhouse, as a columnar database, can reduce disk IO compared to other row-type databases, can accommodate batch insertion, and thus can read related data in faster batches.

Step S405, performing aggregation analysis processing on the first structure data in the target library to obtain a data analysis result, wherein the data analysis result comprises IP (Internet protocol) associated information and/or operation associated information;

in this embodiment, the aggregation analysis processing is to further calculate and determine the change between different IPs, so that the change can be timely adjusted according to the change condition, and the purpose of accurately monitoring the IP terminal is achieved.

The aggregation analysis is to analyze and calculate data from different angles according to requirements, for example, calculate an average network processing delay between two IPs, calculate an average network rate between the two IPs, count packet loss between the two IPs, calculate and generate, in unit time, IP associated information such as an IP ranking with the largest occupied network bandwidth, count abnormal external IPs, and the like, and operation associated information related to subsequent operation processing such as an average response time of a DNS resolution domain name, and then may match a corresponding policy according to actual requirements, or perform visual feedback on related data, so as to implement accurate monitoring on a target IP terminal.

It is conceivable that, in addition to the above analysis calculation, other contents, for example, the access frequency per unit time of a specific IP, etc., may be calculated and analyzed in accordance with the actual demand, thereby satisfying the analysis demand in various cases.

Step S406, executing target operation based on the IP related information and/or the operation related information, wherein the target operation at least comprises event visualization processing and/or abnormal event linkage processing.

In this embodiment, the performed visualization processing and/or abnormal event linkage processing may (but is not limited to) generate an IP blacklist, and send the blacklist to a firewall to instruct the firewall to prevent a relevant IP from continuing to access, or send the IP blacklist to a working area to generate a work order for instructing a worker to perform IP facility maintenance or manual review, or send the IP list to a digital panel to be displayed, so as to implement visualization monitoring on the IP list, thereby implementing linkage with other IT facilities, and solving the problems of high operation and maintenance cost and high operation and maintenance difficulty caused by that effective linkage with other IT facilities cannot be performed in the prior art.

It should be noted that the first condition may also be that other conditions capable of reflecting the IP monitoring requirement, such as the access frequency to the external data center in the unit time of the target IP, the access frequency, the network transmission efficiency between two target IPs, and the packet loss threshold between two IPs, may be used as the first condition of the present application, and are not described herein again.

In an alternative embodiment of the method according to the invention,

after the unpacking the second data packet based on the preset unpacking program to obtain the structured data, the method further includes:

step S4031, based on the time stamp contained in the structured data, the structured data is transmitted to a target message queue in a multithread mode;

the consuming the structured data to obtain the first structured data comprises:

step S4041, based on the timestamp, sequentially obtaining the structure data from the target message queue; and processing the structured data to obtain the first structural data.

In this embodiment, since the parsed structured data further includes the timestamp, it is not necessary to design an additional thread to sequence the protocol information during subsequent identification or processing, and the protocol information can be directly identified and determined according to the timestamp, so that computational power is further saved, and data processing efficiency is also improved.

In an optional embodiment, the processing the structured data to obtain the first structured data comprises:

step S4031, IP information contained in the structured data is obtained;

step S4032, based on the IP attribution information, perform identification processing on the structured data to obtain the first structured data.

In this embodiment, the IP information is obtained and the identification processing is performed on the IP information, so as to facilitate classification according to attributes such as the type of the IP information, and thus the corresponding IP attribute can be quickly identified in the subsequent processing.

It should be noted that the analyzed IP information may originally include identification information, but in order to avoid malicious or abnormal IP disguising from invading the external data center, it is necessary to perform secondary identification through consumption aggregation analysis; it is contemplated that alignment steps or identification steps may be added to the process to further enhance the prophylactic function.

In an optional embodiment, before the obtaining the first data packet through the first port, the method further comprises:

step S4011, when a second port receives an initial data packet, copy the initial data packet through a target program to obtain a first data packet, and transmit the first data packet to the first port, where the second port is in communication connection with the external data center, and the initial data packet includes data packets entering and exiting the external data center.

In this embodiment, the copying and transmitting of the initial data packet by the target program is configured to expand the use range and adapt to a connection mode of a common server, and if the servers are connected by other network architectures, the step is not necessarily performed, for example, the network architecture of the kubernets cluster is connected, because there are many virtual network cards on the servers of the kubernets cluster, and all network traffic needs to perform data communication with other servers in the cluster through the physical network card of the server, so that the step is not required.

The target program may be (but is not limited to) a SPAN function program on a core egress network device, where SPAN technology is mainly used to monitor data streams on switches and is roughly divided into two types, a Local Switched Port Analyzer (SPAN) and a Remote SPAN program (Remote SPAN, RSPAN), and by using SPAN technology, we can COPY or mirror COPY a COPY of a data stream COPY of some ports (hereinafter referred to as controlled ports) on a core egress network device such as a switch to a traffic Analyzer connected to a monitoring Port, such as IDS (intrusion detection system) of some routers, and it should be noted that a Port executing SPAN may be on the same switch (Local SPAN) or on different switches (Remote SPAN), for example, a controlled Port and a monitoring Port of a PC equipped with a SNIFFER tool are located on different switches.

In an optional embodiment, the obtaining, through the first port, the first data packet in the case that it is determined that the signal connection is established with the first port of the core egress network device includes:

step S4012, under the condition that the signal connection with the first port is established, performing network card scanning operation on a target network card through a first program to determine network card information of the target network card, wherein the target network card is connected with the first port;

step S4013, the first program obtains the first data packet from the first port according to the network card information, and sends a start instruction to a second program to instruct the second program to perform archive processing on the first data packet based on a preset target function, so as to obtain a second data packet in a target format.

In this embodiment, the scanning of the network card of the first port is to determine the condition of the network card of the first port for data transmission, so that related data can be received and transmitted in order according to the network card information, one-to-one correspondence is achieved, and the network card of the first port can be ensured to transmit data accurately; and after the network card information is determined, the first port is indicated to perform packet capturing operation, so that the error packet capturing caused by direct excitation of the network card information which does not meet the requirement is avoided, for example, the first data packet is incomplete, or the packet capturing efficiency caused by the fact that the network card transmission efficiency does not meet the requirement is low, and the like.

The network card scanning of the first port can be dynamic scanning or other scanning modes; the corresponding network card information includes (but is not limited to) the serial number of the network card, transmission efficiency, IP path, transmission time, transmission protocol, and so on.

For example, as shown in fig. 3 and fig. 5, the packet capturing program (corresponding to the first program) first dynamically scans the network card name of the physical server A1, wherein, in order to improve the packet capturing efficiency, one capturing program A2 (corresponding to the second program) may be started for each network card of A1, and then the capturing program A2 performs packet capturing operation on the first port.

In an optional embodiment, the performing target operation based on the IP association information and/or the operation association information includes at least one of:

based on the IP associated information and/or the operation associated information, performing information visualization operation;

determining the network quality between the areas where the target objects are located based on the IP associated information and/or the operation associated information;

sending a configuration instruction to a target server based on the IP associated information and/or operation associated information to indicate the target server to generate a first list;

and sending a work order operation order to a target object based on the IP associated information and/or the operation associated information to indicate the target object to generate a target work order and send the target work order.

In this embodiment, the determination of the network quality between the regions where the target object is located is to facilitate optimization of the related network under the condition that the network quality does not meet the requirement, and it is conceivable that, after the network quality is determined, the network quality may be transmitted to a visualization platform to facilitate visual observation; similarly, the first list is generated to facilitate the call query, so as to improve the processing efficiency of the data packet, wherein the generated first list may be (but is not limited to) a black list or a list that needs to be manually processed; the target work order can be generated to directly enable the target object to be manually operated or otherwise processed according to the target work order, so that linkage with other IT facilities is achieved, for example, workers can optimize a network, manually check abnormal IPs or perform other forms of maintenance and the like according to the target work order.

The target object may be, but is not limited to, a server, an operation and maintenance person, and the like.

According to another embodiment of the present invention, as shown in fig. 6, there is provided a packet analyzing apparatus including:

the data capturing module 61 is configured to, in a case that it is determined that a signal connection is established with a first port of a core egress network device, obtain a first data packet through the first port, where the first data packet is obtained by processing, by the core egress network device, an initial data packet that enters and exits an external data center;

the data archiving module 62 is configured to archive the first data packet based on a preset objective function to obtain a second data packet in a target format;

a data unpacking module 63, configured to unpack the second data packet based on a preset unpacking program to obtain protocol information of a target protocol layer included in the second data packet;

a data consumption module 64, configured to perform consumption processing on the structured data to obtain first structure data, and store the first structure data in a target repository;

an aggregation analysis module 65, configured to perform aggregation analysis processing on the first structure data in the target repository to obtain a data analysis result, where the data analysis result includes IP related information and/or operation related information;

and a target operation module 66, configured to execute a target operation based on the IP related information and/or the operation related information, where the target operation at least includes event visualization processing and/or abnormal event linkage processing.

In an alternative embodiment of the method according to the invention,

the data unpacking module 63 further includes: a data transmission unit 631, configured to transmit the structured data to a target message queue in multiple threads based on a timestamp included in the structured data after the second data packet is unpacked based on a preset unpacking program to obtain the structured data;

the data consumption module 64 further includes: the data processing unit 641 is configured to sequentially acquire the structure data from the target message queue based on the timestamp; and processing the structured data to obtain the first structural data.

In an alternative embodiment, the data processing unit 641 includes:

an IP attribution subunit 6411, configured to obtain IP attribution information included in the structured data;

an identifying subunit 6412, configured to perform identification processing on the structured data based on the IP attribution information to obtain the first structured data.

In an optional embodiment, the apparatus further comprises:

an instruction receiving module 601, configured to receive a packet capturing instruction before the first port obtains a first data packet, where the packet capturing instruction is sent by the first port when the first data packet is received, the first data packet is obtained by copying and transmitting an initial data packet received by a second port of the core egress network device by using a target program, the second port is in communication connection with an external data center, and the initial data packet is transmitted to the second port by using the external data center.

In an alternative embodiment, the data capture module 61 includes:

a network card scanning unit 611, configured to, when it is determined that signal connection is established with the first port, perform a network card scanning operation on a target network card through a first program to determine network card information of the target network card, where the target network card is connected with the first port;

a data capture unit 612, configured to obtain the first data packet from the first port according to the network card information through the first program, and send a start instruction to a second program, so as to instruct the second program to perform archive processing on the first data packet based on a preset target function, so as to obtain a second data packet in a target format.

In an optional embodiment, the performing target operation based on the IP association information and/or the operation association information includes at least one of:

based on the IP associated information and/or the operation associated information, executing information visualization operation;

determining the network quality between the areas where the target objects are located based on the IP associated information and/or the operation associated information;

sending a configuration instruction to a target server based on the IP associated information and/or the operation associated information to indicate the target server to generate a first list;

and sending a work order operation order to a target object based on the IP associated information and/or the operation associated information to indicate the target object to generate a target work order and send the target work order.

The present invention is illustrated by the following specific examples.

As shown in fig. 3 and 5, in an embodiment, a method for analyzing a data packet mainly includes the following steps:

s31, connecting a physical server to a port on a switch, and starting a SPAN function (if the application scene of the network data packet analysis of the kubernets cluster is adopted, please skip S1, and start from S2);

s32, a packet capturing program: generating a pcap file by grabbing a physical network card of a server;

s33, uploading a program: grabbing a pcap file, and uploading the pcap file to a message queue;

s34, consumption program: consuming, processing and outputting the analyzed data to a database clickhouse or an elastic search;

s35 polymerization analysis program: analyzing data by the aggregation function of clickhouse, elastic search;

and S36, visualization and event linkage: and based on the result of the aggregation analysis, a digital panel is realized, and meanwhile, based on the abnormal event, the digital panel is linked with the firewall rule, the nginx blacklist and the IT work order center.

Among the above steps, the detailed step of S31 is as follows:

as shown in fig. 3 and 5, all network packets entering and exiting the data center need to be forwarded through the network device at the core exit, so we need to start a SPAN function on the network device at the core exit, copy the network packet at the source port at position 1 in fig. 2 to the destination port at position 2, connect a server to the destination port (the server runs the packet capture program of the present invention), and the function of copying the packet by the SPAN technology runs in a bypass mode, which does not affect the performance of the network device.

For another application scenario: the application scenario of the network data packet analysis of the kubernets cluster can skip the step, and because a plurality of virtual network cards are arranged on the servers of the kubernets cluster, all network flow needs to be in data communication with other servers in the cluster through the physical network cards of the servers, so that SPAN technology intervention is not needed. For such a scenario, step S31 may be skipped and the process may be performed directly from step S32.

Among the above steps, the detailed step of S32 is as follows:

the physical network card of the packet capturing server is connected with the destination port at the position 2 in fig. 3, and when the packet capturing program A2 runs, all data packets entering and exiting from the network device at the core outlet can be captured.

The following capture details need to be explained in detail with reference to the details of the packet capture program A2 in fig. 5:

firstly, a capture program can dynamically scan the name of a network card of a physical server A1, a capture program A2 can be started for each network card of A1, then A2 files a network data packet into a data packet file with a suffix of a pcap every 1 minute through a function based on libpcap, and after capture is finished, a data file path of the pcap is informed to an A4 unpacking program in a message transfer A3 mode.

Then, A4 unpacks the program, based on the TCP/IP four-layer structure: data link layer, network layer, transport layer, application layer. The data packet protocol of each layer is analyzed, such as data link layer protocol ARP and RARP, and the network layer protocol: IP protocol, ICMP protocol, transport layer protocol: TCP, UDP, application layer protocol: http, ssh, dns, etc., parsing each packet into a row of json data.

Among the above steps, the detailed step of S33 is as follows:

the analyzed data has a timestamp, and is uploaded to a message queue in a multithreading mode without ensuring the sequence.

Among the above steps, the detailed step of S34 is as follows:

the consumption program is responsible for consuming data, processing data and outputting data. There are three layers of pipelines in the consumption data, which are executed sequentially:

input: responsible for consuming data (must choose);

the Filter is responsible for processing data (optional);

optionally, the processing may be performed, if the IP is in the data center, field identifiers such as an attribution service system, the data center, and an available area may be added, and if the IP is in the external IP, a country, a province, a city, an operator, a user principal, and the like of the attribution may be identified.

Output is responsible for outputting data (optional);

the data after processing, which needs to be persisted, can be selected from the following databases:

1、clickhouse；

2、elsticsearch；

3、Mongodb；

in practical use, clickhouse is preferentially used as a columnar database, and compared with a row database, the number of disk IO is reduced, batch insertion is realized, and batch reading is faster.

Among the above steps, the detailed step of S35 is as follows:

writing clickhouse of network analysis data is completed at S34, and the aggregate analysis based on these data includes the following:

1. average network processing delay between two ips;

2. average network rate between two ips;

3. counting packet loss between two ips;

4. in unit time, IP top 10 ranking list occupying the largest network bandwidth;

5. statistics of abnormal external IP;

6. dns resolves the average response time of the domain name;

among the above steps, the detailed step of S36 is as follows:

based on the data analysis result of the above step S35, the following problems can be solved:

1. the state of the network is displayed in a digital panel mode, and operation and maintenance personnel of the data center can conveniently watch and debug.

2. And the network flow data statistics can provide data basis for network charging of the data center.

3. Distributed network quality detection: in different regional cities, the network delay of the data center can be detected in a ping detection mode, and after packet capture analysis, the network quality from the IP of different regions to the data center can be analyzed.

4. The security event, such as dns hijacking, can have detailed records to form a report and provide data basis for copying.

5. The abnormal IP may trigger a firewall or an internal web server to configure the blacklist.

6. The abnormal IP can trigger the IT work order center and dispatch the work order to the operation and maintenance personnel of the data center in time.

It should be noted that, in this embodiment, the software deployment manner includes:

1. data center egress, and packet analysis application scenarios of core network devices need to be deployed on servers connected to the switch.

2. The application scenario of the kubernets cluster data packet analysis needs to be operated on kubernets computing nodes in a daemonset containerization mode.

For the specific definition of the packet analysis device, reference may be made to the above definition of the packet analysis method, which is not described herein again. The modules in the packet analysis device may be implemented in whole or in part by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In an embodiment, a computer device is provided, where the computer device provided in the embodiment of the present application may be a server or a client: fig. 7 is a schematic structural diagram of a computer device according to an embodiment of the present application.

The processor 1701, the memory 1702, the bus 1705, the interface 1704, the processor 1701 is connected with the memory 1702 and the interface 1704, the bus 1705 is respectively connected with the processor 1701, the memory 1702 and the interface 1704, the interface 1704 is used for receiving or sending data, and the processor 1701 is a single-core or multi-core central processing unit, or a specific integrated circuit, or one or more integrated circuits configured to implement the embodiments of the present invention. The memory 1702 may be a Random Access Memory (RAM), or may be a non-volatile memory (non-volatile memory), such as at least one hard disk memory. The memory 1702 is used to store computer-executable instructions. Specifically, the computer executable instructions may include a program 1703.

In this embodiment, when the processor 1701 invokes the program 1703, the management server in fig. 7 may perform the operation of the packet analysis method, which is not described herein again.

It should be understood that the processor provided in the above embodiments of the present application may be a Central Processing Unit (CPU), or may be other general-purpose processors, digital Signal Processors (DSPs), application-specific integrated circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and so on. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It should also be understood that the number of processors in the computer device in the above embodiments in the present application may be one or more, and may be adjusted according to the practical application scenario, and this is merely an exemplary illustration and is not limited. The number of the memories in the embodiment of the present application may be one or multiple, and may be adjusted according to an actual application scenario, and this is merely an exemplary illustration and is not limited.

It should be further noted that, when the computer device includes a processor (or a processing unit) and a memory, the processor in this application may be integrated with the memory, or the processor and the memory are connected through an interface, and may be adjusted according to an actual application scenario, and is not limited.

The present application provides a chip system comprising a processor for enabling a computer device (client or server) to implement the functionality of the controller involved in the above method, e.g. to process data and/or information involved in the above method. In one possible design, the system-on-chip further includes a memory, the memory being used to hold the necessary program instructions and data. The chip system may be formed by a chip, or may include a chip and other discrete devices.

In another possible design, when the chip system is a chip in a user equipment or an access network, the chip includes: a processing unit, which may be, for example, a processor, and a communication unit, which may be, for example, an input/output interface, a pin or a circuit, etc. The processing unit may execute computer-executable instructions stored by the storage unit to cause a chip within the client or management server or the like to perform the steps of S401-S404. Alternatively, the storage unit may be a storage unit in the chip, such as a register, a cache, and the like, and the storage unit may also be a storage unit located outside the chip in the client or the management server, such as a read-only memory (ROM) or another type of static storage device that can store static information and instructions, a Random Access Memory (RAM), and the like.

The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a computer, implements the method flows executed by the controller of the client or the management server in any of the method embodiments described above. Correspondingly, the computer may be the computer device (client or server) described above.

It should be understood that the controller or processor mentioned in the above embodiments of the present application may be a Central Processing Unit (CPU), and may also be one or a combination of other general purpose processors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It should also be understood that the number of processors or controllers in the computer device (client or server) or chip system or the like in the above embodiments in the present application may be one or more, and may be adjusted according to the actual application scenario, and this is merely an example and is not limited. The number of the memories in the embodiment of the present application may be one or multiple, and may be adjusted according to an actual application scenario, and this is merely an exemplary illustration and is not limited.

It should also be understood that the memory or the readable storage medium and the like mentioned in the computer device (client or server) and the like in the above embodiments of the present application may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. The non-volatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable EPROM (EEPROM), or a flash memory. Volatile memory can be Random Access Memory (RAM), which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous Dynamic Random Access Memory (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), SLDRAM (synchronous DRAM), and direct rambus RAM (DR RAM).

Those of ordinary skill in the art will appreciate that the steps performed by a computer device (client or server) or processor in whole or in part to implement the embodiments described above may be performed by hardware or a program instructing associated hardware to perform the steps. The program may be stored in a computer-readable storage medium, which may be read only memory, random access memory, or the like. Specifically, for example: the processing unit or processor may be a central processing unit, a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

When implemented in software, the method steps described in the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the present application are all or partially generated when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The available media may be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., DVD), or semiconductor media, among others.

The terms "first," "second," and the like in the description and claims of the present application and in the above-described drawings are used for distinguishing between

Similar objects, but not necessarily for describing a particular order or sequence. It is to be understood that the terms so used are interchangeable under appropriate circumstances and are merely descriptive of the various embodiments of the application and how objects of the same nature can be distinguished. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the embodiments of the present application, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that in the description of the present application, unless otherwise indicated, "/" indicates a relationship where the objects associated before and after are an "or", e.g., a/B may indicate a or B; in the present application, "and/or" is only an association relationship describing an associated object, and means that there may be three relationships, for example, a and/or B, and may mean: a exists alone, A and B exist simultaneously, and B exists alone, wherein A and B can be singular or plural.

The word "if" or "if" as used herein may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrases "if determined" or "if detected (a stated condition or event)" may be interpreted as "when determined" or "in response to a determination" or "when detected (a stated condition or event)" or "in response to a detection (a stated condition or event)", depending on the context.

The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.

Claims (10)

1. A method for packet analysis, the method comprising:

under the condition that signal connection with a first port of core outlet network equipment is established, acquiring a first data packet through the first port, wherein the first data packet is obtained by processing an initial data packet which enters and exits an external data center through the core outlet network equipment;

based on a preset target function, filing the first data packet to obtain a second data packet in a target format;

unpacking the second data packet based on a preset unpacking program to obtain structured data;

consuming the structured data to obtain first structured data, and storing the first structured data to a target library;

performing aggregation analysis processing on the first structure data in the target library to obtain a data analysis result, wherein the data analysis result comprises IP (Internet protocol) associated information and/or operation associated information;

and executing target operation based on the IP related information and/or the operation related information, wherein the target operation at least comprises event visualization processing and/or abnormal event linkage processing.

2. The method of claim 1,

after the unpacking the second data packet based on the preset unpacking program to obtain the structured data, the method further includes: transmitting the structured data to a target message queue in multiple threads based on timestamps contained in the structured data;

the consuming the structured data to obtain the first structured data comprises: sequentially acquiring the structural data from the target message queue based on the timestamp; and processing the structured data to obtain the first structural data.

3. The method of claim 2, wherein said processing the structured data to obtain the first structured data comprises:

acquiring IP attribution information contained in the structured data;

and performing identification processing on the structured data based on the IP attribution information to obtain the first structured data.

4. The method of claim 1, wherein prior to said retrieving a first packet through said first port, said method further comprises:

under the condition that a second port receives an initial data packet, copying the initial data packet through a target program to obtain a first data packet, and transmitting the first data packet to the first port, wherein the second port is in communication connection with an external data center, and the initial data packet comprises data packets entering and exiting the external data center.

5. The method of claim 1, wherein obtaining the first packet through the first port of the core egress network device in the case that it is determined that the signal connection is established with the first port comprises:

under the condition that signal connection with the first port is established, network card scanning operation is carried out on a target network card through a first program so as to determine network card information of the target network card, wherein the target network card is connected with the first port;

and the first program acquires the first data packet from the first port according to the network card information and sends a starting instruction to a second program to indicate the second program to carry out filing processing on the first data packet based on a preset target function so as to obtain a second data packet in a target format.

6. The method of claim 1, wherein performing the target operation based on the IP association information and/or the operation association information comprises at least one of:

based on the IP associated information and/or the operation associated information, executing information visualization operation;

determining the network quality between the areas where the target objects are located based on the IP associated information and/or the operation associated information;

sending a configuration instruction to a target server based on the IP associated information and/or the operation associated information to indicate the target server to generate a first list;

and under the condition that the IP associated information and/or the operation associated information are determined to be in an abnormal state, sending a work order operation command to a target object to indicate the target object to generate a target work order and send the target work order.

7. A packet analysis device, the device comprising:

the data capturing module is used for acquiring a first data packet through a first port of core outlet network equipment under the condition that the signal connection with the first port of the core outlet network equipment is established, wherein the first data packet is obtained by processing an initial data packet which enters and exits an external data center through the core outlet network equipment;

the data archiving module is used for archiving the first data packet based on a preset target function to obtain a second data packet in a target format;

the data unpacking module is used for unpacking the second data packet based on a preset unpacking program to obtain structured data;

the data consumption module is used for performing consumption processing on the structured data to obtain first structure data and storing the first structure data to a target library;

the aggregation analysis module is used for performing aggregation analysis processing on the first structure data in the target library to obtain a data analysis result, wherein the data analysis result comprises IP (Internet protocol) associated information and/or operation associated information;

and the target operation module is used for executing target operation based on the IP related information and/or the operation related information, wherein the target operation at least comprises event visualization processing and/or abnormal event linkage processing.

8. The apparatus of claim 1, further comprising:

an instruction receiving module, configured to receive a packet capturing instruction before the first port obtains a first data packet, where the packet capturing instruction is sent by the first port when the first data packet is received, the first data packet is obtained by copying and transmitting an initial data packet received by a second port of the core egress network device by using a target program, the second port is in communication connection with an external data center, and the initial data packet is transmitted to the second port by using the external data center.

9. A computer device comprising a memory and a processor coupled to the memory, wherein at least one program instruction or code is stored in the memory and loaded into and executed by the processor to cause the computer device to implement the packet analysis method of claims 1-6.

10. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed, realizes the steps of the method of any of claims 1-6.

Images