CN117955854A

CN117955854A - Data processing method and system

Info

Publication number: CN117955854A
Application number: CN202410117706.XA
Authority: CN
Inventors: 周悦; 钱世俊; 戴志勇
Original assignee: Ant Blockchain Technology Shanghai Co Ltd
Current assignee: Ant Blockchain Technology Shanghai Co Ltd
Priority date: 2024-01-26
Filing date: 2024-01-26
Publication date: 2024-04-30

Abstract

The embodiment of the specification provides a data processing method and a system, wherein the data processing method is applied to a data processing system and comprises the following steps: the client generates network data to be processed based on eBPF programs, and uploads the network data to be processed to a server, wherein the network data to be processed is used for determining network monitoring data; the server acquires target service application data according to the network data to be processed, generates network monitoring data according to the network data to be processed and the target service application data, and stores the network monitoring data, wherein the network monitoring data is used for analyzing network state information of the target service application. According to the method provided by the embodiment of the specification, network data are collected through eBPF, the network data to be processed for obtaining the network monitoring data in the data packet are extracted and uploaded to the server, the server finds out the target service application data corresponding to the target service application according to the network data to be processed, and relevant target service information is enriched for subsequent network state analysis.

Description

Data processing method and system

Technical Field

The embodiment of the specification relates to the technical field of computers, in particular to a data processing method.

Background

With development and application of virtualization technology and micro-service architecture, various resource components such as virtual machines, containers, load balancing and the like and small and cohesive distributed micro-service application, more complex network transmission and application calling relations are brought. In this scenario, in order to ensure and optimize interaction between applications, the operation and maintenance personnel need to know network connection and performance conditions between applications, and topology discovery can automatically sense and acquire connection relations between resource components and applications in the environment, and network monitoring can provide index data such as network flow, so as to help the operation and maintenance personnel to quickly locate network problems.

In general, the network packet capturing tool collects information such as IP and ports and indexes such as network traffic, and can represent the network condition of a certain resource assembly. But additional enhancements to the raw network data collected are also needed to observe network connectivity and performance in the application dimension. For example, when a certain IP is used by a certain container, how to supplement the network data with application data corresponding to the container is also a problem to be solved by the technician.

Disclosure of Invention

In view of this, the present embodiments provide a data processing method. One or more embodiments of the present specification are also directed to a data processing system, a computing device, a computer readable storage medium, and a computer program to address the deficiencies of the prior art.

According to a first aspect of embodiments of the present specification, there is provided a data processing method applied to a data processing system, the data processing system including a client and a server, including:

The client generates network data to be processed based on eBPF programs and uploads the network data to be processed to the server, wherein the network data to be processed is used for determining network monitoring data;

The server acquires target service application data corresponding to the network data to be processed according to the network data to be processed, generates network monitoring data corresponding to the network data to be processed according to the network data to be processed and the target service application data, and stores the network monitoring data, wherein the network monitoring data is used for analyzing network state information of target service application.

According to a second aspect of embodiments of the present specification, there is provided

The data processing system comprises a client and a server; wherein,

The client is configured to generate network data to be processed based on eBPF programs and upload the network data to be processed to the server, wherein the network data to be processed is used for determining network monitoring data;

The server is configured to obtain target service application data corresponding to the network data to be processed according to the network data to be processed, generate network monitoring data corresponding to the network data to be processed according to the network data to be processed and the target service application data, and store the network monitoring data, wherein the network monitoring data is used for analyzing network state information of target service applications.

According to a third aspect of embodiments of the present specification, there is provided a data processing method, applied to a server, including:

receiving network data to be processed sent by a client;

Acquiring target business application data corresponding to the network data to be processed according to the network data to be processed;

Generating network monitoring data corresponding to the network data to be processed according to the network data to be processed and the target service application data, wherein the network monitoring data is used for analyzing network state information of the target service application;

And storing the network monitoring data.

According to a fourth aspect of embodiments of the present specification, there is provided a computing device comprising:

a memory and a processor;

The memory is configured to store computer executable instructions that, when executed by the processor, perform the steps of the data processing method described above.

According to a fifth aspect of embodiments of the present specification, there is provided a computer readable storage medium storing computer executable instructions which, when executed by a processor, implement the steps of the data processing method described above.

According to a sixth aspect of the embodiments of the present specification, there is provided a computer program, wherein the computer program, when executed in a computer, causes the computer to perform the steps of the data processing method described above.

One embodiment of the present specification provides a data processing method applied to a data processing system, where the data processing system includes a client and a server, the method includes: the client generates network data to be processed based on eBPF programs and uploads the network data to be processed to the server, wherein the network data to be processed is used for determining network monitoring data; the server acquires target service application data corresponding to the network data to be processed according to the network data to be processed, generates network monitoring data corresponding to the network data to be processed according to the network data to be processed and the target service application data, and stores the network monitoring data, wherein the network monitoring data is used for analyzing network state information of target service application.

According to the method provided by the embodiment of the specification, network data is collected through the eBPF program running in the kernel mode, and the data packet is processed based on the eBPF program running in the kernel mode, so that multiple context switches in the process of grabbing the packet are reduced. And directly processing the data packet in the kernel mode, extracting the network data to be processed for acquiring the network monitoring data in the data packet, uploading the network data to a server, searching the target service application data corresponding to the target service application according to the network data to be processed by the server, combining the network data to be processed into network detection data, and enriching relevant target service information for subsequent network state analysis aiming at the target service application.

Drawings

FIG. 1 is a flow chart of a data processing method provided in one embodiment of the present disclosure;

FIG. 2 is a schematic diagram of a data processing method according to an embodiment of the present disclosure;

FIG. 3 is a process flow diagram of a data processing method according to one embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a data processing system according to one embodiment of the present disclosure;

FIG. 5 is a flow chart of a data processing method applied to a server according to one embodiment of the present disclosure;

FIG. 6 is a schematic diagram of a data processing apparatus according to one embodiment of the present disclosure;

FIG. 7 is a block diagram of a computing device provided in one embodiment of the present description.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many other forms than described herein and similarly generalized by those skilled in the art to whom this disclosure pertains without departing from the spirit of the disclosure and, therefore, this disclosure is not limited by the specific implementations disclosed below.

The terminology used in the one or more embodiments of the specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the specification. As used in this specification, one or more embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any or all possible combinations of one or more of the associated listed items.

It should be understood that, although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first may also be referred to as a second, and similarly, a second may also be referred to as a first, without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "in response to a determination" depending on the context.

It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in this specification are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the relevant laws and regulations and standards of the relevant area, and be provided with corresponding operation entries for the user to select authorization or rejection.

First, terms related to one or more embodiments of the present specification will be explained.

Classical release: applications and services are deployed into a traditional virtual or physical machine environment, and traffic typically runs in a process.

And (3) publishing the cloud primordial: the services typically run in a container based on the way the Kubernetes application deploys and manages.

EBPF: full scale extended Berkeley PACKET FILTER is a technology that enables sandboxed programs to run in the operating system kernel. The kernel can be dynamically programmed to achieve efficient networking, observability, and security functions.

Metadata: or meta information, data providing information of other data for describing, managing and counting other data. Metadata in a cloud service scenario generally includes information about tenants of resources such as clusters.

With development and application of virtualization technology and micro-service architecture, various resource components such as virtual machines, containers, load balancing and the like and small and cohesive distributed micro-service application, more complex network transmission and application calling relations are brought. In this scenario, in order to guarantee and optimize interactions between applications, the operation and maintenance personnel need to know the network connection between applications and their performance. The topology discovery can automatically sense and acquire the connection relation between each resource component and application in the environment, and the network monitoring can provide index data such as network flow and the like so as to help operation and maintenance personnel to quickly locate network problems.

In conventional network packet-grabbing tools, the data packets need to be copied multiple times to reach the application. For example, the data packet needs to be copied from the network card driver to the kernel buffer, and then copied from the kernel buffer to the user-state application buffer, where the data packet is subjected to multiple context switches, and some unnecessary information in the data packet is also copied, so that a large performance overhead is caused.

In the process, the packet grabbing tool sends out packet grabbing instructions for a plurality of times, and the context switching is carried out for a plurality of times. The packet grabbing tool operates in a user state, and sends out a plurality of packet grabbing instructions in the whole network data packet grabbing process, so that the system kernel can be interrupted for a plurality of times, multiple system calls are generated, multiple context switching can occur, and larger performance cost is brought.

The information such as IP, port and the like and the index data such as network flow and the like collected by the network packet grasping tool can represent the network condition of a certain resource assembly. But additional enhancements to the raw network data collected are also needed to observe network connectivity and performance in the application dimension. For example, a certain IP is used by a certain container belonging to a certain application, and by associating such metadata, service information such as an application is supplemented to the network data.

In the scheme of performing service application network monitoring and topology discovery, there are the following problems:

1. The network data is collected by using a traditional technical tool, and in the whole unpacking process, the data package can reach the application program only through multiple memory copies and context switching, and in the process, some unnecessary information in the data package can participate in copying, so that larger performance cost is brought.

2. In the current scheme of network monitoring and topology discovery, network connection quadruple (source IP, source port, target IP, target port) information, network flow information and the like can be acquired, and various acquired network data can be displayed as network monitoring data and connection topology of various resources, but the processing scheme lacks related service application information, and cannot display network monitoring and topology relation of application dimension.

Based on this, in the present specification, a data processing method is provided, and the present specification relates to a data processing system, a computing device, and a computer-readable storage medium, which are described in detail in the following embodiments one by one.

With reference to fig. 1, fig. 1 shows a flowchart of a data processing method according to an embodiment of the present specification, where the method is applied to a data processing system, and the data processing system includes a client and a server, and the method specifically includes the following steps.

Step 102: the client side generates network data to be processed based on eBPF programs and uploads the network data to be processed to the server, wherein the network data to be processed is used for determining network monitoring data.

Extended Berkeley PACKET FILTER (eBPF) is a virtual machine that can run in a Linux kernel. It evolved from Berkeley PACKET FILTER (BPF) and was primarily used initially for network packet filtering. However, as technology evolves, eBPF has far exceeded network packet filtering, which can now be used for many system and network observation and control tasks.

In the embodiment provided in the present specification, the network data to be processed is generated at the client based on eBPF programs, and the network data to be processed is uploaded to the server. Specifically, the network data to be processed specifically refers to network data that is subsequently used to determine network monitoring data, so as to perform network analysis. In practical applications, the network data to be processed specifically refers to statistical data information obtained after processing by eBPF programs, and the network data to be processed is obtained after processing the original network data. Further, the network data to be processed includes data obtained by processing the original network data, such as source IP, source port, destination IP, destination port, process ID, packet size, etc.

In conventional network packet-grabbing tools, the data packets need to be copied multiple times to reach the application. For example, the data packet needs to be copied from the network card driver to the kernel buffer first, and then copied from the kernel buffer to the user-mode application buffer. In this process, the data of all the data packets is copied, which causes a large performance overhead.

Based on this, the client generates network data to be processed based on eBPF program, and uploads the network data to be processed to the server, including:

The client generates network data to be processed based on eBPF programs, and stores the network data to be processed in a preset storage position;

And reading the network data to be processed from the preset storage position, and uploading the network data to be processed to a server.

EBPF is a main feature of being able to run custom programs in the kernel without modifying the kernel source code or loading the kernel module. This means that the user can customize the program to collect and process data in the kernel to enable deep viewing and control of the system. Meanwhile, the eBPF virtual machine uses a specially designed safe sandbox, so that the running program can be ensured not to cause harm to the system.

EBPF programs are typically compiled and loaded into the kernel by tools in user space. They may be attached to many types of kernel objects such as system calls, network interfaces, task switches, etc. When these events occur, eBPF programs are executed, which can read and modify the data in the kernel and even change the behavior of the system.

Further, generating network data to be processed based on eBPF programs, and storing the network data to be processed in a preset storage location, including:

Acquiring initial network data based on eBPF programs in a kernel mode, analyzing the initial network data, and generating network data to be processed;

storing the network data to be processed in a preset storage position of the kernel mode;

Correspondingly, the reading the network data to be processed from the preset storage position comprises the following steps:

And the agent program based on the user mode reads the network data to be processed from the preset storage position.

In an operating system, programs are typically executed in two different modes, kernel space and user space, also referred to as kernel mode and user mode. Kernel mode and user mode are two main operating environments in an operating system.

The User Mode is a normal running program of the program, most of the User programs (such as a text editor, a browser, a database application, etc.) are executed in the User Mode, and in the User Mode, the program cannot directly access the memory space of the kernel of the operating system, hardware or other programs, and only can request the kernel to complete operations requiring special rights, such as reading and writing files, sending network data, etc., by initiating a system call.

Kernel Mode (Kernel Mode), also known as system Mode, operates in this Mode in which the Kernel has direct access to hardware and memory to perform tasks such as managing memory, responding to interrupts, handling system calls, etc. When the user mode program initiates the system call, the system will switch to the kernel mode, and after the kernel finishes the requested service, the user mode will switch. This mode switching mechanism is to protect the security and stability of the system. By limiting the behavior of user-mode programs, malicious or errant code can be prevented from damaging the operating system kernel or other programs. When the program needs to execute special authority operation, the program is switched to the kernel mode in a system calling mode, and the kernel is used for completing the operation instead of the kernel, so that the normal operation and the safety of the system are ensured.

The programs running in the two spaces of kernel mode and user mode are distinguished as follows:

Rights: the kernel-mode running programs have higher rights to directly access hardware and kernel data structures, while the user-space programs need to request the kernel to provide services through system calls, which means that the kernel-mode running programs can do more things and also means that there is a greater risk that an erroneous operation may destroy the entire operating system.

Performance: programs in kernel mode generally run faster than programs in user mode because programs running in kernel mode can directly access hardware and kernel data structures without the overhead of system calls, which is important for tasks that require high performance.

Safety: in a user mode, the program runs in a limited environment, and cannot directly access hardware or influence other programs, which provides a certain security guarantee for an operating system, while in a kernel mode, the program needs to be more careful, because incorrect operation may cause the operating system to crash.

Therefore, when selecting the kernel mode or the user mode to run the program, the trade-off needs to be made according to specific requirements and risks, and eBPF programs provide a security mechanism for running the custom program in the kernel mode, so that the user can enjoy the high performance in the kernel mode without worrying about the security problem of the operating system.

EBPF the program is divided into a kernel mode and a user mode, the eBPF program in the user mode is used for loading eBPF in the kernel mode into the kernel, and the eBPF program in the kernel mode is used for generating network data to be processed according to a network data packet. The eBPF program running in the kernel mode obtains initial network data in the process of calling the network system, wherein the initial network data is the network data in the process of calling the network system, such as kernel structures of sk_buff, socket and the like. Compared with a packet grabbing tool running in a user mode, the eBPF program running in the kernel mode can directly acquire the data packet in the kernel mode, reduces multiple context switching operations in the packet grabbing process, saves computing resources and reduces performance cost.

The method comprises the steps of obtaining network data to be processed from initial network data, wherein the network data to be processed are used for determining network monitoring data, relevant business application information is added in the method provided by the embodiment of the specification, and network monitoring and topological relations are enriched from the dimension of the application information. Therefore, the network data to be processed needs to include information that can directly locate the corresponding business process.

In the cloud primary scene, one container corresponds to one application, so the corresponding target application can be positioned based on the target IP in the network connection quadruple (source IP, source port, target IP and target port) reported by the acquisition Agent, and therefore, at least the target IP is included in the network data to be processed in the cloud primary scene and is used for positioning the target application subsequently, so that application information is acquired.

For a classical release scene, if the mode in the cloud native scene is directly sleeved to the classical release scene, the situation that the target application cannot be positioned can occur. Because in the cloud native scenario, the container can be matched through the IP, so as to find the application corresponding to the container. However, in a classical release scenario, an application usually runs in a process manner, and there are usually multiple applications on a host that use the same IP to communicate, where only IP information cannot be located to a specific service application. Therefore, the network data to be processed in the classical distribution scene at least needs to comprise the process ID. And positioning to the target application through the process ID, so as to acquire application information.

Based on the above, after the initial network data is obtained, the initial network data may be parsed by the eBPF program to obtain the network data to be processed in the initial network data. For example, in the case of a network system call, network message data may be input into the kernel protocol stack, and the eBPF program may obtain the network message data in the kernel protocol stack, and parse the network message data to be processed from the network message data. And then the network data to be processed is stored to a designated position, and then the network data to be processed is transmitted to a user state, so that the size of the data quantity in the transmission process is reduced, and the performance cost is further reduced.

In practical application, at least data information such as source IP, source port, destination IP, destination port, process ID, packet size, etc. needs to be acquired based on network packet data, so as to generate network data to be processed. Specifically, the four-tuple information of the source IP, the source port, the target IP and the target port carried in the network message data can be obtained by analyzing the network message data. The kernel mode of the operating system also stores a port process mapping table, and the port process mapping table stores the corresponding relation between the port number and the process ID, and the process ID corresponding to the target port can be queried through the port process mapping table.

EBPF also supports a kernel data structure named "Maps" that can be used to store and share data. In the method provided in the present specification, the preset storage location specifically refers to eBPF Maps. eBPF programs can read and modify the data in Maps, as can programs in user space. This allows efficient data exchange between the user space program and the kernel space eBPF program. eBPF is a powerful tool that can be used to implement the observation and control tasks of various systems and networks.

In practical applications, eBPF processes acquire network data to be processed for subsequent network analysis, and store the network data to be processed in a preset storage location (eBPF Maps).

Specifically, acquiring initial network data based on eBPF programs in a kernel mode includes:

Initial network data is acquired in a kernel mode based on eBPF programs through hook technology.

In practice, eBPF processes acquire initial network data based on HOOK technology (HOOK). HOOK is a programming term that generally refers to changing or extending the behavior of an operating system, software component, or system by modifying, adding, or intercepting function calls, messages, or events, and in some cases, may also be used to refer to low-level access to hardware devices. While one program (the object program) is running, the other program (the HOOK program) may set a HOOK to intercept certain function calls, messages or events of the object program, when the function called by the HOOK, or a specified message or event occurs, control may be transferred to the HOOK program, which may choose to modify parameters of these calls, messages or events, or to block them, or to add some additional behavior without affecting the object program. Common situations where HOOK technology is applied include debugging and reverse engineering, creating macros, automating tasks, adding new functions or features, and so forth.

In the method provided in the present specification, a HOOK point is set in a kernel protocol stack, so that eBPF programs can obtain network message data written in the kernel protocol stack through a HOOK technology, where the kernel protocol stack is the target program. In practical applications, network system calls may exist as system calls across hosts, and may exist as local system calls. When the network system call occurs, corresponding network message data is written into the kernel protocol stack, the HOOK program is provided with a HOOK point in the kernel protocol stack, and eBPF program can obtain the network message data and analyze the network message data under the condition that the network message data exists in the kernel protocol stack, so as to obtain the network data to be processed carried in the network message data. Generating the network data to be processed according to a preset data storage format of the network data to be processed, and storing the network data to be processed in a eBPF Maps in a kernel mode.

In a specific embodiment provided in the present disclosure, taking a cloud native scenario as an example for explanation, if a certain service application needs to call POD1, network packet data related to POD1 will be added to the kernel protocol stack. The HOOK program obtains the data information of source IP, source port, target IP, target port, data packet size and the like in the network message data from the kernel protocol stack, thereby generating the network data to be processed.

In another specific embodiment provided in the present specification, taking a classical release scenario as an example for explanation, if an application wants to call a process a, network message data about the service call is added to a kernel protocol stack. The HOOK program obtains the data information of source IP, source port, target IP, target port and the like in the network message data from the kernel protocol stack, then determines the process ID corresponding to the target port through the port process mapping table according to the target port, and generates the network data to be processed according to the information of source IP, source port, target IP, target port, process ID and the like.

In practical application, the preset storage location is located in a kernel mode, the network data to be processed needs to be fetched from the preset storage location in the kernel mode, and further, an Agent program (Agent) is operated in a user mode, and the Agent program operated in the user mode reads the network data to be processed from eBPF Maps.

In a cloud-native scenario, an Agent is typically a lightweight process or service deployed on each node, responsible for collecting and reporting information of that node, such as hardware status, system performance metrics, application logs, etc., which are used to monitor and manage the entire cloud environment. The main tasks of the Agent are data collection, event response, execution command and the like. The data collection refers to that an Agent periodically collects various information of a node, such as CPU usage, memory usage, disk I/O, network traffic, and the like, and sends the data to a central monitoring system. Event responses refer to the fact that when certain specific events occur on a node (such as resource exhaustion, service breakdown, etc.), an Agent will immediately report to the central monitoring system. Executing commands, which may include restarting services, application updates, configuration changes, etc., refers to agents that are also able to receive instructions from the central monitoring system and execute on the nodes.

The Agent program is usually started automatically when the node is started and runs in the background, so that the influence on the system performance is small. Agents are very important for maintaining the health and stability of the cloud's native environment, e.g., in Kubernetes, there is Kubelet such agents running in each node, responsible for communicating with the Master node, receiving instructions, and managing Pods on the nodes.

In the method provided in the specification, an Agent program in a cloud primary scene is referenced, and the Agent program is deployed in a terminal. Specifically, an Agent program is deployed in a user state, and the to-be-processed network data is read from a preset storage position (eBPF Maps) by the Agent program in the user state and is sent to a server.

Step 104: the server acquires target service application data corresponding to the network data to be processed according to the network data to be processed, generates network monitoring data corresponding to the network data to be processed according to the network data to be processed and the target service application data, and stores the network monitoring data, wherein the network monitoring data is used for analyzing network state information of target service application.

After receiving the network data to be processed sent by the client, the server acquires target service application data corresponding to the network data to be processed according to the network data to be processed.

The target service application data specifically refers to application data related to a target service application, for example, index information corresponding to the target service application, an upstream service node, a downstream service node, and the like. In one or more embodiments provided herein, the target service application data specifically refers to application data related to a target application for performing network topology.

Specifically, obtaining target service application data corresponding to the network data to be processed according to the network data to be processed includes:

Determining a target service application corresponding to the network data to be processed;

And collecting target business application data corresponding to the target business application.

In practical application, an Agent program in a user state in the client consumes the network data to be processed from eBPF Maps, and sends the network data to be processed to the server. And running a gateway service in the server, and acquiring corresponding target business application data according to the network data to be processed in the gateway service.

In the cloud native environment, the main function of the gateway is to provide an entry point for managing network traffic entering and exiting the K8s cluster, and the gateway makes interaction between internal services and management of external traffic simpler and more efficient. The main functions of the cloud native gateway are routing, load balancing, security, current limiting, protocol conversion and the like. Wherein routing means that the gateway can route requests from outside to an appropriate internal service according to predefined rules, e.g. find the corresponding internal service based on URL path or request header information; load balancing means that the gateway can distribute requests among the services of multiple instances, which helps to improve the availability and performance of the system; security means that the gateway can provide some security measures, such as authentication and authorization, preventing unauthorized access.

In practice Kubernetes Ingress is a common cloud-native gateway, envoy and Istio also provide more complex and powerful gateway functions that can be used with other cloud-native technologies such as service grids, providing a complete and consistent network management solution. In the specific embodiment provided in the specification, the gateway is not only applied to a cloud native scene, but also can be applied to a classical release scene. The gateway can receive the network data to be processed and supplement and acquire the target business application data corresponding to the target business application.

Specifically, the network data to be processed includes application identification information, and the corresponding determination of the target service application corresponding to the network data to be processed includes:

Extracting application identification information in the network data to be processed;

and determining the target business application according to the application identification information.

In practical application, after receiving the network data to be processed, the gateway extracts application identification information stored in the network data to be processed, wherein the application identification information specifically refers to identification information capable of being positioned to a target service application. After the application identification information is determined, the target service application can be positioned according to the application identification information, so that the target service application data corresponding to the target service application is obtained.

In a specific embodiment provided in the present specification, extracting the application identification information in the network data to be processed includes:

and extracting a process identifier or a container identifier in the network data to be processed.

In practical application, if in the cloud primary scene, the network data to be processed includes a container identifier (container IP); in a classical publishing scenario, a process identification (process ID) is included in the network data to be processed. Through the container IP or process ID, a corresponding target business application can be located.

Because of the container technology, network division, virtual network cards and other network conditions of the cloud primary scene are complex, but Yun Yuansheng is a micro-service application, and only one application in each container is usually running, so that the corresponding relation between a target IP address in network data to be processed and the application is clearer in the cloud primary scene, and the target service application can be positioned according to the container identification.

In a classical release scenario, a plurality of applications may be running in a host, if the corresponding target service application cannot be accurately located only by means of the target IP address in the network data to be processed, therefore, in the classical release scenario, PID data is included in the network data to be processed, and the target service application corresponding to the network data to be processed can be accurately located through the PID data.

In a specific embodiment provided in the present specification, collecting target service application data corresponding to the target service application includes:

and inquiring target business application data corresponding to the target business application in a business application metadata base according to the application identification information.

In the cloud primary scene, the kernel of the operating system is located in each independent virtual machine or container, the virtual machines or containers run on the host machine and provide own operating system environment through a certain isolation mechanism, each virtual machine or container contains a complete user space environment including a process space, a file system and the like, but the virtual machines have own kernels, and the containers share the kernels of the host machine.

Under this arrangement, pooled resources (e.g., CPUs, memory, storage, and networks) are managed by the operating system kernel and provided to the virtual machines or containers running therein via virtualization techniques, allowing flexible allocation and scheduling of resources among the virtual machines or containers. For example, kubernetes is a common cloud-native scenario, consisting of a set of interconnected nodes (nodes), each running an operating system kernel, and a Scheduler (Scheduler) schedules Pod (a set of containers containing one or more containers) to run on different nodes, during which the resources of each Node are pooled and managed by the operating system kernel.

Taking the cloud primitive scenario as an example, using Kubernetes as a tool for container arrangement, each Pod and Node will have its own IP address, pod (container group) is the smallest deployment unit in K8s, and may contain one or more containers, in K8s, each Pod will be assigned an IP address, and all the containers in the Pod share this IP address and network namespace. The Node is a working Node in the K8s, can be a physical machine or a virtual machine, and each Node also has an own IP address, and the IP address is used for other components in the K8s cluster to communicate with. The design enables the Pod to perform network communication like a physical host, greatly simplifies network management, and is convenient for realizing advanced network functions such as service discovery, load balancing and the like.

The gateway locates the target service application according to the application identification information in the network data to be processed, specifically, according to the process identification or the container identification. In the cloud primary scene, the resources such as Pod or Node and the like can be positioned through the container IP, and the corresponding target business application is found according to the application label of the resources; under a classical scene, a process can be matched through the IP of a host and the process ID, and a corresponding target service application is found according to an application matching rule of the process.

After the target business application is determined, the target business application data corresponding to the target business application can be queried from the business application metadata base. In the subsequent network monitoring process, network monitoring data of application dimension can be utilized, the dimension of the network monitoring data is enriched, and the richness of network topology is improved.

After the network data to be processed and the target service application data of the target service application are obtained, the network data to be processed and the target service application data are fused, and network monitoring data corresponding to the network protection tool to be processed are generated. The network monitoring data specifically refers to application data associated with a target service application, and is used for carrying out corresponding service network analysis according to actual requirements of a service side.

In practical applications, the network data to be processed includes information such as an active IP, a source port, a target IP, a target port, a process ID, a packet size, etc., and the target service application data may include a target service application identifier, target service application traffic information, target service application network parameter information, a reference service application identifier corresponding to the target service application, reference service application identifier traffic information, reference service application network parameter information, etc. And splicing and fusing the network data to be processed and the target service application data to generate network traffic monitoring data.

After the network monitoring data is obtained, the network monitoring data can be saved for subsequent network monitoring data, and the network monitoring data is used for subsequent analysis of network state information of target service application.

In a specific embodiment provided in the present specification, storing the network monitoring data includes:

And storing the network monitoring data into a non-relational database.

In practical application, the network monitoring data are data acquired in real time, and in order to facilitate data analysis at any time in the following, the network monitoring data can be stored in a database. Because the network monitoring data includes business information related to the target business process, the network monitoring data is non-relational data, and therefore, the non-relational database can be used for storing the network monitoring data. Compared with a relational database, a non-relational database is more flexible and is suitable for processing a large amount of scattered data and high concurrent access. In practical applications, the non-relational database may be ELASTICSEARCH (ES) database, mongoDB, redis, memcache, or the like. Preferably, the non-relational database is ELASTICSEARCH (ES) data storage services. The ES database is a document-oriented database, meaning that the database no longer requires the table field constraints of a determinant. The ES will store the entire constructed data or document, however not just the data, so that each data in the document can be identified and thus retrieved.

In a specific embodiment provided in the present specification, further includes:

receiving a network analysis request for the target service application;

Acquiring at least one network monitoring data corresponding to the target service application;

And generating the network state information of the target service application according to the network monitoring data.

After the network monitoring data is stored in the non-relational database, in subsequent processing, a network analysis request sent by the front-end service side may be received, where the network analysis request may be a service application network topology, traffic monitoring for a target service application, and the like, and in the embodiment provided in this specification, a specific form of the network analysis request is not limited, and the actual application is in control.

After receiving the network analysis request for the target service application, at least one network monitoring data corresponding to the target service application can be obtained, specifically, at least one network monitoring data corresponding to the target service application can be queried from the non-relational database, and network state information corresponding to the target service application can be generated according to each network monitoring data.

In a specific embodiment provided in the present disclosure, taking a network analysis request as an example of a network topology request for a target service application, network monitoring data corresponding to the target service application is obtained from a non-relational database, and network topology information for the target service application is generated through analysis processing of the network monitoring data.

In another specific embodiment provided in the present specification, taking a network analysis request as an example to analyze a traffic situation of a target service application in a certain time interval, acquiring network monitoring data of the target service application in the time interval from an ES database, and generating a network traffic situation of the target service application in the time interval through analysis processing of the network monitoring data.

And visually displaying the network state information.

In practical application, after the network state information is generated by analysis, a corresponding visual mode can be selected to perform visual display on the network state information, wherein the visual mode can be a histogram, a mesh map, a topological map, a line graph and the like, and in the method provided by the specification, the visual display mode is not limited and the practical application is in order. Visual display, which can generate visual display view in the server and send to the front end; the network status information may be sent to the front end by the server, and may be visually displayed on the front end, which is not limited in the embodiment provided in the present specification.

The data processing method provided by the embodiment of the specification comprises the steps of generating network data to be processed based on eBPF programs in a kernel mode, and storing the network data to be processed in a preset storage position, wherein the network data to be processed is used for determining network monitoring data; reading the network data to be processed from the preset storage position in a user mode, and acquiring target business application data corresponding to the network data to be processed according to the network data to be processed; generating network monitoring data corresponding to the network data to be processed according to the network data to be processed and the target business application data; and storing the network monitoring data, wherein the network monitoring data is used for analyzing network state information of the target service application.

According to the method provided by the embodiment of the specification, network data are acquired through eBPF technology, data packets are directly processed in a kernel mode, to-be-processed network data used for acquiring network monitoring data in the data packets are extracted and uploaded to a server, the server searches target service application data corresponding to target service applications according to the to-be-processed network data and combines the target service application data with the to-be-processed network data into network detection data, and relevant target service information is enriched for subsequent network state analysis aiming at the target service applications. By adding the network monitoring data with application dimensions, the dimensions of the network monitoring data are enriched, and the richness of the network topology is improved.

Secondly, in the process of collecting network data through eBPF technology, the data packet is directly processed in a kernel mode, the network data to be processed for obtaining network monitoring data in the data packet is extracted, and the network data to be processed is subjected to subsequent processing, so that the data volume of data copy in traditional collection is greatly reduced, and the processing efficiency is improved.

In addition, in the process of positioning the target business reference through the application identification information, the method is suitable for a cloud primary scene and a classical release scene, in the cloud primary scene, resources such as Pod or Node and the like can be matched through IP, and corresponding target business application can be found according to the application label of the resources; under a classical release scenario, a process can be matched through a host IP and a process ID, and a corresponding target business reference is positioned according to the process. The method provided by the specification can be applied to cloud native scenes or classical release scenes, and has universality.

The application of the data processing method provided in the present specification to a network topology is taken as an example, and the data processing method is further described below with reference to fig. 2 and fig. 3. Fig. 2 is a schematic structural diagram of a data processing method provided in an embodiment of the present disclosure, as shown in fig. 2, a eBPF program is deployed in a kernel mode of a terminal, network packet data is captured from a kernel protocol stack through a eBPF program hook to the kernel protocol stack, and statistical data such as source IP, source port, target IP, target port, process ID, packet size and the like are obtained based on the network packet data, combined into network data to be processed, and added into eBPF Maps.

And (3) an Agent program is deployed in a user mode of the terminal, the Agent program consumes the network data to be processed from eBPF Maps in a kernel mode, and the network data to be processed is reported to a gateway service of the server.

In gateway service of server, according to application identification information in network data to be processed, obtaining target service application data corresponding to target service application from metadata base of server according to application identification information. And fusing the acquired target business application data with the network data to be processed to generate network monitoring data, and storing the network monitoring data into an elastic search data storage service of the server.

The front end requests the network topology of the service application from the server, the server inquires the elastic search data storage service, calculates a directed graph of the network topology of the service application and index information such as the flow of nodes and edges of the directed graph based on network monitoring data stored in the ES data storage service, generates a corresponding network topology graph, and feeds the network topology graph back to the front end for display.

Fig. 3 shows a process flow chart of a data processing method according to an embodiment of the present disclosure, which specifically includes the following steps.

Step 302: the client obtains initial network data based on eBPF programs in kernel mode.

Step 304: and the eBPF program of the client analyzes the initial network data, generates network data to be processed, and stores the network data to be processed in eBPF Maps.

Step 306: the client side reads the network data to be processed from eBPF Maps according to the user mode agent program, and reports the network data to be processed to the gateway service of the server.

Step 308: and extracting application identification information in the network data to be processed from gateway service of a server, and determining a target service application according to the application identification information.

Step 310: and the server inquires target business application data corresponding to the target business application in a business application metadata base according to the application identification information.

Step 312: and the server fuses the target service application data and the network data to be processed to generate network monitoring data.

Step 314: the server saves the network monitoring data to the elastic search data storage service.

Step 316: the server receives a network topology request for the service application sent by the front end.

Step 318: the server acquires at least one network monitoring data corresponding to a target service application from the elastic search data storage service, and generates network state information of the target service application according to each network monitoring data.

Step 320: and the server feeds the network state information back to the front end so that the network state information is visually displayed at the front end.

The method provided by the embodiments of the present description,

Network data are acquired through eBPF technology, data packets are directly processed in a kernel mode, network data to be processed for acquiring network monitoring data in the data packets are extracted and uploaded to a server, the server finds target service application data corresponding to target service applications according to the network data to be processed, the target service application data and the network data to be processed are combined into network detection data, and relevant target service information is enriched for subsequent network state analysis aiming at the target service applications. By adding the network monitoring data with application dimensions, the dimensions of the network monitoring data are enriched, and the richness of the network topology is improved.

Corresponding to the above method embodiments, the present disclosure further provides an embodiment of a data processing system, and fig. 4 shows a schematic structural diagram of a data processing system provided in one embodiment of the present disclosure. As shown in fig. 4, the system includes a client 402 and a server 404:

The client 402 is configured to generate network data to be processed based on eBPF programs, and upload the network data to be processed to the server, where the network data to be processed is used for determining network monitoring data;

The server 404 is configured to obtain target service application data corresponding to the network data to be processed according to the network data to be processed, generate network monitoring data corresponding to the network data to be processed according to the network data to be processed and the target service application data, and store the network monitoring data, where the network monitoring data is used to analyze network state information of a target service application.

Optionally, the client 402 is further configured to:

Generating network data to be processed based on eBPF programs, and storing the network data to be processed in a preset storage position;

Optionally, the client 402 is further configured to:

Optionally, the server 404 is further configured to:

Optionally, the network data to be processed includes application identification information;

the server 404 is further configured to:

Optionally, the server 404 is further configured to:

And storing the network monitoring data into a non-relational database.

Optionally, the server 404 is further configured to:

receiving a network analysis request for the target service application;

Optionally, the server 404 is further configured to:

And visually displaying the network state information.

According to the data processing system provided by the embodiment of the specification, network data are acquired through eBPF technology, data packets are directly processed in a kernel mode, to-be-processed network data used for acquiring network monitoring data in the data packets are extracted and uploaded to the server, the server searches target service application data corresponding to target service applications according to the to-be-processed network data and combines the target service application data with the to-be-processed network data into network detection data, and relevant target service information is enriched for subsequent network state analysis aiming at the target service applications. By adding the network monitoring data with application dimensions, the dimensions of the network monitoring data are enriched, and the richness of the network topology is improved.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for a data processing system, the description is relatively simple, since it is substantially similar to the data processing method embodiment, as relevant to the description of the data processing method embodiment.

Fig. 5 shows a data processing method according to an embodiment of the present disclosure, where the method is applied to a server, and specifically includes the following steps:

step 502: and receiving the network data to be processed sent by the client.

Step 504: and acquiring target business application data corresponding to the network data to be processed according to the network data to be processed.

Step 506: and generating network monitoring data corresponding to the network data to be processed according to the network data to be processed and the target service application data, wherein the network monitoring data is used for analyzing network state information of the target service application.

Step 508: and storing the network monitoring data.

The data processing method provided by the embodiment of the specification is applied to a server, the server receives the network data to be processed sent by the client, obtains the corresponding target service application data according to the network data to be processed, combines the target service application data and the network data to be processed to generate network detection data, and enriches relevant target service information for subsequent network state analysis aiming at the target service application.

Corresponding to the above method embodiments, the present disclosure further provides an embodiment of a data processing system, and fig. 6 shows a schematic structural diagram of a data processing apparatus according to one embodiment of the present disclosure. As shown in fig. 6, the apparatus includes:

A receiving module 602, configured to receive network data to be processed sent by a client;

An obtaining module 604, configured to obtain target service application data corresponding to the network data to be processed according to the network data to be processed;

A generating module 606, configured to generate network monitoring data corresponding to the network data to be processed according to the network data to be processed and the target service application data, where the network monitoring data is used for analyzing network state information of the target service application;

a save module 608 configured to save the network monitoring data.

The data processing device is applied to a server, the server receives network data to be processed sent by a client, obtains corresponding target service application data according to the network data to be processed, combines the target service application data and the network data to be processed to generate network detection data, and enriches relevant target service information for subsequent network state analysis aiming at the target service application.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the data processing apparatus, since it is substantially similar to the data processing method embodiment, the description is relatively simple, and the relevant points are referred to in the description of the data processing method embodiment.

Fig. 7 illustrates a block diagram of a computing device 700 provided in accordance with one embodiment of the present description. The components of computing device 700 include, but are not limited to, memory 710 and processor 720. Processor 720 is coupled to memory 710 via bus 730, and database 750 is used to store data.

Computing device 700 also includes access device 740, access device 740 enabling computing device 700 to communicate via one or more networks 760. Examples of such networks include public switched telephone networks (PSTN, public Switched Telephone Network), local area networks (LAN, local Area Network), wide area networks (WAN, wide Area Network), personal area networks (PAN, personal Area Network), or combinations of communication networks such as the internet. The access device 740 may include one or more of any type of network interface, wired or wireless, such as a network interface card (NIC, network interface controller), such as an IEEE802.11 wireless local area network (WLAN, wireless Local Area Network) wireless interface, a worldwide interoperability for microwave access (Wi-MAX, worldwide Interoperability for Microwave Access) interface, an ethernet interface, a universal serial bus (USB, universal Serial Bus) interface, a cellular network interface, a bluetooth interface, near Field Communication (NFC).

In one embodiment of the present description, the above-described components of computing device 700, as well as other components not shown in FIG. 7, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device illustrated in FIG. 7 is for exemplary purposes only and is not intended to limit the scope of the present description. Those skilled in the art may add or replace other components as desired.

Computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smart phone), wearable computing device (e.g., smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or personal computer (PC, personal Computer). Computing device 700 may also be a mobile or stationary server.

Wherein the processor 720 is configured to execute computer-executable instructions that, when executed by the processor, perform the steps of the data processing method described above.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for computing device embodiments, the description is relatively simple, as it is substantially similar to data processing method embodiments, with reference to the partial description of data processing method embodiments.

An embodiment of the present disclosure also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data processing method described above.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for computer readable storage medium embodiments, since they are substantially similar to data processing method embodiments, the description is relatively simple, and reference is made to the description of data processing method embodiments in part.

An embodiment of the present specification also provides a computer program, wherein the computer program, when executed in a computer, causes the computer to perform the steps of the data processing method described above.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the computer program embodiments, the description is relatively simple, since it is substantially similar to the data processing method embodiments, and reference is made to the description of the data processing method embodiments in part.

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

The computer instructions include computer program code that may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the content of the computer readable medium can be increased or decreased appropriately according to the requirements of the patent practice, for example, in some areas, according to the patent practice, the computer readable medium does not include an electric carrier signal and a telecommunication signal.

It should be noted that the foregoing describes specific embodiments of the present invention. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Further, those skilled in the art will appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily all required for the embodiments described in the specification.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are merely used to help clarify the present specification. Alternative embodiments are not intended to be exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the teaching of the embodiments. The embodiments were chosen and described in order to best explain the principles of the embodiments and the practical application, to thereby enable others skilled in the art to best understand and utilize the invention. This specification is to be limited only by the claims and the full scope and equivalents thereof.

Claims

1. A data processing method applied to a data processing system, the data processing system comprising a client and a server, comprising:

2. The method of claim 1, the client generating pending network data based on eBPF programs and uploading the pending network data to the server, comprising:

3. The method of claim 2, generating network data to be processed based on eBPF programs and saving the network data to be processed in a preset storage location, comprising:

4. The method of claim 3, obtaining initial network data based on eBPF programs in a kernel mode, comprising:

5. The method of claim 1, obtaining target service application data corresponding to the network data to be processed according to the network data to be processed, comprising:

6. The method of claim 5, the network data to be processed comprising application identification information;

determining the target service application corresponding to the network data to be processed comprises the following steps:

7. The method of claim 6, extracting application identification information in the network data to be processed, comprising:

8. The method of claim 5, collecting target business application data corresponding to the target business application, comprising:

9. The method of claim 1, storing the network monitoring data, comprising:

And storing the network monitoring data into a non-relational database.

10. The method of claim 1, further comprising:

receiving a network analysis request for the target service application;

11. The method of claim 10, further comprising:

And visually displaying the network state information.

12. A data processing system comprising a client and a server; wherein,

13. The system of claim 12, the client further configured to:

14. The system of claim 13, the client further configured to:

15. The system of claim 14, the client further configured to:

16. The system of claim 12, the server further configured to:

17. The system of claim 16, the network data to be processed comprising application identification information;

The server is further configured to:

18. The system of claim 17, the server further configured to:

19. The system of claim 16, the server further configured to:

20. The system of claim 12, the server further configured to:

And storing the network monitoring data into a non-relational database.

21. The system of claim 12, the server further configured to:

receiving a network analysis request for the target service application;

22. The system of claim 21, the server further configured to:

And visually displaying the network state information.

23. A data processing method is applied to a server and comprises the following steps:

receiving network data to be processed sent by a client;

And storing the network monitoring data.

24. A computing device, comprising:

a memory and a processor;

The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions which, when executed by the processor, perform the steps of the method of claim 23.

25. A computer readable storage medium storing computer executable instructions which when executed by a processor perform the steps of the method of claim 23.