WO2019137208A1

WO2019137208A1 - Underlying data monitoring method, medium, device and apparatus

Info

Publication number: WO2019137208A1
Application number: PCT/CN2018/123664
Authority: WO
Inventors: 王景春; 杜贤昌; 苗辉
Original assignee: 贵州白山云科技股份有限公司
Priority date: 2018-01-11
Filing date: 2018-12-26
Publication date: 2019-07-18
Also published as: CN109873734A; CN109873734B

Abstract

Disclosed are an underlying data monitoring method, a medium, a device and an apparatus. The method comprises: identifying a connection establishment request initiated by a user terminal to an edge server; after a connection corresponding to the connection establishment request is established successfully, collecting data packet transmission parameter data on the connection; sending the collected data packet transmission parameter data on the connection to a processing module of an application layer; and analyzing the collected data packet transmission parameter data on the application layer. According to the disclosure, a request can be divided according to user's requirements so as to perform data collection and analysis, and entire analysis processing is carried out on an edge server, so that the processing rate, timeliness and promptness can be effectively improved. In the disclosure, a distributed computation principle is used, and data processing is distributed to all devices that require data collection, so that the requirements for the computation performance of a central processing server are reduced, and thus the architecture layout is more flexible.

Description

Underlying data monitoring method, medium, device and device

The present application claims priority to Chinese Patent Application No. 201810026584.8, filed on Jan. 11, 2018, the entire disclosure of which is incorporated herein by reference. Combined in this application.

Technical field

Embodiments of the present invention relate to, but are not limited to, the field of Internet technologies, and in particular, to an underlying data monitoring method, medium, device, and device.

Background technique

As the application of the network becomes more and more extensive, the scale also increases, and the business content carried in the network is also more and more abundant. In order to provide better network services for network users and optimize network performance, the detection and configuration of various network parameters are also increasingly valued, with the most typical monitoring of network traffic. Network traffic indicates the amount of data transmitted on the network. Through the detection and analysis of network traffic, it helps network administrators to provide network rules, network optimization, network monitoring, traffic trend analysis, etc., to provide normal, stable and reliable operation of the network. Guarantee. Moreover, by timely grasping the characteristics of network traffic, it is possible to optimize network bandwidth configuration and solve network performance problems in time.

At present, on the content distribution network, the existing database that supports the S/C mode can connect the server to the client, configure the network traffic rules in the database, change the network traffic detection rules, and detect the need. The network packet is detected, the network traffic is obtained, and the actual demand for detecting the network traffic is realized, and the flexible configuration of the network traffic detection rule is realized.

Most of the existing customer service monitoring is based on user access log generation, and monitoring through network logs has the following disadvantages:

1. The generation time of the record is lagging; since the log is recorded in a data packet or transmitted according to the size of the file, the recording time will be delayed.

2. The recorded data is inaccurate; since the log is mainly implemented on the application layer, there is no network transmission involved, but the network transmission has delays, jitters, etc., so the recorded data is not accurate enough.

3. Monitoring through the log There is no way to record for a single request; the method used in the prior art is to perform one record for one request, and it is not possible to refine a request corresponding to multiple records for statistics.

Summary of the invention

The following is an overview of the topics detailed in this document. This Summary is not intended to limit the scope of the claims.

The underlying data monitoring methods provided in this article include:

Identifying a connection establishment request initiated by the user end to the edge server; after the connection establishment establishing the connection request is successful, collecting the data packet transmission parameter data on the connection; and transmitting the data packet transmission parameter data collected on the connection to the application layer processing module ;

The collected data transmission parameter data is analyzed at the application layer.

The above underlying data monitoring method also has the following characteristics:

The data packet transmission parameters on the collection connection include: collecting data packet transmission parameters in a preset collection period or collecting data packet transmission parameters in a process of transmitting a preset data amount;

The data packet transmission parameter includes at least one of the following parameters: the speed at which the client receives the data, the speed at which the connected server transmits the data, the amount of data received by the client, the amount of data sent by the connected server, and the round trip of the data packet. The amount of data that delays and retransmits data.

The data packet transmission parameters on the collection connection include: setting a transmission parameter hook function in the connected data response function, and after receiving the data response message, calling the transmission parameter hook function to obtain the corresponding transmission parameter.

After the method for establishing a connection initiated by the client to the edge server, the method further includes: recording the client address and/or the target address, obtaining the domain name corresponding to the connection, and recording the client address and the connection after the connection establishment request is established. And the access target address corresponds to the corresponding relationship of the domain name, and the data transmission parameter data collected on the connection is sent to the application layer processing module, and the corresponding relationship is sent to the application layer processing module;

The analysis of the collected data packet transmission parameter data at the application layer includes: analyzing the collected data packet transmission parameter data according to the correspondence between the client address and the domain name.

After the connection established by the connection request is successfully established, the method further includes: collecting the connection data, and performing data aggregation on the connection data and the data transmission parameter data.

The computer readable storage medium provided by the embodiment of the present invention stores a computer program, and when the program is executed by the processor, the steps of the foregoing method are implemented.

A computer device provided by an embodiment of the present invention includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the steps of the foregoing method.

The underlying data monitoring apparatus provided by the embodiment of the present invention includes: an underlying processing module and an application layer processing module; the bottom layer processing module includes an identification module, an acquisition module, and a sending module; and the application layer processing module includes an analysis module;

An identification module, configured to identify a connection establishment request initiated by the user end to the edge server;

The acquiring module is configured to collect data packet transmission parameter data on the connection after the connection established by the connection request is successfully established;

a sending module, configured to send the data packet transmission parameter data collected on the connection to the application layer processing module;

The analysis module is configured to analyze the collected data transmission parameter data.

The above-mentioned underlying data monitoring device also has the following features:

The acquisition module is further configured to collect data transmission parameter data on the connection by using the following method: collecting data packet transmission parameters in a preset collection period or collecting data packet transmission parameters in a process of transmitting a preset data amount;

The acquisition module is also used to collect the data packet transmission parameter data on the connection by setting the transmission parameter hook function in the connected data response function, and after receiving the data response message, calling the transmission parameter hook function to obtain the corresponding transmission parameter.

The device further includes a recording module, configured to record the client address and/or the access target address, obtain the domain name corresponding to the connection after the connection established by the connection request is successfully established, and record the client address and/or the target address corresponding to the domain name. Correspondence relationship

a sending module, configured to send the data transmission parameter data collected on the connection to the application layer processing module, and send the corresponding relationship to the application layer processing module;

The analysis module is configured to analyze the collected data transmission parameter data according to the correspondence between the client address and the domain name.

The acquisition module is also used to collect the connection data;

The apparatus further includes an aggregation module for performing data aggregation on the connection data and the data transmission parameter data.

Advantages of embodiments of the invention include:

For the shortcomings of the prior art, all the data needs to wait for the user to complete the request and write the log and the log is collected to the central data processing server for analysis. The embodiment of the present invention can perform the data cutting according to the user requirements for the request. And analysis, all analysis and processing at the edge server can effectively improve processing speed, timeliness and timeliness.

Second, the principle of distributed computing is used to spread the data processing to all devices that need to collect data, which reduces the computational performance requirements of the central processing server and makes the architecture layout more flexible.

DRAWINGS

The accompanying drawings are intended to provide a further understanding of the embodiments of the embodiments of the invention Improper limitations. In the drawing:

1 is a flow chart of an underlying data monitoring method in an embodiment;

2 is a structural diagram of an underlying data monitoring apparatus in the embodiment.

Detailed ways

The embodiments of the present invention will be further described with reference to the drawings and specific embodiments.

1 is a flow chart of an underlying data monitoring method in an embodiment, the method comprising:

Step 1: Identify the connection establishment request initiated by the user end to the edge server; after the connection establishment corresponding to the connection establishment request is successfully established, collect the data packet transmission parameter data on the connection; and send the data transmission parameter data collected on the connection to the application. Layer processing module;

In step 2, the collected data transmission parameter data is analyzed at the application layer.

The bottom layer of the method is relative to the application layer. In step 1, the following steps are performed on the bottom layer: identifying the connection establishment request initiated by the user end to the edge server; after the connection establishment establishing the connection request is successful, collecting the connection The data packet is transmitted by the parameter data. The concept of the underlying layer includes the concept of the kernel, and the step of executing the above steps in the kernel in step 1 is a more preferable method.

The data packet transmission parameter on the collection connection in step 1 includes: collecting data packet transmission parameters in a preset collection period (for example, 10 seconds or 1 minute) or collecting and transmitting a preset data amount (for example, 100 KB or 1 MB). The data packet transmission parameter includes at least one of the following parameters: a speed at which the client receives the data, a speed at which the connected server transmits the data, a data amount received by the client, and a data sent by the connected server. The amount of data, the round trip delay of the packet, and the amount of data to retransmit the data.

Specifically, the data packet transmission parameter on the collection connection includes: setting a transmission parameter hook function in the connected data response function, and after receiving the data response message, calling the transmission parameter hook function to obtain the corresponding transmission parameter.

The method further includes: after identifying the connection establishment request initiated by the client to the edge server in step 1, recording the client address (client IP address and/or client port address) and/or the access destination address (destination IP address and/or Or the destination port address), after the connection established by the connection request is successfully established, obtain the domain name corresponding to the connection, record the correspondence between the client address and/or the target address corresponding to the domain name, and collect the data packet transmission parameter on the connection. The data is sent to the application layer processing module and the corresponding relationship is sent to the application layer processing module; in step 2, the collected data packet transmission parameter data is analyzed according to the corresponding relationship. In this method, after the client address, the access destination address, and the domain name are recorded, the same client address can also be used to confirm when the same connection establishment request is initiated again.

When analyzing the collected data transmission parameter data, the data in a connection request may be separately analyzed, and the data in all the connections initiated by the client address may be analyzed for the client address, or the domain name may be analyzed for the domain name. The data in all connections for accessing the target.

When sending data to the application layer processing module in step 1, it can be implemented by using netlink software.

In step 1, the information from the receipt of the connection establishment request to the successful connection establishment is also recorded, and the recorded information and the collected data packet transmission parameters are sent to the application layer processing module, so that the application layer processing module is configured to establish the connection process. The information is analyzed separately and the packet transmission parameters are analyzed separately after the connection is established.

In step 2, all the collected data is stored, which can be stored locally or stored in the cloud.

After step 2, an optional step is further included: after the connection established by the connection request is successfully established, the connection data is collected, and the data is aggregated for the connection data and the data transmission parameter data. Jianlian data includes connection identification and connection time.

In the method, after the analysis is completed, the display analysis results show that the relevant personnel judge the quality of service according to the analysis result and perform subsequent processing.

2 is a structural diagram of an underlying data monitoring apparatus in an embodiment, the apparatus includes: an underlying processing module and an application layer processing module; the bottom processing module includes an identification module, an acquisition module, and a sending module; and the application layer processing module includes an analysis module;

among them,

The collection module is further configured to collect data packet transmission parameter data on the connection by collecting data packet transmission parameters in a preset collection period or collecting data packet transmission parameters in a process of transmitting the preset data volume. The data packet transmission parameter includes at least one of the following parameters: a speed at which the client receives the data, a speed at which the connected server transmits the data, an amount of data received by the client, and a data amount of the connected server transmitting the data. , data packet round-trip delay, data volume of retransmitted data.

The acquisition module is further configured to collect data packet transmission parameter data on the connection by: setting a transmission parameter hook function in the connected data response function, and calling the transmission parameter hook function after receiving the data response message Get the corresponding transmission parameters.

The device further includes a recording module for recording the client address and/or the access target address, and obtaining the domain name corresponding to the connection after the connection corresponding to the establishment of the connection request is successfully established, and recording the client address and/or the access target address corresponding to the domain name. Correspondence relationship. The sending module is configured to send the data transmission parameter data collected on the connection to the application layer processing module, and send the correspondence to the application layer processing module. The analysis module is configured to analyze the collected data transmission parameter data according to the correspondence between the client address and the domain name.

The acquisition module of the device is further configured to collect connection data; the device further includes an aggregation module, configured to perform data aggregation on the connection data and the data transmission parameter data, thereby reducing the amount of final storage data.

Advantages of embodiments of the invention include:

A person skilled in the art should understand that the technical solutions of the embodiments of the present invention may be modified or equivalent, without departing from the spirit and scope of the present invention, and should be included in the scope of the claims.

Those of ordinary skill in the art will appreciate that all or some of the steps, systems, and functional blocks/units of the methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be composed of several physical The components work together. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer readable medium, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and nonvolatile, implemented in any method or technology for storing information, such as computer readable instructions, data structures, program modules or other data. Sex, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or may Any other medium used to store the desired information and that can be accessed by the computer. Moreover, it is well known to those skilled in the art that communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and can include any information delivery media. .

Industrial applicability

In this paper, all the data in the prior art needs to wait for the user to complete the request and write the log and the log is collected to the central data processing server for analysis. The data can be cut and analyzed according to the user's requirements. The edge server performs all analysis and processing, which can effectively improve the processing speed, timeliness and timeliness; and utilizes the principle of distributed computing to spread the data processing to all devices that need to collect data, reducing the computational performance requirements for the central processing server. To make the architecture layout more flexible.

Claims

An underlying data monitoring method, including:

Identifying a connection establishment request initiated by the user end to the edge server; after the connection establishment corresponding to the establishment connection request is successful, collecting data packet transmission parameter data on the connection; collecting data packet transmission parameter data on the connection Send to the application layer processing module;

The collected data packet transmission parameter data is analyzed at the application layer.
The underlying data monitoring method according to claim 1, wherein

The collecting the data packet transmission parameter on the connection includes: collecting a data packet transmission parameter in a preset collection period or collecting a data packet transmission parameter in a process of transmitting the preset data volume;

The data packet transmission parameter includes at least one of the following parameters: a speed at which the client receives the data, a speed at which the connected server transmits the data, a data amount at which the client receives the data, and the connected server sends the data. The amount of data, the round trip delay of the packet, and the amount of data to retransmit the data.
The underlying data monitoring method according to claim 1, wherein

The collecting the data packet transmission parameter on the connection comprises: setting a transmission parameter hook function in the connected data response function, and after receiving the data response message, calling the transmission parameter hook function to obtain a corresponding transmission parameter.
The underlying data monitoring method according to claim 1, wherein

After the method of establishing a connection initiated by the user to the edge server, the method further includes: recording a client address and/or a target address, and obtaining a domain name corresponding to the connection after the connection establishment of the connection request is successful. Recording a correspondence between the client address and/or the access target address corresponding to the domain name, and transmitting the data packet transmission parameter data collected on the connection to the application layer processing module, and sending the corresponding relationship to the application layer processing module ;

The analyzing, by the application layer, the collected data packet transmission parameter data comprises: analyzing the collected data packet transmission parameter data according to a correspondence between a client address and the domain name.
The underlying data monitoring method according to claim 1, wherein

After the connection establishment of the connection establishment request is successful, the method further includes: collecting connection data, and performing data aggregation on the connection data and the data transmission parameter data.
A computer readable storage medium having stored thereon a computer program, the program being executed by a processor to perform the steps of the method of any one of claims 1 to 5.
A computer device comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, the processor executing the program to implement any one of claims 1 to 5 The steps of the method.
An underlying data monitoring device includes: an underlying processing module and an application layer processing module; the bottom processing module includes an identification module, an acquisition module, and a sending module; and the application layer processing module includes an analysis module;

The identification module is configured to identify a connection establishment request initiated by the user end to the edge server;

The collecting module is configured to collect data packet transmission parameter data on the connection after the connection establishment of the connection establishment request is successful;

The sending module is configured to send data packet transmission parameter data collected on the connection to an application layer processing module;

The analyzing module is configured to analyze the collected data packet transmission parameter data.
The underlying data monitoring device according to claim 8, wherein

The collecting module is further configured to collect data packet transmission parameter data on the connection by using the following method: collecting a data packet transmission parameter in a preset collection period or collecting a data packet transmission parameter in a process of transmitting a preset data amount;

The data packet transmission parameter includes at least one of the following parameters: a speed at which the client receives the data, a speed at which the connected server transmits the data, a data amount at which the client receives the data, and the connected server sends the data. The amount of data, the round trip delay of the packet, and the amount of data to retransmit the data.
The underlying data monitoring device according to claim 8, wherein

The collecting module is further configured to collect data packet transmission parameter data on the connection by: setting a transmission parameter hook function in the connected data response function, and calling the transmission after receiving the data response message The parameter hook function gets the corresponding transmission parameters.
The underlying data monitoring device according to claim 8, wherein

The device further includes a recording module, configured to record a client address and/or an access target address, obtain a domain name corresponding to the connection after the connection establishment connection corresponding to the connection establishment request is successful, and record a client address and/or a target address. Corresponding to the correspondence of domain names;

The sending module is configured to send the data transmission parameter data collected on the connection to the application layer processing module, and send the corresponding relationship to the application layer processing module;

The analyzing module is configured to analyze the collected data packet transmission parameter data according to the correspondence between the client address and the domain name.
The underlying data monitoring device according to claim 8, wherein

The collection module is further configured to collect connection data;

The apparatus further includes an aggregation module configured to perform data aggregation on the connection data and the data transmission parameter data.