CN115701049A

CN115701049A - Communication delay estimation method and related device

Info

Publication number: CN115701049A
Application number: CN202110864757.5A
Authority: CN
Inventors: 徐明伟; 席照炜; 王旸旸; 周禹; 付洁; 缪丽华
Original assignee: Tsinghua University; Huawei Technologies Co Ltd
Current assignee: Tsinghua University; Huawei Technologies Co Ltd
Priority date: 2021-07-29
Filing date: 2021-07-29
Publication date: 2023-02-07

Abstract

The embodiment of the application provides a communication time delay estimation method and a related device, wherein the method comprises the following steps: the network equipment acquires flow information from the received first data packet, wherein the flow information is identification information of a first flow to which the first data packet belongs; then, determining a first recording unit of the first stream map based on the stream information; the first recording unit comprises the maximum time delay of a single data packet of the target stream passing through the target equipment and the number of the data packets of the target stream received by the network equipment; the target stream includes a stream mapped to the first recording unit; then, updating the number of the packets, and updating the first time delay to the time delay of the first data packet passing through the target equipment under the condition that the time delay of the first data packet passing through the target equipment is larger than the first time delay; in the case where the number of packets is less than the threshold, the first delay is used to estimate a tail delay for the first stream to pass through the target device. The method and the device can save storage and processing resources of each flow, so that tail time delay estimation of the full-flow can be performed on the existing resources.

Description

Communication delay estimation method and related device

Technical Field

The present invention relates to the field of communication network measurement technologies, and in particular, to a communication delay estimation method and a related apparatus.

Background

Network performance detection has always been an important sub-issue in the network field of network measurement, which has great application value. Among them, the measurement of time delay is always a hot point of academic research, and is also a field in which the industry continuously pursues innovation. Conventional tools for measuring end-to-end delay using sampling or sending probes (e.g., internet packet explorer ping, traceroute) have been deployed and applied in large numbers and have played a significant role for a long period of time. Currently, with the rapid expansion of network scale and the continuous improvement of network device capability, users and products put more delicate requirements on network operation and maintenance. Network administrators want to improve the ability of the network to quickly detect and locate anomalies, such as quickly discovering distributed denial of service (DDos) attacks, or being able to detect potential micro-bursts (micro-bursts) in a data center, etc. These more challenging problems rely on more fine-grained measurements of the flow performance metrics, such as the tail delay of the flow.

The measurement techniques for tail delay of a stream commonly used in the current industry can be mainly classified into two types: one type is an in-network measurement technology represented by network flow NetFlow, which supports screening, sampling and copying related message data in network equipment, and then uploading the message data to a controller for tail time delay measurement; the other is an in-band operation administration and maintenance (iOAM) technology represented by an in-band network telemetry (INT) technology. Such techniques encapsulate required operations, administration and maintenance (OAM) information (e.g., measurement data of delay) directly in packets, and forward OAM data removal to the controller for tail delay measurement at the last hop of the monitoring domain, i.e., the edge node. However, the NetFlow technology supports the network device to acquire the relevant information of the data packet and send the information to the controller, but brings large extra load and resource consumption, so that sampling or flow selection is often required in actual use, and tail delay measurement of all flows passing through the network device is difficult to cover; the iOAM technology represented by INT carries required information at the head of a data packet, and uploads collected information to a controller at the last hop, which reduces the uploading cost, but also occupies a link bandwidth, causes phenomena such as increase of flow completion time and throughput reduction, and therefore, the flow cannot be deployed packet by packet, which also causes difficulty in implementing tail delay measurement of all flows.

In summary, how to reduce the resource overhead of the network device in the tail delay measurement process of each flow is a technical problem that needs to be solved urgently by those skilled in the art.

Disclosure of Invention

The embodiment of the application provides a communication delay estimation method and a related device, which can reduce the resource overhead of network equipment in the tail delay measurement process of each flow, so that the network equipment can measure and estimate the tail delays of all flows passing through the network equipment based on the existing storage and processing resources.

In a first aspect, the present application provides a communication delay estimation method, including: the network equipment acquires flow information from a received first data packet, wherein the flow information is identification information of a first flow to which the first data packet belongs; the network device determines a first recording unit of the first stream map based on the stream information; the first recording unit comprises a first time delay and a packet number, wherein the first time delay is the maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is the number of the data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit; the network equipment updates the number of the packets, and updates the first time delay to the time delay of the first data packet passing through the target equipment under the condition that the time delay of the first data packet passing through the target equipment is larger than the first time delay; in the case where the number of packets is less than a threshold, the first delay is used to estimate a tail delay of the first stream through the target device.

Optionally, the target device is the network device; alternatively, the network device is a device on a data plane in a network, the network device is a last hop device of the target stream in the data plane, and the target device includes a plurality of devices through which the single data packet passes in the data plane.

In this embodiment of the present application, the network device is any network device in a data plane for forwarding data, and the first recording unit is configured for each stream received by the network device in the network device to record a maximum delay of a data packet in each stream passing through the target device, so as to estimate a tail delay of each stream passing through the target device when the number of packets of each stream is less than a threshold. In the embodiment of the present application, each flow is mapped to a corresponding first recording unit, and the first recording unit only records the maximum time delay of a data packet in the flow passing through the target device, instead of recording the time delays of all data packets, so that a storage resource is saved. In addition, in the embodiment of the present application, resources required for measuring and estimating the tail delay of a single stream are saved, so that it is not necessary to perform screening and sampling on the stream during the measurement of the tail delay of the stream, that is, compared with the existing scheme, the present application can measure and estimate the tail delays of all streams passing through the network device based on the existing storage and processing resources of the network device.

In a possible implementation, the network device is a device on a data plane in a network, the network device is a last hop device of a packet of the target stream in the data plane, and the target device includes multiple devices through which the single packet passes in the data plane; the time delay of the first data packet passing through the target device is the time delay of the first data packet passing through the plurality of devices; the first data packet includes a delay of the first data packet through other devices, which are other network devices except the network device among the plurality of devices.

In this embodiment of the present application, a delay obtaining module may be disposed on all network devices through which the first stream passes in the data plane, and configured to obtain a delay of each data packet of the first stream through each network device, and then carry the delays in the data packet and send the data packet to the device of the last hop, where the device of the last hop also obtains a delay of each data packet through itself, then counts a total delay of each data packet through all network devices, and records a maximum delay in the first recording unit. The implementation mode can make the recording and processing of information such as time delay and the like only need to be carried out on the last hop device, and each network device through which the first stream passes does not need to have a record, thereby saving the storage and processing resources of the network devices.

In a possible implementation, the network device is a device on a data plane in a network, the network device is a last hop device of a packet of the target stream in the data plane, and the target device includes multiple devices through which the single packet passes in the data plane; the time delay of the first data packet passing through the target device is the time delay of the first data packet passing through the plurality of devices; the first data packet includes a timestamp of an entry of a first hop device that the first data packet passes through in the data plane, and the timestamp is used for calculating a time delay of the first data packet passing through the plurality of devices.

In this embodiment of the present application, an entry timestamp of a data packet of a stream may be obtained on a first device through which the first stream passes in a data plane, and the timestamp is carried in the data packet and sent to the last-hop device, after the last-hop device receives the data packet, a timestamp sent by the data packet from the last-hop device may be obtained, where a time delay of the data packet passing through the target device is obtained by subtracting a previous timestamp from a subsequent timestamp, and then the maximum time delay of the data packet passing through the target device in the stream is recorded in the first recording unit. Similarly, the implementation manner can make it only necessary to record and process information such as time delay at the last hop device, and it is not necessary for each network device that the first stream passes through to have a record, so that storage and processing resources of the network devices can be saved.

In a possible implementation, the data packet of the target stream received by the network device is a data packet received by the network device within an estimation window, where the estimation window is a time window used for estimating a tail delay of the first stream passing through the target device; the first delay is the maximum delay of a single data packet of the target stream passing through the target device, and comprises the following steps: the first time delay is the maximum time delay of a single data packet of the target stream passing through the target device in the time window; the tail delay of the first stream passing through the target device is the tail delay of the first stream passing through the target device within the time window.

In the embodiment of the application, by dividing the transmission process of the stream into one or more time windows and estimating the tail delay of the stream in each time window, the implementability and operability of the network device can be improved.

In a possible embodiment, the first stream map has a plurality of first recording units; the method further comprises the following steps: after the estimation window is ended, under the condition that the number of the packets recorded by at least one unit in the plurality of first recording units is smaller than the threshold value, the network device determines the first time delay recorded in the first unit as a tail time delay estimation value of the first stream passing through a target device in the estimation window, and sends the first time delay recorded in the first unit to a controller; the first unit is a unit with the minimum first time delay recorded in the plurality of first recording units.

In this embodiment of the present application, when a data packet of the target stream received by the network device within the estimation window is smaller than a threshold, the minimum first time delay recorded in the plurality of first recording units may be used to estimate a tail time delay of the first stream passing through the target device, which is simple to implement and has strong operability. This is because in the case where the number of packets is small, the difference between the tail delay and the maximum delay of the data packet through the destination device recorded in the first recording unit is small, so that the maximum delay can be used to estimate the tail delay of the stream.

In one possible embodiment, the first stream map includes a plurality of first recording units; the method further comprises the following steps: in a case where the estimation window ends, the network device transmits information recorded in the plurality of first recording units to the controller.

In this embodiment, the network device may send the information recorded by the first recording unit to the controller, and the controller performs estimation calculation on the tail delay, thereby reducing the processing load of the network device.

In a possible implementation, in a case that the number of packets recorded in the first recording unit is greater than or equal to the threshold, the method further includes: the network equipment allocates a second recording unit for the first recording unit; the second recording unit is configured to record a time delay of each of n data packets passing through the destination device, where the n data packets are data packets in a data packet of the destination stream received by the network device after the first data packet, and n is an integer greater than 1; the n delays are used to estimate a tail delay of the first stream through the target device.

In this embodiment of the present application, when the number of packets is greater than the threshold, the tail delay is different from the maximum delay of the data packet recorded in the first recording unit through the target device by a relatively large amount, so that a delay needs to be recorded additionally to estimate the tail delay of the stream, so that the second recording unit is configured to record the delay of a part of the data packet of the subsequent received stream through the target device, and then estimate the tail delay of the stream based on the delay information recorded by the second recording unit, so as to improve the accuracy of estimation.

In one possible embodiment, the method further comprises: the network equipment receives a second data packet, wherein the second data packet is a data packet of the target stream; the network equipment acquires a second time delay of the n time delays of the second recording unit according to a polling mode; and under the condition that the time delay of the second data packet passing through the target device is greater than the second time delay, the network device updates the second time delay in the second recording unit to the time delay of the second data packet passing through the target device.

In this embodiment of the present application, the time delay in the second recording unit may be updated in a polling manner, so as to ensure that the n time delays are the maximum n time delays.

In one possible embodiment, the method further comprises: the network device receives a third data packet, where the third data packet is a data packet of the target stream, and before the third data packet and after the second data packet, the network device receives m data packets of the target stream, where m is an integer greater than 1; the network equipment randomly acquires a third time delay from the n time delays of the second recording unit; and under the condition that the time delay of the third data packet passing through the target device is less than the third time delay, the network device updates the third time delay in the second recording unit to the time delay of the third data packet passing through the target device.

When the number of packets of a stream received in the estimation window is very large, there is also a large error in estimating the tail delay of the stream using the maximum n delays. For example, if the number of the packets reaches 10000, n is 10, and if the 100 delays are the largest 10 delays, when estimating the 99% fractional tail delay, the 9900 th delays, which are sorted from small to large, of the delays of the 10000 data packets passing through the target device are the accurate 99% fractional tail delay, and each delay of the 10 largest delays is greatly different from the 9900 th delay, so that the 10 delays cannot be used to estimate the 99% fractional tail delay. Therefore, in the embodiment of the present application, in the process of updating the second recording unit, on the basis of the polling update mode, one of the n time delays in the second recording unit is randomly selected to be updated to a smaller time delay every certain number of data packets, so that the n time delays recorded in the second recording unit are no longer the maximum time delay, but time delay information of other sub-bits as many as possible is recorded in the second recording unit according to a certain probability, thereby reducing an estimation error of the tail time delay.

In one possible embodiment, the method further comprises: in case the estimation window ends, the network device estimates a tail delay of the first stream through the target device within the estimation window based on the n delays in the second recording unit.

Optionally, the tail delay of the first stream passing through the target device is p fractional tail delay, where p is greater than 80 and less than 100; the network device estimates, based on the n delays in the second recording unit, a tail delay of the first stream passing through the target device, including: and the network equipment determines the w-th time delay of the n time delays as a tail time delay estimated value of the first stream passing through the target equipment in the estimation window, wherein the w-th time delay is the time delay ordered from small to large as w, and the w is an integer which is greater than 0 and less than or equal to n.

In this embodiment of the application, based on the manner of updating the second recording unit, the p-fractional-bit tail delay may be estimated by using a delay of n delays in the second recording unit, and w may be determined by calculating a mathematical expectation of the number of delays greater than the p-fractional-bit tail delay in the n delays.

In one possible embodiment, the method further comprises: in a case where the estimation window ends, the network device transmits information recorded in the first recording unit and the second recording unit to a controller.

In this embodiment, the network device may send the information recorded by the first recording unit and the second recording unit to the controller, and the controller performs estimation calculation of the tail delay, thereby reducing the processing load of the network device.

In one possible embodiment, the network device determining the first recording unit of the first stream map based on the stream information includes: the network device maps the flow information to a plurality of bits in a bitmap through a plurality of hash functions; one bit in the bitmap is associated with one first recording unit; the network device determines the first recording unit of the first stream map based on the plurality of bits.

In the embodiment of the application, the flow is mapped to the corresponding first recording unit through hash calculation, so that the calculation is simple, the processing is convenient, and the storage resource can be saved.

In a possible implementation manner, the first recording unit further includes an xor result of the stream information and a number of streams, where the number of streams is the number of the target streams, and the xor result of the stream information is a result obtained by performing an xor operation on the stream information of the target stream.

In the embodiment of the present application, the xor result of the number of streams and the stream information is recorded in the first recording unit, which may be used to mark the recording unit in which the hash collision occurs, and may be used to subsequently decode the stream information in the hash collision.

In a second aspect, the present application provides a communication delay estimation method, including: the network equipment acquires flow information from the received first data packet, wherein the flow information is used for indicating that the first data packet is a data packet in a first flow; the network device determining a first recording unit of the first stream map based on the stream information; the first recording unit is configured to record a first time delay and a packet number, where the first time delay is a maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is a number of data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit; the network equipment updates the number of the packets, and updates the first time delay to the time delay of the first data packet passing through the target equipment under the condition that the time delay of the first data packet passing through the target equipment is larger than the first time delay; the network device allocates a second recording unit to the first recording unit when the number of packets is greater than or equal to a threshold; the second recording unit is configured to record a time delay of each of n data packets passing through the target device, where the n data packets are partial data packets in a data packet of the target stream received by the network device after the first data packet, and n is an integer greater than 1; the n delays are used to estimate the tail delay of the first stream through the target device.

In a third aspect, the present application provides a communication delay estimation method, including: the method comprises the steps that a controller receives information recorded in a plurality of first recording units, wherein the information recorded in each first recording unit comprises a first time delay and a packet number, the first time delay is the maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is the number of the data packets of the target stream received by a network device; the target stream includes a stream mapped to the first recording unit, the target stream including a first stream; in the case that the number of packets is smaller than a threshold value, the controller estimates a tail delay of the first stream passing through the target device based on the first delays recorded in the plurality of first recording units.

In this embodiment of the application, when the controller performs the estimation calculation of the tail delay, and when a data packet of the target stream received by the network device within the estimation window is smaller than a threshold, the first delays recorded in the plurality of first recording units may be used to estimate the tail delay of the first stream passing through the target device, which is simple to implement and has strong operability. This is because in the case where the number of packets is small, the difference between the tail delay and the maximum delay of the data packet through the destination device recorded in the first recording unit is small, so that the maximum delay can be used to estimate the tail delay of the stream.

In one possible embodiment, the method further comprises: the controller further receives an identifier of an estimation window indicating that information recorded in the plurality of first recording units is recorded within the estimation window, the estimation window being a time window for estimating a tail delay of the first stream passing the target device; the controller estimates a tail delay of the first stream passing through the target device based on the first delays recorded in the plurality of first recording units, including: the controller determines the first delay recorded in the first unit as an estimate of the tail delay of the first stream through the target device within the estimation window; the first unit is a unit with the minimum first time delay recorded in the plurality of first recording units.

In the embodiment of the application, by dividing the transmission process of the stream into one or more time windows and estimating the tail delay of the stream in each time window, the implementability and operability of the network device can be improved. On this basis, the minimum first time delay recorded in the plurality of first recording units can be used for estimating the tail time delay of the first stream passing through the target device, because if the first time delays in the plurality of first recording units are different, it is indicated that hash collision occurs in a recording unit, and only the time delay larger than the recorded first time delay can be updated into the first recording unit, so that the minimum first time delay in the plurality of first recording units of the first stream mapping is a more accurate tail time delay estimation of the first stream, and the estimation accuracy can be improved by adopting the minimum first time delay to estimate the tail time delay of the first stream.

In a fourth aspect, the present application provides a communication delay estimation method, including: the controller receives information recorded in a first recording unit and a second recording unit, wherein the information recorded in the first recording unit comprises a first time delay and a packet number, the first time delay is the maximum time delay of a single data packet of a first stream passing through a target device, and the packet number is the number of the data packets of the first stream received by the network device; the second recording unit is associated with the first recording unit, the information recorded in the second recording unit includes a time delay of each of n data packets in the first stream passing through the target device, and n is an integer greater than 1; the controller estimates a tail delay of the first stream through the target device based on the n delays.

In this embodiment of the present application, when the number of packets of the target stream received by the network device in the estimation window is greater than the threshold, the recorded information may also be sent to the controller, and the estimation calculation of the tail delay is performed in the controller.

In one possible embodiment, the method further comprises: the controller further receives an identifier of an estimation window indicating that information recorded in the first recording unit and the second recording unit is recorded within the estimation window, the estimation window being a time window for estimating a tail delay of the first stream passing the target device;

the tail delay of the first stream passing through the target device is p fractional tail delay, and p is more than 80% and less than 100%; the controller estimates a tail delay of the first stream through the target device based on the n delays, including:

the controller determines a w-th time delay of the n time delays as an estimated tail time delay value of the first stream passing through the target device in the estimation window, wherein the w-th time delay is a time delay ordered from small to large and is a time delay ordered from w to n, and the w is an integer greater than 0 and less than or equal to n.

In this embodiment, the p decibitwise tail delay may be estimated by using a delay of n delays in the second recording unit, and the w may be determined by calculating a mathematical expectation of the number of delays greater than the p decibitwise tail delay of the n delays.

In a fifth aspect, the present application provides a network device, including a processing unit configured to:

acquiring flow information from a received first data packet, wherein the flow information is identification information of a first flow to which the first data packet belongs;

determining a first recording unit of the first stream map based on the stream information; the first recording unit comprises a first time delay and a packet number, wherein the first time delay is the maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is the number of the data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit;

updating the number of the packets, and updating the first time delay to the time delay of the first data packet passing through the target device under the condition that the time delay of the first data packet passing through the target device is larger than the first time delay; in the case where the number of packets is less than a threshold, the first delay is used to estimate a tail delay of the first stream through the target device.

In a possible implementation, the network device is a device on a data plane in a network, the network device is a last-hop device of a packet of the target stream in the data plane, and the target device includes multiple devices through which the single packet passes in the data plane; the time delay of the first data packet passing through the target device is the time delay of the first data packet passing through the plurality of devices; the first data packet includes a delay of the first data packet through other devices, which are other network devices except the network device among the plurality of devices.

In a possible implementation, the network device is a device on a data plane in a network, the network device is a last hop device of a packet of the target stream in the data plane, and the target device includes multiple devices through which the single packet passes in the data plane; the time delay of the first data packet passing through the target device is the time delay of the first data packet passing through the plurality of devices; the first data packet includes a timestamp of an entry of a first hop device that the first data packet passes through in the data plane, and the timestamp is used for calculating a time delay of the first data packet passing through the multiple devices.

In a possible implementation, the data packet of the target stream received by the network device is a data packet received by the network device within an estimation window, where the estimation window is a time window used for estimating a tail delay of the first stream passing through the target device; the first delay is the maximum delay of a single data packet of the target stream passing through the target device, and comprises the following steps: the first delay is the maximum delay of a single data packet of the target stream passing through the target device within the time window; the tail delay of the first stream passing through the target device is the tail delay of the first stream passing through the target device within the time window.

In a possible embodiment, the first recording unit of the first stream map is plural; the processing unit is further configured to: after the estimation window is ended, under the condition that the number of the packets recorded by at least one unit in the plurality of first recording units is smaller than the threshold value, determining that the first time delay recorded in the first unit is a tail time delay estimation value of the first stream passing through a target device in the estimation window, and sending the first time delay recorded in the first unit to a controller; the first unit is a unit with the smallest first time delay recorded in the plurality of first recording units.

In one possible embodiment, the first stream map includes a plurality of first recording units; the network device further includes a transmitting unit configured to transmit information recorded in the plurality of first recording units to the controller in a case where the estimation window ends.

In a possible implementation manner, in a case that the number of packets recorded in the first recording unit is greater than or equal to the threshold, the processing unit is further configured to:

allocating a second recording unit to the first recording unit; the second recording unit is configured to record a time delay of each of n data packets passing through the target device, where the n data packets are data packets in a data packet of the target stream received by the network device after the first data packet, and n is an integer greater than 1; the n delays are used to estimate the tail delay of the first stream through the target device.

In a possible implementation manner, the network device further includes a receiving unit, configured to receive a second data packet, where the second data packet is a data packet of the target stream;

the processing unit is further configured to obtain a second time delay of the n time delays of the second recording unit according to a polling manner; and updating the second time delay in the second recording unit to be the time delay of the second data packet passing through the target device under the condition that the time delay of the second data packet passing through the target device is larger than the second time delay.

In a possible implementation manner, the receiving unit is further configured to receive a third data packet, where the third data packet is a data packet of the target stream, and before the third data packet and after the second data packet, the receiving unit receives m data packets of the target stream, where m is an integer greater than 1;

the processing unit is further configured to randomly acquire a third time delay from the n time delays of the second recording unit; and updating the third time delay in the second recording unit to be the time delay of the third data packet passing through the target device under the condition that the time delay of the third data packet passing through the target device is smaller than the third time delay.

In a possible implementation, the processing unit is further configured to: and under the condition that the estimation window is ended, estimating the tail time delay of the first stream passing through the target device in the estimation window based on the n time delays in the second recording unit.

Optionally, the tail delay of the first stream passing through the target device is p fractional tail delays, where p is greater than 80 and less than 100; the processing unit estimates a tail delay of the first stream passing through the target device based on the n delays in the second recording unit, and includes: determining a w-th time delay of the n time delays as a tail time delay estimated value of the first stream passing through the target device in the estimation window, wherein the w-th time delay is a time delay ordered from small to large and is a time delay ordered from w to n, and the w is an integer which is greater than 0 and less than or equal to n.

In a possible implementation, the network device further includes a sending unit, configured to send information recorded in the first recording unit and the second recording unit to the controller in case that the estimation window ends.

In one possible embodiment, the determining, by the processing unit, the first recording unit of the first stream map based on the stream information includes: mapping the stream information to a plurality of bits in a bitmap by a plurality of hash functions; one bit in the bitmap is associated with one of the first recording units; the network device determines the first recording unit of the first stream map based on the plurality of bits.

In a possible implementation manner, the first recording unit further includes an xor result of the stream number and the stream information, where the stream number is the number of the target stream, and the xor result of the stream information is a result obtained by performing an xor operation on the stream information of the target stream.

In a sixth aspect, the present application provides a network device, including a processing unit configured to:

acquiring flow information from the received first data packet, wherein the flow information is used for indicating that the first data packet is a data packet in a first flow;

determining a first recording unit of the first stream map based on the stream information; the first recording unit is configured to record a first time delay and a packet number, where the first time delay is a maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is a number of data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit;

updating the number of the packets, and updating the first time delay to the time delay of the first data packet passing through the target device under the condition that the time delay of the first data packet passing through the target device is larger than the first time delay;

allocating a second recording unit to the first recording unit when the number of packets is greater than or equal to a threshold; the second recording unit is configured to record a time delay of each of n data packets passing through the target device, where the n data packets are partial data packets in a data packet of the target stream received by the network device after the first data packet, and n is an integer greater than 1; the n delays are used to estimate a tail delay of the first stream through the target device.

In a seventh aspect, the present application provides a controller comprising:

a receiving unit, configured to receive information recorded in multiple first recording units, where the information recorded in each first recording unit includes a first time delay and a packet number, the first time delay is a maximum time delay when a single data packet of a target stream passes through a target device, and the packet number is a number of data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit, the target stream includes a first stream;

and the processing unit is used for estimating the tail time delay of the first stream passing through the target device based on the first time delay recorded in the plurality of first recording units under the condition that the number of the packets is less than a threshold value.

Optionally, the target device is the network device; alternatively, the network device is a device on a data plane in a network, the network device is a last hop device of the target flow in the data plane, and the target device includes a plurality of devices through which the single packet passes in the data plane.

In a possible implementation manner, the receiving unit is further configured to receive an identifier of an estimation window, where the identifier indicates that the information recorded in the plurality of first recording units is recorded in the estimation window, and the estimation window is a time window for estimating a tail delay of the first stream passing through the target device;

the processing unit estimates a tail delay of the first stream passing through the target device based on the first delays recorded in the plurality of first recording units, including: determining the first delay recorded in the first unit as a tail delay estimation value of the first stream passing through the target equipment in the estimation window; the first unit is a unit with the minimum first time delay recorded in the plurality of first recording units.

In an eighth aspect, the present application provides a controller comprising:

a receiving unit, configured to receive information recorded in a first recording unit and a second recording unit, where the information recorded in the first recording unit includes a first time delay and a packet number, the first time delay is a maximum time delay when a single data packet of a first stream passes through a target device, and the packet number is a number of data packets of the first stream received by the network device; the second recording unit is associated with the first recording unit, the information recorded in the second recording unit includes a time delay of each of n data packets in the first stream passing through the target device, where n is an integer greater than 1;

and the processing unit is used for estimating the tail time delay of the first stream passing through the target equipment based on the n time delays.

In a possible implementation, the receiving unit is further configured to receive an identifier of an estimation window, where the identifier indicates that the information recorded in the first recording unit and the second recording unit is recorded in the estimation window, and the estimation window is a time window used for estimating a tail time delay of the first stream passing through the target device;

the tail delay of the first stream passing through the target device is p fractional tail delay, and p is more than 80% and less than 100%; the estimating, by the processing unit, a tail delay of the first stream passing through the target device based on the n delays includes:

determining a w-th time delay of the n time delays as a tail time delay estimated value of the first stream passing through the target device in the estimation window, wherein the w-th time delay is a time delay ordered from small to large and is a time delay ordered from w to n, and the w is an integer which is greater than 0 and less than or equal to n.

In a ninth aspect, an embodiment of the present application provides a network device, where the network device may include: the device comprises a memory, and a processor, a sending interface and a receiving interface which are coupled with the memory. Wherein the sending interface is configured to support the network device to perform the step of sending the message and/or the data packet in the communication delay estimation method provided by the first aspect. The receiving interface is configured to support the network device to perform the step of receiving the message and/or the data packet in the communication delay estimation method provided in the first aspect. Wherein, the transmitting interface and the receiving interface can be integrated into a transceiver. The processor is configured to support the network device to perform other processing operations besides sending and receiving in the communication method provided by the first aspect.

It should be noted that the transmitting interface and the receiving interface in the embodiment of the present application may be integrated together, or may be coupled through a coupler. The memory is used for storing a computer program of the communication delay estimation method described in the first aspect, and the processor is used for executing the computer program stored in the memory. The memory and the processor may be integrated together or may be coupled by a coupler.

In addition, the computer program in the memory in the present application may be stored in advance, or may be downloaded from the internet when the apparatus is used, and the source of the computer program in the memory in the present application is not particularly limited. The coupling in the embodiments of the present application is an indirect coupling or connection between devices, units or modules, which may be in an electrical, mechanical or other form, and is used for information interaction between the devices, units or modules.

In one possible implementation, the processor is configured to execute the program instructions stored in the memory to cause the network device to:

acquiring flow information from a received first data packet, wherein the flow information is identification information of a first flow to which the first data packet belongs; determining a first recording unit of the first stream map based on the stream information; the first recording unit comprises a first time delay and a packet number, wherein the first time delay is the maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is the number of the data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit; updating the number of the packets, and updating the first time delay to the time delay of the first data packet passing through the target device under the condition that the time delay of the first data packet passing through the target device is larger than the first time delay; in the case where the number of packets is less than a threshold, the first delay is used to estimate a tail delay of the first stream through the target device.

In a tenth aspect, an embodiment of the present application provides a network device, where the network device may include: the device comprises a memory, and a processor, a sending interface and a receiving interface which are coupled with the memory. Wherein, the sending interface is used to support the network device to execute the step of sending the message and/or the data packet in the communication delay estimation method provided by the second aspect. The receiving interface is configured to support the network device to perform the step of receiving the message and/or the data packet in the communication delay estimation method provided by the second aspect. Wherein, the transmitting interface and the receiving interface can be integrated into a transceiver. The processor is configured to support the network device to perform other processing operations besides transmitting and receiving in the communication method provided by the second aspect.

It should be noted that the transmitting interface and the receiving interface in the embodiment of the present application may be integrated together, or may be coupled through a coupler. The memory is used for storing a computer program of the communication delay estimation method described in the second aspect, and the processor is used for executing the computer program stored in the memory. The memory and the processor may be integrated together or may be coupled by a coupler.

In addition, the computer program in the memory in the present application may be stored in advance, or may be downloaded from the internet and stored when the apparatus is used. The coupling in the embodiments of the present application is an indirect coupling or connection between devices, units or modules, which may be in an electrical, mechanical or other form, and is used for information interaction between the devices, units or modules.

acquiring flow information from the received first data packet, wherein the flow information is used for indicating that the first data packet is a data packet in a first flow; determining a first recording unit of the first stream map based on the stream information; the first recording unit is configured to record a first time delay and a packet number, where the first time delay is a maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is a number of data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit; updating the number of the packets, and updating the first time delay to the time delay of the first data packet passing through the target device under the condition that the time delay of the first data packet passing through the target device is larger than the first time delay; allocating a second recording unit to the first recording unit if the number of packets is greater than or equal to a threshold; the second recording unit is configured to record a time delay of each of n data packets passing through the target device, where the n data packets are partial data packets in a data packet of the target stream received by the network device after the first data packet, and n is an integer greater than 1; the n delays are used to estimate a tail delay of the first stream through the target device.

In an eleventh aspect, embodiments of the present application provide a controller, where the controller may include: the device comprises a memory, and a processor, a sending interface and a receiving interface which are coupled with the memory. Wherein, the sending interface is used to support the controller to execute the step of sending the message and/or the data packet in the communication delay estimation method provided by the third aspect. The receiving interface is configured to support the controller to perform the step of receiving the message and/or the data packet in the communication delay estimation method provided by the third aspect. Wherein, the transmitting interface and the receiving interface can be integrated into a transceiver. The processor is configured to support the controller to perform other processing operations besides transmitting and receiving in the communication method provided by the third aspect.

It should be noted that, the sending interface and the receiving interface in the embodiment of the present application may be integrated together, and may also be coupled through a coupler. The memory is used for storing a computer program of the communication delay estimation method described in the third aspect, and the processor is used for executing the computer program stored in the memory. The memory and the processor may be integrated together or may be coupled by a coupler.

In one possible embodiment, the processor is configured to execute program instructions stored in the memory to cause the controller to:

receiving information recorded in a plurality of first recording units, wherein the information recorded in each first recording unit comprises a first time delay and a packet number, the first time delay is the maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is the number of the data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit, the target stream including a first stream; and under the condition that the number of the packets is less than a threshold value, estimating the tail time delay of the first stream passing through the target device based on the first time delays recorded in the plurality of first recording units.

In a twelfth aspect, an embodiment of the present application provides a controller, which may include: the device comprises a memory, and a processor, a sending interface and a receiving interface which are coupled with the memory. Wherein, the sending interface is used for supporting the controller to execute the step of sending the message and/or the data packet in the communication delay estimation method provided by the fourth aspect. The receiving interface is configured to support the controller to perform the step of receiving the message and/or the data packet in the communication delay estimation method provided in the fourth aspect. Wherein, the transmitting interface and the receiving interface can be integrated into a transceiver. The processor is configured to support the controller to perform other processing operations besides transmitting and receiving by the controller in the communication method provided in the fourth aspect.

It should be noted that, the sending interface and the receiving interface in the embodiment of the present application may be integrated together, and may also be coupled through a coupler. The memory is used for storing a computer program of the communication delay estimation method described in the fourth aspect, and the processor is used for executing the computer program stored in the memory. The memory and the processor may be integrated together or may be coupled by a coupler.

receiving information recorded in a first recording unit and a second recording unit, wherein the information recorded in the first recording unit comprises a first time delay and a packet number, the first time delay is the maximum time delay of a single data packet of a first stream passing through a target device, and the packet number is the number of the data packets of the first stream received by the network device; the second recording unit is associated with the first recording unit, the information recorded in the second recording unit includes a time delay of each of n data packets in the first stream passing through the target device, and n is an integer greater than 1; and estimating the tail time delay of the first flow passing through the target equipment based on the n time delays.

In a thirteenth aspect, an embodiment of the present application provides a system, where the system includes a network device and a controller, where the network device is the network device according to the fifth aspect, and the controller is the controller according to the seventh aspect or the eighth aspect; or, the network device is the network device of the sixth aspect, and the controller is the controller of the eighth aspect; alternatively, the network device is the network device according to the ninth aspect, and the controller is the controller according to the eleventh or twelfth aspect; alternatively, the network device is the network device according to the tenth aspect, and the controller is the controller according to the twelfth aspect.

In a fourteenth aspect, an embodiment of the present application is a computer-readable storage medium, which stores a computer program, where the computer program is executed by a processor to implement the method of any one of the first aspect; alternatively, the computer program is executed by a processor to implement the method of any one of the second aspects described above.

In a fifteenth aspect, an embodiment of the present application is a computer-readable storage medium, in which a computer program is stored, the computer program being executed by a processor to implement the method of any one of the third aspects; alternatively, the computer program is executed by a processor to implement the method of any of the fourth aspects above.

In a sixteenth aspect, an embodiment of the present application is a computer program product, which, when run on a computer, causes the computer to perform the method according to any one of the first aspect above; or causing the computer to perform a method as described in any of the second aspects above.

In a seventeenth aspect, an embodiment of the present application provides a computer program product, which, when run on a computer, causes the computer to perform the method according to any one of the above third aspects; or causing the computer to perform the method of any of the fourth aspects above.

It will be appreciated that the network devices, controllers, systems, computer storage media and computer program products provided above are all for performing the method provided in any one of the first to fourth aspects. Therefore, the beneficial effects achieved by the method can refer to the beneficial effects in the corresponding method, and the details are not repeated here.

Drawings

The drawings to be used in the embodiments of the present application will be described below.

Fig. 1 and fig. 2 are schematic diagrams illustrating a system architecture according to an embodiment of the present application;

FIG. 3 is a schematic flow chart of a method provided by an embodiment of the present application;

FIGS. 4 and 5 are schematic diagrams illustrating information carried in a data packet during transmission;

fig. 6 is a schematic diagram illustrating a processing flow architecture of a network device according to an embodiment of the present application;

fig. 7 is a schematic diagram illustrating a flow direction of information in a network according to an embodiment of the present application;

fig. 8 is a schematic diagram illustrating contents included in a data packet according to an embodiment of the present application;

fig. 9 is a schematic diagram illustrating a simulation result provided in the embodiment of the present application;

fig. 10 to fig. 13 are schematic diagrams illustrating a logical structure of an apparatus according to an embodiment of the present application;

fig. 14 to 17 are schematic hardware structures of an apparatus according to an embodiment of the present disclosure.

Detailed Description

Embodiments of the present application are described below with reference to the drawings.

First, terms related to embodiments of the present application will be described:

1. a data plane: the data plane refers to a forwarding plane in the network, and depicts the system and the capability of forwarding chips and forwarding storage of the devices in the network, and corresponds to a control plane with pervasive computing capability.

2. Time delay: the time delay used in the embodiment of the present application is the transmission time of the data packet in the network, which is calculated based on the hardware clock timestamp provided by the network device. In an actual deployment scenario, the time delay may be a single hop (for example, the processing time of a data packet provided by a network device in the device); or it may be an end-to-end delay in the network (e.g., the time from the ingress of the first hop network device to the egress of the last hop network device).

3. Full flow: full flow refers to all flows in the network, and full flow latency refers to latency information for each of the all flows.

4. Tail time delay: the delay of p% of the data packets in a stream is less than a certain delay, which is called p quantile delay. In this application, when p% takes a value between 80% and 100%, the certain time delay is called tail time delay, or p-decisional tail time delay. For example, if the delay of 95% of the packets in a stream is smaller than a certain delay, the certain delay is called 95 fractional tail delay.

An application scenario of the embodiment of the present application is described below.

Referring to fig. 1, fig. 1 is a schematic diagram of a system architecture of an application scenario according to an embodiment of the present application. As can be seen in fig. 1, the system architecture includes a control plane 110 and a data plane 120. The control plane 110 includes a controller 111 therein, and the controller 111 may be a Software Defined Network (SDN) controller or an Operation and Maintenance Center (OMC) or the like. The controller 111 may be deployed in one or more servers, which may form a server cluster.

The data plane 120 includes g network devices 121, where g is an integer greater than 1. The network device 1 and the network device g shown in fig. 1 may further include a plurality of network devices, and the plurality of network devices may be connected in any topology. The network device 121 is used to forward data in the data plane. The network device 121 may be, for example, a forwarding device such as a switch, router, or access device, which may be a base station or wireless access point, etc.

In one possible implementation, referring to fig. 2, the network devices in the data plane have high resources and processing capabilities, and may implement their own control logic in addition to the logic for implementing data forwarding, so that the capability for implementing their own control logic may also be referred to as the device control plane, which may also belong to the capability in the control plane 110. In this application, the device control plane may be configured to process the delay information of the collected data packet to estimate the tail delay of the corresponding flow.

It should be noted that the system architectures shown in fig. 1 and fig. 2 are only one example, and the system architecture to which the embodiment of the present application is applicable is not limited to the system architecture described above.

The measurement techniques for tail delay of a stream commonly used in the current industry can be mainly classified into two types: one type is an in-network measurement technology represented by network flow NetFlow, which supports screening, sampling and copying related message data in network equipment, and then uploading the message data to a controller for tail time delay measurement; the other is an in-band operation administration and maintenance (iOAM) technology represented by an in-band network telemetry (INT) technology. Through analysis of the existing schemes, the main challenge of the full-flow delay measurement task is that the storage resources and logic capacity of the network device on the data plane are insufficient, or the full-flow delay measurement in the existing schemes needs to consume more storage resources and processing resources, so that the resource overhead is very large. The concrete points are as follows:

on one hand, the throughput of the current network is very large, if a full-flow delay measurement task is implemented on a control plane, huge extra bandwidth load (copy) is often brought or coverage rate is reduced, and full-flow (sampling/screening) cannot be achieved, and the number of concurrent flows in the current network is very large, 100K-1M-level concurrent flows often exist in a data center at the same time, so that if stream-by-stream information is required to be recorded, the available storage space of each flow is very limited by the limited storage capacity of the current data plane.

On the other hand, the logic resources and supported operations of the network equipment are more limited than those of a general computer, so that although the current algorithm theory research already has a better research conclusion on the stream model quantile estimation, how to apply and deploy is difficult, and the existing method has strong assumption on the equipment capacity and is actually difficult to deploy; or abandon the precision and adopt the estimation with coarser granularity; or more resources are used to implement a compromise version of the existing algorithm, giving up the scalability and thus making it difficult to support flow-by-flow delay measurements.

Therefore, the scheme of the application focuses on network equipment with limited resources and capacity, and achieves the aim of high-precision measurement of the tail delay of the low-overhead full-volume flow on the data plane, namely, the scheme focuses on reducing the resource overhead of the network equipment in the tail delay measurement process of the flow. The following are specific implementation examples.

Referring to fig. 3, a communication delay estimation method provided by the embodiment of the present application is shown, which is used to solve the above technical problem. The method can be applied to the system architecture shown in fig. 1 or fig. 2. The method may include, but is not limited to, the steps of:

301. the first network device obtains flow information from the received first data packet, wherein the flow information is a flow identifier of a first flow to which the first data packet belongs.

In a specific embodiment, the first network device may be any one of the network devices in the data plane 120 in fig. 1 or fig. 2. The first network device may be configured to forward packets of one or more flows.

Specifically, after receiving a data packet (for convenience of description, the data packet is referred to as a first data packet), the first network device may parse the first data packet to obtain the flow information in the first data packet. Illustratively, the flow information may be, for example, a quad, a quintuple, a heptatuple, or the like. The quadruplet comprises a source Internet Protocol (IP) address, a destination IP address, a source port and a destination port of the stream; the five-tuple comprises a source IP address, a destination IP address, a protocol number, a source port and a destination port; the seven-tuple includes a source IP address, a destination IP address, a protocol number, a source port, a destination port, a service type, and an interface index. After obtaining the flow information in the first packet, the first network device may obtain a flow to which the first packet belongs (for convenience of description, the flow is referred to as a first flow in the following).

In a possible implementation, the first network device may encode the obtained stream information. Illustratively, the flow information may be represented by a symbol or value that may uniquely identify the first flow within the first network device.

302. The first network device determines a first recording unit of the first stream map based on the stream information; the first recording unit includes a first delay and a packet number, where the first delay is a maximum delay of a single data packet mapped to the stream in the first recording unit passing through a target device, and the packet number is a number of data packets mapped to the stream in the first recording unit received by the first network device.

In a specific embodiment, the first network device includes a data structure bitmap (bitmap), where the bitmap includes a plurality of bits, and in this embodiment, the bitmap may be used to determine whether a stream is a new stream. Specifically, after obtaining the flow information of one flow, the first network device may perform hash calculation on the flow information through a plurality of hash functions to obtain a plurality of hash values, where the plurality of hash values correspond to position sequence numbers of a plurality of bits in a bitmap, and therefore, the plurality of hash values may map the flow corresponding to the flow information into the plurality of bits. The plurality of hash functions may be two or more hash functions.

The initial value of each bit in the bitmap may be a first value, and after mapping the stream to the plurality of bits based on the computed plurality of hash values, the values in the plurality of bits may be set to a second value. For example, alternatively, assuming that the initial value of each bit in the bitmap is 0, after mapping the stream to a plurality of bits based on the calculated hash values, the value of the plurality of bits may be set to 1. Alternatively, assuming that the initial value of each bit in the bitmap is 1, after mapping the stream to a plurality of bits based on the calculated hash values, the values in the plurality of bits may be set to 0.

Based on the logic, the first network device performs hash calculation on the flow information of the first flow through the hash functions to obtain a plurality of hash values, maps the first flow to a plurality of bits in a bitmap based on the hash values, and if a value in at least one of the bits is the first value, it may be determined that the first flow is a new flow, and the first packet is a first packet in the first flow received by the first network device. The new stream refers to a stream that has not yet been mapped into the bitmap. If the value in each of the plurality of bits is the second value, it indicates that the first stream is not a new stream, and the first packet is a packet in the first stream received again by the first network device.

In the embodiment of the present application, each bit in the bitmap is associated with a first recording unit, and then the first recording unit associated with a bit is also the unit of the stream map mapped to the bit. The first recording unit may be configured to record the number of packets and the first delay.

The number of packets is the number of data packets mapped to the stream in the first recording unit received by the first network device. Optionally, for a certain first recording unit, if the stream mapped to the first recording unit is one, the number of packets in the first recording unit is the number of data packets of the one stream received by the first network device. Optionally, for the certain first recording unit, if there are multiple streams mapped to the first recording unit, that is, hash collision occurs, the number of packets in the first recording unit is the sum of the number of data packets of the multiple streams received by the first network device.

The first delay is a maximum delay of a single packet of the stream mapped to the first recording unit through the destination device.

In a possible implementation, the target device is the first network device. Alternatively, in another possible implementation manner, the first network device is a last hop device of the stream mapped to the first recording unit in the data plane, and the target device includes a plurality of devices, where the plurality of devices are devices through which a single packet of the stream mapped to the first recording unit passes in the data plane.

Optionally, whether the target device is the first network device or the target device includes multiple devices, compared with a substantially fixed link delay, a processing delay on the network device, especially a queuing delay waiting for processing in the network device, is a main factor causing delay fluctuation, and therefore in this embodiment of the present application, a processing time of the network device may be used as the delay, where the processing time is a time between receiving a data packet from the network device and sending the data packet from the network device, that is, the processing time is a delay of the data packet through the network device. In a case where the target device is the first network device, the first delay is a maximum delay of the single packet through the first network device. In a case where the target device includes a plurality of devices, the first delay is a maximum value of a sum of delays of the single packet passing through each of the plurality of devices. For example, assuming that the plurality of devices are device 1 and device 2, and device 2 receives the first packet and the second packet in the first stream, the delay of the first packet through device 1 and device 2 is 1 millisecond and 0.9 millisecond, respectively, and the delay of the second packet through device 1 and device 2 is 1.1 millisecond and 1 millisecond, respectively, then the first delay in the first recording unit of the first stream map in device 2 is 2.1 millisecond. Based on the foregoing description, the sum of the processing time of the data packet by all the network devices that the data packet passes through in the data plane is the time delay of the whole transmission process of the data packet in the data plane. How to obtain the time delay will be described later, and will not be detailed here.

Or, optionally, for a case that the target device includes multiple devices, a transmission delay of a data packet on a transmission path in a data plane may be taken as a delay of an entire transmission process of the data packet in the data plane, where the transmission delay includes a link delay in the transmission path and a delay caused by processing time in the multiple devices in the transmission path, and the transmission delay is also referred to as a delay through the multiple devices in this application, and since the multiple devices are the target devices, the transmission delay is a delay through the target devices. That is, in this case, the first delay may be a maximum value of a delay of a single packet in the stream mapped to the first recording unit in the entire transmission process. For example, if the first network device receives the first packet and the second packet in the first stream, the delay of the first packet in the entire transmission process is 2.5 msec, and the delay of the second packet in the entire transmission process is 2.8 msec, then the first delay in the first recording unit of the first stream map in the first network device is 2.8 msec. The transmission delay is the delay between the ingress network device receiving the data packet and the egress network device sending the data packet. The ingress network device is a first hop device that forwards the data packet in the data plane, and the egress network device is a last hop device that forwards the data packet in the data plane. How to obtain the time delay will be described in detail, and will not be described in detail here.

In a possible implementation manner, the first recording unit may be further configured to record an exclusive or result of the stream information and the number of streams. The number of streams is the number of streams mapped to the first recording unit, and the exclusive-or result of the stream information is the exclusive-or result of the stream information of the streams mapped to the first recording unit. Specifically, if the number of streams is 1, the xor result of the stream information is the stream information itself, or when the number of streams mapped to the first recording unit is one, the stream information itself is recorded in the position of the first recording unit where the xor result of the stream information is recorded. If the number of streams is greater than 1, that is, multiple streams are mapped to the same first recording unit, the xor result of the stream information is the result obtained by performing xor operation on the multiple streams. In the computer, all information is represented by binary data composed of 0 and 1, and therefore, the result of the exclusive or of the stream information is also binary data composed of 0 and 1.

Based on the above description, the first network device maps the first stream into a plurality of bits in a bitmap, and a plurality of first recording units of the first stream map may be determined by the plurality of bits.

By mapping the stream into the bitmap, storage resources for the first network device can be saved and computational difficulty can be reduced to save computational resources.

303. And the first network equipment updates the number of the packets, and updates the first time delay to the time delay of the first data packet passing through the target equipment under the condition that the time delay of the first data packet passing through the target equipment is greater than the first time delay.

In an embodiment, after determining the plurality of first recording units of the first stream map, the first network device may adaptively update information in the plurality of first recording units, which will be described in the following.

In the first case, the first stream is a new stream and no hash collision occurs in the bits of the first stream map. In this case, if no information is recorded in the first recording units of the first stream map or the recorded information is an original default value, the first network device may set a value of a position where the number of packets is recorded in the first recording units to 1. In addition, for the first time delay in the plurality of first recording units, the first network device may obtain the time delay of the first data packet passing through the target device, and then record the time delay to a position where the first time delay is recorded in the plurality of first recording units.

Specifically, a delay obtaining module may be deployed in the first network device, and the delay obtaining module may obtain a delay of the first data packet passing through the target device.

Optionally, for a case that the target device is the first network device, the delay obtaining module may obtain, through measurement and calculation, a processing time of the first data packet in the first network device, where the processing time is a delay of the first data packet through the first network device. Specifically, the first network device may measure a timestamp of the first data packet arriving at the first network device, estimate a timestamp of the first data packet sent from the first network device, and subtract the previous timestamp from the next timestamp to obtain an amount of time, where the amount of time is a processing time, i.e., a time delay, of the first data packet in the first network device. Then, the first network device records the calculated time delay to a position where the first time delay is recorded in the plurality of first recording units.

It should be noted that the timestamp of the first data packet sent from the network device is a timestamp calculated according to a condition of a data packet sending queue, in this embodiment of the present application, the delay calculated by the delay obtaining module allows an acceptable error to exist, or in this embodiment of the present application, a delay calculated by the delay obtaining module may be regarded as a correct delay of a single data packet through the network device. Optionally, the time delay obtaining module may estimate time accurate to nanoseconds, thereby improving the estimation accuracy of the time delay.

Optionally, for the case that the target device includes multiple devices, in a possible implementation, the processing time of the network device may be used as the time delay. Then, for the first packet, processing logic for calculating the processing time of the first packet in the network device is set on the network device that needs to forward the first packet in the data plane.

Specifically, for the ingress network device, after calculating the delay of the first data packet passing through the ingress network device, the delay is added to the packet header of the first data packet, and the first data packet is sent to the next hop network device. Optionally, the packet header may be a packet header added by self-definition and specifically used for carrying the delay. After the next hop network device receives the first data packet carrying the time delay, the processing time of the first data packet in the next hop network device, namely the time delay, is calculated according to the logic, the calculated time delay is added to the time delay in the packet header of the received first data packet to obtain a new time delay, and the new time delay is updated to the packet header of the first data packet and is continuously sent to the next hop device. Until the first data packet reaches the last hop device in the data plane, that is, the first network device obtains the time delay in the packet header of the first data packet, calculates the time delay of the first data packet in the first network device, and then adds the two time delays to obtain a total time delay, where the total time delay is the time delay of the first data packet passing through the multiple devices. After obtaining the total delay, the first network device may remove a header of the first data packet that is dedicated to carrying the delay. This adding and sending of the delay to the existing delay in the data packet in each hop network device may be implemented, for example, using the INT technique described above. Then, the first network device records the total time delay to the position of the first recording units where the first time delay is recorded. For ease of understanding of this embodiment, reference may be made to fig. 4 by way of example.

Fig. 4 shows a simple topology diagram of a network device in a data plane, and shows information carried in a data packet during transmission, assuming that a transmission path of the first data packet in the data plane is: network device 1-network device 2-network device 5-network device 6. The time delay carried in the first data packet sent by the network device 1 is the time delay a of the first data packet passing through the network device 1. Then, the first data packet is sent to the network device 2, the network device 2 obtains the time delay b of the first data packet, and then encapsulates the time delay a + b into the first data packet. Then, the first data packet is sent to the network device 5, the network device 5 obtains the time delay c of the first data packet passing through itself, and then encapsulates the time delay a + b + c into the first data packet. Then, the first data packet carrying the time delay a + b + c is sent to the network device 6, the network device 6 obtains the time delay d of the first data packet passing through itself, and adds d to the time delay a + b + c to obtain the time delay a + b + c + d, and then records the time delay a + b + c + d to the position where the first time delay is recorded in the plurality of first recording units.

It should be noted that, in the embodiment of the present application, although the delay carried in the first packet forwarded in different network devices is different, as long as the load (payload) in the first packet is not changed, the packet carrying different delay information is still referred to as the first packet.

Optionally, for a case that the target device includes multiple devices, in a possible implementation, a transmission delay of the data packet on a transmission path in the data plane may be adopted as a delay of the entire transmission process of the data packet in the data plane. Then, for the first data packet, on an ingress network device that forwards the first data packet in the data plane, the ingress network device may measure a timestamp of the arrival of the first data packet at the ingress network device, and then add the timestamp to the first data packet header, and forward the first data packet up to an egress network device of the data plane, i.e., the first network device. Optionally, the packet header may be a custom added packet header specifically for carrying the timestamp. The first network device may obtain the timestamp in the first data packet, estimate the timestamp of the first data packet from the first network device, and subtract the previous timestamp from the next timestamp to obtain an amount of time, which is the delay of the first data packet through the target device. After obtaining the amount of time, the first network device may remove the header of the first data packet that is dedicated to carrying the timestamp. This operation of adding a timestamp to a data packet in an ingress network device and transmitting it with the data packet to an egress network device may be implemented, for example, using the INT technique described above. Then, the first network device records the time delay to a position where the first time delay is recorded in the plurality of first recording units. In this implementation, the clock synchronization of the network devices in the data plane is required, so that the calculated time delay is accurate. To facilitate understanding of this embodiment, reference may be made to fig. 5 for example.

Fig. 5 shows a simple topology diagram of a network device in a data plane, and shows information carried in a data packet during transmission, assuming that a transmission path of the first data packet in the data plane is: network device 1-network device 2-network device 5-network device 6. If the network device 1 is the above-mentioned ingress network device, the first data packet sent out from the network device 1 carries a timestamp of the time when the first data packet arrives at the network device 1, i.e. the first timestamp shown in fig. 5. Then, the first data packet carrying the first timestamp is sent to the network device 6 through the network device 2 and the network device 5, the network device 6 estimates the timestamp of the first data packet sent from the network device 6, and the latter timestamp is subtracted from the former timestamp to obtain an amount of time, which is the delay of the first data packet through the target device. The network device 6 records the amount of time to the position where the first time delay is recorded in the plurality of first recording units.

In a possible embodiment, the first recording unit may be further configured to record an exclusive or result of the stream information and the number of streams. If the first recording unit includes the xor result of the stream information, since no hash collision occurs in the bits of the first stream map, the first network device may store the stream information of the first stream in the location of the xor result of the stream information recorded in the first recording unit of the first stream map. If the number of streams is included in the first recording unit, the first network device may set the number of streams recorded in the first recording unit of the first stream map to 1.

In a second case, the first stream is a new stream and a hash collision occurs in at least one of the bits of the first stream map.

In this case, for the first recording unit mapped by the bit without hash collision in the plurality of bits, the corresponding description in the first case may be referred to for updating the number of packets and the first delay in the first recording unit, and details are not repeated here. For the at least one first recording unit of the at least one bit map in which the hash collision occurs, the first network device may increase the number of packets in the at least one first recording unit by 1 count, that is, replace the original number of packets with the number of packets obtained after increasing the 1 count. For the first delay in the at least one first recording unit, the first network device may obtain the delay of the first data packet passing through the target device according to the method described in the first scenario, compare the obtained delay with the first delay recorded in each of the at least one first recording unit, and replace the smallest first delay in the at least one first recording unit with the obtained delay if the obtained delay is greater than the smallest first delay in the at least one first recording unit.

In a possible implementation manner, the first recording unit may be further configured to record an exclusive or result of the stream information and the number of streams.

If the first recording unit includes the xor result of the stream information, the first network device may store, for the first recording unit to which the bits of the plurality of bits are mapped without hash collision, the stream information of the first stream in a position of the xor result of the stream information recorded in the first recording unit. For the at least one first recording unit of the at least one bit map in which the hash collision occurs, the first network device may perform an exclusive or operation on the stream information of the first stream and information (may be information of one stream or may be an exclusive or result of stream information of a plurality of streams) in a position in the first recording unit in which an exclusive or result of the stream information is recorded, obtain an exclusive or result, and then store the calculated exclusive or result in a position in the exclusive or result of the stream information recorded in the first recording unit.

If the first recording unit includes the number of streams, the first network device may set, to 1, the number of streams recorded in the first recording unit of the first stream map for the first recording unit of the bit map without the hash collision among the plurality of bits. For the at least one first recording unit of the at least one bitmap in which the hash collision occurs, the first network device may add 1 count to the number of streams in the at least one first recording unit, that is, replace the original number of streams with the number of streams obtained after the 1 count is added.

In a third case, the first stream is not a new stream and no hash collision occurs in the bits of the first stream map.

In this case, it indicates that the first packet is a packet in the first stream received again by the first network device, and the number of packets and the first delay of the packet of the first stream received before are recorded in the plurality of first recording units of the first stream map. The first network device may increase the number of packets in the first recording unit of the first stream map by 1 count, that is, replace the original number of packets with the number of packets obtained after the 1 count is increased. In addition, for the first time delay in the plurality of first recording units, the first network device may obtain the time delay of the first data packet passing through the target device, then compare the obtained time delay with the first time delay recorded in the first recording unit, and if the obtained time delay is greater than the first time delay recorded in the first recording unit, the first network device updates the first time delay in the plurality of first recording units to the obtained time delay. If the acquired time delay is not greater than the first time delay recorded in the first recording unit, the first time delays in the plurality of first recording units remain unchanged.

In a possible implementation manner, the first recording unit may be further configured to record an exclusive or result of the stream information and the number of streams. If the first recording unit includes the xor result of the stream information, the stream information of the first stream is still recorded in the positions of the plurality of first recording units of the first stream map where the xor result of the stream information is recorded, and remains unchanged, because no hash occurs in the plurality of bits of the first stream map. If the first recording unit includes the number of streams, since no hash collision occurs in the bits of the first stream map, the corresponding streams mapped in the plurality of first recording units only have the first stream, and thus, the number of streams recorded in the plurality of first recording units of the first stream map is maintained as 1.

In a fourth case, the first stream is not a new stream and a hash collision occurs on at least one of the bits of the first stream map.

In this case, for the first recording unit mapped by the bit without hash collision in the plurality of bits, the corresponding description in the third case may be referred to for updating the number of packets and the first delay in the first recording unit, and details are not described here again. For the at least one first recording unit of the at least one bit map with hash collision, the corresponding description in the second case may be referred to for updating the number of packets and the first time delay, and details are not repeated here.

In a possible implementation manner, the first recording unit may be further configured to record an exclusive or result of the stream information and the number of streams. Since the first stream is not a new stream, the xor result and the number of streams of the stream information in the plurality of first recording units of the first stream map remain unchanged, which can be specifically referred to the corresponding description in the third case above.

In a possible implementation manner, the first data packet is a data packet received by the first network device within a preset time window, where the size of the preset time window may be any size between 10 milliseconds and 100 milliseconds; in this case, the first network device estimates the delay of the first stream in segments by using the time window as an estimated duration, and then the first delay is the maximum delay of a single data packet of the stream mapped into the first recording unit passing through the target device within the time window.

Or, in another possible implementation, the first data packet is any one of data packets in the first stream received by the first network device, in this case, the first network device estimates a delay of the first stream by using the overall transmission time of the first stream as an estimated duration, and then the first delay is a maximum delay of a single data packet of the stream passing through the target device within the overall transmission time of the stream mapped in the first recording unit.

In the embodiment of the present application, the preset time window or the entire transmission time of the stream may be collectively referred to as an estimation window, that is, the estimation window is a time window for estimating the delay of the stream. In the embodiment of the present application, the purpose is to estimate the tail delay of the stream, and then, the estimation window may also be said to be a time window for estimating the tail delay of the stream. The information recorded in the first recording unit is also estimated within the estimation window.

In this embodiment of the application, the estimation of the tail delay of the first stream passing through the target device includes two different processing manners, and a specific processing manner may be determined by determining whether the number of data packets of the first stream received by the first network device in the estimation window is greater than a threshold value. Specifically, after the first network device updates the plurality of first recording units of the first stream map, it may be determined whether the number of packets recorded in the plurality of first recording units is greater than a threshold, and of course, the data packets corresponding to the number of packets are received by the first network device in the estimation window. The threshold may be, for example, 100, 1000, or 10000, and may be set according to a desired measurement bit and a desired accuracy, which is not limited in the present application.

In a possible implementation manner, if the number of packets recorded in the plurality of first recording units is less than the threshold, and if the estimation window is not ended, the first network device continues to receive the data packets of the first stream, and then updates the information in the plurality of first recording units of the first stream map, where the specific update may refer to the corresponding description in step 303. After the estimation window is ended, the first network device may find the multiple first recording units of the first stream mapping according to the hash value of the stream information of the first stream, and the first network device obtains multiple first delays recorded in the multiple first recording units of the first stream mapping, and then takes a minimum value of the multiple first delays as an estimated value of a tail delay of the first stream passing through the target device in the estimation window.

In the processing method, since the number of the data packets of the first stream in the estimation window is smaller than the threshold, it indicates that the number of the data packets is relatively small, then the tail delay difference between the 80-min tail delay and the 100-min tail delay of the first stream in the estimation window is small, and the first delay recorded in the first recording unit of the first stream map is the maximum delay of a single data packet passing through the target device in the estimation window, therefore, the first delay recorded in the first recording unit of the first stream map can be used to estimate any tail delay between the 80-min tail delay and the 100-min tail delay of the first stream in the estimation window, and this estimation error is small and is within an acceptable range. And because hash collision may exist, if the first time delay recorded by the unit with hash collision in the plurality of first recording units of the first stream map is different from the first time delay recorded by the unit without hash collision, based on the above principle of replacing the smaller time delay with the larger time delay, the first time delay recorded by the unit with hash collision is not smaller than the first time delay recorded by the unit without hash collision, and therefore, it is more accurate to select the smallest first time delay in the plurality of first recording units of the first stream map as the estimated value of the tail time delay of the first stream passing through the target device in the estimation window.

In a possible implementation manner, for at least one first recording unit of the bit map in which the hash collision exists in the bits of the first stream map, after the estimation window is ended, the first network device may perform an xor operation on the xor result of the stream information in the at least one first recording unit and the stream information of the first stream again in the process of decoding the information in the first recording unit, and record the xor operation result into the at least one first recording unit, so that the stream information of the first stream may be deleted from the at least one first recording unit. Meanwhile, the first network device may subtract 1 from the number of streams in the at least one first recording unit, and subtract the number of packets of the first stream received within the estimation window from the number of packets, but the first delay is not changed. Through these processes, the related information of the first stream can be deleted from the at least one first recording unit, and in the case where there are a plurality of estimation windows, the at least one first recording unit can prepare the related information for recording the first stream within the next estimation window.

For the first recording units of the bit map without hash collision in the bits of the first stream map, after the estimation window ends, the first network device may clear all the information recorded in the first recording units, and for the case of multiple estimation windows, the first recording units may prepare the related information for recording the first stream in the next estimation window.

In a possible implementation, if the number of packets recorded in the first recording units is greater than or equal to the threshold, then, when the number of packets is equal to the threshold, if the estimation window has not ended, the first network device configures an associated second recording unit for each of the first recording units, and the first recording unit and the second recording unit are associated by an index, and the first network device may configure a plurality of index recording units for exclusively recording index information between the first recording unit and the second recording unit, and each index recording unit records index information between one first recording unit and one second recording unit. N time delays may be recorded in the second recording unit, where n is an integer greater than 1. The n time delays are time delays of n data packets in the target stream received by the first network device passing through the target device after the number of the packets is equal to the threshold. If the number of packets recorded in the first recording unit just reaches the threshold after the first network device receives the first data packet, the n data packets are n data packets in the data packets of the target stream received by the first network device after the first data packet. The target stream is the stream mapped into the first recording unit, and if the stream mapped into the first recording unit only has the first stream, the target stream is the first stream; if the stream mapped to the first recording unit includes a plurality of streams including the first stream, the target stream is the plurality of streams.

Specifically, when the second recording unit is configured, the initial values of n time delays may be configured in the second recording unit by default, or the second recording unit is empty at first. In this embodiment of the present application, n time delays in the second recording unit may be updated in a polling manner. After the number of packets recorded in the first recording unit is equal to the threshold, the time delay of the first n packets of the target stream received by the first network device passing through the target device may be sequentially recorded in the position where the n time delays are recorded in the second recording unit. Then, after the n data packets, the first network device may determine, in a polling manner, a time delay in the second recording unit every time the first network device receives a data packet of the target stream, and then replace, in a case where the time delay is smaller than a time delay of the received data packet passing through the target device, the time delay of the received data packet passing through the target device by a time delay in the second recording unit.

The polling mode refers to that from one time delay of the n time delays, each time delay is sequentially inquired one by one according to a preset sequence until the inquiry of the n time delays is completed, then from the other time delay, each time delay is sequentially inquired one by one according to the preset sequence until the inquiry of the n time delays is completed again, and the inquiry operation is executed in a circulating mode. For ease of understanding, the following is exemplified. See table 1.

TABLE 1

Location of time delay	1	2	3	4	5
						Value of time delay	0.3 milliseconds	0.35 ms	0.28 millisecond	0.32 milliseconds	0.26 milliseconds

Table 1 exemplarily shows the time delay recorded in one second recording unit. Assuming that the second recording unit can record the time delay of the data packets of 5 target streams passing through the target device, the time delay at the corresponding position is shown in table 1. It is assumed that the 5 time delays recorded in table 1 are time delays of the first 5 data packets of the first stream received by the first network device passing through the target device after the number of packets recorded in the first recording unit is equal to the threshold. In addition, it is assumed that the polling manner is to sequentially search for the time delay from the time delay of the location 1 to the time delay of the location 5, then to round the search for the time delay of the location 1 and then to sequentially search for the time delay of the location 5, and so on.

Based on table 1, the first network device receives the 1 st packet of the target stream after the number of the packets is equal to the threshold, and obtains the delay of the 1 st packet through the target device, which is 0.32 ms. Then, based on the polling manner, the first network device first queries the time delay in the location 1 in the second recording unit, where the time delay is 0.3 ms. Since 0.32 msec is greater than 0.3 msec, the first network device replaces the 0.3 msec delay in this location 1 with 0.32 msec. After a round of operation, that is, after the first network device receives 5 packets of the first stream after the number of packets is equal to the threshold value, and polls to update the second recording unit, the 5 time delays in the second recording unit are shown in table 2.

TABLE 2

Location of time delay	1	2	3	4	5
						Value of time delay	0.32 milliseconds	0.35 msec	0.3 milliseconds	0.32 milliseconds	0.28 ms

Based on table 2, the first network device receives the 6 th packet of the target stream after the number of the packets is equal to the threshold, and obtains the delay of the 6 th packet through the target device, which is 0.25 ms. Then, the first network device queries the time delay in the position 1 in the second recording unit based on the polling mode, wherein the time delay is 0.32 milliseconds. Since 0.25 ms is less than 0.32 ms, the first network device keeps the delay of 0.32 ms in location 2 unchanged, i.e. the delay in location 1 is not updated this time.

Optionally, the tail delay of the target stream passing through the target device in the estimation window is subsequently estimated based on the n delays recorded in the second recording unit, if the number of packets of the target stream is very large, the n delays in the second recording unit are updated according to the polling manner, then the finally obtained n tail delays are the maximum n tail delays, and at this time, the n tail delays may be used to more accurately estimate the tail delay close to 100 decitex, but are used to estimate tail delays such as 80 decitex tail delay, 85 decitex tail delay, 90 decitex tail delay or 95 decitex tail delay, even 99 decitex tail delay, and the estimated tail delay is inaccurate due to a large error. In order to solve this problem, the embodiments of the present application provide the following operations:

on the basis of updating the second recording unit in the polling manner, after the number of the packets is equal to the threshold, the first network device receives a data packet of the target stream again after receiving a preset number of data packets of the target stream, may randomly query a time delay in the second recording unit, and then compare the time delay of the data packet passing through the target device with the queried time delay. And when the time delay of the data packet passing through the target equipment is smaller than the time delay of the query, replacing a time delay record of the query with the time delay of the data packet passing through the target equipment in the second recording unit, and otherwise, not updating the time delay of the query.

The preset number may be 10, 50, or 100, and the like, and the specific value of the preset number is not limited in the present application. For ease of understanding, the following is exemplified.

Still further based on the above table 1 as an example, assuming that the preset number is 8, after the number of the packets is equal to the threshold, the first network device receives 8 packets of the target stream, and polls and updates the second recording unit, and then 5 time delays in the second recording unit are referred to in table 3.

TABLE 3

Location of time delay	1	2	3	4	5
						Value of time delay	0.32 milliseconds	0.35 ms	0.31 ms	0.32 milliseconds	0.28 millisecond

Based on table 3, the first network device receives the 9 th packet of the target stream after the number of packets is equal to the threshold, and obtains the delay of the 9 th packet through the target device, which is 0.27 ms. Then, the first network device randomly queries a time delay among the time delays of the 5 positions of the second recording unit, and it is assumed that the time delay of the position 5 is randomly queried to be 0.28 ms. Since 0.27 ms is less than 0.28 ms, the first network device replaces 0.28 ms for 0.27 ms in this location 5, as can be seen in table 4.

TABLE 4

Location of time delay	1	2	3	4	5
						Value of time delay	0.32 milliseconds	0.35 ms	0.31 ms	0.32 milliseconds	0.27 ms

It should be noted that, because of the hash collision, the number of packets of the bit mapped unit having the hash collision is greater than the number of packets of the bit mapped unit having no hash collision among the plurality of first recording units. For the unit of bit mapping with hash collision, as long as the number of packets reaches the threshold, a second recording unit may also be configured for the unit, and then the second recording unit is used to record the time delay of a part of data packets received by the first network device passing through the target device after the number of packets is greater than the threshold, where the operation of updating the second recording unit may refer to the above description, and is not described herein again.

Based on the update method for the second recording unit, after the estimation window is ended, the first network device finds the associated second recording unit based on the plurality of first recording units of the first stream map, and specifically, may find the second recording unit associated with the first recording unit through the index information recorded in the index recording unit. Then, the first network device obtains the n time delays recorded in the second recording unit, and estimates the tail time delay of the first stream passing through the target device in the estimation window based on the n time delays. Specifically, assuming that the n delays are ordered from small to large, a p-fractional-bit tail delay of the first stream within the estimation window may be estimated by using a delay ordered as w from the n delays, where the delay ordered before the w is smaller than or equal to the p-fractional-bit tail delay, and the w is an integer greater than 0 and smaller than or equal to n.

In one possible implementation, to avoid data confusion caused by hash collision, n delays in the second recording unit associated with the first recording unit with the stream number of 1 or the smallest stream number may be selected to estimate the tail delay of the first stream passing through the target device within the estimation window.

In a possible implementation, it is assumed that N packets of the first flow are received by the first network device within the estimation window, where N is an integer greater than or equal to N. Then, w may be n (1-1/n) ^N(1-p) That is, the sequence of the n time delays is n (1-1/n) ^N(1-p) Estimating p fractional bits of the first stream within the estimation window with a large time delayTail time delay, the nth (1-1/n) ^N(1-p) The large delay is an estimated value of the nth × p large delay among the N delays through the target device corresponding to the N packets in the first stream. The specific estimation process can be exemplarily seen in the following calculation formula:

wherein, a _K A is a time delay of K-th time delay of the N time delays passing through the target device _j Represents the jth delay of the n delays, C represents the set of n delays in the second recording unit, and C _i Indicating the time delay of recording the ith position in the second recording unit, P (a) _K E.c) represents a _K Probability of being recorded by any position of the second recording unit, a _K meet c _i Denotes a _K Comparison in polling with the time delay recorded at the i-th position in the second recording unit, [ lambda ] denotes "and ″, a _K ＝max(a _j |a _j meet c _i ) Denotes a _K Equal to the maximum of all the delays compared to the delay of the ith position record. Formula calculation of _K The probability of being recorded.

Suppose a _K meet c _i Has a probability of 1/n, i.e. a _K The probability of encountering each number in C is the same, and the difference between the number of times of actually encountering the delays of different locations in the above-described polling scheme is at most 1, and therefore, it is assumed here that a _K meet c _i The error of 1/n is small. Based on this assumption, then:

then

Then

Wherein represents c _i >a _N*p It represents that the time delay of the ith position record is larger than the time delay of the p fractional tail. E (Num (c) _i >a _N*p ) A mathematical expectation, E (Num (c), representing the number of delays of the n delays greater than the p fractional tail delay _i ≤a _N*p ) A mathematical expectation that the n delays are not greater than the number of delays of the p fractional tail delays, i.e. theoretically, the expectation of the p fractional tail delay is the first of the n delays

Large recording delay, so w can be n (1-1/n) ^N(1-p) 。

In a possible implementation manner, for the case that the number of packets of the first stream is very large, the above-mentioned method of recording n time delays in the second recording unit uses a manner of randomly replacing the original large time delay with a small time delay every certain number of packets, so as to reduce the final estimation error. Then, the matched tail delay estimation processing method can be described as follows:

it is assumed that N packets of the first stream are received by the first network device within the estimation window, where N is an integer greater than or equal to N. Then, w may be n (1-q/n) ^N(1-p) That is, n (1-q/n) can be set as the sequence of the n delays ^N(1-p) Estimating the p fractional tail delay of the first stream within the estimation window by the large delay, the nth (1-q/n) ^N(1-p) The large delay is an estimated value of the nth × p large delay among the N delays through the target device corresponding to the N packets in the first flow, and q is a real number greater than 0 and less than 1. The specific estimation process can be exemplarily seen in the following calculation formula:

the meanings of the related characters can be referred to the related description in the above formula (1), and are not described herein again.

Similarly, assume a _K meet c _i Has a probability of 1/n, i.e. a _K The probability of encountering each number in C is the same, and the difference between the number of times the delays of different locations are actually encountered in the above-described polling scheme is at most 1, so it is assumed here that a _K meet c _i The error of 1/n is small. The accurate analysis of the problem is related to the time sequence of the arrival of the time delay, the solution method is complex, and the accurate solution is difficult, therefore, a mode of randomly replacing the original large time delay by the small time delay at every certain packet number and a default recording mode of the second recording unit are mutually independent, namely, a mode of replacing the original large time delay by the random small time delay by the q probability is executed by assuming that the default recording mode of the second recording unit is executed when polling comparison is performed every time, and the original large time delay is replaced by the random small time delay by the 1-q probability, wherein q is a real number which is set to be greater than 0 and less than 1. Based on this assumption, then:

then

Then

Wherein, q represents a default recording mode for executing the second recording unit with a probability of q, q is a set real number greater than 0 and smaller than 1, and the meaning represented by the related characters can refer to the related description in the above formula (1), and is not described herein again.

In a possible embodiment, regarding the processing of the information recorded in the plurality of first recording units of the first stream map after the end of the estimation window, the corresponding description in the case where the number of packets is less than the threshold value can be referred to. Regarding the plurality of second recording units associated with the plurality of first recording units, for the second recording units without hash collision, the first network device may release the storage space of these second recording units for subsequent recording storage of other information.

Based on the above-described method for estimating the tail delay, the first network device may estimate the size of the tail delay of the first stream passing through the target device within the estimation window. For the case that the estimation window is the preset time window, the transmission process of the first stream may include a plurality of preset time windows, and the first network device may estimate, in each of the plurality of preset time windows, a tail delay of the first stream passing through the target device in the window, where the specific estimation process may refer to the foregoing description. The first network device may send the tail delay to the controller for subsequent summary analysis and processing each time the tail delay of the first flow passing through the target device within a preset time window is estimated. Or the first network device may send the multiple tail delays to the controller together for subsequent summary analysis and processing after estimating the tail delays of the first stream passing through the target device in the multiple preset time windows. For the case that the estimation window is the entire transmission time of the stream, the first network device may send the estimated tail delay of the first stream passing through the target device to the controller after estimating the tail delay of the first stream passing through the target device, and the controller further performs summary analysis and processing according to the tail delay information.

The controller may receive, in addition to the tail delay information of the first stream from the first network device passing through the target device, tail delay information of the first stream from other network devices through which the first stream passes in the data plane, where estimation and processing operations of each of the other network devices with respect to the tail delay of the first stream passing through the target device are the same as those of the first network device, and thus, are not described again.

After receiving the tail delay information of the first stream passing through the target device, sent by the first network device and the other network devices, the controller may summarize the tail delay information in a time sequence, for example, the tail delay estimation data in a longer time window (up to the stream life cycle) may be obtained by weighting and dividing the bits by using the number of packets as a weight. And then, based on the tail delay estimation data, tail delay mutation early warning can be carried out according to the requirement of network operation and maintenance, so that the abnormity such as congestion is detected, the network space drawing of the tail delay can be carried out, and the network slice analysis and the network bottleneck detection based on the delay are obtained. According to the obtained full-volume stream tail delay data, various network measurement tools facing users and operation and maintenance personnel can be deployed.

In a possible embodiment, the operation of estimating the tail delay of the first stream in the corresponding estimation window based on the information recorded in the first recording unit or the first recording unit and the second recording unit may be implemented by a controller. Specifically, the first network device records relevant information in the first recording unit of the first stream mapping or the first recording unit and the second recording unit in the estimation window, and the specific recorded information refers to the foregoing description, and is not described herein again. Then, after the estimation window ends, the first network device sends the information recorded in the first recording unit, or the first recording unit and the second recording unit, to the controller, and optionally, the information in the index recording unit that records the index information between the first recording unit and the second recording unit is also sent to the controller together. The controller estimates, based on the information, a tail delay of the first stream passing through the target device in the corresponding estimation window, and the specific estimation manner may refer to the estimation manner of the first network device, which is not described herein again.

Optionally, for a case that only the first time delay and the number of packets are recorded in the first recording unit, when the first network device sends information in the first recording unit to the controller, the first network device may send stream information of the stream mapped to the first recording unit to the controller together, so that the controller associates the information recorded in the first recording unit with the corresponding stream.

For a better understanding of the communication delay estimation method described above, it will be further illustrated with reference to the accompanying drawings.

Referring to fig. 6, fig. 6 is a schematic diagram illustrating the processing flow architecture of the first network device. In fig. 6, the whole process involves a bloom filter, a stream encoder, a long stream check unit, a long stream delay recording unit, and an information recording unit.

Specifically, the first network device hashes the stream information of the first stream into a plurality of bits in a bitmap through a bloom filter, and a plurality of first recording units in the information recording unit mapped by the first stream can be determined through the plurality of bits.

The first network device performs the xor operation on the introduction stream information by the stream encoder, records the result of the xor operation into the first recording unit, or may record the stream information of the first stream into the first recording unit, and may further record the delay information, the number of packets, and the number of streams into a plurality of first recording units of the first stream map, and return the number of packets recorded in each of the plurality of first recording units.

The first network device checks whether the number equal to the threshold value exists in the plurality of packet numbers returned by the stream encoder through the long stream checking unit, if so, allocates a second recording unit to the first recording unit corresponding to the packet number equal to the threshold value, allocates an index recording unit for recording index information between the first recording unit and the second recording unit, and returns information of the specific position of the second recording unit in the information recording unit.

When the number of packets recorded in the first recording unit corresponding to the number of packets equal to the threshold is greater than the threshold, the first network device records the delay of the larger data packet passing through the target device into the second recording unit through the long-stream delay recording unit in the polling manner described above. Optionally, the first network device may further randomly select one of the time delays recorded by the second recording unit to be replaced with a smaller time delay through the long flow time delay recording unit according to the above-described procedure for replacing every certain number of data packets.

For specific implementation of the operations executed by the bloom filter, the stream encoder, the long stream check unit, and the long stream delay recording unit, reference may be made to the foregoing description, and details are not described herein again.

In fig. 6, it can also be seen that the total storage size allocated to all first recording units in an information recording unit is k (9 + x) bytes (B), where k is the number of bits in the bitmap, since there is one first recording unit mapped to each bit, and k is an integer greater than 1. In addition, four items of information, i.e., an exclusive or result of stream information, the number of streams, the number of packets, and the first delay, may be included in each first recording unit, and illustratively, the number of streams may be recorded in 1 byte, the number of packets may be recorded in 4 bytes, the first delay may be recorded in 4 bytes, and the exclusive or result of stream information may be recorded in x bytes, where x may be configured as needed. From this, the total storage size allocated to all the first recording units can be calculated as k (9 + x) bytes. In addition, the total memory size allocated to all index recording units in an information recording unit is k × log2 (M)/8 bytes, and the total memory size allocated to all second recording units is M × n × 4 bytes. The number M is the number of the second recording units, which may be estimated according to the number of concurrent streams that may exceed a threshold, specifically, the number may be estimated by using the existing stream size distribution and the result of the network slice, where n is the number of delay information that can be recorded by each second recording unit at most, the size of each second recording unit is n × 4 bytes, and each delay information is recorded by 4 bytes.

It should be noted that, the specific number of bytes given here is only an example, and does not constitute a limitation to the embodiments of the present application, and the specific implementation may be to record the information by using bytes of other sizes.

Referring to fig. 7, in fig. 7, each network device may be the first network device described above, and then each network device may implement the operations implemented by the first network device described above, or the first network device is any one of the network devices shown in fig. 7. Specifically, a control information channel is first established between each network device and the controller, the controller sets controller information such as an estimation window, a quantile point, a monitoring threshold value, and the like, and then the control information is sent to each network device through the control information channel. Then, the each network device and the controller establish a data transfer channel, and the controller transmits a data signal of the tail delay measurement of the stream to the each network device. And each network device starts to record the information related to the time delay of the received stream according to the data signal, wherein the specific recorded information refers to the information recorded in the first recording unit and the second recording unit.

After the first recording unit, or the first recording unit and the second recording unit, record relevant information, in one possible implementation, each network device sends the information recorded by the recording unit to the controller through its own device control plane, the controller performs stream tail delay estimation based on the information, and then the controller performs summary analysis on the estimated tail delay, and uses the analyzed data for relevant processing needs such as operation and maintenance. The specific estimation process and the summary analysis can be referred to the related description, and are not described herein. As for the process of transmitting the information recorded in the recording unit to the controller, reference may be made to fig. 8 as an example.

Referring to fig. 8, in the example shown in fig. 8, each network device in the data plane may perform the operations performed by the first network device, and therefore, each network device records information about packet latency. Or taking the first network device as an example, after the corresponding estimation window ends, the first network device transfers the information recorded by the recording unit to its own device control plane, specifically, the device control plane may obtain the information in the recording units through pull operation, or the first network device pushes a register group formed by the recording units to its own device control plane. Specifically, the recording units may include information recorded in the first recording unit, the index recording unit, and the second recording unit of the first stream map, and fig. 8 illustrates three first recording units of the first stream map and three corresponding second recording units, but the number of the first and second recording units is not limited in specific implementation.

In addition, in the embodiment shown in fig. 8, assuming that the estimation of the tail delay of the first flow passing through the target device is performed by the controller, the device control plane of the first network device may send the obtained information of the recording unit to the controller in the form of a data packet through the established data transmission channel. The format of a specific transmitted packet may be referred to as the format shown in fig. 8, and the packet includes a network device identification (sid), an estimation window identifier (tw), a data type (type), and information recorded in a recording unit. The network device identifies the network device used for sending the data packet, the estimation window identifier is used for identifying which estimation window the information in the sent recording unit is recorded in, and the data type is used for identifying whether the load data in the data packet is the information in the first recording unit, the information in the index recording unit or the information in the second recording unit. In fig. 8, for example, a data type of 1 indicates that the payload data in the data packet is information in the first recording unit, a data type of 2 indicates that the payload data in the data packet is information in the index recording unit, and a data type of 3 indicates that the payload data in the data packet is information in the second recording unit.

In another possible implementation, after the corresponding estimation window is ended, the recorded information may be sent to the own device control plane to perform corresponding tail delay estimation, and for specific estimation, reference may be made to the description of the first network device estimating the tail delay of the first stream passing through the target device. Then, each network device sends the estimated tail delay information of the stream to the controller, the controller collects and analyzes the tail delay information, and the analyzed data is used for relevant processing requirements such as operation and maintenance. For example, or taking the first network device as an example, the specific manner in which the first network device transfers the information recorded by the recording unit to the device control plane of the first network device may refer to the corresponding description in fig. 8, and details are not described here again. After the device control plane of the first network device obtains the information, the device control plane estimates and obtains the tail delay of the first flow passing through the target device in the corresponding estimation window, and then the tail delay can be sent to the controller through the established data transmission channel. Specifically, the tail delay is also encapsulated in a data packet and sent, and in order to identify that the data in the data packet is a data packet with tail delay, a TCP reserved field special value may be set in the packet header of the data packet, so that after the controller receives the data, it finds that the TCP reserved field special value exists in the packet header, and may determine that the data packet is a data packet including the tail delay of the stream. In addition, the data packet also includes the above-mentioned network device identification (sid), estimation window identifier (tw), and data type (type), and may further include information such as stream information of the first stream and the number of packets of the first stream received within the corresponding estimation window.

Optionally, for a case that the number of packets of the target stream received by the first network device in the estimation window is smaller than the threshold, since the first recording unit mapped by the target stream does not have an associated second recording unit, the information passed by the first network device to the device control plane of the first network device does not include information of the second recording unit and the index recording unit, and the rest of the information is still sent, which may specifically refer to the content described above.

In the above-described embodiment, most of the first stream and the first packet are described as an example, however, the first network device may record information of each stream passing through itself in the above-described manner, so that estimation of tail delay of each stream, that is, estimation of tail delay of a full-volume stream may be implemented.

In summary, in the embodiment of the present application, measurement of tail delay information of a stream-by-stream may be implemented, and tail delay information of a full-volume stream of a fine granularity is obtained. Also, the measurement of 99-quantile latency for 1M streams can be supported without exceeding 30 MB. Meanwhile, the data plane logic of the embodiment of the application can be deployed in a programmable data plane, for example, on a bareboot Tofino switch, belongs to the preset capability of network equipment, and has better deployability. In addition, under the condition that the number of the packets is larger than the threshold, the information of the second recording unit is updated in a mode that the time delay in the second recording unit is randomly selected to be replaced by a smaller time delay at intervals of a certain number of the packets through the polling mode, and compared with a random sketch algorithm with the same space overhead, the measurement accuracy of the tail time delay of the stream is remarkably improved. For example, an NS3 simulation platform is used in a simulation experiment, a network topology is a Fat Tree Fat-Tree topology common to a data center, open source data of a large data center is used for flow distribution data, a network load is set at 50%, and flows at the same end satisfy poisson arrival. N =10 for the second recording unit, the random sketch algorithm uses the same memory space as the embodiments of the present application under this condition for sampling and storing the stream delay. The estimation window is set to 100ms, and other network settings are also reasonable values, and the simulation result shown in fig. 9 can be obtained in this experimental environment. In a simulation result, for the 99-minute tail delay, the measurement error is reduced by 96.5%; and for the tail time delay of 95 minutes, the measurement error is reduced by 80.7 percent.

Compared with the existing tail time delay measurement scheme, in the embodiment of the application, the relevant information of the tail time delay of the stream is acquired in the estimation window, and the various recording units are adopted to record the information, so that the recording of unnecessary information is reduced, and the effective information is finally obtained, thereby effectively reducing the repeated or invalid information reporting, and greatly reducing the generated bandwidth extra load.

In summary, the embodiment of the present application can reduce the resource overhead of the network device in the tail delay measurement process of each flow, so that the network device can support the flow-by-flow tail delay measurement task of a large number of concurrent flows in a large-scale network in limited storage resources, and simultaneously, no huge extra load is generated, and good measurement accuracy is obtained.

The foregoing mainly introduces a communication delay estimation method provided in this embodiment of the present application. It is understood that each device comprises corresponding hardware structures and/or software modules for executing each function in order to realize the corresponding function. The elements and steps of the various examples described in connection with the embodiments disclosed herein may be embodied as hardware or a combination of hardware and computer software. Whether a function is performed in hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the embodiments of the present application.

In the embodiment of the present application, the device may be divided into the functional modules according to the method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that the division of the modules in the embodiment is illustrative, and is only one division of logic functions, and there may be another division manner in actual implementation.

In the case of dividing each functional module according to each function, fig. 10 shows a schematic diagram of a possible logical structure of an apparatus, where the apparatus may be the first network device described in the foregoing method embodiment, or may be a chip in the first network device, or may be a processing system in the first network device, or the like. The apparatus 1000 includes a processing unit 1001, and optionally, may further include a transmitting unit 1002 and/or a receiving unit 1003. Wherein:

a processing unit 1001 for:

acquiring flow information from a received first data packet, wherein the flow information is identification information of a first flow to which the first data packet belongs; the processing unit 1001 may be configured to implement the operation of acquiring the flow information in step 301 in fig. 3;

determining a first recording unit of the first stream map based on the stream information; the first recording unit comprises a first time delay and a packet number, wherein the first time delay is the maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is the number of the data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit; the processing unit 1001 may be configured to implement the operation of determining the first recording unit of the first stream map in step 302 in fig. 3;

updating the number of the packets, and updating the first time delay to the time delay of the first data packet passing through the target device under the condition that the time delay of the first data packet passing through the target device is larger than the first time delay; the processing unit 1001 may be configured to implement the updating operation in step 303 in fig. 3; in the case where the number of packets is less than a threshold, the first delay is used to estimate a tail delay of the first stream through the target device.

In a possible embodiment, the first stream map has a plurality of first recording units; the processing unit 1001 is further configured to: after the estimation window is finished, under the condition that the number of the packets recorded by at least one unit in the plurality of first recording units is smaller than the threshold value, determining that the first time delay recorded in the first unit is a tail time delay estimation value of the first stream passing through target equipment in the estimation window, and sending the first time delay recorded in the first unit to a controller; the first unit is a unit with the minimum first time delay recorded in the plurality of first recording units.

In one possible embodiment, the first stream map includes a plurality of first recording units; the network device further comprises a sending unit 1002 for sending the information recorded in the plurality of first recording units to the controller in case the estimation window ends.

In a possible implementation manner, in a case that the number of the packets recorded in the first recording unit is greater than or equal to the threshold, the processing unit 1001 is further configured to:

allocating a second recording unit to the first recording unit; the second recording unit is configured to record a time delay of each data packet of n data packets passing through the target device, where the n data packets are data packets in the data packet of the target stream received by the network device after the first data packet, and n is an integer greater than 1; the n delays are used to estimate the tail delay of the first stream through the target device.

In a possible implementation manner, the network device further includes a receiving unit 1003, configured to receive a second data packet, where the second data packet is a data packet of the target stream;

the processing unit 1001 is further configured to obtain a second time delay of the n time delays of the second recording unit according to a polling manner; and updating the second time delay in the second recording unit to be the time delay of the second data packet passing through the target device under the condition that the time delay of the second data packet passing through the target device is larger than the second time delay.

In a possible embodiment, the receiving unit 1003 is further configured to receive a third data packet, where the third data packet is a data packet of the destination stream, and before the third data packet and after the second data packet, the receiving unit 1003 receives m data packets of the destination stream, where m is an integer greater than 1;

the processing unit 1001 is further configured to randomly acquire a third time delay from the n time delays of the second recording unit; and updating the third time delay in the second recording unit to be the time delay of the third data packet passing through the target device under the condition that the time delay of the third data packet passing through the target device is smaller than the third time delay.

In a possible implementation, the processing unit 1001 is further configured to: and under the condition that the estimation window is ended, estimating the tail time delay of the first stream passing through the target device in the estimation window based on the n time delays in the second recording unit.

Optionally, the tail delay of the first stream passing through the target device is p fractional tail delay, where p is greater than 80 and less than 100; the processing unit 1001 estimates a tail delay of the first stream passing through the target device based on the n delays in the second recording unit, and includes: determining a w-th time delay of the n time delays as a tail time delay estimation value of the first stream passing through the target device in the estimation window, wherein the w-th time delay is a time delay ordered from small to large and is an integer which is larger than 0 and smaller than or equal to n.

In a possible implementation, the network device further includes a sending unit 1002, configured to send information recorded in the first recording unit and the second recording unit to the controller in case that the estimation window ends.

In one possible embodiment, the processing unit 1001, based on the stream information, determines the first recording unit of the first stream map, and includes: mapping the stream information to a plurality of bits in a bitmap by a plurality of hash functions; a bit in the bitmap is associated with one of the first recording units; the network device determines the first recording unit of the first stream map based on the plurality of bits.

For specific operations and benefits of each unit in the apparatus 1000 shown in fig. 10, reference may be made to the corresponding description in fig. 3 and its possible method embodiment, which is not described herein again.

In the case of dividing each functional module according to each function, fig. 11 shows a schematic diagram of a possible logical structure of an apparatus, where the apparatus may be the first network device described in the foregoing method embodiment, or may be a chip in the first network device, or may be a processing system in the first network device, and the like. The apparatus 1100 comprises a processing unit 1101 and a receiving unit 1102. Wherein:

a processing unit 1101 for:

acquiring flow information from a first packet received by the receiving unit 1102, where the flow information is identification information of a first flow to which the first packet belongs; the processing unit 1001 may be configured to implement the operation of acquiring the flow information in step 301 in fig. 3;

updating the number of the packets, and updating the first time delay to the time delay of the first data packet passing through the target device under the condition that the time delay of the first data packet passing through the target device is larger than the first time delay; the processing unit 1001 may be configured to implement the updating operation in step 303 in fig. 3;

allocating a second recording unit to the first recording unit when the number of packets is greater than or equal to a threshold; the second recording unit is configured to record a time delay of each of n data packets passing through the destination device, where the n data packets are partial data packets in a data packet of the destination stream received by the network device after the first data packet, and n is an integer greater than 1; the n delays are used to estimate a tail delay of the first stream through the target device.

In the case of dividing each functional module according to each function, fig. 12 shows a schematic diagram of a possible logical structure of an apparatus, which may be the controller described in the above method embodiment, or may be a chip in the controller, or may be a processing system in the controller, and the like. The apparatus 1200 comprises a receiving unit 1201 and a processing unit 1202. Wherein:

a receiving unit 1201, configured to receive information recorded in multiple first recording units, where the information recorded in each first recording unit includes a first time delay and a packet number, the first time delay is a maximum time delay when a single data packet of a target stream passes through a target device, and the packet number is a number of data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit, the target stream includes a first stream;

a processing unit 1202, configured to estimate a tail delay of the first stream passing through the target device based on the first delay recorded in the plurality of first recording units when the number of packets is less than a threshold.

In a possible implementation, the receiving unit 1201 is further configured to receive an identifier of an estimation window, where the identifier indicates that the information recorded in the plurality of first recording units is recorded in the estimation window, and the estimation window is a time window used for estimating a tail delay of the first stream passing through the target device;

the estimating, by the processing unit 1202, a tail delay of the first stream passing through the target device based on the first time delays recorded in the plurality of first recording units includes: determining the first delay recorded in the first unit as a tail delay estimate of the first stream passing through a target device within the estimation window; the first unit is a unit with the minimum first time delay recorded in the plurality of first recording units.

In the case of dividing each functional module according to each function, fig. 13 shows a schematic diagram of a possible logical structure of an apparatus, which may be the controller described in the above method embodiment, or may be a chip in the controller, or may be a processing system in the controller, and the like. The apparatus 1300 comprises a receiving unit 1301 and a processing unit 1302. Wherein:

a receiving unit 1301, configured to receive information recorded in a first recording unit and a second recording unit, where the information recorded in the first recording unit includes a first time delay and a packet number, the first time delay is a maximum time delay of a single data packet of a first flow passing through a target device, and the packet number is a number of data packets of the first flow received by the network device; the second recording unit is associated with the first recording unit, the information recorded in the second recording unit includes a time delay of each of n data packets in the first stream passing through the target device, and n is an integer greater than 1;

a processing unit 1302, configured to estimate a tail delay of the first stream passing through the target device based on the n delays.

In a possible implementation manner, the receiving unit 1301 is further configured to receive an identifier of an estimation window, where the identifier indicates that the information recorded in the first recording unit and the second recording unit is recorded in the estimation window, and the estimation window is a time window used for estimating a tail delay of the first stream passing through the target device;

the tail delay of the first stream passing through the target device is p fractional tail delay, and p is more than 80% and less than 100%; the processing unit 1302 estimates a tail delay of the first stream passing through the target device based on the n delays, including:

Fig. 14 is a schematic diagram illustrating a possible hardware structure of an apparatus according to an embodiment of the present application, where the apparatus may be the first network device described in the foregoing embodiment, or may be a chip in the first network device, or may be a processing system in the first network device, and the like. The apparatus 1400 comprises: a processor 1401, a memory 1402, and a communication interface 1403. The processor 1401, the communication interface 1403 and the memory 1402 may be interconnected with each other or via a bus 1404.

Illustratively, the memory 1402 is used for storing computer programs and data of the apparatus 1400, and the memory 1402 may include, but is not limited to, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), and the like.

The communication interfaces 1403 include a transmitting interface and a receiving interface, and the number of the communication interfaces 1403 may be multiple, which are used for supporting the apparatus 1400 to perform communication, such as receiving or transmitting data or messages.

The processor 1401 may be, for example, a central processing unit, a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, transistor logic, a hardware component, or any combination thereof. A processor may also be a combination of computing functions, e.g., a combination comprising one or more microprocessors, a digital signal processor and a microprocessor, or the like. The processor 1401 may be configured to read the program stored in the memory 1402, so as to enable the apparatus 1400 to perform the operations performed by the first network device in any one of the communication delay estimation methods described in fig. 3 and the possible embodiments thereof.

In one possible implementation, the processor 1401 may be configured to perform the following operations: acquiring flow information from a received first data packet, wherein the flow information is identification information of a first flow to which the first data packet belongs; determining a first recording unit of the first stream map based on the stream information; the first recording unit comprises a first time delay and a packet number, wherein the first time delay is the maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is the number of the data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit; updating the number of the packets, and updating the first time delay to the time delay of the first data packet passing through the target device under the condition that the time delay of the first data packet passing through the target device is larger than the first time delay; in the case that the number of packets is less than a threshold, the first delay is used to estimate a tail delay of the first stream through the target device.

For specific operations and benefits of each unit in the apparatus 1400 shown in fig. 14, reference may be made to the corresponding description in fig. 3 and possible method embodiments thereof, which are not described herein again.

Fig. 15 is a schematic diagram illustrating a possible hardware structure of an apparatus according to an embodiment of the present application, where the apparatus may be the first network device described in the foregoing embodiment, or may be a chip in the first network device, or may be a processing system in the first network device, or the like. The apparatus 1500 includes: a processor 1501, memory 1502, and a communication interface 1503. The processor 1501, communication interface 1503, and memory 1502 may be connected to each other or to each other through a bus 1504.

Illustratively, the memory 1502 is used for storing computer programs and data of the apparatus 1500, and the memory 1502 may include, but is not limited to, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or portable read-only memory (CD-ROM), etc.

The communication interface 1503 includes a sending interface and a receiving interface, and the number of the communication interfaces 1503 may be multiple, which are used for supporting the apparatus 1500 to perform communication, such as receiving or sending data or messages.

Illustratively, the processor 1501 may be a central processing unit, a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, transistor logic, a hardware component, or any combination thereof. A processor may also be a combination of computing functions, e.g., a combination of one or more microprocessors, a digital signal processor and a microprocessor, or the like. The processor 1501 may be configured to read the program stored in the memory 1502, so that the apparatus 1500 executes the operations performed by the first network device in any one of the communication delay estimation methods described in the above fig. 3 and its possible embodiments.

In one possible implementation, the processor 1501 may be configured to perform the following operations: acquiring flow information from the received first data packet, wherein the flow information is used for indicating that the first data packet is a data packet in a first flow; determining a first recording unit of the first stream map based on the stream information; the first recording unit is configured to record a first time delay and a packet number, where the first time delay is a maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is a number of data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit; updating the number of the packets, and updating the first time delay to the time delay of the first data packet passing through the target device under the condition that the time delay of the first data packet passing through the target device is larger than the first time delay; allocating a second recording unit to the first recording unit when the number of packets is greater than or equal to a threshold; the second recording unit is configured to record a time delay of each of n data packets passing through the target device, where the n data packets are partial data packets in a data packet of the target stream received by the network device after the first data packet, and n is an integer greater than 1; the n delays are used to estimate a tail delay of the first stream through the target device.

For specific operations and beneficial effects of each unit in the apparatus 1500 shown in fig. 15, reference may be made to the corresponding description in fig. 3 and its possible method embodiment, which is not described herein again.

Fig. 16 is a schematic diagram illustrating a possible hardware structure of an apparatus according to an embodiment of the present application, where the apparatus may be the controller according to the foregoing embodiment, or may be a chip in the controller, or may be a processing system in the controller, or the like. The apparatus 1600 includes: a processor 1601, a memory 1602, and a communication interface 1603. The processor 1601, the communication interface 1603, and the memory 1602 may be connected to each other or to each other via a bus 1604.

Illustratively, the memory 1602 is used for storing computer programs and data of the apparatus 1600, and the memory 1602 may include, but is not limited to, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), and the like.

The communication interfaces 1603 include a sending interface and a receiving interface, and the number of the communication interfaces 1603 may be multiple, and the communication interfaces are used for supporting the apparatus 1600 to perform communication, such as receiving or sending data or messages.

Illustratively, the processor 1601 may be a central processing unit, a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, transistor logic, a hardware component, or any combination thereof. A processor may also be a combination of computing functions, e.g., a combination of one or more microprocessors, a digital signal processor and a microprocessor, or the like. The processor 1601 is configured to read the program stored in the memory 1602, so that the apparatus 1600 performs the operations performed by the controller in the communication delay estimation method as described in the above fig. 3 and any possible embodiment thereof.

In one possible implementation, the processor 1601 may be configured to perform the following operations: receiving information recorded in a plurality of first recording units, wherein the information recorded in each first recording unit comprises a first time delay and a packet number, the first time delay is the maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is the number of the data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit, the target stream includes a first stream; and under the condition that the number of the packets is less than a threshold value, estimating the tail time delay of the first stream passing through the target device based on the first time delays recorded in the plurality of first recording units.

For specific operations and advantages of each unit in the apparatus 1600 shown in fig. 16, reference may be made to the above description of fig. 3 and corresponding descriptions in possible method embodiments, which are not repeated herein.

Fig. 17 is a schematic diagram illustrating a possible hardware structure of an apparatus according to an embodiment of the present application, where the apparatus may be the controller according to the above-described embodiment, or may be a chip in the controller, or may be a processing system in the controller, or the like. The apparatus 1700 includes: a processor 1701, a memory 1702, and a communication interface 1703. The processor 1701, the communication interface 1703, and the memory 1702 may be connected to each other or to each other through the bus 1704.

Illustratively, the memory 1702 is used for storing computer programs and data of the apparatus 1700, and the memory 1702 may include, but is not limited to, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), and the like.

The communication interface 1703 includes a sending interface and a receiving interface, and the number of the communication interfaces 1703 may be multiple, and is used for supporting the apparatus 1700 to perform communication, such as receiving or sending data or messages.

The processor 1701 may be, for example, a central processing unit, a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, transistor logic, a hardware component, or any combination thereof. A processor may also be a combination of computing functions, e.g., a combination of one or more microprocessors, a digital signal processor and a microprocessor, or the like. The processor 1701 may be configured to read the program stored in the memory 1702 described above, so that the apparatus 1700 performs the operations performed by the controller in any of the communication delay estimation methods described above in fig. 3 and its possible embodiments.

In one possible implementation, the processor 1701 may be configured to perform the following operations: receiving information recorded in a first recording unit and a second recording unit, wherein the information recorded in the first recording unit comprises a first time delay and a packet number, the first time delay is the maximum time delay of a single data packet of a first stream passing through a target device, and the packet number is the number of the data packets of the first stream received by the network device; the second recording unit is associated with the first recording unit, the information recorded in the second recording unit includes a time delay of each of n data packets in the first stream passing through the target device, where n is an integer greater than 1; and estimating the tail time delay of the first flow passing through the target equipment based on the n time delays.

For specific operations and benefits of each unit in the apparatus 1700 shown in fig. 17, reference may be made to the corresponding description in fig. 3 and possible method embodiments thereof, which are not described herein again.

An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored, and the computer program is executed by a processor to implement the operations performed by the first network device in the method described in fig. 3 and any of the possible method embodiments thereof.

An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored, and the computer program is executed by a processor to implement the operations performed by the controller in the method described in fig. 3 and any possible method embodiment thereof.

Embodiments of the present application further provide a computer program product, which when being read and executed by a computer, performs the operations performed by the first network device in the method described in any of the above fig. 3 and possible method embodiments.

Embodiments of the present application further provide a computer program product, which when being read and executed by a computer, performs the operations performed by the controller in the method described in any of the above-mentioned fig. 3 and possible method embodiments.

To sum up, the embodiment of the present application can reduce the resource overhead of the network device in the tail delay measurement process of each flow, so that the network device can measure and estimate the tail delays of all flows passing through the network device based on the existing storage and processing resources.

In this application, the terms "first," "second," and the like are used for distinguishing identical or similar items with substantially identical functions and functionalities, and it should be understood that "first," "second," and "n" have no logical or temporal dependency, and no limitation on the number or execution order. It will be further understood that, although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first latency may be referred to as a second latency, and similarly, a second latency may be referred to as a first latency, without departing from the scope of various described examples. Both the first and second time delays may be time delays, and in some cases, may be separate and distinct time delays.

It should also be understood that, in each embodiment of the present application, the sequence number of each process does not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiment of the present application.

It will be further understood that the terms "comprises," "comprising," "includes," and/or "including," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It should also be appreciated that reference throughout this specification to "one embodiment," "an embodiment," or "one possible implementation" means that a particular feature, structure, or characteristic described in connection with the embodiment or implementation is included in at least one embodiment of an embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "one possible implementation" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A communication delay estimation method, the method comprising:

the method comprises the steps that the network equipment acquires flow information from a received first data packet, wherein the flow information is identification information of a first flow to which the first data packet belongs;

the network device determining a first recording unit of the first stream map based on the stream information; the first recording unit comprises a first time delay and a packet number, the first time delay is the maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is the number of the data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit;

the network equipment updates the packet number, and updates the first time delay to the time delay of the first data packet passing through the target equipment under the condition that the time delay of the first data packet passing through the target equipment is larger than the first time delay;

the first latency is used to estimate a tail latency of the first stream passing the target device if the number of packets is less than a threshold.

2. The method of claim 1, wherein the target device is the network device; alternatively, the first and second liquid crystal display panels may be,

the network device is a device on a data plane in a network, the network device is a last hop device of the target flow in the data plane, and the target device includes a plurality of devices through which the single data packet passes in the data plane.

3. The method of claim 1 or 2, wherein the network device is a device on a data plane in a network, wherein the network device is a last hop device of a packet of the target stream in the data plane, and wherein the target device comprises a plurality of devices through which the single packet passes in the data plane;

the time delay of the first data packet passing through the target device is the time delay of the first data packet passing through the plurality of devices;

the first data packet includes a time delay of the first data packet through other devices, and the other devices are other network devices except the network device in the plurality of devices.

4. The method of claim 1 or 2, wherein the network device is a device on a data plane in a network, wherein the network device is a last hop device of a packet of the destination flow in the data plane, and wherein the destination device comprises a plurality of devices through which the single packet passes in the data plane;

the first data packet comprises a timestamp of an entry of a first hop device, through which the first data packet passes in the data plane, and the timestamp is used for calculating the time delay of the first data packet through the multiple devices.

5. The method according to any of claims 1-4, wherein the packets of the target stream received by the network device are packets received by the network device within an estimation window, the estimation window being a time window for estimating a tail delay of the first stream passing through the target device;

the first delay is the maximum delay of a single data packet of the target stream passing through the target device, and comprises the following steps: the first delay is the maximum delay of a single data packet of the target stream passing through the target device within the time window;

the tail delay of the first stream passing through the target device is the tail delay of the first stream passing through the target device within the time window.

6. The method according to claim 5, wherein the first recording unit of the first stream map is plural; the method further comprises the following steps:

after the estimation window is ended, under the condition that the number of packets recorded by at least one unit in the plurality of first recording units is smaller than the threshold, the network device determines that the first time delay recorded in the first unit is a tail time delay estimation value of the first stream passing through the target device in the estimation window, and sends the first time delay recorded in the first unit to a controller; the first unit is a unit in which the first time delay recorded in the plurality of first recording units is minimum.

7. The method according to claim 5, wherein the first recording unit of the first stream map is plural; the method further comprises the following steps:

in a case where the estimation window ends, the network device transmits information recorded in the plurality of first recording units to a controller.

8. The method according to claim 5, wherein in a case where the number of packets recorded in the first recording unit is greater than or equal to the threshold value, the method further comprises:

the network equipment allocates a second recording unit to the first recording unit; the second recording unit is configured to record a time delay of each data packet of n data packets passing through the target device, where the n data packets are data packets in a data packet of the target stream received by the network device after the first data packet, and n is an integer greater than 1; the n delays are used to estimate a tail delay of the first stream through the target device.

9. The method of claim 8, further comprising:

the network equipment receives a second data packet, wherein the second data packet is a data packet of the target stream;

the network equipment acquires a second time delay of the n time delays of the second recording unit according to a polling mode;

and under the condition that the time delay of the second data packet passing through the target equipment is greater than the second time delay, the network equipment updates the second time delay in the second recording unit to the time delay of the second data packet passing through the target equipment.

10. The method of claim 8, further comprising:

the network device receives a third data packet, where the third data packet is a data packet of the target stream, and before the third data packet and after the second data packet, the network device receives m data packets of the target stream, where m is an integer greater than 1;

the network equipment randomly acquires a third time delay from the n time delays of the second recording unit;

and under the condition that the time delay of the third data packet passing through the target equipment is smaller than the third time delay, the network equipment updates the third time delay in the second recording unit into the time delay of the third data packet passing through the target equipment.

11. The method according to claim 9 or 10, characterized in that the method further comprises:

and under the condition that the estimation window is ended, the network equipment estimates the tail time delay of the first stream passing through the target equipment in the estimation window based on the n time delays in the second recording unit.

12. The method of claim 11, wherein a tail delay of the first stream through the target device is a p-fractional tail delay, p being greater than 80 and less than 100;

the network device estimates a tail delay of the first stream passing through the target device based on the n delays in the second recording unit, and the method comprises the following steps:

and the network equipment determines the w-th time delay of the n time delays as a tail time delay estimated value of the first stream passing through the target equipment in the estimation window, wherein the w-th time delay is the time delay sequenced from small to large as w in the n time delays, and the w is an integer which is greater than 0 and less than or equal to n.

13. The method according to claim 9 or 10, characterized in that the method further comprises:

in a case where the estimation window ends, the network device transmits information recorded in the first recording unit and the second recording unit to a controller.

14. The method of any of claims 1-11, wherein the network device determining a first recording unit of the first stream map based on the stream information comprises:

the network device maps the flow information to a plurality of bits in a bitmap through a plurality of hash functions; one bit in the bitmap is associated with one of the first recording units;

the network device determines the first recording unit of the first stream map based on the plurality of bits.

15. The method according to any one of claims 1 to 14, wherein the first recording unit further includes an exclusive or result of stream numbers and stream information, the stream numbers are the number of the target streams, and the exclusive or result of the stream information is a result obtained by exclusive or operation on the stream information of the target streams.

16. A method for estimating communication delay, the method comprising:

the network equipment acquires flow information from the received first data packet, wherein the flow information is used for indicating that the first data packet is a data packet in a first flow;

the network device determining a first recording unit of the first stream map based on the stream information; the first recording unit is configured to record a first time delay and a packet number, where the first time delay is a maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is a number of data packets of the target stream received by the network device; the target stream includes a stream mapped to the first recording unit;

the network equipment updates the number of the packets, and updates the first time delay to the time delay of the first data packet passing through the target equipment under the condition that the time delay of the first data packet passing through the target equipment is larger than the first time delay;

when the number of the packets is larger than or equal to a threshold value, the network equipment allocates a second recording unit to the first recording unit; the second recording unit is configured to record a time delay of each data packet of n data packets passing through the target device, where the n data packets are partial data packets in a data packet of the target stream received by the network device after the first data packet, and n is an integer greater than 1; the n delays are used to estimate a tail delay of the first stream through the target device.

17. A communication delay estimation method, the method comprising:

the method comprises the steps that a controller receives information recorded in a plurality of first recording units, wherein the information recorded in each first recording unit comprises a first time delay and a packet number, the first time delay is the maximum time delay of a single data packet of a target stream passing through a target device, and the packet number is the number of the data packets of the target stream received by a network device; the target stream includes a stream mapped to the first recording unit, the target stream includes a first stream;

in a case where the number of packets is less than a threshold value, the controller estimates a tail delay of the first stream passing through the target device based on recording the first delay in the plurality of first recording units.

18. The method of claim 17,

the method further comprises the following steps: the controller further receives an identifier of an estimation window indicating that information recorded in the plurality of first recording units is recorded within the estimation window, the estimation window being a time window for estimating a tail latency of the first stream passing through the target device;

the controller estimates a tail delay of the first stream through the target device based on the first delays recorded in the plurality of first recording units, including: the controller determines that the first delay recorded in the first unit is an estimated value of tail delay of the first stream passing through the target device within the estimation window; the first unit is a unit in which the first time delay recorded in the plurality of first recording units is minimum.

19. A method for estimating communication delay, the method comprising:

the method comprises the steps that a controller receives information recorded in a first recording unit and a second recording unit, wherein the information recorded in the first recording unit comprises a first time delay and a packet number, the first time delay is the maximum time delay of a single data packet of a first stream passing through a target device, and the packet number is the number of the data packets of the first stream received by a network device;

the second recording unit is associated with the first recording unit, the information recorded in the second recording unit includes a time delay of each data packet of n data packets in the first stream passing through the target device, and n is an integer greater than 1;

the controller estimates a tail delay of the first stream through the target device based on the n delays.

20. The method of claim 19, further comprising: the controller further receives an identifier of an estimation window indicating that information recorded in the first recording unit and the second recording unit is recorded within the estimation window, the estimation window being a time window for estimating a tail delay of the first stream through the target device;

the tail delay of the first stream passing through the target device is p fractional tail delay, and p is more than 80% and less than 100%; the controller estimating a tail delay of the first stream through the target device based on the n delays, comprising:

and the controller determines the w-th time delay of the n time delays as a tail time delay estimated value of the first stream passing through the target equipment in the estimation window, wherein the w-th time delay is the time delay sequenced from small to large into w from the n time delays, the time delay sequenced before the w is less than or equal to the p quantile tail time delay, and the w is an integer which is greater than 0 and less than or equal to n.

21. The method according to any one of claims 17 to 20,

the target device is the network device; alternatively, the first and second electrodes may be,

22. A network device, characterized in that the network device comprises means for performing the method of any of claims 1-15.

23. A network device, characterized in that the network device comprises means for performing the method of claim 16.

24. A controller, characterized in that the controller comprises means for performing the method of claim 17 or 18.

25. A controller, characterized in that the controller comprises means for performing the method of claim 19 or 20.

26. A network device, comprising a processor and a memory; wherein the memory is configured to store a computer program and the processor is configured to execute the computer program stored in the memory to cause the network device to perform the method of any of claims 1-15; or cause the network device to perform the method of claim 16.

27. A controller, wherein the controller comprises a processor and a memory; wherein the memory is for storing a computer program and the processor is for executing the computer program stored in the memory to cause the controller to perform the method of claim 17 or 18; or cause the network device to perform the method of claim 19 or 20.

28. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which is executed by a processor to implement the method of any one of claims 1 to 15; alternatively, the computer program is executed by a processor to implement the method of claim 16.

29. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which is executed by a processor to implement the method of claim 17 or 18; alternatively, the computer program is executed by a processor to implement the method of claim 19 or 20.

30. A system comprising a network device according to claim 22 and a controller according to claim 24 or 25; or, the network device is the network device of claim 23, and the controller is the controller of claim 25; alternatively, the network device is the network device of claim 26, and the controller is the controller of claim 27.