CN116032789A - Time stamp obtaining method and device, computer readable storage medium and terminal - Google Patents
Time stamp obtaining method and device, computer readable storage medium and terminal Download PDFInfo
- Publication number
- CN116032789A CN116032789A CN202211675506.3A CN202211675506A CN116032789A CN 116032789 A CN116032789 A CN 116032789A CN 202211675506 A CN202211675506 A CN 202211675506A CN 116032789 A CN116032789 A CN 116032789A
- Authority
- CN
- China
- Prior art keywords
- data
- original data
- piece
- original
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
A method and a device for acquiring a time stamp, a computer readable storage medium and a terminal are provided, wherein the method comprises the following steps: determining received multi-frame original data frames, wherein each frame of original data frame comprises original data, identification information of the original data and a time stamp for receiving the original data; according to the identification information and the data length of the original data, the original data in the original data frames of each frame are recombined to determine one or more recombined data; analyzing each piece of recombined data to determine multi-item mark service data, wherein each item of mark service data belongs to one or more pieces of original data; and determining the original data of the preset byte in each piece of target service data, and taking the timestamp of receiving the original data as the timestamp of receiving the target service data. The scheme can accurately extract the target service data from the original data frame of the data link layer and receive the time stamp of the target service data.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a method and a device for acquiring a time stamp, a computer readable storage medium and a terminal.
Background
In some particular data transmission scenarios, it is important to obtain an accurate timestamp of the receipt of a network data packet by a computer or server. Taking the actual business scenario of the securities and futures industry as an example, the trade bidding principle is price priority and time priority, namely, under the same price, the trade bidding principle is first to first obtain. The essence of securities trade competition is time-lapse competition. The prior market data acquisition becomes a resolution point of the securities market. Thus, for a dealer's quotation system, a delay indicator (e.g., the time delay for acquiring quotation data) is an important item of performance indicators, and the design, construction, and post-monitoring optimization of a low-delay quotation system are not separated from accurate and scientific time delay measures.
To determine the time delay of the ticker system, a problem that is inevitably involved is how to accurately restore the data file in the packet-grabbing format or other switch formats to the target service data (e.g., the ticker data), and in particular how to accurately obtain the time stamp of the received ticker data by the ticker system. In practical applications, the timestamp may include a software timestamp (e.g., a computer operating system timestamp) or a hardware timestamp (e.g., a hardware timestamp of a switch or network card) of different program nodes, terminal devices, or hardware devices in the ticketing system that receive the ticketing data.
However, in the prior art, for a data transmission scenario oriented to a data stream, for example, a quotation system for transmitting data based on a TCP protocol, there is no effective scheme for accurately extracting required quotation data from an original data Frame (the original data Frame mainly refers to an original data Frame (Frame) of a data link layer) and acquiring a time stamp of each quotation data.
Disclosure of Invention
The technical problem solved by the embodiment of the invention is how to accurately extract the target service data from the network data packet and receive the timestamp of the target service data.
To achieve the above object, an embodiment of the present invention provides a method for obtaining a timestamp, including the following steps: determining received multi-frame original data frames, wherein each frame of original data frame comprises original data, identification information of the original data and a time stamp for receiving the original data; according to the identification information and the data length of the original data, the original data in the original data frames of each frame are recombined to determine one or more recombined data; analyzing each piece of recombined data to determine multi-item mark service data, wherein each item of mark service data belongs to one or more pieces of original data; and determining the original data of the preset byte in each piece of target service data, and taking the timestamp of receiving the original data as the timestamp of receiving the target service data.
Optionally, the determining that the multi-frame original data frame is received includes: the method comprises the steps of capturing packets of received data to obtain a plurality of capturing packet files; filtering the plurality of packet capturing files based on a preset source IP address, a target IP address and a target port identifier to obtain a packet capturing file to be analyzed; and carrying out file format analysis on the packet capturing file to be analyzed to determine a plurality of frames of the original data frames.
Optionally, the original data is TCP data, and the identification information of the original data is a TCP serial number; and reorganizing the original data in the original data frames of each frame according to the identification information and the data length of the original data to determine one or more reorganized data, wherein the reorganizing data comprises the following steps: sequencing the original data in the original data frames of each frame according to the sequence from small to large of the TCP serial number of the original data, and obtaining sequenced original data; comparing the sum of the TCP serial number and the data length of the first piece of ordered original data with the TCP serial number of the second piece of original data, and splicing the first piece of original data with the second piece of original data to obtain first spliced data when the sum of the TCP serial number and the data length of the first piece of original data is larger than or equal to the TCP serial number of the second piece of original data; continuously adopting the sum of the TCP serial number and the data length of the N-1 th piece of original data after sequencing, comparing the sum with the TCP serial number of the N-1 th piece of original data, when the sum of the TCP serial number and the data length of the N-1 th piece of original data is greater than or equal to the TCP serial number of the N-1 th piece of original data, splicing the N-1 th piece of spliced data with the N-1 th piece of original data to obtain the N-1 th piece of spliced data, and adopting the N-1 th piece of spliced data as single recombined data until the sum of the TCP serial number and the data length of the N-1 th piece of original data is smaller than the TCP serial number of the N-th piece of original data; wherein N and N are positive integers, and N < 1 and N.
Optionally, when the sum of the TCP serial number and the data length of the nth original data is equal to the TCP serial number of the nth original data, directly splicing the nth-1 spliced data and the nth original data to obtain the nth spliced data; wherein N and N are positive integers, and N < 1 and N.
Optionally, whenever the sum of the TCP serial number and the data length of the nth original data is greater than the TCP serial number of the nth original data, determining an intersection of the nth original data and the nth original data, intercepting the intersection in the nth original data, and splicing the nth-1 spliced data and the intercepted nth original data to obtain spliced data; wherein N and N are positive integers, and N < 1 and N.
Optionally, the reorganized data is stored in a cache; the determining the original data of the preset bytes in each piece of target service data comprises the following steps: determining a cache address of a preset byte in the target service data; judging whether the end address of each piece of original data in the recombined data is larger than or equal to the cache address one by one from the cache head address of the recombined data; when the end address is greater than or equal to the cache address, the original data of the end address is taken as the original data of the preset byte in the target service data.
Optionally, the preset byte is the first byte of the target service data.
The embodiment of the invention also provides a device for acquiring the time stamp, which comprises the following steps: the device comprises an original data frame determining module, a receiving module and a receiving module, wherein the original data frame determining module is used for determining received multi-frame original data frames, and each frame of original data frame comprises original data, identification information of the original data and a time stamp for receiving the original data; the data reorganization module is used for reorganizing the original data in the original data frames of each frame according to the identification information and the data length of the original data so as to determine one or more reorganized data; the target service data determining module is used for analyzing each piece of recombined data to determine multi-item target service data, wherein each item of target service data belongs to one or more pieces of original data; and the time stamp determining module is used for determining the original data of the preset byte in each piece of target service data, and taking the time stamp of receiving the original data as the time stamp of receiving the target service data.
The embodiment of the invention also provides a computer readable storage medium, on which a computer program is stored, which when being executed by a processor, performs the steps of the above-described time stamp obtaining method.
The embodiment of the invention also provides a terminal, which comprises a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor executes the steps of the time stamp obtaining method when running the computer program.
Compared with the prior art, the technical scheme of the embodiment of the invention has the following beneficial effects:
the embodiment of the invention provides a time stamp obtaining method, which is used for reorganizing each piece of original data according to the identification information and the data length of the original data contained in a received multi-frame original data frame to determine one or more pieces of reorganized data; then analyzing each piece of recombined data to obtain multi-label business data; and determining the original data of the preset byte in each piece of target service data, and taking the timestamp of receiving the original data as the timestamp of receiving the target service data.
Due to reasons of sticking, packetizing and the like in data transmission, one or more pieces of standard service data may be included in a single frame of original data frame, a part of one piece of standard service data may also be included, and problems of disorder, retransmission, packet loss and the like of the received original data frame may also exist due to characteristics of a transmission protocol (such as a TCP protocol facing a data stream) or various factors of network transmission. Therefore, in the embodiment of the invention, by reorganizing and analyzing the original data in each frame of original data frame, the accurate and effective target service data can be restored as far as possible, and further the time stamp of receiving each item of target service data can be accurately determined, so that accurate data support is provided for the subsequent delay measurement.
Further, in the embodiment of the present invention, the original data is TCP data, the identification information of the original data is a TCP sequence number, and in the process of reorganizing the original data, the original data is ordered according to the sequence from the small TCP sequence number to the large TCP sequence number, so as to obtain ordered original data; and then, from the first piece of ordered original data, successively comparing the sum of the TCP serial number and the data length of the current piece of original data with the TCP serial number of the next piece of original data, and then, performing different modes of splicing processing on the data according to the comparison result (for example, directly performing front-back splicing when the data is normally transmitted, intercepting a retransmission part when the data is retransmitted, then performing splicing, ending the current splicing process and determining a single piece of recombined data when the data is lost, and then, circulating the judging and splicing steps to obtain a plurality of other pieces of recombined data). Therefore, the data with the problems of disorder, retransmission or packet loss can be accurately recombined to obtain recombined data containing one or more pieces of standard service data, and the recombined data are provided for a subsequent application layer for analysis, so that the accuracy of analyzed target service data can be improved.
Further, in the embodiment of the invention, the reorganized data is stored in the cache; then, for each piece of extracted target service data, determining a cache address of a preset byte (for example, a first byte) in the target service data; judging whether the end address of each piece of original data in the recombined data is larger than or equal to the cache address one by one from the cache head address of the recombined data; when the end address is greater than or equal to the cache address, the original data of the end address is taken as the original data of the preset byte in the target service data. Therefore, the original data of the preset byte can be determined efficiently and accurately, and the accuracy of the time stamp of the subsequent determination of the receiving target service data can be improved.
Drawings
FIG. 1 is a schematic diagram showing the comparison of the structure of a UDP-based raw data frame and a TCP-based raw data frame in an embodiment of the present invention;
FIG. 2 is a flowchart of a method for obtaining a timestamp according to an embodiment of the invention;
FIG. 3 is a flow chart of one embodiment of step S22 of FIG. 2;
FIG. 4 is a flow chart of one embodiment of step S24 of FIG. 2;
FIG. 5 is a schematic diagram of an application scenario according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of another application scenario of an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a timestamp acquiring apparatus according to an embodiment of the present invention.
Detailed Description
As mentioned in the background, in some specific data transmission scenarios, accurate and scientific latency metrics are important. Among these, the problem involved is how to accurately restore the data file in the packet-grabbing format or other switch formats to the target service data, and especially how to accurately acquire the timestamp of the system receiving the target service data.
In the prior art, some open-source protocol processing tools exist, which can capture packets of data received by a quotation system, and parse the files in a capture packet format to obtain original data (UserData) therein. For example, in a market system that performs data transmission based on the UDP protocol, such as multicast data, if the UDP fragmentation problem is not considered, the original data Frame (Frame) of the data link layer and the market data often have a one-to-one relationship. Thus, existing schemes obtain the Timestamp of the received market data by parsing the Frame format and extracting the Timestamp (Timestamp) in the Frame header.
However, in a data transmission scenario facing to a data stream, for example, a market system for transmitting data based on a TCP protocol may have problems of sticking, packetizing, and out-of-order due to network data transmission, a complete market data may span multiple frames of original data frames (each frame of original data frame includes one TCP data packet, each TCP data packet mainly includes a TCP header+userdata), or one frame of original data frame includes multiple market data, in other words, one market data packet may belong to multiple TCP data packets, and a single TCP data packet may also include multiple market data. In addition, in network data transmission, there are also problems such as data retransmission and data loss caused by other factors. Therefore, the time stamp in the original data frame cannot be directly in one-to-one correspondence with the market data.
For the above situation, there is currently no effective scheme for accurately extracting desired market data from an original data frame and acquiring time stamps of respective market data.
To facilitate understanding of the technical problem described above, the following description is made in connection with fig. 1. Fig. 1 is a schematic diagram showing the comparison of the structure of a UDP-based raw data frame and a TCP-based raw data frame in an embodiment of the present invention.
The upper original data frame in fig. 1 is a partial structure of the original data frame based on UDP, and the lower original data frame is a partial structure of the original data frame based on TCP.
As one non-limiting example, a tcpdump capture tool is used to capture packets in a three-tier network environment to obtain a capture packet file (e.g., a pcap format file), and the capture packet file header is removed. The header of the original data frame contains a time stamp (Timestamp) field, typically in nanoseconds or microseconds, related to the type of network card support and tcpdump parameter selection. The body (body) portion of the original data frame is then formed of, in order: ethernet (ethernet) frame header, IP header, transmission layer (Transmission) TCP header or UDP header, and application layer raw data (UserData).
In the original data frame based on UDP, if the UDP fragmentation problem is not considered, there is usually a one-to-one correspondence between the original data in the original data frame and the target service data, and the time stamp (Timestamp) field in the frame header of the original data frame is extracted to obtain the time when the target service data (e.g. market data) passes through the network card.
However, in the original data frame based on TCP, because TCP has problems of packet sticking and packetization in the network, a complete target service may span multiple TCP packets, and one TCP packet may also contain multiple target service data. As shown in the lower part of fig. 1, a UserData contains a complete piece of target traffic data B1 and also contains a part of the target traffic data B2. Therefore, the relationship between the original data and the target service data in the original data frame based on TCP is not one-to-one, and the timestamp in the frame header of the original data frame cannot be directly used as the time when the target service data B1 or the target service data B2 passes through the network card like the processing of the original data frame based on UDP.
In order to solve the above technical problems, an embodiment of the present invention provides a method for obtaining a timestamp, which specifically includes: determining received multi-frame original data frames, wherein each frame of original data frame comprises original data, identification information of the original data and a time stamp for receiving the original data; according to the identification information and the data length of the original data, the original data in the original data frames of each frame are recombined to determine one or more recombined data; analyzing each piece of recombined data to determine multi-item mark service data, wherein each item of mark service data belongs to one or more pieces of original data; and determining the original data of the preset byte in each piece of target service data, and taking the timestamp of receiving the original data as the timestamp of receiving the target service data.
By the method, in the embodiment of the invention, the original data in each frame of original data frame is recombined and analyzed according to the identification information and the data length of the original data, so that the accurate and effective target service data can be restored as far as possible, further, the time stamp of receiving each item of target service data can be accurately determined, and accurate data support is provided for subsequent delay measurement.
In order to make the above objects, features and advantages of the present invention more comprehensible, embodiments accompanied with figures are described in detail below.
Referring to fig. 2, fig. 2 is a flowchart of a method for acquiring a timestamp according to an embodiment of the present invention. The method can be applied to terminal equipment with data receiving and processing functions, wherein the terminal equipment can be a computer, a tablet computer, a mobile phone, a server, a cloud platform and the like. The method may include steps S21 to S24:
step S21: determining received multi-frame original data frames, wherein each frame of original data frame comprises original data, identification information of the original data and a time stamp for receiving the original data;
step S22: according to the identification information and the data length of the original data, the original data in the original data frames of each frame are recombined to determine one or more recombined data;
Step S23: analyzing each piece of recombined data to determine multi-item mark service data, wherein each item of mark service data belongs to one or more pieces of original data;
step S24: and determining the original data of the preset byte in each piece of target service data, and taking the timestamp of receiving the original data as the timestamp of receiving the target service data.
In a specific implementation of step S21, the original data frame may be a protocol data unit (protocal data unit) of a data link layer. In a computer network hierarchy, it generally includes an application layer, a network layer, a transport layer (transport layer protocols mainly include TCP protocol and UDP protocol), a network layer, and a network interface layer.
The network interface layer may in turn include a data link layer and a physical layer. The roles of the data link layer include: physical address addressing, framing of data, flow control, error detection of data, retransmission, etc. At the data link layer, the smallest unit of data is called a "frame," i.e., the original data frame described in embodiments of the present invention.
The specific process of encapsulating the IP datagram of the network layer into frames mainly comprises the following steps: first the network layer passes the IP datagram to the data link layer, which sees the IP datagram as "frame" data and adds a "frame" header and trailer marker to the "frame" data, which are essentially specific bit streams. The advantage of adopting the frame transmission mode is that when the data transmission error is found, only the frame with the error is required to be retransmitted, and the bit stream of all data is not required to be retransmitted, thereby greatly improving the transmission efficiency.
The original data carried by the original data frame may be data sent by an application layer. The identification information of the raw data may include a TCP sequence number, which is typically located in a TCP header.
The TCP-based raw data frame shown in connection with fig. 1 mainly comprises two major parts, a frame header and a body. Wherein the frame header includes a timestamp field for receiving the original data. The body part mainly includes an ethernet (ethernet) frame header, an IP header, a TCP header of a transport layer, and raw data (UserData) of an application layer.
Wherein the Ehternet frame header is usually fixed in length, and the main contents are Mac address and upper layer network protocol type; the IP header is variable in length, and includes complex fields such as an IP address, an IP layer data length, a protocol type, and the like, and needs to be adapted according to the actual situation of the environment to be tested, such as a virtual local area network (Vlan). The TCP header contains a TCP Sequence number (Sequence) and TCP header length information (TCPHeaderLen).
Further, the determining that the multi-frame original data frame is received in step S21 may include: the method comprises the steps of capturing packets of received data to obtain a plurality of capturing packet files; filtering the plurality of packet capturing files based on a preset source IP address, a target IP address and a target port identifier to obtain a packet capturing file to be analyzed; and carrying out file format analysis on the packet capturing file to be analyzed to determine a plurality of frames of the original data frames.
Specifically, the capturing of the received data may specifically be capturing the data received by the network card or the switch on the terminal device. The hardware time stamp or the software time stamp for acquiring the received data when the packet is grabbed can be specified according to the actual application scene requirement. The hardware timestamp may be a timestamp of the packet passing through the network card or the switch, and the software timestamp may be an operating system timestamp.
In a specific implementation, taking tcpdump as an example for network card packet capturing in a Linux environment, for example, network card support, a-J command of tcpdump can be used for querying when the packet is captured, -J designates a timestamp type, and when the timestamp is not designated, the timestamp is used by default. In contrast, hardware time stamping is more accurate than software time stamping, and the difference between the two is tens of microseconds to milliseconds.
In the implementation of step S22, the data length of the original data may be determined according to the total length of the IP contained in the IP header, the data length of the IP header, and the data length of the TCP header contained in the TCP header, where a specific calculation formula is as follows: userDataLen = TotalLen-iphoadererlen-TCPHeaderLen;
wherein TotalLen is the total length of IP, IP header len is the header data length of IP, TCPHeaderLen is the header data length of TCP.
In the streaming concept of TCP, each Byte (Byte) of the original data of the application layer may be uniquely identified by a TCP Sequence number (Sequence) +an offset bit (Length) of the Byte in the original data, and two consecutive TCP data packets TCP1 and TCP2 satisfy that Sequence (1) +userdatain (1) =sequence (2), in other words, for two consecutive frames of original data that are normally transmitted, a sum of a Sequence number of original data in a current frame of original data frame and a data Length of the original data is equal to a Sequence number of original data in a next frame of original data frame, and if Sequence (1) +userdatain (1) > Sequence (2), data retransmission may occur, and if Sequence (1) +userdalen (1) < Sequence (2), data packet loss may occur.
According to the principle, the problems of disorder, retransmission and packet loss of TCP can be solved.
Referring to fig. 3, fig. 3 is a flowchart of one embodiment of step S22 in fig. 2. In the embodiment of the invention, the original data is TCP data, and the identification information of the original data is TCP serial number. The step S22 may include steps S31 to S33, and each step will be described below.
In the implementation of step S31, the original data in the original data frames of each frame is sorted according to the order of the TCP sequence numbers of the original data from small to large, so as to obtain sorted original data.
In the implementation of step S32, the sum of the TCP sequence number and the data length of the first piece of ordered original data is compared with the TCP sequence number of the second piece of original data, and when the sum of the TCP sequence number and the data length of the first piece of original data is greater than or equal to the TCP sequence number of the second piece of original data, the first piece of original data and the second piece of original data are spliced to obtain the first piece of spliced data.
In the implementation of step S33, the sum of the TCP sequence number and the data length of the N-1 th piece of original data after sorting is continuously adopted, the sum is compared with the TCP sequence number of the N-1 th piece of original data, when the sum of the TCP sequence number and the data length of the N-1 th piece of original data is greater than or equal to the TCP sequence number of the N-1 th piece of original data, the N-1 th piece of spliced data and the N-1 th piece of original data are spliced to obtain the N-1 th piece of spliced data, and when the sum of the TCP sequence number and the data length of the N-1 th piece of original data is smaller than the TCP sequence number of the N-1 th piece of original data, the N-1 th piece of spliced data is adopted as a single piece of recombined data.
Wherein N and N are positive integers, and N < 1 and N.
Further, in the step S33, according to the sum of the TCP sequence number and the data length of the n-1 th original data, the n-1 th spliced data and the n-th original data may be spliced in different manners, different from the comparison result of the TCP sequence number of the n-1 th original data.
Case one: when the sum of the TCP serial number and the data length of the n-1 th original data is equal to the TCP serial number of the n-1 th original data, directly splicing the n-1 th spliced data and the n-th original data to obtain the n-th spliced data; wherein N and N are positive integers, and N < 1 and N.
In the first case, the directly splicing may be to splice the tail byte of the n-1 th spliced data and the first byte of the n-th original data.
And a second case: each time the sum of the TCP serial number and the data length of the n-1 th original data is larger than the TCP serial number of the n-1 th original data, determining the intersection of the n-1 th original data and the n-th original data, intercepting the intersection in the n-1 th original data or the n-1 th spliced data, and splicing the intercepted data and the non-intercepted data to obtain spliced data; wherein N and N are positive integers, and N < 1 and N.
In the second case, the splicing the intercepted data and the non-intercepted data may specifically be: and splicing the tail bytes of the non-intercepted n-1 th spliced data with the head bytes of the intercepted n-th original data. It may also be: and splicing the tail bytes of the n-1 th spliced data from which the intersection is intercepted and the head bytes of the n-th original data which are not intercepted.
The specific implementation of the above steps S31 to S34 will be described in detail below using a non-limiting example.
The original data frames of n frames exist in the packet capturing file obtained by packet capturing, the original data contained in each frame of the original data frames are UserData (1), userData (2), userData (3), userData (4), userData (5) … … UserData (n), the Sequence numbers and the data lengths corresponding to each piece of the original data are Sequence (1) and UserDataLen (1), sequence (2) and UserDataLen (2), sequence (3) and UserDataLen (3), sequence (4) and UserDataLen (4), sequence (5) and UserDataLen (5) … … Sequence (n) and UserDataLen (n) respectively.
Firstly, a linked list data structure may be adopted, and each piece of original data is inserted into a linked list according to the Sequence from small to large, so as to obtain an ordered TCP linked list structure, each node of the linked list stores one piece of original data, and in this embodiment, it is assumed that the original data (i.e., the ordered original data) sequentially stored from the beginning to the end of the linked list is: userData (1), userData (2), userData (3), userData (4), userData (5), userData (8), userData (9), userData (7), userData (6) … …).
Then, from the first piece of original data in the chain table head, the sum of the TCP Sequence number Sequence (n-1) of the current piece of original data UserData (n-1) and the data length UserData (n-1) is compared with the TCP Sequence number Sequence (n) of the next piece of original data one by one, and the comparison result and the splicing process are as follows.
(1) In this embodiment, the first piece of original data in the linked list is UserData (1), and Sequence (1) +userdatalen (1) =userdatalen (2), then UserData (1) and UserData (2) may be sequentially added to the first buffer to be parsed, where UserData (1) is added first, and after UserData (2) is added to UserData (1), the first piece of spliced data (UserData (1) +userdata (2)) is obtained;
(2) The second piece of original data in the linked list is UserData (2), and Sequence (2) +userdatalen (2) =userdalen (3), then UserData (3) can be added into the first buffer cache continuously, wherein after UserData (3) is added into UserData (2), second piece of spliced data (UserData (1) +userdata (2) +userdata (3)) is obtained;
(3) The third piece of original data in the linked list is UserData (3), and Sequence (3) +userdatalen (3) > UserDataLen (4), then it can be confirmed that data retransmission occurs between UserData (3) and UserData (4).
In this case, the intersection of UserData (3) and UserData (4) may be determined first, the intersection of UserData (4) or the second piece of spliced data may be cut, and the cut data and the non-cut data may be spliced. For example, if the intersection part in the UserData (4) is truncated, the truncated UserData (4) may be added to the UserData (3) in the first buffer, to obtain a third piece of spliced data (UserData (1) +userdata (2) +userdata (3) +truncated UserData (4)).
(4) The fourth piece of original data in the linked list is UserData (4), and Sequence (4) +userdatalen (4) < UserDataLen (5), it can be confirmed that UserData (4) has data loss.
In this case, adding data to the first buffer is stopped, and splice data existing in the first buffer is adopted as a single piece of reorganized data. In this embodiment, the spliced data is (UserData (1) +userdata (2) +userdata (3) +userdata (4) after interception, and (5) from the fifth piece of original data in the linked list, the steps (1) to (4) may be circularly executed to obtain a plurality of other pieces of reorganized data, which is not repeated herein in the specific implementation process.
In the embodiment of the invention, by adopting the recombination scheme, front and back splicing is directly carried out when data is normally transmitted; when the data is retransmitted, intercepting a retransmission part and then splicing; when the packet loss exists in the data, the current splicing process is ended, a single piece of recombined data is determined, and then the judging and splicing steps are circulated to obtain a plurality of other pieces of recombined data. Therefore, the data with the problems of disorder, retransmission or packet loss can be accurately restored and recombined to obtain recombined data containing one or more pieces of target service data, and the recombined data are provided for a subsequent application layer to analyze, so that the accuracy of analyzed target service data can be improved.
With continued reference to fig. 2, in the implementation of step S23, the method for analyzing each piece of the reorganized data to determine multi-label service data may be an existing application layer analysis scheme. For example, according to a preset data type, each piece of the reorganized data may be parsed byte by byte to determine the type as a number or an english character, and the method for parsing the application layer is not described herein.
In the implementation of step S24, as can be seen in the embodiment described in fig. 3, each piece of reorganized data includes one or more pieces of ordered original data, so that the parsed target service data and the original data in the reorganized data can be corresponding, and any byte in the target service data can find the original data in the reorganized data. And then taking the timestamp of receiving the belonging original data as the timestamp of receiving the target business data.
Referring to fig. 4, fig. 4 is a flowchart of one embodiment of step S24 in fig. 2. In the embodiment of the invention, the reorganized data is stored in a cache. The determining in step S24 that the original data to which the preset byte in each piece of target service data belongs may include steps S41 to S43, and each step is described below.
In step S41, a buffer address of a preset byte in the target service data is determined.
As a non-limiting embodiment, the preset byte is the first byte of the target service data.
It will be appreciated that in a practical application scenario, a single piece of target service data may belong to multiple pieces of original data, in which case the original data to which the first byte of the target service data belongs is the first (also received earliest) in the pieces of original data to which it belongs. Therefore, the time stamp of the original data with the top ranking is used as the time stamp of the item mark business data, so that the time for receiving the item mark business data can be more accurately determined.
In step S42, from the first address of the cache of the reorganized data, it is determined whether the end address of each piece of original data in the reorganized data is greater than or equal to the cache address one by one.
The end address of each piece of original data can be determined according to the first address of the recombined data in the cache and the length of each piece of original data. In connection with the embodiment provided above for fig. 3, for UserData (4), its end address = the first address of the first cache buffer + UserDataLen (1) +userdatalen (2) +userdatalen (3) +userdatalen (4).
In step S43, when there is an end address equal to or greater than the buffer address, the original data to which the end address belongs is used as the original data to which the preset byte in the target service data belongs.
In the embodiment of the invention, by adopting the cache storage and address positioning scheme, the original data of the preset bytes in each piece of target service data can be efficiently and accurately determined, and the accuracy of the subsequent determination of the time stamp of the received target service data can be further improved.
In an implementation, a skip list (skip list) structure may be used to store a time stamp (Timestamp) field in a received multi-frame original data frame, a buffer header address for storing the reassembly data, and a length of UserData in each frame of the original data frame.
The determining the original data to which the preset byte in each piece of target service data belongs may include: determining a buffer address of a first byte of the target service data in a buffer; then determining UserData to which the first byte of the item target service data belongs based on the cache address, the Buffer first address stored in the skip list and the length of each piece of UserData; and acquiring the time stamp of the original data frame to which the UserDate belongs as the time stamp for receiving the business data of the item mark. Thus, the rapid and accurate acquisition of the timestamp of the target business data can be realized.
Further, the timestamp of receiving the target service data may be a timestamp of receiving the same target service at different nodes, for example, may be a timestamp T of receiving a piece of quotation data by a network card at an entrance of a quotation system a Or the time stamp T of the network card at the exit of the quotation system for receiving a certain quotation data b Thereby, the time delay DeltaT=T of the quotation system can be calculated b -T a 。
Referring to fig. 5, a schematic diagram of an application scenario of an embodiment of the present invention is given.
In the market data transmission, the market data is transmitted through optical fibers among the three systems, and is received or forwarded through hardware devices such as a network card, a switch and the like.
Fig. 5 shows a scenario of latency measurement for directly adopting a network card to capture a packet and introducing an optical splitter to capture the packet.
The input network card of the quotation system is provided with a time stamp T1, the output network card is provided with a time stamp T2, and the time delay delta T1 = T2-T1 of the quotation system can be obtained. Because only the software time stamp (such as the time stamp of the operating system) of the quotation system can be grabbed for the outlet flow, for the same clock source, the T1 can only use the software time stamp, so that the precision of the time stamps T1 and T2 is lower, and the network card packet grabbing scheme can only be used for detecting the penetration time delay of each link in a scene with lower precision.
In addition, if the field programmable gate array (Field Programmable Gate Array, FPGA) is used for hardware distribution of market data, because the FPGA adopts a low-delay hardened network protocol stack, the data packet is not transmitted through the network card, the timestamp of the data cannot be obtained through the network card packet capturing tool, and only one-hop links can be added at the upstream and downstream of the FPGA to perform time delay detection. From this point of view, new detection errors are brought about.
In order to solve the problem, an optical splitter may be introduced to mirror the optical fiber data, as shown in fig. 5, the optical splitter data is subjected to packet capturing analysis on the same low-delay network card, and because of the ingress traffic, network card hardware time stamps, respectively T3 and T4, can be acquired, so that the time delay Δt2=t4-T3 of the quotation system is obtained, and because the time delay problem of clock sources between different devices does not exist on the same network card, the measured time delay precision of the quotation system is higher. The delay measurement scheme introducing the optical splitter is suitable for service scenes with extremely low delay and extremely high precision requirements, but can not be suitable for service scenes with complex system deployment, cross cabinets and even machine rooms.
Referring to fig. 6, a schematic diagram of another application scenario of the embodiment of the present invention in fig. 6 is shown.
Fig. 6 illustrates a scenario of latency metrics for packet grabbing of switch mirror data. Wherein, network data acquisition analysis tools (e.g. tcpdump tools) can be adopted on the analysis tester to capture packets of the switch mirror image data, and capture packet landing files.
The left switch in fig. 6 marks a timestamp T5 on the mirror image data input to the quotation system (in this scheme, the timestamp adopts a switch hardware timestamp), the right switch in fig. 6 marks a timestamp T6 on the mirror image data output from the quotation system, so as to calculate a time delay Δt3=t6-T5 of the transmission of the quotation data between the two switches, and since both T5 and T6 are switch hardware timestamps, the calculated time delay Δt3 has higher precision. Tests show that under the condition of dotting by adopting the same switch, the error can be controlled to be in the order of 10 nanoseconds. In addition, in the measurement scheme, mirror image data are acquired from the switch, and the switch is only required to be opened for corresponding functions, for example, the Cisco Nexus 3548p switch is required to be opened for an ERSPAN function, and the market system to be measured is basically noninductive and low in invasiveness.
It should be noted that if the system under test is large in scale, the deployment structure is complex, such as across multiple cabinets or across multiple rooms, further optimization of the solution in fig. 6 is required. For a service scenario across multiple cabinets, the introduction of a test access point or terminal access point (Test Access Point or Terminal Access Point, TAP) switch, dedicated for mirroring data dotting, can be considered, avoiding clock source synchronization problems between different devices; or clock synchronization is performed by adopting a high-precision time protocol (Precision Time Protocol, PTP), so that errors caused by clock synchronization are reduced to the greatest extent.
Referring to fig. 7, fig. 7 is a schematic diagram of a time stamp obtaining apparatus according to an embodiment of the present invention. The apparatus may include:
an original data frame determining module 71, configured to determine received multi-frame original data frames, where each frame of original data frame includes original data, identification information of the original data, and a timestamp for receiving the original data;
a data reorganizing module 72, configured to reorganize original data in the original data frames of each frame according to the identification information and the data length of the original data, so as to determine one or more reorganized data;
a target service data determining module 73, configured to parse each piece of the reorganized data to determine multiple pieces of target service data, where each piece of target service data belongs to one or more pieces of the original data;
the timestamp determining module 74 is configured to determine original data to which a preset byte in each piece of targeted service data belongs, and use a timestamp of receiving the original data to be the timestamp of receiving the targeted service data.
Regarding the principle, implementation and advantageous effects of the timestamp obtaining apparatus, please refer to the foregoing and the related descriptions of the timestamp obtaining method shown in fig. 2 to 4, which are not repeated herein.
The embodiment of the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the timestamp acquisition method shown in fig. 2 to 4 described above. The computer readable storage medium may include non-volatile memory (non-volatile) or non-transitory memory, and may also include optical disks, mechanical hard disks, solid state disks, and the like.
Specifically, in the embodiment of the present invention, the processor may be a central processing unit (central processing unit, abbreviated as CPU), and the processor may also be other general purpose processors, digital signal processors (digital signal processor, abbreviated as DSP), application specific integrated circuits (application specific integrated circuit, abbreviated as ASIC), off-the-shelf programmable gate arrays (field programmable gate array, abbreviated as FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and so on. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
It should also be appreciated that the memory in embodiments of the present application may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically erasable ROM (electrically EPROM, EEPROM), or a flash memory. The volatile memory may be a random access memory (random access memory, RAM for short) which acts as an external cache. By way of example but not limitation, many forms of random access memory (random access memory, abbreviated as RAM) are available, such as static random access memory (static RAM), dynamic Random Access Memory (DRAM), synchronous Dynamic Random Access Memory (SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, abbreviated as DDR SDRAM), enhanced Synchronous Dynamic Random Access Memory (ESDRAM), synchronous Link DRAM (SLDRAM), and direct memory bus random access memory (direct rambus RAM, abbreviated as DR RAM).
The embodiment of the invention also provides a terminal, which comprises a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor executes the steps of the time stamp obtaining method shown in the figures 2 to 4 when running the computer program. The terminal can include, but is not limited to, terminal equipment such as a mobile phone, a computer, a tablet computer, a server, a cloud platform, and the like.
It should be understood that the term "and/or" is merely an association relationship describing the associated object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In this context, the character "/" indicates that the front and rear associated objects are an "or" relationship.
The term "plurality" as used in the embodiments herein refers to two or more.
The first, second, etc. descriptions in the embodiments of the present application are only used for illustrating and distinguishing the description objects, and no order division is used, nor does it indicate that the number of the devices in the embodiments of the present application is particularly limited, and no limitation on the embodiments of the present application should be construed.
It should be noted that the serial numbers of the steps in the present embodiment do not represent a limitation on the execution sequence of the steps.
Although the present invention is disclosed above, the present invention is not limited thereto. Various changes and modifications may be made by one skilled in the art without departing from the spirit and scope of the invention, and the scope of the invention should be assessed accordingly to that of the appended claims.
Claims (10)
1. A time stamp obtaining method, comprising:
determining received multi-frame original data frames, wherein each frame of original data frame comprises original data, identification information of the original data and a time stamp for receiving the original data;
according to the identification information and the data length of the original data, the original data in the original data frames of each frame are recombined to determine one or more recombined data;
analyzing each piece of recombined data to determine multi-item mark service data, wherein each item of mark service data belongs to one or more pieces of original data;
and determining the original data of the preset byte in each piece of target service data, and taking the timestamp of receiving the original data as the timestamp of receiving the target service data.
2. The method of claim 1, wherein said determining that a multi-frame original data frame was received comprises:
The method comprises the steps of capturing packets of received data to obtain a plurality of capturing packet files;
filtering the plurality of packet capturing files based on a preset source IP address, a target IP address and a target port identifier to obtain a packet capturing file to be analyzed;
and carrying out file format analysis on the packet capturing file to be analyzed to determine a plurality of frames of the original data frames.
3. The method of claim 1, wherein the original data is TCP data, and the identification information of the original data is a TCP sequence number;
and reorganizing the original data in the original data frames of each frame according to the identification information and the data length of the original data to determine one or more reorganized data, wherein the reorganizing data comprises the following steps:
sequencing the original data in the original data frames of each frame according to the sequence from small to large of the TCP serial number of the original data, and obtaining sequenced original data;
comparing the sum of the TCP serial number and the data length of the first piece of ordered original data with the TCP serial number of the second piece of original data, and splicing the first piece of original data with the second piece of original data to obtain first spliced data when the sum of the TCP serial number and the data length of the first piece of original data is larger than or equal to the TCP serial number of the second piece of original data;
Continuously adopting the sum of the TCP serial number and the data length of the N-1 th piece of original data after sequencing, comparing the sum with the TCP serial number of the N-1 th piece of original data, when the sum of the TCP serial number and the data length of the N-1 th piece of original data is greater than or equal to the TCP serial number of the N-1 th piece of original data, splicing the N-1 th piece of spliced data with the N-1 th piece of original data to obtain the N-1 th piece of spliced data, and adopting the N-1 th piece of spliced data as single recombined data until the sum of the TCP serial number and the data length of the N-1 th piece of original data is smaller than the TCP serial number of the N-th piece of original data;
wherein N and N are positive integers, and N < 1 and N.
4. The method of claim 3, wherein the step of,
each time the sum of the TCP serial number and the data length of the n-1 th original data is equal to the TCP serial number of the n-1 th original data, directly splicing the n-1 th spliced data and the n-th original data to obtain the n-th spliced data;
wherein N and N are positive integers, and N < 1 and N.
5. The method of claim 3, wherein the step of,
each time the sum of the TCP serial number and the data length of the n-1 th original data is larger than the TCP serial number of the n-1 th original data, determining the intersection of the n-1 th original data and the n-th original data, intercepting the intersection in the n-1 th original data or the n-1 th spliced data, and splicing the intercepted data and the non-intercepted data to obtain spliced data;
Wherein N and N are positive integers, and N < 1 and N.
6. The method of claim 1, wherein the reorganized data is stored in a cache;
the determining the original data of the preset bytes in each piece of target service data comprises the following steps:
determining a cache address of a preset byte in the target service data;
judging whether the end address of each piece of original data in the recombined data is larger than or equal to the cache address one by one from the cache head address of the recombined data;
when the end address is greater than or equal to the cache address, the original data of the end address is taken as the original data of the preset byte in the target service data.
7. The method of claim 6, wherein the predetermined byte is a first byte of the target service data.
8. A time stamp obtaining apparatus, comprising:
the device comprises an original data frame determining module, a receiving module and a receiving module, wherein the original data frame determining module is used for determining received multi-frame original data frames, and each frame of original data frame comprises original data, identification information of the original data and a time stamp for receiving the original data;
the data reorganization module is used for reorganizing the original data in the original data frames of each frame according to the identification information and the data length of the original data so as to determine one or more reorganized data;
The target service data determining module is used for analyzing each piece of recombined data to determine multi-item target service data, wherein each item of target service data belongs to one or more pieces of original data; and the time stamp determining module is used for determining the original data of the preset byte in each piece of target service data, and taking the time stamp of receiving the original data as the time stamp of receiving the target service data.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, performs the steps of the time stamp obtaining method according to any of claims 1 to 7.
10. A terminal comprising a memory and a processor, the memory having stored thereon a computer program executable on the processor, characterized in that the processor executes the steps of the time stamp obtaining method according to any of claims 1 to 7 when the computer program is executed by the processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211675506.3A CN116032789A (en) | 2022-12-26 | 2022-12-26 | Time stamp obtaining method and device, computer readable storage medium and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211675506.3A CN116032789A (en) | 2022-12-26 | 2022-12-26 | Time stamp obtaining method and device, computer readable storage medium and terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116032789A true CN116032789A (en) | 2023-04-28 |
Family
ID=86077158
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211675506.3A Pending CN116032789A (en) | 2022-12-26 | 2022-12-26 | Time stamp obtaining method and device, computer readable storage medium and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116032789A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116561018A (en) * | 2023-07-12 | 2023-08-08 | 天津金城银行股份有限公司 | Service system counting method, device, terminal equipment and storage medium |
-
2022
- 2022-12-26 CN CN202211675506.3A patent/CN116032789A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116561018A (en) * | 2023-07-12 | 2023-08-08 | 天津金城银行股份有限公司 | Service system counting method, device, terminal equipment and storage medium |
| CN116561018B (en) * | 2023-07-12 | 2023-09-01 | 天津金城银行股份有限公司 | Service system counting method, device, terminal equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20090122805A1 (en) | Instrumenting packet flows | |
| US7668107B2 (en) | Hardware implementation of network testing and performance monitoring in a network device | |
| US8811410B1 (en) | Method and apparatus for measuring system latency using global time stamp | |
| US20140006610A1 (en) | Real-time highly accurate network latency measurement with low generated traffic or data requirements | |
| CN109617743B (en) | Network performance monitoring and service testing system and testing method | |
| CN108075935B (en) | Method and device for measuring time delay | |
| CN113242158B (en) | Real-time monitoring method and system based on switch hardware timestamp | |
| US9424268B2 (en) | System and method for aligning data frames in time | |
| WO2010057398A1 (en) | Device and method for implementing clock transparent transmission | |
| CN113328902B (en) | Network performance detection method and device and network equipment | |
| US20100128770A1 (en) | Measuring Delay in a Network Segment and/or through a Network Communications Device | |
| CN106027414A (en) | HDFS-oriented parallel network message reading method | |
| US20140293825A1 (en) | Timestamping data packets | |
| CN116032789A (en) | Time stamp obtaining method and device, computer readable storage medium and terminal | |
| US8055612B2 (en) | System and method for aligning data frames in time | |
| US8194552B1 (en) | System and method for trace replay using parallelized streams | |
| WO2023123956A1 (en) | Io full-lifecycle latency monitoring method and related apparatus | |
| CN111385163A (en) | Flow analysis and detection method and device | |
| CN115277510A (en) | Method for automatically identifying equipment, equipment interface and equipment path in network session | |
| CN115174676A (en) | Convergence and shunt method and related equipment thereof | |
| CN112968747B (en) | Time calibration method and device, computer equipment and storage medium | |
| CN112822722B (en) | Data message transmission method and device | |
| CN110602524B (en) | Method, device and system for synchronizing multi-channel digital streams and storage medium | |
| US11025519B2 (en) | Systems, methods and computer-readable media for external non-intrusive packet delay measurement | |
| US20130315251A1 (en) | Network system and communication device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |