CN111309403A

CN111309403A - Asynchronous dotting acquisition assembly for application layer delay data

Info

Publication number: CN111309403A
Application number: CN201911309042.2A
Authority: CN
Inventors: 张海荣; 张勇; 方义
Original assignee: Shanghai Financial Futures Information Technology Co ltd
Current assignee: Shanghai Financial Futures Information Technology Co ltd
Priority date: 2019-12-18
Filing date: 2019-12-18
Publication date: 2020-06-19
Anticipated expiration: 2039-12-18
Also published as: CN111309403B

Abstract

The invention discloses an asynchronous dotting acquisition component for application layer delay data, which realizes various cross-process or in-process scenes needing to be counted and delayed, thoroughly solves the reusability problem of common delay counting schemes and the influence on normal delay caused by dotting, and unifies cross-host and single-process internal calling modes. The technical scheme is as follows: the application interfaces of the invention are respectively a starting setting appointed log file and a dotting interface; the invention enables the application layer to call the dotting interface to perform dotting recording at any position of the program where dotting is needed, and the component automatically records tag _ id, guid and current time and falls into a file specified by the application layer at a proper time; in addition, each request of the application layer can record time delay, so that abnormal data can be counted conveniently; meanwhile, the method is friendly to secondary analysis, and the csv format storage of the three fields is convenient for secondary analysis.

Description

Asynchronous dotting acquisition assembly for application layer delay data

Technical Field

The invention relates to a financial transaction software technology, in particular to an application layer delay asynchronous dotting acquisition component which can be particularly applied to various cross-process or in-process scenes needing statistical delay.

Background

In some high performance applications in the field of financial transaction software, there is often a point of demand for performance statistics. For example, the delay statistics needs to count the time consumed by a network message from one host to the end of processing by another host, and also needs to count the time consumed by a program from an entry to the end, where the former is inter-system delay and the latter is intra-module delay.

The statistical delay often requires recording a thread number or a program execution point in addition to recording time. Common scenarios and solutions that require statistical delay are listed below: the first scenario is that the overall delay of a multi-process cross-network is counted, a common solution is that a network card tcp _ dump catches a packet, in the debugging of a network problem, the tcp _ dump is an indispensable tool, and is a command line type data packet sniffing tool based on a Unix system, which can catch a data packet flowing on the network card, and the defects are that the reusability is not high, and the corresponding analysis development cost is high; the second scenario is to count the time delay of a sub-function in a single module, and a common solution is to apply synchronous or asynchronous logging, which is the first conceivable method in most software development, to record time through a log and then perform manual analysis, but the disadvantages are that the performance is affected due to too high frequency of the log, and the statistical information is inaccurate due to too low frequency of the log.

Disclosure of Invention

The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

The invention aims to solve the problems, provides an asynchronous dotting acquisition component for application layer delay data, realizes various cross-process or in-process scenes needing to count delay, thoroughly solves the reusability problem of a common delay counting scheme and the influence on normal delay caused by dotting by providing a very simplified calling interface, and unifies cross-host and single-process internal calling modes.

The technical scheme of the invention is as follows: the invention discloses an asynchronous dotting acquisition component for application layer delay data, which comprises:

the two application interfaces are respectively an initialization interface executed at an entrance and a dotting interface called at the entrance and the exit of each program module, wherein the initialization interface is used for initialization work including starting and setting a designated log file, and the dotting interface is used for calling an application layer at a place needing dotting recording so as to perform dotting tracking;

the storage module is used for storing CSV format files with three parameter fields for secondary calculation and statistics, wherein the three parameter fields comprise a dotting identification parameter, a global user identity authentication parameter and a time parameter, the dotting identification parameter is used for identifying a program dotting position, the global user identity authentication parameter is used for marking a service message or configuring an identifier of a complete life cycle of operation, and the time parameter is used for recording dotting time.

According to an embodiment of the application layer delay data asynchronous dotting collection component of the present invention, the initialization interface and the dotting interface are provided by a C + + class implemented by an asynchronous component, and the C + + class implemented by the asynchronous component further includes two components: the system comprises a tag _ cache pointer array and a tag _ writer class, wherein the tag _ cache pointer array is used for storing cache region pointers corresponding to dotting identification parameters, the cache region is used for accommodating dotting records, each dotting record is a message body, the message body comprises three members which are three parameter fields in a CSV format file respectively, and the tag _ writer class is used for landing logs, managing the cache region and starting threads.

According to an embodiment of the asynchronous dotting acquisition component for the application layer delayed data, the component is configured to execute the following management flow of tag _ cache pointer arrays:

in an application program, different dotting identification parameters are used at different dotting points, a component stores node records corresponding to the same dotting identification parameter in the same tag _ cache pointer array until the tag _ cache pointer array is fully written and landed in a log file, the tag _ cache pointer array is recycled at the same time, a new tag _ cache pointer array is applied to record the node with the same dotting identification parameter to store data, a tag _ cache buffer area is constructed in the period after initialization, the head address of the cache area is added into a lock-free queue, wherein the lock-free queue is a lock-free circular queue which is realized based on CAS operation and is safe in multithreading read and write, the component manages the cache area, the component provides two interfaces for managing the cache area, one interface is used for applying for the cache area, the other interface is used for recycling the cache area into the lock-free queue after the data in the cache area land, the component applies for a tag _ cache pointer array for a dotting identification parameter, and when a dotting interface at the dotting identification parameter is called every time, if the tag _ cache pointer array corresponding to the dotting identification parameter is not recovered, the tag _ cache pointer array directly returns to the previous tag _ cache pointer array, otherwise, a new cache area is obtained from the lock-free queue.

According to an embodiment of the asynchronous dotting acquisition component for application layer delayed data of the present invention, the component is configured to execute the following processes of dotting identification parameters:

after the dotting interface is called, firstly, a tag _ cache pointer array distributed before is obtained according to the dotting identification parameters, if the obtaining is unsuccessful, and for the condition that the dotting identification parameters are applied for the first time, a new cache area is obtained from the lock-free queue to be used by the current dotting identification parameters until the current dotting identification parameters fall to the ground and are recycled. After a tag _ cache pointer array is obtained, recording the identification information in a cache region, when the cache region of the tag _ cache pointer array is full or the time for operating the tag _ cache pointer array last time exceeds a set value, performing file writing operation, writing all the identification information in the tag _ cache pointer array into a specified file, and recovering the tag _ cache pointer array for next distribution.

Compared with the prior art, the invention has the following beneficial effects: firstly, the application interface of the invention is single and extremely simple, the use cost is extremely low, and the application interface has two interfaces, wherein one interface is a starting setting designated log file, and the other interface is a dotting interface. It is extremely convenient for the user and does not need to consider the internal implementation details. Secondly, the method can support any seamless use, is friendly to developers, an application layer can call a dotting interface to perform dotting recording at any position of a program where dotting is needed, and the components automatically record tag _ id, guid and current time and fall into files specified by the application layer at proper time. Thirdly, the invention can reduce the performance overhead caused by dotting to the maximum extent (the request overhead of one lock-free queue can be generated by dotting for two ten thousand times, and all asynchronous output of printing records is simultaneously realized), and the time consumed by dotting or counting needs to be reduced as much as possible no matter all the programs with delay counting requirements in the cross-process or the process, so that the invention can ensure the accuracy of service delay to the maximum extent. Fourthly, each request of the application layer in the invention can record time delay, which is convenient for counting abnormal data; meanwhile, the method is friendly to secondary analysis, and the csv format storage of three fields is convenient for secondary analysis of languages such as excel or python.

Drawings

The above features and advantages of the present disclosure will be better understood upon reading the detailed description of embodiments of the disclosure in conjunction with the following drawings. In the drawings, components are not necessarily drawn to scale, and components having similar relative characteristics or features may have the same or similar reference numerals.

Fig. 1 shows an interface schematic diagram of an embodiment of an application layer delayed data asynchronous dotting acquisition component according to the present invention.

FIG. 2 illustrates a class diagram of an embodiment of an application layer latency data asynchronous dotting acquisition component of the present invention.

FIG. 3 shows a schematic diagram of the tag _ write class management and allocation of the tag _ cache pointer array shown in FIG. 2.

FIG. 4 shows a flow diagram of tag _ cache pointer array management as shown in FIG. 2.

Fig. 5 shows the main flow chart of tag dotting shown in fig. 2.

Detailed Description

The invention is described in detail below with reference to the figures and specific embodiments. It is noted that the aspects described below in connection with the figures and the specific embodiments are only exemplary and should not be construed as imposing any limitation on the scope of the present invention.

The dotting parameters of the asynchronous dotting acquisition component for the application layer delayed data are respectively as follows: tag _ id, guid, time. Wherein, tag _ id is 16-bit integer, is a parameter of dotting identification, is used for identifying the dotting position of the program, and is defined by the user. The guid is a 64-bit integer, which is a parameter of a global UID (User Identification) attached to a thread, and is used to mark a service message or configure an identifier of a complete life cycle of an operation, for example, all subsequent processes triggered by a basic market in the field of financial futures in which the present invention is located are connected in series by a uniform guid number, which is convenient for a monitoring program or a statistical program to perform service analysis and Identification. time is 64-bit integer in microseconds, and is a time parameter of the dotting time. The three parameters of guid, tag _ id and time can be combined to clearly record the time of the service data in each life cycle at the dotting point, and the service delay can be monitored through a plurality of points.

Fig. 1 shows an interface schematic diagram of an application layer delayed data asynchronous dotting acquisition component according to the present embodiment.

As shown in fig. 1, the application layer delay data asynchronous dotting acquisition component of this embodiment has two external interfaces: the initialization interface (start _ tag interface) and the dotting interface (tag interface), for example, are both interfaces implemented in C + +. The start _ tag interface is used for initializing the dotting acquisition component, such as starting a setting specified log file; the tag interface is mainly a dotting interface which is called by an application layer at a place where dotting recording is needed, a plurality of tag _ ids can be defined so as to be called by different definitions of the application layer, and the dotting acquisition component is stored in a CSV format of three parameter fields of tag _ id + guid + time so as to facilitate secondary calculation and statistics.

The definition form of the interface is as follows:

void start_tag(const char*file_name)；

void tag(uint16_t tag_id)。

as shown in FIG. 1, the start _ tag interface executes once at the entry, sets the file path and starts the dotting component thread. The execution of the application program code is divided into 3 program modules, the entry and the exit of each program module respectively call a tag interface to perform dotting tracking, for example, two dotting tag _ ids in the program module 1 are respectively 0x0001 and 0x0002, two dotting in the program module 2 are respectively 0x0003 and 0x0004, and two dotting in the program module 3 are respectively 0x0005 and 0x 0006. Each time the program is executed to the dotting point, tag _ id, guid and time information of the dotting acquisition component are recorded and written into a file (file _ name) at an appropriate time.

Fig. 2 is a class diagram of an application layer delay data asynchronous dotting collection component in the embodiment in a C + + implementation, where tag _ tool is an asynchronous component implementation class, and is used to provide a start _ tag interface and a tag interface, and includes two components: the tag _ cache pointer array and the tag _ writer class. And the tag _ cache pointer array is used for storing a cache pointer corresponding to the tag _ id. the tag _ cache pointer array is a cache area for node tag records, and one tag _ cache can store 20480 tag nodes. the tag _ writer class is mainly used for log landing, buffer management and thread start.

Fig. 3 is a schematic diagram of the tag _ write class management and distribution tag _ cache pointer array shown in fig. 2, for example, 256 tag _ cache pointer arrays (cache areas) are maintained in the asynchronous dotting component of this embodiment, where each cache area can hold 20480 tag records (e.g., tag 1-tag 20480 shown in fig. 3), each tag record (dotting record) is actually a message body in the asynchronous dotting collection component, and the message body includes three members, which are tag _ id, guid, and time respectively. After a user calls a tag (tag _ id) interface, the asynchronous dotting component encapsulates the tag _ id + guid + time into a tag node which is stored in a currently used tag _ cache (cache region) for subsequent asynchronous landing files.

FIG. 4 shows a flow diagram of tag _ cache pointer array management. In an application program, different tag _ ids are used at different dotting points, an asynchronous dotting acquisition component stores node records corresponding to the same tag _ id in the same tag _ cache pointer array until the tag _ cache pointer array is fully written and falls to a log file, the tag _ cache pointer array is recycled at the same time, and a system applies a new tag _ cache pointer array to record the same tag _ id node for data storage, so that the advantage of preventing multithreading from writing the same tag _ cache pointer array is achieved. 256 tag _ cache buffers in fig. 3 are constructed in the post-initialization period, and the cache first address is added into a lock-free queue (lock _ free _ queue < tag _ cache > empty _ tags), where the lock-free queue is a lock-free circular queue implemented based on CAS operation and with multithread read-write security, and is used for an asynchronous dotting collection component to manage the cache. The asynchronous dotting acquisition component management cache region provides two interfaces, one is an application cache region interface; and the other is to recycle the buffer area to the lock-free queue after the data in the buffer area falls to the ground. And the asynchronous dotting component applies for a tag _ id for a tag _ cache pointer array, a dotting interface at the tag _ id is called every time, if the tag _ cache pointer array corresponding to the tag _ id is not recovered, the previous tag _ cache pointer array is directly returned, and otherwise, a new cache area is obtained from the lock-free queue.

Fig. 5 is a main flow chart of tag (dotting mark) dotting. After a tap interface tag (tag _ id) is called, firstly, a previously allocated tag _ cache is obtained according to the tag _ id, if the obtaining is unsuccessful, two conditions exist, the first condition is that the tag _ cache corresponding to the tag _ id has finished the file landing and is recovered last time, and the second condition is that the tag _ id is applied for the first time, a new cache buffer area needs to be obtained from a lock-free queue to be used for the current tag _ id until the current tag _ id lands and is recovered. After the tag _ cache is obtained, the current tag information, including tag _ id, guid and macro _ time, needs to be recorded in the cache area. When the cache area of the tag _ cache is full or the time for operating the tag _ cache last time exceeds 30 seconds, notifying a tag _ writer to write a file at the moment, writing all tags in the tag _ cache into a specified file, and recycling the tag _ cache for next distribution.

The invention is applied to financial derivative transaction software at present, the transaction software has extremely high requirements on service delay, and the components of the invention can complete the record statistics of the service delay under the condition of extremely low self overhead. For example, the pricing module performs delay statistics, and the pricing is to perform correct estimation on options of related financial derivatives according to the original market conditions, and the process of calculating the theoretical price is generally called pricing. In the option trading software system, option pricing belongs to a core business process, commonly used pricing models comprise a BSM model and a Black76 model, and each model has obvious requirements on delay statistics in the process of calculating a theoretical price. If the pricing system records a time point at the place where the original quotation is received, the time point is recorded as tag (0x0001), a time point tag (0x0003) is recorded before the pricing model processes, a time point tag (0x0003) is recorded after the theoretical price is calculated, and a time point tag (0x0004) is recorded after the theoretical price is issued. The time processing process from tracking an original quotation to calculating the theoretical price can be realized by the above 4 points. All original quotations in a day are distinguished by different guids, and the triggered tag points of theoretical price calculation fall into a file in a three-field CSV format. Tag data for three original quotations are shown in the following table:

TABLE 1

According to the data information in the table, the system can analyze and display the data by using an excel or python monitoring program, such as average delay of a pricing module within 5 minutes, average delay within 30 minutes, average delay of one hour and average delay of the whole day, etc. can be counted.

While, for purposes of simplicity of explanation, the methodologies are shown and described as a series of acts, it is to be understood and appreciated that the methodologies are not limited by the order of acts, as some acts may, in accordance with one or more embodiments, occur in different orders and/or concurrently with other acts from that shown and described herein or not shown and described herein, as would be understood by one skilled in the art.

Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a web site, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk (disk) and disc (disc), as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks (disks) usually reproduce data magnetically, while discs (discs) reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An asynchronous dotting acquisition component for application layer delay data, characterized in that the component comprises:

2. The application layer delay data asynchronous dotting collection component according to claim 1, wherein the initialization interface and the dotting interface are provided by a C + + class implemented by an asynchronous component, and the C + + class implemented by the asynchronous component further includes two components: the system comprises a tag _ cache pointer array and a tag _ writer class, wherein the tag _ cache pointer array is used for storing cache region pointers corresponding to dotting identification parameters, the cache region is used for accommodating dotting records, each dotting record is a message body, the message body comprises three members which are three parameter fields in a CSV format file respectively, and the tag _ writer class is used for landing logs, managing the cache region and starting threads.

3. The asynchronous dotting acquisition component for application layer delayed data according to claim 2, wherein the component is configured to execute the following management flow of tag _ cache pointer array:

4. The application-layer delayed data asynchronous dotting acquisition component of claim 3, wherein the component is configured to perform the following process of dotting identification parameters dotting: