CN111309403B

CN111309403B - Asynchronous dotting acquisition assembly for delay data of application layer

Info

Publication number: CN111309403B
Application number: CN201911309042.2A
Authority: CN
Inventors: 张海荣; 张勇; 方义
Original assignee: Shanghai Financial Futures Information Technology Co ltd
Current assignee: Shanghai Financial Futures Information Technology Co ltd
Priority date: 2019-12-18
Filing date: 2019-12-18
Publication date: 2023-05-12
Anticipated expiration: 2039-12-18
Also published as: CN111309403A

Abstract

The invention discloses an application layer delay data asynchronous dotting acquisition component, which realizes various scenes needing to count delay in a cross-process or a process, thoroughly solves the reusability problem of a common delay counting scheme and the influence on normal delay caused by dotting, and unifies the calling modes in the cross-host and single process. The technical proposal is as follows: the application interfaces of the invention are respectively a starting setting appointed log file and a dotting interface; the invention enables the application layer to call the dotting interface to perform dotting record at any place of the program needing dotting, and the component automatically records tag_id, guid and current time and falls into a file appointed by the application layer at a proper time; in addition, each request of the application layer can record delay, so that abnormal data can be counted conveniently; meanwhile, the method is friendly to secondary analysis, and the csv format storage of three fields is convenient for secondary analysis.

Description

Asynchronous dotting acquisition assembly for delay data of application layer

Technical Field

The invention relates to a financial transaction software technology, in particular to an application layer delay asynchronous dotting acquisition component which can be particularly applied to various scenes needing statistic delay in a cross-process or a process.

Background

In some high performance applications in the field of financial transaction software, there is often a point of need for performance statistics. If delay statistics need to count the time spent by a network message from one host to the end of processing by another host, it also needs to count the time spent by a program from the entrance to the end, the former is inter-system delay, and the latter is intra-module delay.

The statistical delay often needs to record the thread number or the program execution point in addition to the recording time. Common scenarios and solutions requiring statistical delay are listed below: the first scene is to count the overall delay of multiple processes across a network, the common solution is to grasp the packets by using a network card tcp_dump, in the debugging of network problems, tcp_dump is an indispensable tool, is a command line type data packet sniffing tool based on a Unix system and can grasp the data packets flowing on the network card, and the disadvantage is that the reusability is low and the corresponding analysis and development cost is high; the second scenario is to count the internal sub-function delay of a single module, and a common solution is to apply synchronous or asynchronous journaling, which is the first thought method in most software development, record time through journaling and then manually analyze, but the disadvantage is that the too high journaling frequency affects the performance, and the too low journaling frequency has inaccurate statistical information.

Disclosure of Invention

The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

The invention aims to solve the problems, provides an application layer delay data asynchronous dotting acquisition component, realizes various scenes needing statistics delay in a cross-process or a process, thoroughly solves the reusability problem of a common delay statistics scheme and the influence on normal delay caused by dotting by providing a very simplified calling interface, and unifies the calling modes in the cross-host and the single process.

The technical scheme of the invention is as follows: the invention discloses an application layer delay data asynchronous dotting acquisition assembly, which comprises:

the two application interfaces are an initialization interface executed at the entrance and a dotting interface called at the entrance and the exit of each program module respectively, wherein the initialization interface is used for initializing work including starting a set appointed log file, and the dotting interface is used for calling an application layer at a place needing dotting record so as to carry out dotting tracking;

the storage module is used for storing CSV format files with three parameter fields for secondary calculation and statistics, wherein the three parameter fields comprise a dotting identification parameter, a global user identification parameter and a time parameter, the dotting identification parameter is used for identifying the dotting position of a program, the global user identification parameter is used for marking a service message or an identifier of a complete life cycle of configuration operation, and the time parameter is used for recording dotting time.

According to an embodiment of the application layer delay data asynchronous dotting acquisition component of the present invention, the initialization interface and the dotting interface are provided by a c++ class implemented by an asynchronous component, the c++ class implemented by the asynchronous component further comprises two components: the system comprises a tag_cache pointer array and a tag_writer class, wherein the tag_cache pointer array is used for storing a buffer area pointer corresponding to a dotting identification parameter, the buffer area is used for accommodating dotting records, each dotting record is a message body, the message body comprises three members which are respectively three parameter fields in a CSV format file, and the tag_writer class is used for log landing, buffer area management and thread starting.

According to an embodiment of the application layer delay data asynchronous dotting acquisition component of the present invention, the component is configured to execute the following management flow of tag_cache pointer array:

in an application program, different dotting positions use different dotting identification parameters, a component stores node records corresponding to the same dotting identification parameters in the same tag_cache pointer array until the tag_cache pointer array is fully written into a log file, meanwhile, the tag_cache pointer array is recycled, a new tag_cache pointer array is applied to the node records of the same dotting identification parameters for storing data, a tag_cache buffer area is built in an unlocked queue in an initialized period, a first address of the buffer area is added to the unlocked queue, the unlocked queue is a multi-thread read-write safe lock-free circular queue based on CAS operation, the component is used for managing the buffer area, two interfaces are provided for managing the buffer area, one interface is used for applying for the buffer area, the other interface is used for recovering the buffer area into the unlocked queue after the data in the buffer area falls into the ground, the component applies for a tag_cache pointer array for one dotting identification parameter each time, the tag_cache pointer array is called in the buffer area, and if the corresponding tag_cache array is not called, and if the tag_cache array is not directly returned to the buffer area, otherwise, the tag_cache array is not directly obtained.

According to an embodiment of the application layer delay data asynchronous dotting acquisition component of the invention, the component is configured to execute the following procedure of dotting identification parameters:

after the dotting interface is called, firstly acquiring a tag_cache pointer array allocated before according to the dotting identification parameter, if the acquisition is unsuccessful, and acquiring a new buffer area from the lock-free queue for the current dotting identification parameter to use until the current dotting identification parameter is landed and recovered for the first time. After the tag_cache pointer array is obtained, recording the identification information in a cache region, and when the cache region of the tag_cache pointer array is full or the time for operating the tag_cache pointer array last time exceeds a set value, performing file writing operation, writing all the identification information in the tag_cache pointer array into a designated file, and recovering the tag_cache pointer array for next allocation.

Compared with the prior art, the invention has the following beneficial effects: firstly, the application interface of the invention is simple and extremely low in use cost, and two interfaces are provided, wherein one interface is a starting setting appointed log file, and the other interface is a dotting interface. It is extremely convenient for the user, without regard to the implementation details of the interior. Secondly, the invention can support any seamless use, is friendly to a developer, the application layer can call a dotting interface to perform dotting record at any place of the program needing dotting, and the component automatically records tag_id, guid and current time and falls into a file appointed by the application layer at a proper time. Thirdly, the invention can reduce the performance cost brought by dotting to the greatest extent (the request cost of a lock-free queue can be generated when dotting is performed for twice, and all asynchronous output is printed and recorded), and the invention can ensure the accuracy of service delay to the greatest extent, regardless of the procedure with delay statistics requirements in a cross-procedure or a process, as far as possible, by reducing the time consumed by dotting or statistics. Fourth, each request of the application layer can record delay, so that abnormal data can be counted conveniently; meanwhile, the method is friendly to secondary analysis, and the csv format of the three fields is stored, so that the secondary analysis of the languages such as excel or python is facilitated.

Drawings

The above features and advantages of the present invention will be better understood after reading the detailed description of embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily to scale and components having similar related features or characteristics may have the same or similar reference numerals.

FIG. 1 illustrates an interface diagram of one embodiment of an application layer latency data asynchronous dotting acquisition component of the present invention.

FIG. 2 illustrates a class diagram of one embodiment of an application layer latency data asynchronous dotting acquisition component of the present invention.

FIG. 3 is a schematic diagram illustrating the tag_write class management and allocation tag_cache pointer array shown in FIG. 2.

FIG. 4 shows a flow chart of tag_cache pointer array management shown in FIG. 2.

Fig. 5 shows a main flow chart of the tag dotting shown in fig. 2.

Detailed Description

The invention is described in detail below with reference to the drawings and the specific embodiments. It is noted that the aspects described below in connection with the drawings and the specific embodiments are merely exemplary and should not be construed as limiting the scope of the invention in any way.

The dotting parameters of the asynchronous dotting acquisition component of the delay data of the application layer in the embodiment are respectively as follows: tag_ id, guid, time. The tag_id is a 16-bit integer and is a parameter of dotting identification, and is used for identifying the dotting position of a program and is defined by a user. guid is a 64-bit integer, which is a parameter of a global UID (user identification ) attached to a thread, and is an identifier for marking a complete life cycle of a service message or a configuration operation, for example, all subsequent processes of a basic market trigger in the financial futures field of the present invention are connected in series through a unified guid number, so that a monitoring program or a statistics program can conveniently perform service analysis and identification. time is a 64-bit integer in microseconds, which is the time parameter at the moment of dotting. By combining the three parameters of guid, tag_id and time, the time of the business data in each life cycle at the dotting position can be clearly recorded, and the business delay can be monitored through a plurality of points.

Fig. 1 shows an interface schematic diagram of an application layer delay data asynchronous dotting acquisition component of the present embodiment.

As shown in fig. 1, the application layer delay data asynchronous dotting acquisition component of this embodiment provides two interfaces for external use: the initialization interface (start_tag interface) and the dotting interface (tag interface) are both interfaces implemented by c++, for example. The start_tag interface is used for initializing the dotting acquisition component, such as starting a setting appointed log file; the tag interface is mainly a dotting interface which is called by an application layer at a place where dotting records are needed, a plurality of tag_ids can be defined so as to call at different definitions of the application layer, and the dotting acquisition component is stored in a CSV format of three parameter fields, namely tig_id+guid+time, so that secondary calculation and statistics are facilitated.

The interface is defined as follows:

void start_tag(const char*file_name)；

void tag(uint16_t tag_id)。

as shown in fig. 1, the start tag interface executes once at the entry, sets the file path and starts the dotting component thread. The execution of the application program code is divided into 3 program modules, the entry and the exit of each program module respectively call the tag interface to perform dotting tracking, for example, two dotting tag_ids in the program module 1 are respectively 0x0001 and 0x0002, two dotting in the program module 2 are respectively 0x0003 and 0x0004, and two dotting in the program module 3 are respectively 0x0005 and 0x0006. Every time the program is executed to the above dotting point, tag_id, guid and time information of the dotting acquisition component are recorded and written into a file (file_name) at an appropriate time.

Fig. 2 is a class diagram of an application layer delay data asynchronous dotting acquisition component in the embodiment on a c++ implementation, where tag_tool is an asynchronous component implementation class, and is configured to provide a start_tag interface and a tag interface, and includes two components at the same time: tag_cache pointer array and tag_writer class. the tag_cache pointer array is used for storing the cache region pointer corresponding to the tag_id. the tag_cache pointer array is a cache area of node tag records, and one tag_cache can store 20480 tag nodes. the tag_writer class is mainly used for log landing, buffer management and thread starting.

FIG. 3 is a schematic diagram of the tag_write class management and allocation tag_cache pointer array shown in FIG. 2, for example, maintaining 256 tag_cache pointer arrays (buffers) in the asynchronous dotting component of this embodiment, where each buffer can hold 20480 tag records (such as tag 1-tag 20480 shown in FIG. 3), each tag record (dotting record) is actually a message body in the asynchronous dotting acquisition component, and the message body contains three members, namely tag_id, guid and time. After a user calls a tag (tag_id) interface, an asynchronous dotting component packages tag_id+guid+time into tag nodes to be stored in a currently used tag_cache (cache region) for a subsequent asynchronous floor file.

FIG. 4 shows a flow chart of tag_cache pointer array management. In an application program, different tag_ids are used at different dotting positions, the asynchronous dotting acquisition component stores node records corresponding to the same tag_id in the same tag_cache pointer array until the tag_cache pointer array is fully written into a log file, and meanwhile, the tag_cache pointer array is recycled, and the system applies for a new tag_cache pointer array to store data for the node records of the same tag_id. 256 tag_cache buffers in fig. 3 are constructed in the period after initialization, and the first addresses of the buffers are added into a lock_free_queue (log_cache) which is a multi-thread read-write safe lock-free circular queue realized based on CAS operation for the asynchronous dotting acquisition component to manage the buffers. The asynchronous dotting acquisition component manages the buffer area to provide two interfaces, one is an application buffer area interface; and the other is to recycle the buffer area to the lock-free queue after the data in the buffer area is landed. The asynchronous dotting component applies for a tag_cache pointer array for a tag_id, a dotting interface at the tag_id is called each time, if the tag_cache pointer array corresponding to the tag_id is not recovered yet, the previous tag_cache pointer array is directly returned, otherwise, a new cache buffer area is obtained from the lock-free queue.

Fig. 5 is a main flowchart of tag (dotting mark) dotting. After a dotting interface tag (tag_id) is called, firstly, acquiring a previously allocated tag_cache according to the tag_id, and if the acquisition is unsuccessful, obtaining two cases, wherein the first case is that the tag_cache corresponding to the tag_id has completed file landing and is recycled last time, and the second case is that the tag_id is applied for the first time, and then a new cache area is required to be acquired from a lock-free queue for the current tag_id until the file landing and the recovery are completed. After the tag_cache is obtained, the tag information of this time needs to be recorded in the cache area, including tag_id, guid and macro_time. When the tag_cache buffer area is full or the time for last operation of the tag_cache is more than 30 seconds, the tag_writer is informed to perform file writing operation, all tags in the tag_cache are written into a designated file, and the tag_cache is recycled for next allocation.

The invention is applied to financial derivative transaction software at present, the transaction software has extremely high requirement on service delay, and the component of the invention can complete record statistics on the service delay under the condition of extremely low self-spending. For example, the time delay statistics of the pricing module are that the pricing is to correctly estimate the options of related financial derivatives according to the original quotation, and the process of calculating the theoretical price is generally called pricing. In the option trading software system, option pricing belongs to a core business process, and a common pricing model comprises a BSM model and a Black76 model, and the demand of each model for delay statistics is remarkable in the process of calculating theoretical prices. If the pricing system receives the original market place and records a time point, we record the time point as tag (0 x 0001), the pricing model records a time point tag (0 x 0003) before processing, calculates the theoretical price and records a time point tag (0 x 0003), and issues the theoretical price to record a time point tag (0 x 0004). The 4 points can track the time processing process from the original quotation to the calculation of the theoretical price. All original quotations in one day can be distinguished by different guids, and the triggered tag points calculated by theoretical price can fall into a file in a three-field CSV format. The tag data of three original quotations are shown in the following table:

TABLE 1

Based on the data information in the table above, the system can analyze and present the data using excel or python monitors, such as counting the average delay of the positioning module within 5 minutes, the average delay within 30 minutes, the average delay of one hour, the average delay of the whole day, etc.

While, for purposes of simplicity of explanation, the methodologies are shown and described as a series of acts, it is to be understood and appreciated that the methodologies are not limited by the order of acts, as some acts may, in accordance with one or more embodiments, occur in different orders and/or concurrently with other acts from that shown and described herein or not shown and described herein, as would be understood and appreciated by those skilled in the art.

Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a web site, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk (disk) and disc (disk) as used herein include Compact Disc (CD), laser disc, optical disc, digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks (disk) usually reproduce data magnetically, while discs (disk) reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An application layer delay data asynchronous dotting acquisition assembly, the assembly comprising:

2. The application-layer-delayed-data asynchronous dotting acquisition component of claim 1, wherein the initialization interface and the dotting interface are provided by a c++ class implemented by an asynchronous component, the c++ class implemented by the asynchronous component further comprising two components: the system comprises a tag_cache pointer array and a tag_writer class, wherein the tag_cache pointer array is used for storing a buffer area pointer corresponding to a dotting identification parameter, the buffer area is used for accommodating dotting records, each dotting record is a message body, the message body comprises three members which are respectively three parameter fields in a CSV format file, and the tag_writer class is used for log landing, buffer area management and thread starting.

3. The application-layer-delayed data asynchronous dotting acquisition component of claim 2, wherein the component is configured to perform the following management flow of the tag_cache pointer array:

4. An application-layer-delayed-data asynchronous dotting acquisition component as claimed in claim 3, wherein the component is configured to perform the following procedure of dotting identification parameters:

after the dotting interface is called, firstly acquiring a tag_cache pointer array allocated before according to the dotting identification parameter, if the acquisition is unsuccessful, and acquiring a new buffer area from the lock-free queue for the current dotting identification parameter to use until the current dotting identification parameter is landed and recovered for the first time; after the tag_cache pointer array is obtained, recording the identification information in a cache region, performing file writing operation when the cache region of the tag_cache pointer array is full or the time for operating the tag_cache pointer array last time exceeds a set value, writing all the identification information in the tag_cache pointer array into a designated file, and recycling the tag_cache pointer array for next allocation.