CN117370290A - Software for financial data compression - Google Patents

Abstract

The invention provides software for financial data compression, which uses two compression algorithms, namely a first compression algorithm for data differential compression and a second compression algorithm for data compression of the whole memory block, and develops a system comprising a secondary transmission driving module, a delay module and a landing module; of these two algorithms, the currently popular compression algorithm can be adopted with respect to the second compression mode, wherein the first data differential compression is divided into two schemes: distinguishing by field and distinguishing by byte; the invention can simulate the market information of the outbound site, has no problem of losing data or the fact that the data is not paired with the market, has small recorded data volume, has a two-layer compression mechanism, has high speed of simulating the market, has high modularization degree and is convenient to maintain.

Description

Software for financial data compression

Technical Field

The invention relates to the field of financial data compression, in particular to software for financial data compression.

Background

The financial market data stream of the exchange is transmitted in real time, and in many cases, the data needs to be played back to realize: system testing, problem duplication, algorithm back testing and the like, but the real-time market data is stored, the data volume is extremely huge, and a large amount of disk space is occupied.

The current simulation system source simulation quotation system cannot truly record the change process of data in a determined time, often has incorrect data playing time points and also has the condition of data loss, the recorded and replayed data volume is huge, and the recorded original data occupies real-time quotation processing time, so that quotation disclosure is slow.

Therefore, the method has the advantages of not losing data, ensuring that the data quantity is not overlarge, and ensuring accurate market revealing time.

Disclosure of Invention

In view of this, the present invention provides software for financial data compression.

In order to achieve the above purpose, the present invention provides a software for financial data compression, which comprises a secondary transmission driving module, a delay module and a landing module, wherein the secondary transmission driving module comprises processing control, push distribution, a console, a global, data module, transaction processing, parameters and communication functions, and the landing module comprises processing control, market recording and data recording functions.

Preferably, the process control is coupled to a console, a global, transaction processing, communication, parameters, and data modules, and distributes data push to a landing module and a transport platform, the console and global being coupled to the transport platform, and the communication is coupled to the transaction processing, parameters, and remote transport platform.

Preferably, the data processing control of the pushing and distributing is followed by market recording and data recording.

Preferably, the two-stage transmission driving module determines whether to load the landing disc module by configuring a loading mark, and under the configuration, the recorded data file name and the storage time interval of the landing disc file are also provided.

Preferably, in the driving module of the secondary transmission module, a user can decide whether to load the landing recording data module according to the configuration, and the landing module, the delay module and the simulated market record simulation are all decided whether to load according to the configuration.

Preferably, the disc-drop module is provided with a snapshot data for recording the market data pushed by the secondary transmission, and the disc-drop module sequentially stores the snapshot data into the recorded data file according to the configured time, and when the recorded data is stored, the data all use the original data of the exchange.

Preferably, the system also comprises an original market data record and an original market data play, wherein the original market data record is required to be hung under the secondary transmission of the market by a simulation market module of the market, and the function of the original market data record is started through the configuration of the secondary transmission; the original quotation data is played, the simulation quotation server is required to be mounted, the simulation quotation module simulates a secondary transmission source at the moment, and the quotation data is sent out through the playing simulation file.

Preferably, the system has two compression algorithms, the first compression algorithm is data differential compression, the second compression algorithm is compression on data of the whole memory block, and the first data differential compression is divided into two schemes: distinguishing by field and distinguishing by byte:

distinction by field: the data structure compares various variable fields to determine which field data have differences, and the composed data consists of a difference head and a difference body;

difference head: the data structure is in one-to-one correspondence with byte bits in the difference head according to a variable field, 1 or 0 on the byte bits indicates whether the fields in the following data exist or not, a first byte first bit indicates a first field, a second bit indicates a second field, and so on, if the first bit is 0, the first field is the same as the last time, and the second bit is 1, the first data in the difference body is the content of the second field, and the difference head also comprises the position of the next difference head of 2 bytes and the current difference head length;

differential: the field without difference will not exist, only the field with difference with the last data exists in the difference body, wherein the existence of difference is determined by the difference head;

examples: there are 10 variable fields "30 bytes total" for one structure, with the third field "field type: unsigned long "and fifth field" field types: double' has data difference, the difference head is 2 bytes, the difference is 12 bytes, and the whole difference data is 14 bytes. Wherein the 3 rd bit and the 5 th bit of the first byte in the difference header are 1 and the others are 0;

the advantages are that: it can be known what the specific value should be for that field to change;

disadvantages: the data type and length of each field in the structure must be known; when a field with 4 bytes of "unsigned long" is changed only by one byte, 4 bytes are needed to be used for representing;

byte-wise differentiation:

comparing the data structure with each byte of the binary string to determine which byte data are different, wherein the data consist of a difference head and a difference body;

difference head: mapping each byte in the data structure to one bit in the difference head, and using 1 or 0 on the bit to indicate whether the byte is changed or not, wherein the difference head also comprises the position of the next difference head of 2 bytes and the length of the difference head;

differential: the byte without difference will not exist, only the byte with difference with the last data exists in the difference body, wherein the existence of difference is determined by the difference head;

examples: there are 10 variable fields "30 bytes total" for one structure, with the third field "field type: there are data differences and a fifth field type in 2 bytes in unsigned long: the data difference exists in 2 bytes in double ", the difference head is 4 bytes, the difference body is 4 bytes, and the whole difference data is 8 bytes. If the 1 st, 2 nd and 4 th fields are long type, namely 4 bytes of each field, the first byte in the difference head is 0, the 1 st and 2 nd bits of the second byte are 1, the 1 st and 2 nd bits of the third byte are 1, and the other bytes are 0;

the advantages are that: the method is irrelevant to specific data and structure of the user, and has high universality;

disadvantages: the difference header is large, it is impossible to distinguish which data has changed, and it is also not known what the value of a specific field is, and the probability that some fields in the difference header are all 0 is large.

Compared with the prior art, the invention can simulate real quotation information, has no problem of losing data or actually unpaired data and market, has small recorded data volume, has a two-layer laminating mechanism, can simulate quotation, has high modularization degree and is convenient to maintain.

Drawings

FIG. 1 is a diagram of a lower hanger of a transmission platform according to the present invention;

FIG. 2 is a diagram of a two-level transmission architecture of the present invention;

FIG. 3 is a schematic diagram of a market place holder module according to the present invention.

Detailed Description

The technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Example 1

Original market data record

1. Calling and starting an original data recording interface, wherein the original data recording interface needs to be transmitted into a market ID, a file storage catalog, a file storage data mark, a log callback interface and a transaction request interface: the successful return of the start is greater than or equal to 0, and the failed return is less than 0:

a) After checking the market ID, the system initializes the original data file system, such as market information, code table file and original data file, establishes own code table and code table snapshot sequence, changes the memory block (index number is needed to be put into one memory block), and pays attention to: subsequent snapshots must be taken in this order;

b) Initializing an original recorded data block, taking out the current snapshot and storing the current snapshot in a block head.

2. When the first push data packet arrives, calling an original data recording interface, wherein the interface protocol number is the beginning of the data packet, and transmitting market time:

adding a packet start flag to the block, followed by a time interval from the last packet; the incoming market is not the same as the last market in time, and a market time change protocol is added.

3. When the push data is released, the original data recording interface is called, the transmitted protocol number is obtained according to the relation between the current push data protocol number and the protocol number, and the data and the length are transmitted:

a) Checking whether the protocol number is between 0 and 19, if so, executing the next step, and if not, jumping to f);

b) According to the structure corresponding to the protocol number, the transmitted data is fetched, and if the incoming length is inconsistent with the current structure, the error is reported and the data is withdrawn; otherwise, the code is taken out to inquire the sequence number, and then the position of the data in the snapshot memory is searched out according to the relation between the sequence number and the protocol number;

c) Establishing a BITMAP table according to the size of the structure body, wherein the size of the BITMAP table is as follows: (size+15)/16, then BITMap is all cleared 0;

d) And comparing the transmitted data double bytes with the designated snapshot position, and when the data double bytes are found to be different, taking out different data, adding the different data to the BITMAP table, and updating snapshot data. Note that when the data length is single, the last comparison will not be compared in double bytes, and must be compared and processed in one byte;

e) Adding the generated BITMAP and data into a memory block according to a format;

f) Analog protocol numbers and data are directly added into the memory block.

4. And (5) cycling the previous step until the current data packet is completely decoded.

5. After the unpacking is completed, calling an original data recording interface, and transmitting a data packet ending protocol:

a) Adding a packet end mark to the memory block;

b) Checking whether the current block exceeds 10 minutes, and the recorded data packets also exceed 2000 data packets, or the reason recorded data is about to exceed 5M (initializing the maximum block memory), and returning if the condition is not met; if the condition is satisfied, performing as follows;

c) The data blocks are compressed once and stored in the file, if the data blocks are needed to be stored in a divided mode and the current file is more than 10 blocks, the current file is needed to be closed, and then the next file is initialized;

d) Initializing an original recorded data block, taking out the current snapshot and storing the current snapshot in a block head.

6. The system exits or the last time the disc is received, the operation is as follows:

a) Compressing the data block once, storing the data block into a file, setting a last file mark in a file header, and closing the current file;

b) Releasing the current resource.

Example 2

Original quotation data playing

1. Initializing a playing interface: the parameters of the transmission are: playing the file and the current date of the market.

a) Firstly, opening a file, and performing validity check on the file, for example: checking a file mark, checking the market ID in the file and the current module market ID, and reporting an error to exit if the checking is not passed; taking out all the currently played block numbers;

b) And taking out the current market information and the code table, and recovering the snapshot of the first block of data.

2. Play is started: the parameters of the transmission are: play position (number of data blocks), play speed;

if the current block number is different from the playing position, jumping to the appointed block number, starting a playing thread, wherein the playing thread works as follows:

a) Loading current block data and decompressing the current block data;

b) Restoring the snapshot of the data to a snapshot memory;

c) Taking the current data, checking whether the protocol number is a packet start mark, and executing the next step if the protocol number is the packet start mark; otherwise jump to g);

d) Taking out the waiting time, and calculating the waiting time and the playing speed (if the speed is 0, dividing the waiting time by the speed; equal to 0, the same is true; less than 0, the waiting time is multiplied by the absolute value of the speed), the difference between the current time interval pen-up time and the real waiting time is calculated, if the difference is greater than 15 milliseconds, the waiting time is set (if the waiting time is greater than 500 milliseconds, the waiting time is set to be 500 milliseconds), and the next step is executed; otherwise, moving the data fetch position, and jumping to the f);

e) Waiting for the appointed time, checking whether the playing state is playing, if not, circularly waiting, and if so, jumping to the step c);

f) Checking whether the current protocol number is market time, if so, updating the market time, moving the data access position, and jumping to c); otherwise, the next step is to;

g) Checking whether the protocol number is a packet end mark, if so, pushing out the data in the pushing pool, moving the data taking position, and then jumping to c); otherwise, the next step is to;

h) Checking whether the protocol number is 20 or above, if so, storing the current data in a push pool, moving a data fetch position, and then jumping to c); otherwise, the next step is to;

i) Checking whether the protocol number is between 0 and 19, if so, the next step; otherwise reporting error, moving the data position to jump to c);

j) Taking out the sequence number, and then calculating the position of the snapshot where the data is located according to the sequence number;

k) At the snapshot specified location, then processed according to the word (double word byte) in the snapshot to which the following BITMAP BIT corresponds, when the BIT is set, to update the data in the snapshot to the data following the BITMAP, note: when the data length is single and the last bit is 1, only one byte is updated;

l) storing the currently designated snapshot data in a push pool, moving the data position, and jumping to c);

m) note that: when a push pool exists, the protocol number needs to be converted with the protocol number when pushing.

3. Pause the playback: and setting the current playing state as stop.

4. And (5) resuming playing: and setting the current playing state as playing.

5. And (3) adjusting the playing speed: and adjusting the playing speed.

6. And (5) ending playing:

a) Closing the file and the connection;

b) And releasing the resource.

7. Obtaining the number of data blocks: and returning to the total number of the current playing quotation blocks.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (8)

1. The software for financial data compression is characterized by comprising a secondary transmission driving module, a time delay module and a landing module, wherein the secondary transmission driving module comprises processing control, pushing and distributing, a control console, a global system, a data module, transaction processing, parameters and communication functions, and the landing module comprises processing control, quotation recording and data recording functions.

2. The software for financial data compression of claim 1, wherein the process control is coupled to a console, a global, transaction processing, communication, parameters, and data modules, and distributes data push to a landing module and a transmission platform, the console and global being coupled to the transmission platform, and the communication being coupled to the transaction processing, parameters, and the remote transmission platform.

3. A software for financial data compression according to claim 2, wherein the data processing control of the push distribution is followed by a market recording and recording of data.

4. The software for financial data compression of claim 1, wherein the secondary transmission driver module determines whether to load the landing disc module by configuring a load flag, and further wherein the recorded data file name and the save time interval of the landing disc file are provided under the configuration.

5. The software for financial data compression according to claim 4, wherein the secondary transmission module driver module is configured to determine whether to load the landing recording data module, and the landing module, the delay module, and the simulated market record simulation are all configured to determine whether to load.

6. The software for financial data compression according to claim 5, wherein the landing module has a snapshot data for recording the market data pushed by the secondary transmission, and the landing module stores the snapshot data in the recorded data file in sequence according to the configured time, and the data is the transaction original data when the recorded data is stored.

7. The software for financial data compression of claim 1, further comprising an original market data record and an original market data play, wherein the original market data record requires to hang down a market analog market module under a secondary transmission of the market, and the function of the original market data record is started through configuration of the secondary transmission; the original quotation data is played, the simulation quotation server is required to be mounted, the simulation quotation module simulates a secondary transmission source at the moment, and the quotation data is sent out through the playing simulation file.

8. The software for compressing financial data according to any one of claims 1-7, wherein the system has two compression algorithms, the first is differential compression of data, the second is compression of data of the whole memory block, the second compression mode is compression algorithm, and the differential compression of the first data is divided into two schemes: distinguishing by field and distinguishing by byte:

distinction by field: the data structure compares various variable fields to determine which field data have differences, and the composed data consists of a difference head and a difference body;

difference head: the data structure is in one-to-one correspondence with byte bits in the difference head according to a variable field, 1 or 0 on the byte bits indicates whether the fields in the following data exist or not, a first byte first bit indicates a first field, a second bit indicates a second field, and so on, if the first bit is 0, the first field is the same as the last time, and the second bit is 1, the first data in the difference body is the content of the second field, and the difference head also comprises the position of the next difference head of 2 bytes and the current difference head length;

differential: the field without difference will not exist, only the field with difference with the last data exists in the difference body, wherein the existence of difference is determined by the difference head;

byte-wise differentiation:

comparing the data structure with each byte of the binary string to determine which byte data are different, wherein the data consist of a difference head and a difference body;

difference head: mapping each byte in the data structure to one bit in the difference head, and using 1 or 0 on the bit to indicate whether the byte is changed or not, wherein the difference head also comprises the position of the next difference head of 2 bytes and the length of the difference head;

differential: the byte without the difference will not exist, and only the byte with the difference from the last data exists in the difference body, wherein the existence of the difference is determined by the difference head.