CN112003814A

CN112003814A - Market data processing method and device, terminal equipment and storage medium

Info

Publication number: CN112003814A
Application number: CN201910445836.5A
Authority: CN
Inventors: 郑继翔; 王先平; 张勇; 张立生; 李小波
Original assignee: China Merchants Securities Co ltd
Current assignee: China Merchants Securities Co ltd
Priority date: 2019-05-27
Filing date: 2019-05-27
Publication date: 2020-11-27
Anticipated expiration: 2039-05-27
Also published as: CN112003814B

Abstract

The embodiment of the application discloses a market data processing method, a market data processing device, terminal equipment and a storage medium, wherein the market data processing method comprises the following steps: acquiring market quotation snapshot data; setting corresponding positions in a bitmap according to the difference between each non-array numerical field of the quotation snapshot data and the corresponding non-array numerical field of the quotation snapshot data at a time point on the same security, and compressing the difference to obtain first compressed data; generating primitive coding information according to the relation between each array numerical field of the market data and the corresponding array numerical field of the market snapshot data at the last time point, and compressing the primitive coding information to obtain second compressed data; and obtaining a quotation data compressed message according to the set bitmap, the first compressed data and the second compressed data. The embodiment of the application can greatly reduce the transmitted data volume, can ensure the real-time transmission of the quotation snapshot data, and can also ensure the compression ratio of the quotation snapshot data.

Description

Market data processing method and device, terminal equipment and storage medium

Technical Field

The application belongs to the technical field of finance, and particularly relates to a market data processing method and device, terminal equipment and a computer readable storage medium.

Background

With the continuous development of society, the trading volume of securities trading banks is increasing, and market data is huge. On the other hand, with the rise of programmed trading and strategic trading, the requirement on real-time property of market data pushing is higher and higher.

The method for pushing huge market data occupies large bandwidth resources, has long transmission time and low pushing real-time performance. In order to improve the push real-time performance and save bandwidth resources, the market data is generally compressed and then transmitted. The current market data compression algorithms include protobuf compression algorithm and 7z compression algorithm. However, in practical market data applications, the compression rate of the protobuf compression algorithm is not ideal, and the requirement of high compression rate cannot be met. And the real-time market data are all small message data transmission, and if the 7z compression algorithm needs to reach a high compression ratio, the market data need to be cached in the compression process. The real-time performance of the data is sacrificed by caching the market data, and the requirement of the real-time performance of the data cannot be met.

That is, the current market data compression technology cannot combine the compression ratio and the real-time data transmission.

Disclosure of Invention

In view of this, embodiments of the present application provide a market data processing method and apparatus, a terminal device, and a storage medium, so as to solve the problem that the market data compression technology in the prior art cannot combine the compression ratio and the real-time data transmission.

A first aspect of an embodiment of the present application provides a market data processing method, including:

acquiring market quotation snapshot data, wherein the market quotation snapshot data comprise an array numerical field and a non-array numerical field;

setting corresponding positions in a bitmap according to the difference between each non-array numerical field of the quotation snapshot data and the corresponding non-array numerical field of the quotation snapshot data at a time point on the same security, and compressing the difference to obtain first compressed data; the compressed message comprises a message header and compressed data content, wherein the message header comprises the bitmap, and each bit of the bitmap corresponds to a non-array numerical field of the quotation snapshot data;

generating primitive coding information according to the relation between each array numerical field of the market data and the corresponding array numerical field of the market snapshot data at the last time point, and compressing the primitive coding information to obtain second compressed data; the primitive coding information is information representing the change of the gear information of the market quotation snapshot data compared with the gear information of the market quotation snapshot data at the last time point;

and obtaining a compressed message of the market data according to the set bitmap, the first compressed data and the second compressed data.

With reference to the first aspect, in a possible implementation manner, the setting a corresponding position in a bitmap according to a difference between each non-array value field of the market snapshot data and a corresponding non-array value field of the market snapshot data at a time point on the same security, and compressing the difference to obtain first compressed data includes:

respectively calculating the difference between the non-array numerical field of the quotation snapshot data and the corresponding non-array numerical field of the quotation snapshot data at the same security at a time point;

judging whether each difference value is zero or not;

when the difference value is zero, the corresponding position in the bitmap is not positioned;

and when the difference value is not zero, setting a corresponding position in the bitmap, and compressing the difference value to obtain first compressed data.

With reference to the first aspect, in one possible implementation manner, the primitive includes an insert primitive, a delete primitive, a modify primitive, and a complete primitive; the insertion primitive comprises a market position, a price increment value and a quantity value at the last time point; the modification primitive comprises a last time point market gear and a quantity increment value; the deletion primitive comprises a market situation gear at the last time point; the completion primitive includes end flag information;

generating primitive coding information according to the relationship between each array value field of the market data and the corresponding array value field of the market snapshot data at the last time point, and compressing the primitive coding information to obtain second compressed data, wherein the generating primitive coding information comprises:

judging whether the price of each gear of the array numerical field of the market quotation snapshot data is equal to the price of the corresponding gear of the array numerical field of the market quotation snapshot data at the last time point;

when the prices are equal, calculating a difference value of the entrusted quantity between the corresponding gear of the market quotation snapshot data and the corresponding gear of the market quotation snapshot data at the last time point;

judging whether the difference value of the entrusted quantity is zero or not;

when the difference value of the entrusted quantity is not zero, generating the modification primitive, and compressing the modification primitive to obtain first primitive compressed data;

when the prices are not equal, judging the relationship between the quotation information of the corresponding gear of the quotation snapshot data and the corresponding gear of the quotation snapshot data at the last time point;

when the relation is that the gear is newly increased, generating the insertion primitive, and compressing the insertion primitive to obtain second primitive compressed data;

when the relation is gear movement, generating the deletion primitive, and compressing the deletion primitive to obtain third primitive compressed data;

when the primitive compression process is finished, generating the completion primitive, and compressing the completion primitive to obtain fourth primitive compressed data;

wherein the second compressed data includes at least one of the first primitive compressed data, the second primitive compressed data, the third primitive compressed data, and the fourth primitive compressed data.

A second aspect of the embodiments of the present application provides a market data processing method, including:

acquiring a compressed message of market data, wherein the compressed message of market data comprises compressed data content and a message header, the message header comprises bitmap information, and the compressed data content comprises first compressed data corresponding to a non-array numerical field and second compressed data corresponding to an array numerical field;

decompressing the first compressed data to obtain incremental market data;

obtaining first quotation data of a non-array numerical field according to the bitmap information, the incremental quotation data and the quotation snapshot data at a time point on the same security;

decompressing the second compressed data to obtain primitive coding information;

obtaining second market data of an array numerical field according to the primitive coding information and the market snapshot data at the last time point;

and obtaining decompressed market quotation snapshot data according to the first market quotation data and the second market quotation data.

With reference to the second aspect, in a possible implementation manner, the obtaining, according to the bitmap information, the incremental quotation data, and the quotation snapshot data at a time point on the same security, first quotation data of a non-array value field includes:

judging whether the numerical value of the corresponding bit of the non-array numerical field of the quotation snapshot data in the bitmap is a first numerical value or not according to the bitmap information;

when the data is the first numerical value, taking the quotation snapshot numerical value of the corresponding non-array numerical value field of the quotation snapshot data at the last time point as the first quotation numerical value of the corresponding non-array numerical value field of the quotation snapshot data;

when the data is a second numerical value, adding the incremental market data of the corresponding non-array numerical value field and the market snapshot numerical value of the corresponding field of the market snapshot data at the last time point to obtain a second market numerical value of the corresponding non-array numerical value field;

and obtaining the first quotation data according to the first quotation value and the second quotation value.

With reference to the second aspect, in a possible implementation manner, the obtaining second market data of an array value field according to the primitive coding information and the market snapshot data of the last time point includes:

and according to the primitive coding information, carrying out corresponding operation on the volume price information of the corresponding gear of the market quotation snapshot data at the last time point to obtain the second market quotation data of the array numerical value field.

A third aspect of the embodiments of the present application provides a market data processing apparatus, including:

the system comprises a quotation snapshot data acquisition module, a quotation snapshot data processing module and a quotation snapshot processing module, wherein the quotation snapshot data acquisition module is used for acquiring quotation snapshot data which comprises an array numerical value field and a non-array numerical value field;

the first compression module is used for setting corresponding positions in a bitmap according to the difference between each non-array numerical field of the quotation snapshot data and the corresponding non-array numerical field of the quotation snapshot data at a time point on the same security, and compressing the difference to obtain first compressed data; the compressed message comprises a message header and compressed data content, wherein the message header comprises the bitmap, and each bit of the bitmap corresponds to a non-array numerical field of the quotation snapshot data;

the second compression module is used for generating primitive coding information according to the relation between each array value field of the market data and the corresponding array value field of the market snapshot data at the last time point, and compressing the primitive coding information to obtain second compressed data; the primitive coding information is information representing the change of the gear information of the market quotation snapshot data compared with the gear information of the market quotation snapshot data at the last time point;

and the generation module is used for obtaining the compressed message of the market data according to the set bitmap, the first compressed data and the second compressed data.

A fourth aspect of the embodiments of the present application provides a market data processing apparatus, including:

the system comprises a quotation data compressed message acquisition module, a quotation data compressed message acquisition module and a quotation data compressed message processing module, wherein the quotation data compressed message comprises compressed data content and a message header, the message header comprises bitmap information, and the compressed data content comprises first compressed data corresponding to a non-array numerical field and second compressed data corresponding to an array numerical field;

the first decompression module is used for decompressing the first compressed data to obtain incremental market data;

the first calculation module is used for obtaining first quotation data of a non-array numerical field according to the bitmap information, the incremental quotation data and the quotation snapshot data at the same security at the last time point;

the second decompression module is used for decompressing the second compressed data to obtain primitive coding information;

the second calculation module is used for obtaining second market quotation data of the array numerical value field according to the primitive coding information and the market quotation snapshot data at the last time point;

and the decompressed data generation module is used for obtaining decompressed market quotation snapshot data according to the first market quotation data and the second market quotation data.

A fifth aspect of embodiments of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the method according to the first aspect or the second aspect when executing the computer program.

A sixth aspect of embodiments of the present application provides a computer-readable storage medium, in which a computer program is stored, which, when executed by a processor, performs the steps of the method according to the first or second aspect.

Compared with the prior art, the embodiment of the application has the advantages that:

in the embodiment of the application, each bit of the bitmap is used for representing each non-array numerical field of the market quotation snapshot data, when the market quotation numerical difference value of a certain non-array numerical field is zero, only 1bit is needed to transmit the market quotation data of the non-array numerical field without change, and when the market quotation numerical difference value of a certain non-array numerical field is not zero, the difference compression value of the field is transmitted, so that the transmitted data volume can be reduced; in addition, for the array numerical field of the market snapshot data, the change information of the gear quantity price of the field is represented by generating primitive coding information, and the compressed primitive coding information is transmitted to replace the information of the gear quantity price, so that the transmitted data quantity can be greatly reduced. The data volume of transmission is greatly reduced, the real-time transmission of the quotation snapshot data is ensured, and the compression ratio of the quotation snapshot data is also considered.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.

Fig. 1 is a schematic block diagram of a flow of a method for compressing market data according to an embodiment of the present disclosure;

FIG. 2 is a schematic block diagram of a compression process for a non-array value field according to an embodiment of the present disclosure;

FIG. 3 is a schematic block diagram of a compression process of primitive coding information provided by an embodiment of the present application;

fig. 4 is a schematic block diagram of a flow of a method for decompressing market data according to an embodiment of the present application

FIG. 5 is a block diagram illustrating a decompression process for a non-array value field according to an embodiment of the present disclosure;

fig. 6 is a schematic block diagram of a market data compression device according to an embodiment of the present application

Fig. 7 is a schematic block diagram of a market data decompression device according to an embodiment of the present application;

fig. 8 is a schematic diagram of a terminal device according to an embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

In order to explain the technical solution described in the present application, the following description will be given by way of specific examples.

Example one

The present embodiment will explain a market data compression process.

Referring to fig. 1, a schematic flow chart of a method for compressing market data according to an embodiment of the present application is shown, where the method includes the following steps:

step S101, market quotation snapshot data are obtained, and the market quotation snapshot data comprise array value fields and non-array value fields.

It can be understood that, a trading exchange (e.g., a filter depth trading exchange) generates corresponding market snapshot data at regular intervals, and the interval between two market snapshots is generally 3 seconds at present, i.e., one market snapshot is generated every 3 seconds. And after the quotation snapshot data is decoded and recoded by the dealer quotation system, the quotation snapshot data with a fixed data structure can be obtained. The market snapshot data includes an array value field and a non-array value field.

Typically, the array value field includes a bid price, a bid amount, a bid price, and a bid amount. The unarrated numerical field includes message ID, market, data generation time, transaction status, yesterday closing price, opening price, maximum price, minimum price, latest price, number of trades, total committed bids, total committed asks, weighted average committed bids, IOPV net worth estimate, return to maturity rate, stop-and-go price, and stop-and-go price. The format of the current market snapshot message is generally shown in table 1 below.

TABLE 1

And respectively carrying out corresponding compression processing on the array numerical field and the non-array numerical field after acquiring the market quotation snapshot data of the market quotation system to obtain a market quotation data compression message.

Step S102, setting corresponding positions in a bitmap according to difference values between each non-array numerical field of the quotation snapshot data and corresponding non-array numerical fields of the quotation snapshot data at a previous time point on the same security, and compressing the difference values to obtain first compressed data; the compressed message comprises a message header and compressed data content, wherein the message header comprises the bitmap, and each bit of the bitmap corresponds to a non-array numerical field of the market situation snapshot data.

It should be noted that, the exchange generates corresponding market snapshot data at regular intervals, and the time intervals of any two consecutive market snapshots are equal. The quotation snapshot data at the previous time point of the same security refers to the quotation snapshot data at the previous time interval, and the quotation snapshot data is the quotation snapshot data closest to the current time point. For example, a ten-gear snapshot is generated every 3 seconds at the exchange, the current time point is 11 points, 11 minutes and 13 seconds, the generation time of the previous ten-gear snapshot is 11 points, 11 minutes and 10 seconds, and the ten-gear snapshot generated by 11 points, 11 minutes and 10 seconds is the previous time point situation snapshot data. The same security refers to a target security having the same security code.

After the market snapshot data is compressed, the market snapshot data can be transmitted in a compressed message form. The compressed message comprises a message header and compressed data content, wherein the message header comprises a bitmap and can also comprise a message type and an incremental compression ID. The message type can represent the message type of the compressed message, the incremental compression ID can ensure the continuity of the incremental compressed message, and if the message is discontinuous, the data obtained by subsequent decompression can be wrong. Each security code corresponds to a separate incremental message ID. The compressed data content includes delta compressed content for non-array value fields and primitive compressed content for array value fields.

In this embodiment, the definition of the compact packet may be as shown in table 2 below.

TABLE 2

The message type occupies 16 bits, the increment compression ID 16 bits, the bitmap 32 bits, and the compression data content has an indefinite length.

It should be noted that each non-array value field corresponds to one bit in the bitmap, and the corresponding bit in the bitmap is set according to a difference between each non-array value field of the current market snapshot data and the corresponding field of the market snapshot data at the previous time point. If the difference is zero, the corresponding position of the field is not set, if the difference is not zero, the corresponding position of the field is set, and the difference is compressed. Only 1bit is needed to transmit the market data of the unchanged field, and for the market data of the changed field, the compressed difference value is transmitted. Thus, the data transmission amount can be reduced, and the real-time performance of data transmission is improved.

For example, the value of the lowest-priced field in the current market snapshot data of a certain security is 100000(10.0000 yuan), the value of the highest-priced field is 100300(10.0300 yuan), the value of the lowest-priced field in the market snapshot data of the last time point of the security is 100000(10.0000 yuan), and the value of the highest-priced field is 100500(10.0500 yuan). For the lowest price field, if the difference value between the current market and the market at the last time point is zero, the corresponding position of the lowest price field in the bitmap is not positioned, and the value of the corresponding position is 0; and for the highest price field, if the difference value between the current market and the market at the last time point is not zero, setting the corresponding position of the highest price field in the bitmap, wherein the value of the corresponding position is 1, and compressing the difference value to obtain the compressed difference value.

The first compressed data refers to data of difference compressed data including all non-array value fields of the market snapshot data.

In some embodiments, referring to the schematic block diagram of the compression process of the unarrated numerical value field shown in fig. 2, the specific process of obtaining the first compressed data by setting a corresponding position in the bitmap according to a difference between each unarrated numerical value field of the market snapshot data and a corresponding unarrated numerical value field of the market snapshot data at a time point on the same security and compressing the difference may include:

step S201, respectively calculating a difference between the non-array value field of the market snapshot data and the corresponding non-array value field of the market snapshot data at a previous time point on the same security.

Step S202, judging whether each difference value is zero or not; when the difference is zero, the process proceeds to step S203, whereas when the difference is not zero, the process proceeds to step S204.

And step S203, not positioning the corresponding position in the bitmap.

And step S204, setting corresponding positions in the bitmap, and compressing the difference to obtain first compressed data.

Wherein, after the corresponding position in the bitmap is set, the value is 1, and if not set, the value is 0.

Specifically, the stock code is used as an index to search the previous-time-point market snapshot data buffered in the local memory. Respectively calculating difference values between the non-array numerical value fields based on the current market quotation snapshot data and the market quotation snapshot data at the last time point; judging whether the difference value is zero; if the difference value is zero, the data of the field is not changed, the market data of the field is not transmitted, and the corresponding position of the field in the bitmap is not positioned; if the difference is not zero, the data of the field is changed, the corresponding of the field in the bitmap is set to be 1, and the difference is compressed to obtain corresponding compressed data. And after all the differential value compression is finished, obtaining first compressed data of the non-array value field.

The compression algorithm used for compressing the difference value may be any, and in this embodiment, the field difference value may be compressed by using a Zigzag numerical compression algorithm.

Step S103, generating primitive coding information according to the relation between each array value field of the market data and the corresponding array value field of the market snapshot data at the last time point, and compressing the primitive coding information to obtain second compressed data; the primitive coding information is information representing the change of the gear information of the market snapshot data compared with the gear information of the market snapshot data at the last time point.

It should be noted that, the exchange generally takes one market snapshot data every 3 seconds, and there are three cases of price and data information of each gear within 3 seconds: no change, the data of the deal on the 1 gear is changed, the first gears are all handed over, and the subsequent gears move forward. And based on the gear data characteristics of the market, predefining an operation primitive, and representing the change of the gear information of the current market snapshot data compared with the market gear information at the last time point by using the operation primitive.

In this embodiment, four operation primitives are defined, which respectively include: insert primitive, delete primitive, modify primitive, and complete primitive. The primitive definitions may be as shown in table 3 below.

TABLE 3

Wherein 0x10, 0x20, 0x40 and 0x70 are numerical values represented in a 16-ary system. 0x00 indicates an end flag, i.e., when 0x00 is read, it indicates that the data processing process is ended.

One market snapshot data generally includes multiple-stage market information, and currently, the market snapshot data is mainly 10 stages (ten-stage snapshot). And generating primitive coding information corresponding to each gear according to the quantity and price change conditions of the current market snapshot data and each gear of the snapshot market snapshot data at the last time point.

When the current price of a certain gear of a certain array value field is equal to the price of the last time point, calculating the difference between the quantities; judging whether the difference value is zero, if the difference value is zero, indicating that the price and the quantity of the gear are not changed, and not generating primitive coding information; if the difference is not zero, the price on the gear is not changed, and the quantity is changed, generating primitive coding information: modify | the market gear | number increment difference at a previous point in time and compress the primitive coding information.

When the current price of a certain gear of a certain array numerical value field is not equal to the price of the last time point, judging the relation between the price of the gear and the market price of the last time point; if the new addition is made, generating primitive coding information: inserting the quantity of the market gear price increment value of the previous time point, and compressing the primitive coding information; if the gear is moved, deleting the price and the quantity of the corresponding gear at the last time point, moving the market gear at the last time point down by one gear, and generating primitive coding information: and deleting the market gear of the previous time point.

In some embodiments, the primitives include an insert primitive, a delete primitive, a modify primitive, and a complete primitive; inserting primitives including last-time market gear, price increment value and quantity value; modifying the primitive to comprise the market quotation gear at the last time point and the quantity increment value; deleting the primitive to comprise a market gear at the last time point; the completion primitive includes end flag information.

It is to be understood that the last-time-point-in-schedule gear in each primitive refers to a gear required to perform a corresponding operation.

Typically, the array value field includes a bid price, a bid amount, a bid price, and a bid amount. The bid price and ask price both include ten-grade prices, for example, the ten-grade price for ask price is 100000(10.0000 yuan), 100200(10.0200 yuan), 100300(10.0300 yuan), 100400(10.0400 yuan), 100600(10.0600 yuan), 100800(10.0800 yuan), 100900(10.0900 yuan), 101000(10.1000 yuan), 101100(10.1100 yuan) and 101200(10.1200 yuan), wherein the prices are all expressed as integers in the algorithm. Each gear has a corresponding quantity, which is the bid quantity or the ask quantity, respectively. The primitive coding information is used for representing the quantity and price change of each gear in the current latest market situation snapshot data and the market situation snapshot data of the last time interval, for example, if a certain gear of the current latest market situation snapshot data has no price and the quantity is more or less than the same gear of the previous market situation snapshot data, the quantity change of the gear can be represented by a modification primitive.

At this time, referring to the schematic block diagram of the compression process of the primitive coding information shown in fig. 3, the specific process of generating the primitive coding information according to the relationship between each array value field of the market data and the corresponding array value field of the market snapshot data at the last time point, and compressing the primitive coding information to obtain the second compressed data may include:

step S301, judging whether the price of each gear of the array numerical field of the market quotation snapshot data is equal to the price of the corresponding gear of the array numerical field of the market quotation snapshot data at the last time point; when the prices are equal, the process proceeds to step S302, whereas when the prices are not equal, the process proceeds to step S305.

Specifically, whether the prices of all the gears of each array value field of the current market snapshot data and the market snapshot data at the last time point are equal or not is sequentially compared. When the prices are equal, further judging whether the consignation quantity of the gear is changed; and when the prices are not equal, further judging the relation between the quotation data of the gear and the quotation data of each gear of the previous time point quotation snapshot data.

For example, the buy 1, buy 2, and buy 3 files of the current market snapshot data of a certain security are 100100(10.0100 yuan), 100200(10.0200 yuan), and 100600(10.0600 yuan), respectively, and the prices of the buy 1, buy 2, and buy 3 files of the market snapshot data at the last time point are 100100(10.0100 yuan), 100300(10.0300 yuan), and 100500(10.0500 yuan). Comparing the prices of the current latest buying 1 gear with the prices of the buying 1 gear at the last time point, wherein the prices of the two buying 1 gears are 100100(10.0100 yuan), and at the moment, further judging whether the entrusted quantity of the buying 1 gear changes. And if the prices of buying the 2 gear and buying the 3 gear are not equal, further judging the latest relationship between the numerical values of buying the 2 gear and buying the 3 gear and the gear at the last time point.

Step S302, calculating a difference value of the consignation quantity between the corresponding gear of the market quotation snapshot data and the corresponding gear of the market quotation snapshot data at the last time point.

Step S303, judging whether the difference value of the entrustment quantity is zero or not; and when the difference value of the consignment quantity is not zero, the step S304 is carried out, otherwise, when the difference value of the consignment quantity is zero, the fact that the quantity and the price of the corresponding gear are not changed is indicated, and the primitive coding information is not generated.

Specifically, for the gear positions with equal prices, the difference of the consignment quantities corresponding to the gear positions is further calculated. If the difference is zero, it indicates that there is no change in the number of gear positions, the primitive coding information is not generated, and if the difference is not zero, it indicates that there is a change in the number of gear positions, the modification primitive is generated.

For example, the prices before and after buying 1 and buying 2 for a certain security are equal, the requested quantity of buying 1 for the current latest market snapshot data is 1000, the requested quantity of buying 2 is 2000, the requested quantity of buying 1 for the market snapshot data at the previous time point is 500, the requested quantity of buying 2 is 2000, at this time, the difference of the requested quantity of buying 1 is 500, and a modification primitive is generated: 0x20 buy 1 st gear 500. And if the difference value of the consignment quantity of buying the 2 files is zero, the primitive coding information does not need to be generated.

Step S304, generating a modification primitive, and compressing the modification primitive to obtain the first primitive compressed data.

In a specific application, the compression algorithm for compressing the modification primitive may be, but is not limited to, a Zigzag numerical compression algorithm.

Step S305, judging the relation between the quotation information of the corresponding gear of the quotation snapshot data and the corresponding gear of the quotation snapshot data at the last time point. When the relationship is a new gear, the process proceeds to step S306, and when the relationship is a gear shift, the process proceeds to step S307.

Step S306, generating an inserting primitive and compressing the inserting primitive to obtain the second primitive compressed data.

And step S307, generating a deletion primitive, and compressing the deletion primitive to obtain third primitive compressed data.

The compression algorithm for compressing the insert primitive and the delete primitive and the complete primitive may be, but is not limited to, a Zigzag value compression algorithm.

It should be noted that, when the prices of the corresponding gears are not equal, the relationship between the quotation information of the gear of the current quotation snapshot data and the quotation information of the corresponding gear of the previous quotation snapshot data generally includes two types, namely, new addition and movement. The new addition means that the value of a certain gear of the current market quotation snapshot data is not included in the market quotation snapshot data at the last time point, and the gear is newly added. For example, the buy 1, buy 2, and buy 3 files of the current market snapshot data of a certain security are 100100(10.0100 yuan), 100200(10.0200 yuan), and 100600(10.0600 yuan), respectively, and the prices of the buy 1, buy 2, and buy 3 files of the market snapshot data at the last time point are 100100(10.0100 yuan), 100300(10.0300 yuan), and 100500(10.0500 yuan); at this time, the prices 10.0200 yuan and 10.0600 yuan of buying 2 and buying 3 files of the current market snapshot data are not included in the market snapshot data at the last time point, the gear positions 100200(10.0200 yuan) and 100600(10.0600 yuan) of the current market snapshot data are newly added, and at this time, if the consignation quantity of each gear position is 1000, an insertion primitive may be generated: 0x10| buy 2 nd gear |1| 1000; of course, in this case, the buy 2 file and the buy 3 file of the market snapshot data at the previous time point need to be deleted, and the buy 3 file of the current market snapshot data needs to be inserted by inserting the primitive.

The gear shift refers to shifting up or down a certain gear, and generally, a gear with a high price moves towards a gear with a low price. For example, the buy 1, buy 2, and buy 3 files of the current market snapshot data of a certain security are 100100(10.0100 yuan), 100300(10.0300 yuan), and 100600(10.0600 yuan), respectively, and the buy 1, buy 2, and buy 3 files of the market snapshot data at the last time point are 100100(10.0100 yuan), 100200(10.0200 yuan), and 100300(10.0300 yuan), respectively. If the current latest market snapshot data does not have 100200(10.0200 meta) gear, deleting 100200(10.0200 meta) gear in the market snapshot data at the last time point, and moving 100300(10.0300 meta) gear to buy 2 gear, at this time, generating a deletion primitive: buy 2 nd gear 0x40 |.

Step S308, when the primitive compression process is finished, a finishing primitive is generated and compressed to obtain fourth primitive compressed data; wherein the second compressed data includes at least one of the first primitive compressed data, the second primitive compressed data, the third primitive compressed data, and the fourth primitive compressed data.

Specifically, after primitive compression encoding is finished, a corresponding end primitive may be generated: 0x70|0x 00.

It can be seen that the primitive coding information is used to represent the quantity and price change information of each gear of the array value field, and in the snapshot structured market data, the price quantity information of the gear occupies 70% of the whole structure, so that the primitive coding information is used to replace the price quantity information of the gear to be transmitted, and the transmitted data quantity can be greatly reduced, so as to ensure the real-time performance and the compression ratio of the market snapshot data.

It should be noted that, in a specific application, the process of generating the primitive code according to the market snapshot gear change is defined as an LDPE (level-differential-private-encoding) compression algorithm, and the LDPE compression algorithm is used for compressing market snapshot data. And the LDPE compression algorithm can be implemented as a loop process, and generates corresponding primitive coding information by comparing the price quantity information of each gear in a loop mode. During the circulation judgment, whether the latest market gear currently processed and the market gear at the last time interval are both smaller than 10 can be judged, and if the new market gear is smaller than 10, the insertion primitive can be generated by the subsequent gear in a newly increased form.

It should be noted that the execution sequence between step S102 and step S103 may be executed simultaneously or sequentially, and the execution sequence is arbitrary.

And step S104, obtaining a compressed message of the market data according to the set bitmap, the first compressed data and the second compressed data.

Specifically, security codes are output to a buffer in a serialized mode, after all fields are compressed, the obtained bitmap is stored to the head of a compressed message, first compressed data and second compressed data are output to the buffer in a serialized mode and stored to the compressed data content part of the message, and the compressed message of market data is obtained. Then, the compressed message of the market data can be pushed out.

In specific application, when the Zigzag numerical compression algorithm is adopted in the processes of differential compression, primitive compression and the like, the embodiment of the application compresses the market snapshot data through the bitmap and LDPE compression algorithm to obtain compressed data, and the theoretical compression ratio is as follows: the full compression ratio is 1 and the compression ratio is between 58.5 when there is no change in the market snapshot. The average compression ratio in practical applications is around 10. And the compression process is summarized and all the compression is small message real-time data compression.

Therefore, the embodiment of the application can greatly reduce the transmitted data volume, ensure the real-time transmission of the quotation snapshot data and also give consideration to the compression ratio of the quotation snapshot data.

Example two

This embodiment will describe a decompression process corresponding to the compression process of the first embodiment.

Referring to fig. 4, a schematic block diagram of a flow of a method for decompressing market data according to an embodiment of the present application is provided, where the method may include the following steps:

step S401, a compressed message of market data is obtained, the compressed message of market data includes compressed data content and a message header, the message header includes bitmap information, and the compressed data content includes first compressed data corresponding to a non-array numerical field and second compressed data corresponding to an array numerical field.

It will be appreciated that a compact message includes a message header and a compact data content, the message header including a bitmap, a message type, and an incremental message ID. The compressed data content includes the difference compressed content of the non-array value field and the primitive coding information of the array value field.

According to the message type and the incremental message ID, the message can be judged to be effective, if the message is discontinuous, the incremental message is not available, and the full message can be retransmitted at the moment.

And S402, decompressing the first compressed data to obtain incremental market data.

Specifically, the first compressed data is decompressed through a corresponding decompression algorithm, and incremental data of the non-array numerical field is obtained. Wherein the decompression algorithm corresponds to the compression algorithm. For example, when the compression algorithm is a Zigzag numerical compression algorithm, decompression is also performed using the Zigzag numerical compression algorithm herein. The incremental market data refers to the incremental data of each non-array value field.

Step S403, obtaining first quotation data of the non-array numerical field according to the bitmap information, the incremental quotation data and the quotation snapshot data at the same security at the same time point.

The quotation snapshot data at the previous time point is the quotation snapshot data of the same security at the previous time point stored locally. In the specific application, the stock code information is obtained by deserializing the stock code, and the market quotation snapshot data stored at the last local time point is searched out according to the stock code.

Whether the corresponding non-array value field is changed or not can be judged through the numerical value of each bit in the bitmap, and the changed quantity can be known through the incremental market data.

In some embodiments, referring to the schematic block diagram of the decompression process of the unarrated numerical field shown in fig. 5, the specific process of obtaining the first quotation data of the unarrated numerical field according to the bitmap information, the incremental quotation data and the quotation snapshot data at a time point on the same security may include:

step S501, judging whether the value of the corresponding bit of the non-array value field of the market quotation snapshot data in the bitmap is a first value or not according to the bitmap information.

Step S502, when the data is the first numerical value, taking the quotation snapshot numerical value of the corresponding non-array numerical value field of the quotation snapshot data at the last time point as the first quotation numerical value of the corresponding non-array numerical value field of the quotation snapshot data.

And S503, when the data is the second numerical value, adding the incremental market data of the corresponding non-array numerical value field and the market snapshot numerical value of the corresponding field of the market snapshot data at the last time point to obtain the second market numerical value of the corresponding non-array numerical value field.

Step S504, obtaining the first quotation data according to the first quotation value and the second quotation value.

The first value may be 0, and the second value is 1 accordingly. The first value refers to a value corresponding to when the market data is not changed, and the second value refers to a value corresponding to when the market data is changed. When the first value is 0 and the second value is 1, and when the value of a certain field in the bitmap is 0, the field is not changed and incremental data does not exist; conversely, if the value is 1, this indicates that the field has changed and incremental data exists.

For the non-array value field which is not changed, the market conditions snapshot value of the market conditions snapshot data at the last time point can be used as the market conditions snapshot value of the current latest market conditions snapshot data. And corresponding to the changed non-array numerical field, the market condition snapshot numerical value at the last time point can be added with the corresponding incremental data to obtain a numerical value which is the latest market condition snapshot data of the field.

And S404, decompressing the second compressed data to obtain primitive coding information.

The decompression algorithm of the first compressed data and the second compressed data may be specifically, but not limited to, a Zigzag numerical compression algorithm.

And S405, obtaining second market data of the array numerical value field according to the primitive coding information and the market snapshot data at the last time point.

In some embodiments, the process of step S405 may be, for example: and according to the primitive coding information, carrying out corresponding operation on the price measuring information of the corresponding gear of the market quotation snapshot data at the last time point to obtain second market quotation data of the array numerical field.

Specifically, according to the primitive type and the information in the primitive coding information, the price measuring information of each gear of the market quotation snapshot data at the last time point is deleted, operated, inserted and the like, so as to obtain the latest market quotation snapshot data.

The specific process can be as follows: initializing a current previous gear and a current new gear, wherein the current previous gear is a gear in the market quotation snapshot data at a previous time point, the current new gear is a gear of the market quotation snapshot data, and the gears are all initialized to be 1; entering circulation, and judging whether the last time gear in the current primitive coding information is equal to the last time gear;

when the gears are not equal, taking the quantity price information of the current previous gear as the quantity price information of the current new gear, moving the current previous gear and the current new gear downwards by one gear, returning to the step of judging whether the previous gear in the current primitive coding information is equal to the current previous gear until entering the next primitive coding information;

when the gear positions are equal, judging the type of the current primitive coding information;

when the primitive is inserted, calculating to obtain the quantity price information of the current new gear according to the price increment value and the quantity value in the current primitive coding information, then taking the next primitive coding information as the current primitive coding information, and returning to the step of judging whether the last gear in the current primitive coding information is equal to the last gear;

when the primitive is modified, calculating to obtain the quantity information of the current new gear according to the quantity increment value in the current primitive coding information, taking the next primitive coding information as the current primitive coding information, and returning to the step of judging whether the gear at the last time in the current primitive coding information is equal to the gear at the last time;

when the primitive is deleted, deleting the quantity and price information of the current gear at the last time;

and circularly judging whether the current new gear is processed and completed or not when the primitive is completed, and copying the price number of the current previous gear to the current new gear if the processing is not completed. If the processing is completed, the current last gear is shifted down by one.

When all primitive coding information is processed, corresponding volume price information in the market quotation snapshot data at the last time point is directly used as volume price information of the current market quotation snapshot data for some volume price information without primitive coding information.

And S406, obtaining decompressed market quotation snapshot data according to the first market quotation data and the second market quotation data.

It should be noted that the decompression process of the array value field and the decompression process of the non-array value field may be executed simultaneously or sequentially, and the execution sequence is arbitrary.

Therefore, by matching the decompression algorithm of the embodiment with the compression algorithm of the first embodiment, the real-time transmission and the compression ratio of the market snapshot data can be ensured.

It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

EXAMPLE III

Referring to fig. 6, a schematic block diagram of a market data compression apparatus according to an embodiment of the present application is provided, where the apparatus may include:

the quotation snapshot data acquisition module 61 is configured to acquire quotation snapshot data, where the quotation snapshot data includes an array value field and a non-array value field;

a first compression module 62, configured to set a corresponding position in a bitmap according to a difference between each non-array numerical field of the market snapshot data and a corresponding non-array numerical field of the market snapshot data at a time point on the same security, and compress the difference to obtain first compressed data; the compressed message comprises a message header and compressed data content, wherein the message header comprises the bitmap, and each bit of the bitmap corresponds to a non-array numerical field of the quotation snapshot data;

a second compression module 63, configured to generate primitive coding information according to a relationship between each array value field of the market data and a corresponding array value field of the market snapshot data at the previous time point, and compress the primitive coding information to obtain second compressed data; the primitive coding information is information representing the change of the gear information of the market quotation snapshot data compared with the gear information of the market quotation snapshot data at the last time point;

and the generating module 64 is configured to obtain the compressed market data packet according to the set bitmap, the first compressed data, and the second compressed data.

In a possible implementation manner, the first compression module is specifically configured to: respectively calculating the difference between the non-array numerical field of the quotation snapshot data and the corresponding non-array numerical field of the quotation snapshot data at the same security at a time point; judging whether each difference value is zero or not; when the difference value is zero, the corresponding position in the bitmap is not positioned; and when the difference value is not zero, setting a corresponding position in the bitmap, and compressing the difference value to obtain first compressed data.

In one possible implementation, the primitives include an insert primitive, a delete primitive, a modify primitive, and a complete primitive; the insertion primitive comprises a market position, a price increment value and a quantity value at the last time point; the modification primitive comprises a last time point market gear and a quantity increment value; the deletion primitive comprises a market situation gear at the last time point; the completion primitive includes end flag information;

the second compression module is specifically configured to: judging whether the price of each gear of the array numerical field of the market quotation snapshot data is equal to the price of the corresponding gear of the array numerical field of the market quotation snapshot data at the last time point; when the prices are equal, calculating a difference value of the entrusted quantity between the corresponding gear of the market quotation snapshot data and the corresponding gear of the market quotation snapshot data at the last time point; judging whether the difference value of the entrusted quantity is zero or not; when the difference value of the entrusted quantity is not zero, generating the modification primitive, and compressing the modification primitive to obtain first primitive compressed data; when the prices are not equal, judging the relationship between the quotation information of the corresponding gear of the quotation snapshot data and the corresponding gear of the quotation snapshot data at the last time point; when the relation is that the gear is newly increased, generating the insertion primitive, and compressing the insertion primitive to obtain second primitive compressed data; when the relation is gear movement, generating the deletion primitive, and compressing the deletion primitive to obtain third primitive compressed data; when the primitive compression process is finished, generating the completion primitive, and compressing the completion primitive to obtain fourth primitive compressed data;

It should be noted that the compression apparatus provided in this embodiment corresponds to the compression method of the first embodiment one to one, and for specific description, reference is made to the above corresponding contents, which is not described herein again.

Therefore, the embodiment can greatly reduce the transmitted data volume, ensure the transmission real-time performance of the quotation snapshot data and also consider the compression ratio of the quotation snapshot data.

Example four

Referring to fig. 7, a schematic block diagram of a market data decompression device provided in an embodiment of the present application is shown, where the device may include:

a quotation data compressed message acquiring module 71, configured to acquire a quotation data compressed message, where the quotation data compressed message includes a compressed data content and a message header, the message header includes bitmap information, and the compressed data content includes first compressed data corresponding to a non-array numerical field and second compressed data corresponding to an array numerical field;

a first decompression module 72, configured to decompress the first compressed data to obtain incremental market data;

the first calculation module 73 is configured to obtain first quotation data of a non-array numerical field according to the bitmap information, the incremental quotation data, and the quotation snapshot data at a time point on the same security;

a second decompressing module 74, configured to decompress the second compressed data to obtain primitive encoding information;

a second calculating module 75, configured to obtain second market data of an array value field according to the primitive coding information and the market snapshot data at the previous time point;

and a decompressed data generating module 76, configured to obtain decompressed market data snapshot data according to the first market data and the second market data.

In a possible implementation manner, the first calculating module is specifically configured to:

In a possible implementation manner, the second calculating module is specifically configured to:

It should be noted that the decompression device provided in this embodiment corresponds to the decompression method of the second embodiment one to one, and the detailed description is given to the corresponding contents above, which is not repeated herein.

Therefore, the embodiment can ensure the transmission real-time property and the compression ratio of the market snapshot data.

EXAMPLE five

Fig. 8 is a schematic diagram of a terminal device according to an embodiment of the present application. As shown in fig. 8, the terminal device 8 of this embodiment includes: a processor 80, a memory 81 and a computer program 82 stored in said memory 81 and executable on said processor 80. The processor 80, when executing the computer program 82, implements the steps in the above-described embodiments of the market data compression method or market data decompression method, such as steps S101 to S104 shown in fig. 1 or steps S401 to S406 in fig. 4. Alternatively, the processor 80, when executing the computer program 82, implements the functions of each module or unit in each device embodiment described above, such as the functions of the modules 61 to 64 shown in fig. 6 or 71 to 76 shown in fig. 7.

Illustratively, the computer program 82 may be partitioned into one or more modules or units that are stored in the memory 81 and executed by the processor 80 to accomplish the present application. The one or more modules or units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution of the computer program 82 in the terminal device 8.

The terminal device 8 may be a computing device such as a cloud server. The terminal device may include, but is not limited to, a processor 80, a memory 81. Those skilled in the art will appreciate that fig. 8 is merely an example of a terminal device 8 and does not constitute a limitation of terminal device 8 and may include more or fewer components than shown, or some components may be combined, or different components, e.g., the terminal device may also include input-output devices, network access devices, buses, etc.

The Processor 80 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The storage 81 may be an internal storage unit of the terminal device 8, such as a hard disk or a memory of the terminal device 8. The memory 81 may also be an external storage device of the terminal device 8, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the terminal device 8. Further, the memory 81 may also include both an internal storage unit and an external storage device of the terminal device 8. The memory 81 is used for storing the computer program and other programs and data required by the terminal device. The memory 81 may also be used to temporarily store data that has been output or is to be output.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus, terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus and the terminal device are merely illustrative, and for example, the division of the module or the unit is only one logical function division, and there may be another division in actual implementation, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated modules or units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow in the method of the embodiments described above can be realized by a computer program, which can be stored in a computer-readable storage medium and can realize the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.

Claims

1. A market data processing method is characterized by comprising the following steps:

2. The method according to claim 1, wherein said setting corresponding positions in a bitmap according to differences between each non-array value field of said market snapshot data and a corresponding non-array value field of market snapshot data at a time point on the same security, and compressing said differences to obtain first compressed data comprises:

respectively calculating the difference between the non-array numerical field of the quotation snapshot data and the corresponding non-array numerical field of the quotation snapshot data at the same security at the same time point;

judging whether each difference value is zero or not;

3. The method according to claim 1 or 2, wherein the primitives include an insert primitive, a delete primitive, a modify primitive, and a complete primitive; the insertion primitive comprises a market position, a price increment value and a quantity value at the last time point; the modification primitive comprises a last time point market gear and a quantity increment value; the deletion primitive comprises a market situation gear at the last time point; the completion primitive includes end flag information;

judging whether the difference value of the entrusted quantity is zero or not;

4. A market data processing method is characterized by comprising the following steps:

decompressing the first compressed data to obtain incremental market data;

5. The method of claim 4, wherein said deriving first market data for a non-array value field based on said bitmap information, said incremental market data, and said snapshot of market data at a point in time on the same security comprises:

6. The method as claimed in claim 4 or 5, wherein the obtaining the second market data of the array value field according to the primitive coding information and the market snapshot data of the previous time point comprises:

7. A market data processing apparatus, comprising:

8. A market data processing apparatus, comprising:

9. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 3 or 4 to 6 when executing the computer program.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 3 or 4 to 6.