CN115174698A - Market data decoding method, device, equipment and medium based on table entry index - Google Patents
Market data decoding method, device, equipment and medium based on table entry index Download PDFInfo
- Publication number
- CN115174698A CN115174698A CN202211091771.7A CN202211091771A CN115174698A CN 115174698 A CN115174698 A CN 115174698A CN 202211091771 A CN202211091771 A CN 202211091771A CN 115174698 A CN115174698 A CN 115174698A
- Authority
- CN
- China
- Prior art keywords
- field
- module
- target
- market data
- decoded
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/0001—Systems modifying transmission characteristics according to link quality, e.g. power backoff
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Abstract
The invention relates to the technical field of big data, and provides a market data decoding method, a device, equipment and a medium based on table index, wherein the method is applied to a system comprising modules built based on a hardware platform and comprises the following steps: the stop bit decoding module identifies a field to be decoded in the market data, the field mapping module extracts a PMAP value in the field to be decoded by using a state machine and transmits the PMAP value to the table entry index module, the table entry index module queries in a field ID index table by taking the PMAP value as an address and returns a queried target ID to the field mapping module, and the field mapping module outputs the field to be decoded and the target ID to the field post-processing module for processing to obtain the decoded target market data. According to the invention, each functional module can be built based on the hardware platform, so that the decoupling of the module function is realized, and each module is accelerated and executed in parallel based on the hardware platform, so that the decoding efficiency of market data can be effectively improved.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a market data decoding method, device, equipment and medium based on table entry indexes.
Background
In recent years, the domestic quantitative investment scale is gradually increased, the exchange technology system is continuously upgraded, and the instant market information is particularly important for transaction decision makers, such as: the LDDS (Low latency Data Distribution System) System in Exchange takes FAST (FIX attached for Streaming) as a basis and combines with the STEP (Security Trade Exchange Protocol) Protocol of the Chinese standard to design a Data transmission Protocol with native compatibility and high Data compression rate.
At present, most of transaction systems in the domestic financial industry are based on a traditional Central Processing Unit (CPU) architecture, the problem of high time delay exists in market situation protocol decoding with high compression rate, large fluctuation exists in market situation time delay due to system process scheduling and CPU main frequency influence, and CPU load occupation is overhigh.
Disclosure of Invention
In view of the above, it is necessary to provide a method, an apparatus, a device and a medium for decoding market data based on table index, which aim to solve the problem of high delay in decoding market data.
A market data decoding method based on table index is applied to a target system, the target system comprises a stop bit decoding module, a field mapping module, a table index module and a field post-processing module which are built based on a hardware platform, and the method comprises the following steps:
in response to the received market data, the stop bit decoding module identifies a field to be decoded in the market data;
the field mapping module extracts a body field PMAP value and a repeat group field PMAP value in the field to be decoded by using a state machine;
the field mapping module passing the body field PMAP value and the duplicate group field PMAP value to the table entry indexing module;
the table entry indexing module takes the main field PMAP value and the repeat group field PMAP value as addresses, and inquires in a field ID indexing table to obtain a target ID;
the table entry indexing module returns the target ID to the field mapping module;
the field mapping module outputs the field to be decoded and the target ID to the field post-processing module;
and the field post-processing module processes the field to be decoded according to the target ID to obtain decoded target market data.
According to a preferred embodiment of the present invention, the stop bit decoding module identifying the field to be decoded in the market data comprises:
acquiring the highest bit of each byte in the market data;
and when the highest bit of the byte is 1, determining the byte as the field to be decoded.
According to the preferred embodiment of the present invention, the field ID index table is used for storing the field ID of each type of market data in the message template, and the field ID index table is stored in the entry index module;
the table entry indexing module takes the main field PMAP value and the repeat group field PMAP value as addresses, and queries in a field ID index table to obtain a target ID, and the table entry indexing module comprises:
acquiring all fields with the value of 1 in the main field PMAP value and the repeat group field PMAP value;
acquiring a domain name corresponding to each field in all the fields from the field ID index table;
and determining the field ID corresponding to each domain name as the target ID.
According to a preferred embodiment of the present invention, the field post-processing module processes the field to be decoded according to the target ID, and obtaining the decoded target market data includes:
acquiring a domain corresponding to the target ID as a target domain;
determining data attributes according to the target domain;
and carrying out scaling and type conversion on the field to be decoded according to the data attribute to obtain the target market data.
According to a preferred embodiment of the present invention, the field post-processing module processes the field to be decoded according to the target ID, and obtaining the decoded target market data further includes:
determining a recipient of the target market data;
determining the transmission bandwidth and the operational capability corresponding to the receiver;
when the transmission bandwidth and the operational capacity corresponding to the receiver reach a completion condition, acquiring a domain operational character corresponding to the target ID;
determining a completion strategy according to the domain operator corresponding to the target ID;
and completing the default field in the field to be decoded according to the completion strategy to obtain the target market data.
According to a preferred embodiment of the present invention, after obtaining the decoded target market data, the method further comprises:
when new market data of a new type exists, the field mapping module acquires field information of a repeated group of the new market data of the new type;
the field mapping module informs the state machine of the repeat group field information.
According to the preferred embodiment of the present invention, the stop bit decoding module, the field mapping module, the entry indexing module and the field post-processing module accelerate parallel execution based on the hardware platform.
A market data decoding device based on table index runs in a target system, the target system comprises a stop bit decoding module, a field mapping module, a table index module and a field post-processing module, the stop bit decoding module, the field mapping module, the table index module and the field post-processing module are built based on a hardware platform, and the device comprises:
the stop bit decoding module is used for responding to the received market data and identifying a field to be decoded in the market data;
the field mapping module is used for extracting a main field PMAP value and a repeated group field PMAP value in the field to be decoded by utilizing a state machine;
the field mapping module is configured to transfer the body field PMAP value and the duplicate group field PMAP value to the entry indexing module;
the table entry index module is used for taking the main field PMAP value and the repeat group field PMAP value as addresses and inquiring in a field ID index table to obtain a target ID;
the table entry index module is used for returning the target ID to the field mapping module;
the field mapping module is used for outputting the field to be decoded and the target ID to the field post-processing module;
and the field post-processing module is used for processing the field to be decoded according to the target ID to obtain decoded target market data.
A computer device, the computer device comprising:
a memory storing at least one instruction; and
and the processor executes the instructions stored in the memory to realize the market data decoding method based on the table entry index.
A computer-readable storage medium having stored therein at least one instruction for execution by a processor in a computer device to implement the listing index-based market data decoding method.
According to the technical scheme, the stop bit decoding module, the field mapping module, the table entry index module and the field post-processing module can be built based on the hardware platform, so that the functions of the modules are decoupled, and the modules are accelerated and executed in parallel based on the hardware platform, so that the decoding efficiency of market data can be effectively improved.
Drawings
FIG. 1 is a flowchart of a method for decoding market data based on table entry index according to a preferred embodiment of the present invention.
FIG. 2 is a functional block diagram of a device for decoding market data based on table entry index according to a preferred embodiment of the present invention.
FIG. 3 is a schematic structural diagram of a computer device according to a preferred embodiment of the present invention for implementing a method for decoding market data based on table entry indexes.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in detail with reference to the accompanying drawings and specific embodiments.
FIG. 1 is a flowchart illustrating a method for decoding market data based on table entry indexes according to a preferred embodiment of the present invention. The order of the steps in the flow chart may be changed and some steps may be omitted according to different needs.
The quotation data decoding method based on the table entry index is applied to one or more computer devices, wherein the computer devices are devices capable of automatically performing numerical calculation and/or information processing according to preset or stored instructions, and hardware of the computer devices comprises but is not limited to a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device and the like.
The computer device may be any electronic product capable of performing human-computer interaction with a user, for example, a Personal computer, a tablet computer, a smart phone, a Personal Digital Assistant (PDA), a game machine, an interactive web Television (IPTV), an intelligent wearable device, and the like.
The computer device may also include a network device and/or a user device. The network device includes, but is not limited to, a single network server, a server group consisting of a plurality of network servers, or a Cloud Computing (Cloud Computing) based Cloud consisting of a large number of hosts or network servers.
The server may be an independent server, or may be a cloud server that provides basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.
Among them, artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.
The artificial intelligence base technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.
The Network in which the computer device is located includes, but is not limited to, the internet, a wide area Network, a metropolitan area Network, a local area Network, a Virtual Private Network (VPN), and the like.
In this embodiment, the market data decoding method based on table entry index is applied to a target system, where the target system includes a stop bit decoding module, a field mapping module, a table entry index module, and a field post-processing module, which are built based on a hardware platform, and the method includes:
s10, responding to the received market data, and identifying a field to be decoded in the market data by the stop bit decoding module.
In this embodiment, the market data may include market data based on FAST (FIX Adapted for Streaming) market protocol.
In this embodiment, the hardware platform may include, but is not limited to: an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit) chip, and the like.
In this embodiment, the stop bit decoding module, the field mapping module, the entry indexing module, and the field post-processing module accelerate parallel execution based on the hardware platform.
Through parallel execution, decoding and accelerated processing of each functional module can be realized through a hardware platform, and parallel execution among the modules can further improve the data processing speed and enable the decoding of market data to be more efficient.
In this embodiment, the identifying, by the stop bit decoding module, the field to be decoded in the market data includes:
acquiring the highest bit of each byte in the market data;
and when the highest bit of the byte is 1, determining the byte as the field to be decoded.
Specifically, in the FAST protocol, the lower 7 bits of each byte are used for data transmission, and the highest bit is used to indicate the attribution of the next byte, specifically, 0 indicates whether the next byte belongs to the current domain, and 1 indicates that the next byte does not belong to the current domain.
Therefore, when the highest bit of a byte is 1, it indicates that the next byte does not belong to the current field, i.e. the transmission of the current field is finished, and the byte is determined as the field to be decoded.
S11, the field mapping module extracts a main field PMAP (Presence Map) value and a repeat group field PMAP value in the field to be decoded by utilizing a state machine.
It will be appreciated that some market types (e.g., bid market snapshot type data) have a circular repeating group, and the state machine is intended to distinguish between body fields (i.e., non-repeating fields) and repeating group fields, both of which have respective PMAP (Presence Map) values.
S12, the field mapping module transmits the body field PMAP value and the repeat group field PMAP value to the table entry indexing module.
The following of the body field with the duplicate group field is a common structure at present, and therefore, the embodiment extracts the body field PMAP value and the duplicate group field PMAP value, which can ensure the universality of the decoding scheme in this case.
S13, the table item index module takes the main field PMAP value and the repeat group field PMAP value as addresses and inquires in a field ID index table to obtain a target ID.
In this embodiment, the field ID index table is used to store the field ID of each type of market data in the message template, and the field ID index table is stored in the entry index module.
Specifically, the message template may be a message template file in an XML format, and stores each market field ID. The message template is determined prior to the two-way communication, and only the actual content of the message actually needs to be transferred.
That is, the field ID index table is pre-written with the field mapping IDs corresponding to all PMAP values, and all the field IDs (or called field tags) corresponding to the PMAP values can be found by using the input PMAP values as addresses during decoding.
In this embodiment, the table entry indexing module uses the main field PMAP value and the repeat group field PMAP value as addresses, and performs query in a field ID index table to obtain the target ID, including:
acquiring all fields with the value of 1 in the main field PMAP value and the repeat group field PMAP value;
acquiring a domain name corresponding to each field in all the fields from the field ID index table;
and determining the field ID corresponding to each domain name as the target ID.
Specifically, in the FAST protocol, a flag string (BYTE array) is added to the beginning of each message, and each bit indicates whether a template-specific field exists in the following actual data. Wherein, 0 indicates that there is no data of the domain corresponding to the template in the following data, and 1 indicates that there is data of the domain corresponding to the template in the following data. After the marking, the coding and decoding can be omitted by the domain which is not transmitted, thereby reducing the data volume.
In the FAST protocol, the domain name may include, but is not limited to: template ID, setNo, recNo, etc.
S14, the table entry indexing module returns the target ID to the field mapping module.
In this embodiment, the table entry indexing module returns the target ID obtained after the query to the field mapping module, so that the field mapping module transmits the target ID to a subsequent stage.
And S15, the field mapping module outputs the field to be decoded and the target ID to the field post-processing module.
The field to be decoded is the actual content of the transmission message, the inquired target ID and the field to be decoded are transmitted to the field post-processing module as the lower level together, and the field post-processing module can perform corresponding processing on the field to be decoded according to the target ID.
And S16, the field post-processing module processes the field to be decoded according to the target ID to obtain decoded target market data.
For example: after the field 0x3945a4 to be decoded is decoded, the corresponding target market data 942755 can be obtained.
It can be understood that, by performing corresponding processing on the field to be decoded, the field to be decoded can be made to better meet the requirement of the receiving party on the data attribute.
In this embodiment, the processing the field to be decoded by the field post-processing module according to the target ID, and obtaining the decoded target market data includes:
acquiring a domain corresponding to the target ID as a target domain;
determining data attributes according to the target domain;
and carrying out scaling and type conversion on the field to be decoded according to the data attribute to obtain the target market data.
Wherein the data attribute may characterize a unit, a type, etc. of the data.
The magnification and the type conversion are for unifying the units, such as unifying the time related fields into millisecond units, unifying the price related fields into sub-units, and the like.
In this embodiment, the processing the field to be decoded by the field post-processing module according to the target ID to obtain the decoded target market data further includes:
determining a recipient of the target market data;
determining the transmission bandwidth and the operational capability corresponding to the receiver;
when the transmission bandwidth and the operational capacity corresponding to the receiver reach a completion condition, acquiring a domain operational character corresponding to the target ID;
determining a completion strategy according to the domain operator corresponding to the target ID;
and completing the default field in the field to be decoded according to the completion strategy to obtain the target market data.
Wherein, the completion condition can be configured according to actual requirements.
Wherein the completion policy may include performing copy, tail, difference, etc. operations to recover fields according to domain operators.
Among them, in the FAST protocol, domain operators are defined in a data domain to further reduce the actual transmission amount of data. Domain operators as commonly used may include: constant, default, copy, increment.
constant: used for representing fixed value data in the template;
defaults: when no data value is provided, to represent a default value in the template;
copy: used to indicate the case that adjacent data are the same;
increment: the data is used for representing the condition that the data increases regularly before and after;
delta: which is used to indicate the case where there is a small difference between the previous and next data.
Thus, the FAST protocol provides message-level optimization through implicit tags provided by XML-formatted templates, and message-level and tag-level optimization through operators defined for domains within the templates.
For example, when the domain operator corresponding to the target ID is increment and indicates that the previous and subsequent data are regularly incremented (incremented by step 1), the previous field of the default field in the field to be decoded is 9, and the default field may be complemented by 10.
After data completion, the requirement on transmission bandwidth is higher, and the requirement on the processing capacity of a software end is lower.
According to the embodiment, mutual decoupling of the functional modules is realized based on the hardware platform, the universality of the functional modules is improved, a decoding functional module is not required to be configured independently for each market data, and the resource utilization rate is improved.
In this embodiment, after obtaining the decoded target market data, the method further includes:
when new market data of a new type exists, the field mapping module acquires field information of a repeated group of the new market data of the new type;
the field mapping module informs the state machine of the repeat group field information.
Wherein the repeat group field information includes, but is not limited to: whether the new market quotation data has the repeat group, the type of the repeat group, and the like.
In the above embodiment, when new market data of a new type exists, only the state machine needs to be notified of the field information of the repeating group, so as to avoid omission in field ID query, and achieve the purpose of quickly supporting decoding of the new market data by simply changing configuration.
According to the technical scheme, the stop bit decoding module, the field mapping module, the table entry index module and the field post-processing module can be built based on the hardware platform, so that the functions of the modules are decoupled, and the modules are accelerated and executed in parallel based on the hardware platform, so that the decoding efficiency of market data can be effectively improved.
Fig. 2 is a functional block diagram of a device for decoding market data based on table entry index according to a preferred embodiment of the present invention. The apparatus 11 for decoding market data based on entry index comprises a stop bit decoding module 110, a field mapping module 111, an entry index module 112, and a field post-processing module 113. A module/unit as referred to herein is a series of computer program segments stored in a memory that can be executed by a processor and that can perform a fixed function. In the present embodiment, the functions of the modules/units will be described in detail in the following embodiments.
In this embodiment, the market data decoding device 11 based on entry index operates in a target system, where the target system includes the stop bit decoding module 110, the field mapping module 111, the entry index module 112, and the field post-processing module 113 that are built based on a hardware platform, and the device includes:
the stop bit decoding module 110 is configured to identify a field to be decoded in the market data in response to the received market data.
In this embodiment, the market data may include market data based on FAST attached for Streaming (FAST attached for Streaming) market protocol.
In this embodiment, the hardware platform may include, but is not limited to: an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit) chip, and the like.
In this embodiment, the stop bit decoding module 110, the field mapping module 111, the entry index module 112, and the field post-processing module 113 accelerate parallel execution based on the hardware platform.
Through parallel execution, decoding and accelerated processing of each functional module can be realized through a hardware platform, and parallel execution among the modules can further improve the data processing speed and enable the decoding of market data to be more efficient.
In this embodiment, the stop bit decoding module 110 identifying the field to be decoded in the market data includes:
acquiring the highest bit of each byte in the market data;
and when the highest bit of the byte is 1, determining the byte as the field to be decoded.
Specifically, in the FAST protocol, the lower 7 bits of each byte are used for data transmission, and the highest bit is used to indicate the attribution of the next byte, specifically, 0 indicates whether the next byte belongs to the current domain, and 1 indicates that the next byte does not belong to the current domain.
Therefore, when the highest bit of a byte is 1, it indicates that the next byte does not belong to the current field, i.e. the transmission of the current field is finished, and the byte is determined as the field to be decoded.
The field mapping module 111 is configured to extract a body field PMAP (Presence Map) value and a duplicate group field PMAP value in the field to be decoded by using a state machine.
It will be appreciated that some market types (e.g., bid market snapshot type data) have a circular repeating group, and the state machine is intended to distinguish between body fields (i.e., non-repeating fields) and repeating group fields, both of which have respective PMAP (Presence Map) values.
The field mapping module 111 is configured to transfer the body field PMAP value and the duplicate group field PMAP value to the entry index module 112.
The following of the body field with the duplicate group field is a common structure at present, and therefore, the embodiment extracts the body field PMAP value and the duplicate group field PMAP value, which can ensure the universality of the decoding scheme in this case.
The table entry indexing module 112 is configured to use the main field PMAP value and the repeat group field PMAP value as addresses, and perform query in a field ID index table to obtain a target ID.
In this embodiment, the field ID index table is used to store the field ID of each type of market data in the message template, and the field ID index table is stored in the entry index module.
Specifically, the message template may be a message template file in an XML format, and stores each market field ID. The message template is determined prior to two-way communication, and only the actual content of the message actually needs to be transmitted.
That is, the field ID index table is pre-written with the field mapping IDs corresponding to all PMAP values, and all the field IDs (or called field tags) corresponding to the PMAP values can be found by using the input PMAP values as addresses during decoding.
In this embodiment, the table entry indexing module 112 uses the main field PMAP value and the repeat group field PMAP value as addresses, and performs query in a field ID index table to obtain the target ID, which includes:
acquiring all fields with the value of 1 in the main field PMAP value and the repeat group field PMAP value;
acquiring a domain name corresponding to each field in all the fields from the field ID index table;
and determining the field ID corresponding to each domain name as the target ID.
Specifically, in the FAST protocol, a flag string (BYTE array) is added to the beginning of each message, and each bit indicates whether a template-specific field exists in the following actual data. Wherein, 0 indicates that there is no data of the domain corresponding to the template in the following data, and 1 indicates that there is data of the domain corresponding to the template in the following data. After the marking, the coding and decoding can be omitted by the domain which is not transmitted, thereby reducing the data volume.
In the FAST protocol, the domain name may include, but is not limited to: templateID, setNo, recNo, and the like.
The entry indexing module 112 is further configured to return the target ID to the field mapping module 111.
In this embodiment, the entry indexing module 112 returns the target ID obtained after the query to the field mapping module 111, so that the field mapping module 111 transmits the target ID to a subsequent stage.
The field mapping module 111 is further configured to output the field to be decoded and the target ID to the field post-processing module 113.
The field to be decoded is the actual content of the transmission message, the queried target ID and the field to be decoded are transmitted to the field post-processing module 113 as the lower stage together, so that the field post-processing module 113 can perform corresponding processing on the field to be decoded according to the target ID.
The field post-processing module 113 is configured to process the field to be decoded according to the target ID, so as to obtain decoded target market data.
For example: after the field 0x3945a4 to be decoded is decoded, the corresponding target market data 942755 can be obtained.
It can be understood that, by performing corresponding processing on the field to be decoded, the field to be decoded can be made to better meet the requirement of the receiving party on the data attribute.
In this embodiment, the field post-processing module 113 processes the field to be decoded according to the target ID, and obtaining the decoded target market data includes:
acquiring a domain corresponding to the target ID as a target domain;
determining data attributes according to the target domain;
and carrying out scaling and type conversion on the field to be decoded according to the data attribute to obtain the target market data.
Wherein the data attribute may characterize a unit, a type, etc. of the data.
The magnification and the type conversion are for unifying the units, such as unifying the time related fields into millisecond units, unifying the price related fields into sub-units, and the like.
In this embodiment, the processing the field to be decoded by the field post-processing module 113 according to the target ID to obtain the decoded target market data further includes:
determining a recipient of the target market data;
determining the transmission bandwidth and the operational capability corresponding to the receiver;
when the transmission bandwidth and the operational capacity corresponding to the receiver reach a completion condition, acquiring a domain operational character corresponding to the target ID;
determining a completion strategy according to the domain operator corresponding to the target ID;
and completing the default field in the field to be decoded according to the completion strategy to obtain the target market data.
Wherein, the completion condition can be configured according to actual requirements.
Wherein the completion policy may include performing copy, tail, difference, etc. operations to recover the field according to the domain operator.
Among them, in the FAST protocol, domain operators are defined in a data domain to further reduce the actual transmission amount of data. Domain operators as commonly used may include: constant, default, copy, increment.
constant: used for representing fixed value data in the template;
defaults: when no data value is provided, to represent a default value in the template;
copy: used to indicate the case that the adjacent data are the same;
increment: the condition that the data increase regularly before and after is shown;
delta: which is used to indicate the case where there is a small difference between the previous and next data.
Thus, the FAST protocol provides message-level optimization through implicit tags provided by XML-formatted templates, and message-level and tag-level optimization through operators defined for domains within the templates.
For example, when the domain operator corresponding to the target ID is increment and indicates that the previous and subsequent data are regularly incremented (incremented by step 1), the previous field of the default field in the field to be decoded is 9, and the default field may be complemented by 10.
After data completion, the requirement on transmission bandwidth is higher, and the requirement on the processing capacity of a software end is lower.
According to the embodiment, mutual decoupling of each functional module is realized based on a hardware platform, the universality of each functional module is improved, a decoding functional module is not required to be configured for each market data independently, and the resource utilization rate is improved.
In this embodiment, after the decoded target market data is obtained, when there is new market data of a new type, the field mapping module 111 obtains field information of a repeating group of the new market data of the new type;
the field mapping module 111 informs the state machine of the repeat group field information.
Wherein the repeat group field information includes, but is not limited to: whether the new market quotation data has the repeat group, the type of the repeat group, and the like.
In the above embodiment, when there is new market data of a new type, only the state machine needs to be notified of the field information of the repeating group, so as to avoid omission in field ID query, and achieve the purpose of quickly supporting decoding of the new market data by simply changing configuration.
According to the technical scheme, the stop bit decoding module, the field mapping module, the table entry index module and the field post-processing module can be built based on the hardware platform, so that the functions of the modules are decoupled, and the modules are accelerated and executed in parallel based on the hardware platform, so that the decoding efficiency of market data can be effectively improved.
Fig. 3 is a schematic structural diagram of a computer device for implementing the method for decoding market data based on entry indexes according to the preferred embodiment of the present invention.
The computer device 1 may comprise a memory 12, a processor 13 and a bus, and may further comprise a computer program stored in the memory 12 and executable on the processor 13, such as a market data decoding program based on an entry index.
It will be understood by those skilled in the art that the schematic diagram is merely an example of the computer device 1, and does not constitute a limitation to the computer device 1, the computer device 1 may have a bus-type structure or a star-shaped structure, the computer device 1 may further include more or less other hardware or software than those shown, or different component arrangements, for example, the computer device 1 may further include an input and output device, a network access device, etc.
It should be noted that the computer device 1 is only an example, and other electronic products that are currently available or may come into existence in the future, such as electronic products that can be adapted to the present invention, should also be included in the scope of the present invention, and are included herein by reference.
The memory 12 includes at least one type of readable storage medium, which includes flash memory, removable hard disks, multimedia cards, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disks, optical disks, etc. The memory 12 may in some embodiments be an internal storage unit of the computer device 1, for example a removable hard disk of the computer device 1. The memory 12 may also be an external storage device of the computer device 1 in other embodiments, such as a plug-in removable hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), etc. provided on the computer device 1. Further, the memory 12 may also include both an internal storage unit and an external storage device of the computer device 1. The memory 12 can be used not only for storing application software installed in the computer apparatus 1 and various types of data such as codes of a market data decoding program based on an entry index, etc., but also for temporarily storing data that has been output or is to be output.
The processor 13 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 13 is a Control Unit (Control Unit) of the computer device 1, connects various components of the entire computer device 1 by using various interfaces and lines, and executes various functions and processes data of the computer device 1 by running or executing programs or modules stored in the memory 12 (for example, executing a market data decoding program based on table entry indexes, etc.), and calling data stored in the memory 12.
The processor 13 executes the operating system of the computer device 1 and various installed application programs. The processor 13 executes the application program to implement the steps in each embodiment of the listing index-based market data decoding method, such as the steps shown in fig. 1.
Illustratively, the computer program may be divided into one or more modules/units, which are stored in the memory 12 and executed by the processor 13 to accomplish the present invention. The one or more modules/units may be a series of computer readable instruction segments capable of performing certain functions, which are used to describe the execution of the computer program in the computer device 1. For example, the computer program may be partitioned into a stop bit decoding module 110, a field mapping module 111, an entry indexing module 112, and a field post-processing module 113.
The integrated unit implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a computer device, or a network device) or a processor (processor) to execute the parts of the listing index-based market data decoding method according to the embodiments of the present invention.
The integrated modules/units of the computer device 1 may be stored in a computer-readable storage medium if they are implemented in the form of software functional units and sold or used as separate products. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments may be implemented.
Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U.S. disk, removable hard disk, magnetic diskette, optical disk, computer Memory, read-Only Memory (ROM), random access Memory, etc.
Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
The bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one line is shown in FIG. 3, but that does not indicate only one bus or type of bus. The bus is arranged to enable connection communication between the memory 12 and at least one processor 13 or the like.
Although not shown, the computer device 1 may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 13 through a power management device, so that functions of charge management, discharge management, power consumption management and the like are realized through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The computer device 1 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Further, the computer device 1 may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used for establishing a communication connection between the computer device 1 and other computer devices.
Optionally, the computer device 1 may further comprise a user interface, which may be a Display (Display), an input unit, such as a Keyboard (Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the computer device 1 and for displaying a visualized user interface.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
Fig. 3 shows only the computer device 1 with the components 12-13, and it will be understood by a person skilled in the art that the structure shown in fig. 3 does not constitute a limitation of the computer device 1, and may comprise fewer or more components than shown, or a combination of certain components, or a different arrangement of components.
With reference to fig. 1, the memory 12 in the computer device 1 stores a plurality of instructions to implement a market data decoding method based on table entry index, and is applied to a target system, where the target system includes a stop bit decoding module, a field mapping module, a table entry index module, and a field post-processing module, and the processor 13 can execute the plurality of instructions to implement:
in response to the received market data, the stop bit decoding module identifies a field to be decoded in the market data;
the field mapping module extracts a main field PMAP value and a repeat group field PMAP value in the field to be decoded by using a state machine;
the field mapping module passing the body field PMAP value and the duplicate set field PMAP value to the table entry indexing module;
the table entry indexing module takes the main field PMAP value and the repeat group field PMAP value as addresses, and inquires in a field ID indexing table to obtain a target ID;
the table entry indexing module returns the target ID to the field mapping module;
the field mapping module outputs the field to be decoded and the target ID to the field post-processing module;
and the field post-processing module processes the field to be decoded according to the target ID to obtain decoded target market data.
Specifically, the processor 13 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1 for a specific implementation method of the instruction, which is not described herein again.
It should be noted that all the data involved in the present application are legally acquired.
In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The invention is operational with numerous general purpose or special purpose computing system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
Furthermore, it will be obvious that the term "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the present invention may also be implemented by one unit or means through software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only intended to illustrate the technical solutions of the present invention and not to limit the same, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A market data decoding method based on table index is characterized in that the method is applied to a target system, the target system comprises a stop bit decoding module, a field mapping module, a table index module and a field post-processing module which are built based on a hardware platform, and the method comprises the following steps:
in response to the received market data, the stop bit decoding module identifies a field to be decoded in the market data;
the field mapping module extracts a body field PMAP value and a repeat group field PMAP value in the field to be decoded by using a state machine;
the field mapping module passing the body field PMAP value and the duplicate group field PMAP value to the table entry indexing module;
the table entry index module takes the main field PMAP value and the repeat group field PMAP value as addresses and inquires in a field ID index table to obtain a target ID;
the table entry indexing module returns the target ID to the field mapping module;
the field mapping module outputs the field to be decoded and the target ID to the field post-processing module;
and the field post-processing module processes the field to be decoded according to the target ID to obtain decoded target market data.
2. The method for decoding market data based on entry index of claim 1, wherein the stop bit decoding module identifying the field to be decoded in the market data comprises:
acquiring the highest bit of each byte in the market data;
and when the highest bit of the byte is 1, determining the byte as the field to be decoded.
3. The method for decoding market data based on entry index of claim 1, wherein the field ID index table is used to store the field ID of each type of market data in the message template, and the field ID index table is stored in the entry index module;
the table entry indexing module takes the main field PMAP value and the repeat group field PMAP value as addresses, and queries in a field ID index table to obtain a target ID, and the table entry indexing module comprises:
acquiring all fields with the value of 1 in the main field PMAP value and the repeat group field PMAP value;
acquiring a domain name corresponding to each field in all the fields from the field ID index table;
and determining the field ID corresponding to each domain name as the target ID.
4. The listing index-based market data decoding method of claim 1, wherein the field post-processing module processes the field to be decoded according to the target ID, and obtaining the decoded target market data comprises:
acquiring a domain corresponding to the target ID as a target domain;
determining data attributes according to the target domain;
and carrying out scaling and type conversion on the field to be decoded according to the data attribute to obtain the target market data.
5. The listing index-based market data decoding method of claim 1, wherein the field post-processing module processes the field to be decoded according to the target ID, and obtaining the decoded target market data further comprises:
determining a recipient of the target market data;
determining the transmission bandwidth and the operational capability corresponding to the receiver;
when the transmission bandwidth and the operational capacity corresponding to the receiver reach a completion condition, acquiring a domain operational character corresponding to the target ID;
determining a completion strategy according to the domain operator corresponding to the target ID;
and completing the default field in the field to be decoded according to the completion strategy to obtain the target market data.
6. The method for decoding market data based on entry index according to claim 1, wherein after obtaining the decoded target market data, the method further comprises:
when new market data of a new type exists, the field mapping module acquires field information of a repeated group of the new market data of the new type;
the field mapping module informs the state machine of the repeat group field information.
7. The method of claim 1, wherein the stop-bit decoding module, the field mapping module, the entry indexing module, and the field post-processing module are executed in parallel based on the hardware platform acceleration.
8. The market data decoding device based on the table item index is characterized by running in a target system, wherein the target system comprises a stop bit decoding module, a field mapping module, a table item index module and a field post-processing module which are built based on a hardware platform, and the device comprises:
the stop bit decoding module is used for responding to the received market data and identifying a field to be decoded in the market data;
the field mapping module is used for extracting a main field PMAP value and a repeat group field PMAP value in the field to be decoded by utilizing a state machine;
the field mapping module is configured to transfer the body field PMAP value and the duplicate group field PMAP value to the entry indexing module;
the table entry indexing module is used for taking the main field PMAP value and the repeat group field PMAP value as addresses and inquiring in a field ID (identity) index table to obtain a target ID;
the table entry indexing module is used for returning the target ID to the field mapping module;
the field mapping module is used for outputting the field to be decoded and the target ID to the field post-processing module;
and the field post-processing module is used for processing the field to be decoded according to the target ID to obtain decoded target market data.
9. A computer device, characterized in that the computer device comprises:
a memory storing at least one instruction; and
a processor executing the instructions stored in the memory to implement the listing index-based market data decoding method according to any of claims 1-6.
10. A computer-readable storage medium characterized by: the computer-readable storage medium stores at least one instruction therein, the at least one instruction being executed by a processor in a computer device to implement the listing index-based market data decoding method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211091771.7A CN115174698B (en) | 2022-09-07 | 2022-09-07 | Market data decoding method, device, equipment and medium based on table entry index |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211091771.7A CN115174698B (en) | 2022-09-07 | 2022-09-07 | Market data decoding method, device, equipment and medium based on table entry index |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115174698A true CN115174698A (en) | 2022-10-11 |
| CN115174698B CN115174698B (en) | 2022-12-13 |
Family
ID=83480439
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211091771.7A Active CN115174698B (en) | 2022-09-07 | 2022-09-07 | Market data decoding method, device, equipment and medium based on table entry index |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115174698B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111967244A (en) * | 2020-07-30 | 2020-11-20 | 浪潮(北京)电子信息产业有限公司 | FAST protocol decoding method, device and equipment based on FPGA |
| CN112330456A (en) * | 2020-11-27 | 2021-02-05 | 上海特高信息技术有限公司 | Ultra-low-delay hardware accelerated market data stream analysis system |
| CN112346843A (en) * | 2020-11-26 | 2021-02-09 | 上海金融期货信息技术有限公司 | Analysis method of low-delay FAST protocol |
| CN112650499A (en) * | 2020-12-29 | 2021-04-13 | 国泰君安证券股份有限公司 | System for realizing hardware decoding processing of exchange level-2FAST market based on OpenCL platform |
| CN114238334A (en) * | 2021-12-20 | 2022-03-25 | 武汉联影医疗科技有限公司 | Heterogeneous data encoding method and device, heterogeneous data decoding method and device, computer equipment and storage medium |
-
2022
- 2022-09-07 CN CN202211091771.7A patent/CN115174698B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111967244A (en) * | 2020-07-30 | 2020-11-20 | 浪潮(北京)电子信息产业有限公司 | FAST protocol decoding method, device and equipment based on FPGA |
| CN112346843A (en) * | 2020-11-26 | 2021-02-09 | 上海金融期货信息技术有限公司 | Analysis method of low-delay FAST protocol |
| CN112330456A (en) * | 2020-11-27 | 2021-02-05 | 上海特高信息技术有限公司 | Ultra-low-delay hardware accelerated market data stream analysis system |
| CN112650499A (en) * | 2020-12-29 | 2021-04-13 | 国泰君安证券股份有限公司 | System for realizing hardware decoding processing of exchange level-2FAST market based on OpenCL platform |
| CN114238334A (en) * | 2021-12-20 | 2022-03-25 | 武汉联影医疗科技有限公司 | Heterogeneous data encoding method and device, heterogeneous data decoding method and device, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 李函: "基于FPGA的FAST协议解码金融加速设计", 《中国优秀硕士学位论文全文数据库（信息科技辑）》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115174698B (en) | 2022-12-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2023056943A1 (en) | Internet of things rule engine-based terminal control method and apparatus, and device and medium | |
| CN111428458A (en) | Universal report generation method and device and computer readable storage medium | |
| CN114979120B (en) | Data uploading method, device, equipment and storage medium | |
| CN113890712A (en) | Data transmission method and device, electronic equipment and readable storage medium | |
| CN113806434A (en) | Big data processing method, device, equipment and medium | |
| CN112949278A (en) | Data checking method and device, electronic equipment and readable storage medium | |
| CN114491047A (en) | Multi-label text classification method and device, electronic equipment and storage medium | |
| CN114386509A (en) | Data fusion method and device, electronic equipment and storage medium | |
| CN114880368A (en) | Data query method and device, electronic equipment and readable storage medium | |
| CN113722533B (en) | Information pushing method and device, electronic equipment and readable storage medium | |
| CN113868528A (en) | Information recommendation method and device, electronic equipment and readable storage medium | |
| CN114491646A (en) | Data desensitization method and device, electronic equipment and storage medium | |
| CN113468175A (en) | Data compression method and device, electronic equipment and storage medium | |
| CN115314570B (en) | Data issuing method, device, equipment and medium based on protocol development framework | |
| CN111985194A (en) | Data storage method and device, electronic equipment and storage medium | |
| CN115174698B (en) | Market data decoding method, device, equipment and medium based on table entry index | |
| CN116450723A (en) | Data extraction method, device, computer equipment and storage medium | |
| CN114547011A (en) | Data extraction method and device, electronic equipment and storage medium | |
| CN114840388A (en) | Data monitoring method and device, electronic equipment and storage medium | |
| CN113918305A (en) | Node scheduling method and device, electronic equipment and readable storage medium | |
| CN112328960B (en) | Optimization method and device for data operation, electronic equipment and storage medium | |
| CN115174555A (en) | File transmission method and device, electronic equipment and storage medium | |
| CN114860349B (en) | Data loading method, device, equipment and medium | |
| CN113704616B (en) | Information pushing method and device, electronic equipment and readable storage medium | |
| CN113452785B (en) | Service access method and device based on offline resources, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |