CN110415027B - Big data quotation platform system - Google Patents
Big data quotation platform system Download PDFInfo
- Publication number
- CN110415027B CN110415027B CN201910640372.3A CN201910640372A CN110415027B CN 110415027 B CN110415027 B CN 110415027B CN 201910640372 A CN201910640372 A CN 201910640372A CN 110415027 B CN110415027 B CN 110415027B
- Authority
- CN
- China
- Prior art keywords
- data
- module
- unit
- user
- processing module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
- G06F11/2023—Failover techniques
- G06F11/2028—Failover techniques eliminating a faulty processor or activating a spare
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0201—Market modelling; Market analysis; Collecting market data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/06—Asset management; Financial planning or analysis
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Development Economics (AREA)
- Finance (AREA)
- Databases & Information Systems (AREA)
- Accounting & Taxation (AREA)
- General Engineering & Computer Science (AREA)
- Strategic Management (AREA)
- Data Mining & Analysis (AREA)
- Entrepreneurship & Innovation (AREA)
- Game Theory and Decision Science (AREA)
- General Business, Economics & Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- Software Systems (AREA)
- Technology Law (AREA)
- Operations Research (AREA)
- Computational Linguistics (AREA)
- Human Resources & Organizations (AREA)
- Quality & Reliability (AREA)
- Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a big data quotation platform system, which simplifies quotation receiving of a trading exchange, reduces docking cost and quotation delay, and enables investors to analyze market conditions more comprehensively. The technical proposal is as follows: the difficulty of accessing multiple quotations by a user is reduced through the design of the user API interface. Simple binary coded SBE and internal high performance UDP communication frameworks optimized for low latency incur lower system latency. The invention also provides diversified data such as K lines, time sharing, indexes, other public market data, public opinion data, other analysis data and the like, helps investors to know capital markets more thoroughly, and provides more reference basis for transactions of vast investors. Thanks to the multiple nodes, users can be connected to different prepositions according to load balancing through the trans-forming technology, and the prepositions can acquire data from the different nodes according to the load balancing, so that investors can acquire extremely fast experience.
Description
Technical Field
The invention relates to the field of big data, in particular to a quotation platform system with big data processing capability, which can be applied to a financial securities futures scene.
Background
At the central economic conference held at the end of 2018, the general book clearly suggests that a canonical, transparent, open, viable, flexible capital market is to be created. Over the years, as the capital market has been comprehensively reformed and the open step has advanced, more and more investors have participated in it. In the field of financial transactions, market data plays a vital role in various policy drivers, and market speed determines transaction speed.
However, at present, the market clients acquire the market quotation of each exchange, and have technical thresholds and admission thresholds, and meanwhile, the maintenance cost of the market clients is higher, the reliability is lower, and the delay of the existing market quotation platform is larger. The 2005 international financial information exchange organization (FIX Protocol Limited abbreviated FPL) issued FAST (FIX Adapted for STreaming) encoding methods supporting high throughput data communication between financial institutions. The exchanges using FAST codes mainly include new york stock exchanges, chicago commodity exchanges, shanghai stock exchanges, etc. Most of the market today is based on FAST coding, whose main objective is to highly compress the data, thereby reducing bandwidth usage and thus also causing delay increases.
In recent years, large data, AI (Artificial Intelligence artificial intelligence) and intelligent financial technology applications are developed at a high speed, and the existing system cannot support a historical mass data query function, a large-scale batch computing service and a real-time computing service, so that the requirements of users for analyzing other public market data, public opinion data and the like cannot be met.
Disclosure of Invention
The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.
The invention aims to solve the problems, and provides a big data quotation platform system which solves the problems of quotation acquisition and data enhancement of users in the transaction process, can simplify quotation reception of a plurality of exchanges, reduces the butting cost and greatly reduces the delay of quotation acquisition. Meanwhile, indexes, other public market data, public opinion data and other analysis data can be obtained, so that investors can more comprehensively analyze market conditions, the reliability of the distributed architecture is improved, and the maintenance cost of users is reduced.
The technical scheme of the invention is as follows: the invention discloses a big data quotation platform system, which comprises a core service module, a database module, a cache module, a message bus, a service processing module, a pre-processing module and a user API interface module, wherein:
the core service module comprises an original data receiving and analyzing unit, an SBE encoding unit and a topic distribution unit, wherein the original data receiving and analyzing unit is used for acquiring original quotation data from transactions and analyzing the data, the SBE encoding unit is used for performing SBE encoding on the original quotation data received and analyzed by the original data receiving unit by adopting a simple binary encoding SBE, and the topic distribution unit is used for distributing the quotation data after the SBE encoding into different topics to a message bus;
the cache module is used for acquiring the SBE coded market data from the message bus or acquiring the processed data from the service processing module, storing the data in a self memory and/or database module, receiving a request for reading the data from the service processing module or the pre-processing module, acquiring the data from the self memory and returning the data, and inquiring the database module to return the inquired data if the data is not in the memory;
the user API interface module is used for accessing the system by a user, and the user returns the data processed by the pre-processing module in response to the request to the user through a request including user login, data subscription and data query initiated by the API interface;
the pre-processing module is used for pushing real-time data, inquiring current day data and historical data, receiving a request sent by a user through the user API interface module and responding to the request;
the business processing module is used for acquiring the market data from the message bus, correspondingly processing the acquired storage data from the cache module, sending the processing result to the cache module for storage, receiving a request comprising user authentication and query history data from the pre-processing module, acquiring the data from the cache module and returning the data to the pre-processing module.
According to one embodiment of the big data quotation platform system of the invention, the core service module performs unidirectional data transmission with the message bus, the message bus performs unidirectional data transmission with the pre-processing module, the cache module and the service processing module respectively, the cache module performs bidirectional data transmission with the database module and the service processing module respectively, and the cache module and the pre-processing module perform unidirectional data transmission. And the pre-processing module respectively carries out bidirectional data transmission with the service processing module and the user API interface module.
According to an embodiment of the big data quotation platform system of the present invention, the system further comprises:
and the trans module and the pre-processing module perform bidirectional data transmission and provide load balancing service for the pre-processing module.
According to an embodiment of the big data quotation platform system of the invention, the load balancing service configured in the fets module is to obtain the current load condition of each pre-processing module by communicating with each pre-processing module, select the pre-address with the smallest load in the pre-processing module, and send the pre-address back to the user API interface module for data communication.
According to one embodiment of the big data quotation platform system of the invention, the data after SBE encoding in the system is transmitted by adopting UDP protocol.
According to an embodiment of the big data quotation platform system of the invention, the pre-processing module comprises a general quotation pre-unit, a query data pre-unit and a history data pre-unit, wherein:
the general quotation front-end unit receives the subscription request of the user API interface module, establishes long connection with the user API interface module after the data authentication of the user authentication unit, acquires data from the message bus and the cache module, and pushes different data to the corresponding user API interface module according to the topic subscribed by the user;
the inquiry data preposition unit is used for receiving a request of inquiring the appointed data of the current day of the user API interface module, acquiring the data from the cache module after the authentication of the user authentication unit and returning the data to the user API interface module;
the historical data preposition unit is used for receiving a request of inquiring historical specified data of the user API interface module, acquiring data from the cache module after authentication of the user authentication unit and returning the data to the user API interface module, and if the acquired historical data is before a certain time, directly returning a link downloaded by a file for a client to acquire the data.
According to an embodiment of the big data quotation platform system of the present invention, the business processing module comprises, but is not limited to, a user authentication unit, a history data unit, a K-line quotation unit, an index calculation unit, a data analysis unit, and a return detection unit, wherein:
a user authentication unit for authenticating authority of user access data;
the K line quotation unit is used for calculating and processing the K line and the time sharing;
the data analysis unit is used for analyzing data such as quotation;
the historical quotation unit is used for storing historical quotations;
the index calculation unit is used for calculating various indexes of the quotation;
and the return unit is used for calculating return data including the profitability and the maximum withdrawal rate under certain conditions.
According to one embodiment of the big data quotation platform system, the system is of a distributed architecture, is deployed on a plurality of nodes and realizes automatic fault switching through real-time hot standby.
Compared with the prior art, the invention has the following beneficial effects: the invention greatly reduces the difficulty of accessing multiple quotations by the user through the simple and easy-to-use API. At the same time, the simple binary SBE (Simple Binary Encoding) and internal high performance UDP (User Datagram Protocol ) communication framework, which benefits from optimizing for low latency, introduces lower system latency. Because the standardized financial data exchange protocol SBE issued by the FPL is adopted, when a user selects SBE type data as return, the number of times of copying the data is reduced, so that the data acquisition speed of the whole system is higher. Besides, the invention provides diversified data such as K line, time sharing, index, other public market data, public opinion data and other analysis data besides basic market data, helps investors to know capital market more thoroughly, and provides more reference basis for transactions of vast investors. The invention is a distributed architecture, can flexibly deploy a plurality of nodes, realizes automatic fault switching by hot standby in real time, ensures that the market is not interrupted, improves the reliability of the whole system, and reduces the maintenance cost of users. Thanks to the multiple nodes, users can be connected to different prepositions according to load balancing through the trans (prepositive gateway for the technology responsible for access management and load balancing), and the prepositions can acquire data from different service nodes according to the load balancing, so that investors can acquire extremely fast experience.
Drawings
The above features and advantages of the present invention will be better understood after reading the detailed description of embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily to scale and components having similar related features or characteristics may have the same or similar reference numerals.
FIG. 1 illustrates a system architecture diagram of one embodiment of the big data ticker platform system of this invention.
FIG. 2 illustrates a data subscription flow diagram configured by an embodiment of the big data ticker platform system of the present invention.
FIG. 3 illustrates a data query flow diagram configured by an embodiment of the big data ticker platform system of this invention.
Detailed Description
The invention is described in detail below with reference to the drawings and the specific embodiments. It is noted that the aspects described below in connection with the drawings and the specific embodiments are merely exemplary and should not be construed as limiting the scope of the invention in any way.
FIG. 1 illustrates a system architecture of one embodiment of the big data ticker platform system of this invention. Referring to fig. 1, the system of the present embodiment includes: the system comprises a core service module, a database module, a cache module, a message bus, a business processing module, a pre-processing module, a user API (application program interface) module and a trans module.
The invention is a distributed architecture, can flexibly deploy a plurality of nodes, and realizes automatic fault switching by the hot standby in real time. The core service module performs unidirectional data transmission with the message bus, the message bus performs unidirectional data transmission with the pre-processing module, the cache module and the service processing module respectively, the cache module performs bidirectional data transmission with the database module and the service processing module respectively, and the cache module and the pre-processing module perform unidirectional data transmission. And the pre-processing module respectively carries out bidirectional data transmission with the service processing module, the user API interface module and the trans module.
The core service module further comprises: the system comprises an original data receiving and analyzing unit, an SBE encoding unit and a topic distribution unit.
The original data receiving and analyzing unit acquires the quotation seats of the exchanges, and receives and analyzes the original quotation data from each exchange by using an API (application programming interface) or a network data packet communication protocol provided by each exchange. Individual exchanges include, but are not limited to: shanghai stock exchange, shenzhen stock exchange, china financial futures exchange, shanghai international energy exchange, dalian commodity exchange, zhengzhou commodity exchange and the like.
The SBE encoding unit is used for performing SBE encoding on the original market data received and analyzed by the original data receiving unit by adopting a simple binary encoding SBE.
The topic distribution unit is used for distributing the market data after the SBE codes into different topics to the message bus. The transmission of SBE encoded data in the system is by UDP (User Datagram Protocol ).
The caching module is used for caching and storing data, and specifically processes the data as follows: the cache module acquires the market data of different subjects after the SBE codes from the message bus, decodes the market data and stores the market data in a memory of the cache module, and simultaneously writes the acquired market data into the database module. In addition, the buffer memory module also acquires the data processed by the processing units from each processing unit in the service processing module, and stores the data into the memory and database module, such as K line, time sharing, relative strength index, random index and other data. The data are stored into the internal memory of the cache module, so that the reading and writing speed of the data can be conveniently accelerated, and the data are deleted from the internal memory or simultaneously saved into the database module after the set time period threshold value. The cache module receives a request for reading data from the service processing module or the pre-processing module, acquires the data from the internal memory of the cache module and returns the data to the corresponding module, and if the data which is not needed to be acquired is not in the internal memory, the cache module can inquire from the database module to return the data in the inquired database module.
The business processing module is used for acquiring market data of different subjects after the SBE codes from the message bus and acquiring various data such as user information, K lines and the like from the cache module. And the processing units in the business processing module acquire the market data and then perform corresponding processing, and the processing results are sent to the cache module for storage. In addition, the processing units in the service processing module also obtain the requests of user authentication, historical data inquiry and the like from the pre-processing module, acquire the data from the cache module and return the data to the pre-processing module.
The service processing module further includes, but is not limited to: the system comprises a user authentication unit, a historical data unit, a K line quotation unit, an index calculation unit, a data analysis unit and a return measurement unit.
The user authentication unit is used for authenticating the authority of the user to access the data. The specific process of authentication is as follows: the user authentication unit receives requests sent from all the pre-units (including a general quotation pre-unit, a query data pre-unit or a historical data pre-unit) of the pre-processing module, queries whether the user has corresponding data access rights from the cache module, and returns a query result to the pre-processing module.
The K line quotation unit is used for calculating and processing the K line and the time sharing. The specific processing of the K line quotation unit is as follows: the K line quotation unit obtains quotation data of each exchange from the message bus, performs SBE coding on the data after performing K line and time-sharing corresponding algorithms, sends the data to the general quotation pre-unit in real time, and simultaneously writes the data into the cache module for inquiring the data pre-unit and the historical data pre-unit to obtain the data. The user can obtain the data for front-end display only by subscribing or inquiring the K-wire and time-sharing data through the user API interface module, and the user does not need to calculate.
The data analysis unit is used for analyzing data such as quotation. The data analysis unit specifically processes: the data analysis unit obtains market data, other public market data, public opinion data and the like of each trade from the message bus, obtains historical data from the cache module, calculates according to a preset algorithm, and stores the data of the calculation result into the cache module for inquiring the data preposition unit and obtaining the data by the historical data preposition unit. For example, the data analysis unit analyzes the liquidity of the Chinese financial futures exchange, and the liquidity index as a calculation result can help the user know how the product has liquidity in the market.
The history quotation unit is used for storing history quotations, and is specifically implemented as follows: the historical quotation unit acquires quotation data from the message bus in real time on each transaction day, writes the data into the cache module, and lands a document additionally according to a preset format.
The index calculation unit is used for calculating various indexes of the market, and the specific processing is as follows: the index calculation unit obtains market data of each trade exchange from the message bus, calculates according to different algorithms of various indexes (such as relative strength index, random index, trend index, smooth different average line and the like), and finally writes the calculation result into the cache module. When a user subscribes or inquires to acquire data through the user API interface module, the calculation results can be displayed at the front end together with K lines and the like, and can also be used for strategy analysis.
The return unit is used for calculating data such as profitability, maximum withdrawal rate and the like under a certain condition, and the specific processing is as follows: the return unit acquires historical quotations from the input time point from the buffer module according to the input of the user, performs stock selection according to the designated stock combination, performs buying and selling according to the transaction rule, and finally calculates and outputs data such as profitability, maximum withdrawal rate and the like in a time period to the user.
The pre-processing module further comprises: a general quotation pre-unit, a query data pre-unit and a history data pre-unit.
The general market leading unit is used for pushing real-time data, and specifically processes the real-time data as follows: the general quotation front-end unit receives the subscription request of the user API interface module, establishes long connection with the user API interface module after the data authentication of the user authentication unit, acquires data from the message bus and the cache module, and pushes different data to the corresponding user API interface module according to the topic subscribed by the user. These topics include, but are not limited to: by exchange quotation topic, single contract quotation topic, exchange K-wire topic, single contract K-wire topic, etc.
The inquiry data pre-unit is used for inquiring the data of the current day, and specifically comprises the following steps: the inquiry data preposition unit receives the request of inquiring the appointed data of the current day of the user API interface module, acquires the data from the buffer memory module after passing the authentication of the user authentication unit and returns the data to the user API interface module.
The history data pre-unit is used for inquiring history data, and specifically comprises the following steps: the history data preposition unit receives a request of inquiring history appointed data of the user API interface module, acquires data from the cache module after authentication of the user authentication unit and returns the data to the user API interface module, and if the acquired history data is before a certain time, the data is directly returned to a link downloaded by a file for a client to acquire the data.
The user API interface module is used for user access to the system, and its set interfaces include, but are not limited to: an API management interface, a request interface and a data callback interface. The user API interface module is used for inputting user name, password and ns address by user, initializing after the user inputs the information and sequentially receiving user requests such as user login, data subscription, data inquiry and the like initiated by the user, the data can be returned in the data callback interface, and the user processes the received data in the data callback interface.
The ns module provides load balancing service for the pre-processing module, and the specific processing mode is as follows: and the trans module acquires the current load condition of each pre-processing module through communication with each pre-processing module, selects a pre-address with the minimum load in the pre-processing module, and returns the pre-address to the user API interface module for data communication.
Fig. 2 shows a data subscription flow configured by the system, and main steps of acquiring push data such as quotation by the system are as follows:
step S11: the server side operates: firstly, starting the exchange quotation receiving process of the system, and starting quotation pushing by the exchange. In the process, due to the form of the dynamic link library, the reception of the selectable number of exchanges can be configured for one quotation receiving process, if a plurality of machines respectively deploy one exchange, the exchanges have no influence on each other, and the speed is higher. After receiving the quotation from the exchange, the core service module in the system directly copies and calculates the exchange data, finally uses SBE to encode, and uses UDP (User Datagram Protocol ) to send the encoded data to the pre-processing module. Based on the internal high-performance framework, UDP communication can reach 500 ten thousand QPS (Queries Per Second, query number per second), and communication delay is greatly reduced. The released data is received by the historical data unit at the same time, after the SBE decoding processing, part of hot spot data is stored in the buffer memory module for low-delay data inquiry, and the rest data is stored in the database module and the file.
Step S12: the user invokes the user API interface module of the system: as shown in table 1, the present system provides several easy to use API interfaces. Notably the order of calling the interfaces: (1) inheriting CFfexGmdDyMdUserSpi, realizing 4 interfaces therein, and adding own data processing logic into a callback function; (2) creating an instance using createGmdDyMdUserApi; (3) registering a front server address by using a register front address, and obtaining a version of a current API by using GetVersion; (4) setting the type of the received data by using the SetExtConfig, and selecting a structure body, a text and data after SBE coding; (5) registering required data topics by using a register topic, and subscribing a plurality of topics at the same time; (6) registering the instance implemented in the first step using a RegisterSpi; (7) starting the Start and waiting for the completion of the execution of the API by using the Start and the Join; (8) after the API is used, a Release deletion instance can be called.
Step S13: processing a user request: after the user API interface module is correctly invoked in step S12, the pre-processing module receives the subscription request of the user, and then sends the data to the user authentication unit for verification, and the pre-processing module receives the return. According to the result of verification, if the authority or the user makes mistakes, error information is sent to the user API interface module, and detailed error information is returned in the OnEvent interface of the user.
Step S14: pushing quotation data: in step S13, if the verification is passed, the pre-processing module establishes a connection with the user API interface module, and pushes data according to the theme subscribed by the user, and at this time, the interfaces in cffexgmddydmduserspi inherited and implemented by the user in step S12 will all have data to return. Preferably, the selection of the SBE encoding in the alternative in operation (4) of step S12 results in a faster rate.
FIG. 3 illustrates a data query flow configured by an embodiment of the big data ticker platform system of this invention. Referring to fig. 3, the implementation steps for obtaining query data such as historical quotations in this embodiment are described in detail below.
Step S21: as in step S11 in fig. 2. Namely, the server side operates: firstly, starting the exchange quotation receiving process of the system, and starting quotation pushing by the exchange. In the process, due to the form of the dynamic link library, the reception of the selectable number of exchanges can be configured for one quotation receiving process, if a plurality of machines respectively deploy one exchange, the exchanges have no influence on each other, and the speed is higher. After receiving the quotation from the exchange, the core service module in the system directly copies and calculates the exchange data, finally uses SBE to encode, and uses UDP (User Datagram Protocol ) to send the encoded data to the pre-processing module. Based on the internal high-performance framework, UDP communication can reach 500 ten thousand QPS (Queries Per Second, query number per second), and communication delay is greatly reduced. The released data is received by the historical data unit at the same time, after the SBE decoding processing, part of hot spot data is stored in the buffer memory module for low-delay data inquiry, and the rest data is stored in the database module and the file.
Step S22: the user invokes the user API module of the system. As shown in table 1, the present system provides several easy to use interfaces. Notably the order of calling the interfaces: (1) inheriting CFfexGmdDyMdUserSpi, realizing 4 interfaces therein, and adding own data processing logic into a callback function; (2) creating an instance using createGmdDyMdUserApi; (3) registering a front server address by using a register front address, and obtaining a version of a current API by using GetVersion; (4) setting the type of the received data by using the SetExtConfig, and selecting a structure body, a text and data after SBE coding; (5) calling a SendRequest interface in a self program to request the data required by the user; (6) registering the instance implemented in the first step using a RegisterSpi; (7) starting the Start and waiting for the completion of the execution of the API by using the Start and the Join; (8) after the API is used, a Release deletion instance can be called.
Step S23: the user request is processed. After the user API interface module is correctly invoked in step S22, the pre-processing module receives a request for querying data from a user, and then sends a verification request to the user authentication unit, and after verification is legal, the pre-processing module sends a request for querying data to different processing units in the service processing module according to the data type of the requested query, for example, historical data, K-line calculation, time-sharing calculation, data analysis, and the like. Some of the processing units may be configured to obtain the required data from the file, database module, and cache module. And a plurality of hot spot data are stored in the cache module, so that the query and reading speed is very high, and the delay of data acquisition is reduced. Finally, the pre-processing module obtains the data returned by the query.
Step S24: and returning the query result. In step S23, if the authority verification is illegal, an error message is sent to the user API interface module, and at this time, a detailed error message is returned from the OnEvent interface of the user. If the verification is passed, the data will be returned through the interface of cffexgmddydmduserspi inherited and implemented by the user in step S22.
While, for purposes of simplicity of explanation, the methodologies are shown and described as a series of acts, it is to be understood and appreciated that the methodologies are not limited by the order of acts, as some acts may, in accordance with one or more embodiments, occur in different orders and/or concurrently with other acts from that shown and described herein or not shown and described herein, as would be understood and appreciated by those skilled in the art.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.
In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a web site, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk (disk) and disc (disk) as used herein include Compact Disc (CD), laser disc, optical disc, digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks (disk) usually reproduce data magnetically, while discs (disk) reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (4)
1. The big data quotation platform system is characterized by comprising a core service module, a database module, a cache module, a message bus, a service processing module, a pre-processing module and a user API interface module, wherein:
the core service module comprises an original data receiving and analyzing unit, an SBE encoding unit and a topic distribution unit, wherein the original data receiving and analyzing unit is used for acquiring original quotation data from transactions and analyzing the data, the SBE encoding unit is used for performing SBE encoding on the original quotation data received and analyzed by the original data receiving unit by adopting a simple binary encoding SBE, and the topic distribution unit is used for distributing the quotation data after the SBE encoding into different topics to a message bus;
the cache module is used for acquiring the SBE coded market data from the message bus or acquiring the processed data from the service processing module, storing the data in a self memory and/or database module, receiving a request for reading the data from the service processing module or the pre-processing module, acquiring the data from the self memory and returning the data, and inquiring the database module to return the inquired data if the data is not in the memory;
the user API interface module is used for enabling a user to access the system, enabling the user to initiate a request comprising user login, data subscription and data query through an API interface, and returning the data processed by the pre-processing module in response to the request to the user;
the pre-processing module is used for pushing real-time data, inquiring current day data and historical data, receiving a request sent by a user through the user API interface module and responding to the request;
the business processing module is used for acquiring market data from the message bus, correspondingly processing the acquired stored data from the cache module, sending the processing result to the cache module for storage, receiving a request comprising user authentication and historical data inquiry from the pre-processing module, acquiring the data from the cache module and returning the data to the pre-processing module;
the system comprises a pre-processing module, a loading balancing service, a pre-processing module and a loading balancing service, wherein the pre-processing module and the pre-processing module perform bidirectional data transmission, the loading balancing service configured in the pre-processing module is used for acquiring the current loading condition of each pre-processing module through communication with each pre-processing module, selecting a pre-processing address with the minimum loading in the pre-processing module, and returning the pre-processing address to a user API interface module for data communication;
the preposition processing module comprises a general quotation preposition unit, a query data preposition unit and a history data preposition unit, wherein:
the general quotation front-end unit receives the subscription request of the user API interface module, establishes long connection with the user API interface module after the data authentication of the user authentication unit, acquires data from the message bus and the cache module, and pushes different data to the corresponding user API interface module according to the topic subscribed by the user;
the inquiry data preposition unit is used for receiving a request of inquiring the appointed data of the current day of the user API interface module, acquiring the data from the cache module after the authentication of the user authentication unit and returning the data to the user API interface module;
the historical data preposition unit is used for receiving a request of inquiring historical appointed data of the user API interface module, acquiring data from the cache module after authentication of the user authentication unit and returning the data to the user API interface module, and if the acquired historical data is before a certain time, directly returning a link downloaded by a file for a client to acquire the data;
wherein, the data after SBE coding in the system is transmitted by adopting UDP protocol.
2. The big data quotation platform system of claim 1, wherein the core service module performs unidirectional data transmission with the message bus, the message bus performs unidirectional data transmission with the pre-processing module, the cache module and the service processing module respectively, the cache module performs bidirectional data transmission with the database module and the service processing module respectively, the cache module and the pre-processing module performs unidirectional data transmission, and the pre-processing module performs bidirectional data transmission with the service processing module and the user API interface module respectively.
3. The big data ticker platform system of claim 1, wherein the business processing module includes, but is not limited to, a user authentication unit, a history data unit, a K-wire ticker unit, an index calculation unit, a data analysis unit, a return unit, wherein:
a user authentication unit for authenticating authority of user access data;
the K line quotation unit is used for calculating and processing the K line and the time sharing;
the data analysis unit is used for analyzing the market data, other public market data and public opinion data;
the historical quotation unit is used for storing historical quotations;
the index calculation unit is used for calculating various indexes of the quotation;
and the return unit is used for calculating return data including the profitability and the maximum withdrawal rate under certain conditions.
4. The big data quotation platform system of claim 1, wherein the system is a distributed architecture deployed at a plurality of nodes to implement fail-over via real-time hot standby.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910640372.3A CN110415027B (en) | 2019-07-16 | 2019-07-16 | Big data quotation platform system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910640372.3A CN110415027B (en) | 2019-07-16 | 2019-07-16 | Big data quotation platform system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110415027A CN110415027A (en) | 2019-11-05 |
| CN110415027B true CN110415027B (en) | 2023-05-26 |
Family
ID=68361685
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910640372.3A Active CN110415027B (en) | 2019-07-16 | 2019-07-16 | Big data quotation platform system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110415027B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112116466A (en) * | 2020-06-11 | 2020-12-22 | 上海金融期货信息技术有限公司 | Unified platform system for stock futures financial data exchange |
| CN111709839B (en) * | 2020-06-19 | 2023-11-14 | 上海金融期货信息技术有限公司 | Wind control complex service implementation system and method based on clearing core |
| CN112132681B (en) * | 2020-07-02 | 2024-03-26 | 上海金融期货信息技术有限公司 | Unified access system for quotation sources |
| CN112347020A (en) * | 2020-10-26 | 2021-02-09 | 东方证券股份有限公司 | FAST market analysis system and method based on CGRA |
| CN113065958A (en) * | 2021-02-20 | 2021-07-02 | 上海金融期货信息技术有限公司 | Theme market situation business data streaming system |
| CN114124668B (en) * | 2021-11-03 | 2024-08-02 | 上证所信息网络有限公司 | System and method for guaranteeing consistency of multi-host quotation slices |
| CN114331445A (en) * | 2022-03-15 | 2022-04-12 | 上海金仕达软件科技有限公司 | API (application programming interface), method, storage medium and electronic equipment for accessing massive users |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108269188A (en) * | 2016-12-30 | 2018-07-10 | 上海金融期货信息技术有限公司 | A kind of exchange's quotation information processing method and system based on FPGA |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070288662A1 (en) * | 2006-06-12 | 2007-12-13 | Telepaq Technology Inc. | Distributed push-pull information service system |
| CN103247003B (en) * | 2013-05-15 | 2017-07-07 | 上海大智慧股份有限公司 | Towards the distributed program transaction system of event handling |
| CN106294486B (en) * | 2015-06-04 | 2020-06-05 | 交通银行股份有限公司 | Financial market data processing method and system |
| CN106708815B (en) * | 2015-07-15 | 2021-09-17 | 中兴通讯股份有限公司 | Data processing method, device and system |
-
2019
- 2019-07-16 CN CN201910640372.3A patent/CN110415027B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108269188A (en) * | 2016-12-30 | 2018-07-10 | 上海金融期货信息技术有限公司 | A kind of exchange's quotation information processing method and system based on FPGA |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110415027A (en) | 2019-11-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110415027B (en) | Big data quotation platform system | |
| US11677563B2 (en) | Systems, apparatus and methods for local state storage of distributed ledger data without cloning | |
| JP2021503643A (en) | Computer-readable storage media containing methods, devices, computer programs and computer programs for optimizing high-volume transaction performance on the blockchain. | |
| CN111935227A (en) | Method for uploading file through browser, browser and electronic equipment | |
| US11709815B2 (en) | Retrieving index data from an object storage system | |
| CA2911001A1 (en) | Failover system and method | |
| US11216415B2 (en) | Identification and recommendation of file content segments | |
| WO2022110990A1 (en) | Account management method and related product | |
| CN109377220B (en) | Block chain-based media data transaction method and system and computer storage medium | |
| CN112785408A (en) | Account checking method and device based on Hash | |
| CN111200637B (en) | Cache processing method and device | |
| WO2016171708A1 (en) | Cross-platform messaging application | |
| CN112669158B (en) | Transaction system and control method and device of transaction service thereof | |
| CN105610823A (en) | Stream media processing method and processing system architecture based on task vectors | |
| CN111401566B (en) | Machine learning training method and system | |
| US10296633B1 (en) | Data storage management system | |
| CN113065953A (en) | Futures relay trading system based on distribution | |
| US8239513B1 (en) | Optimizing media distribution using metrics | |
| US7889649B2 (en) | Method and system for gateway communication | |
| CN111209263A (en) | Data storage method, device, equipment and storage medium | |
| JP7498800B2 (en) | Screen mirroring method and apparatus, electronic device and computer-readable storage medium | |
| US11900346B1 (en) | Apparatuses and methods for payment for consumable content | |
| US11556402B2 (en) | Metadata plane for application programming interface | |
| US10254980B1 (en) | Scheduling requests from data sources for efficient data decoding | |
| CN114240579A (en) | Method and device for communication and cooperation of purchasing party and supplier |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |