CN110335148A - Securities data parallel processing system (PPS) and method - Google Patents
Securities data parallel processing system (PPS) and method Download PDFInfo
- Publication number
- CN110335148A CN110335148A CN201910474388.1A CN201910474388A CN110335148A CN 110335148 A CN110335148 A CN 110335148A CN 201910474388 A CN201910474388 A CN 201910474388A CN 110335148 A CN110335148 A CN 110335148A
- Authority
- CN
- China
- Prior art keywords
- data
- market
- module
- securities
- parallel processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 56
- 238000000034 method Methods 0.000 title claims abstract description 16
- 238000006243 chemical reaction Methods 0.000 claims abstract description 19
- 238000004364 calculation method Methods 0.000 claims abstract description 7
- 230000015572 biosynthetic process Effects 0.000 claims abstract 2
- 239000000872 buffer Substances 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- PEDCQBHIVMGVHV-UHFFFAOYSA-N Glycerine Chemical compound OCC(O)CO PEDCQBHIVMGVHV-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012958 reprocessing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
Landscapes
- Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Engineering & Computer Science (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)
Abstract
The invention discloses a kind of securities data parallel processing system (PPS) and its methods: the system includes: the format conversion module sequentially connected, basic processing module, plate index computing module and market AM access module；Format conversion module is for being converted into the market data of unified format after connecting market source data, being decoded to market source data；Basic processing module is used for the market data of reading format conversion module output and is split by stock code interval range, is processed to the market data of the security in specified interval range, the market data of same security are added with unique identification formation intermediate data；The connection of plate index computing module generates final data for reading the intermediate data of basic processing module output, carrying out calculation processing to specified index；Market AM access module connects plate index meter and calculates module, for reading the final data of plate index computing module output and feeding back to external request port.The present invention can be improved concurrency, reduces packet loss, supports the extension of service management.
Description
Technical field
The invention belongs to technical field of data administration, more particularly, to a kind of securities data parallel processing system (PPS), Yi Jiji
In a kind of securities data method for parallel processing that the system is realized.
Background technique
Securities exchange system has the characteristics that index magnanimity, polynary, changeable, complicated.Most common some market index packets
It includes: quotations, point data, time data, minute K line, year K in diurnal month line, all kinds of financial technical indicators, various dimensions row
Sequence, various dimensions statistics etc., these indexs need to carry out statistics calculating by the market data flow binding time stream of exchange.It is existing
Its course of work of quotation system is as follows: step 1, and market metadata is by the market snapshot such as Shanghai Stock Exchange, Shen Zhen Stock Exchange, exchange union
It provides, is sent to MQ message queue after market transcoding service.Step 2, market indication computing system is by subscribing to market member
Data are calculated, and write the result into Redis cluster, and be sent to push proxy in real time.Step 3, user pass through access gateway
Quotation is inquired, which goes Redis cluster to inquire corresponding data by logical proxy.Push proxy in real time to
The quotation that family push is subscribed to.This technical solution the problem is that；(1) single access process can stablize the concurrent of support
Connection number is not high enough, needs the server more in market access to network system deployment.(2) because largely using MQ communication component,
Under high water level, packet loss is higher, and MQ has load balancing difference.(3) configuration file is mostly written in configuration, causes to take
Business management is inflexible, cannot achieve Visualization Service management.Therefore, a kind of new securities data processing system how is developed
System can be improved concurrency, reduces packet loss, supports the extension of service management, be the side that those skilled in the art need to study
To.
Summary of the invention
The object of the present invention is to provide a kind of securities market data parallel processing system (PPS)s, can be improved concurrency, and reduction is lost
Packet rate supports the extension of service management.
The technical scheme adopted is as follows:
A kind of securities data parallel processing system (PPS): comprising: the format conversion module sequentially connected, basic processing module,
Plate index computing module and market AM access module；The format conversion module is for connecting market source data, to market source number
According to the market data for being converted into unified format after being decoded；The basis processing module is exported for reading format conversion module
Market data and market data are split by the code interval range of security, to the security in specified interval range
Market data are processed and add unique identification to the market data of same security, form intermediate data；The plate
Index computing module is used to read the intermediate data of basic processing module output, carries out calculation processing generation to specified index most
Whole data；The market AM access module is used to read the final data of plate index computing module output and feeds back to external request
Port.
By using this technical solution: each basis processing module handles the Number in specified interval range respectively
According to, it can be to the multiple interval ranges of different kinds setting, and then optimized integration processing module is in the lateral nothing of server architecture
Limit extension.By adding unique identification in each data to same branch security, it can be ensured that same data will not be by down-stream
Reprocessing.By being split by pointer type, while a service only calculates specified index, and then realizes plate
Longitudinal wireless extensions of the index computing module in server architecture.Realize the expanded application of service management.
Preferably, in above-mentioned securities data parallel processing system (PPS): the format conversion module, basic processing module, plate
Data transmission is realized by Netty Open Framework between block index computing module and market AM access module.
By using this technical solution: carrying out data communication using Netty Open Framework, assisted based on bottom TCP/IP
View can be realized the non-delay effect that data are transmitted between each module.
It is further preferred that in above-mentioned securities data parallel processing system (PPS): the format conversion module, basic processing module,
Plate index computing module and market AM access module all pass through Google Protocol Buffers protocol realization internal data biography
It is defeated.
By using this technical solution: with GoogleProtocol Buffers format, the scale of construction is small, parsing is fast.
It may further be preferable that in above-mentioned securities data parallel processing system (PPS): the market AM access module is assisted based on Http
Negotiation WebSocket agreement is to external request end mouthful feedback final data.
By using this technical solution:
Based on above system, the present invention also provides a kind of securities market data method for parallel processing.
Its technical solution is as follows:
A kind of securities data method for parallel processing comprising following steps: S1: read in real time different market data sources,
The market data of unified format are decoded and are converted into market source data；S2: market data obtained by step S1 are pressed into security
Code interval range is split and is processed to the market data of the security in specified interval range, simultaneously to same
The market data of one security add unique identification and form intermediate data；S3: by the finger in the intermediate data exported obtained by step S2
Fixed index carries out calculation processing and generates final data；S4: final data obtained by step S3 is fed back into external request port.
Preferably, in above-mentioned securities data method for parallel processing, the step S2 is specifically included: S22: service starting,
Configuration information is read to configuration center, and monitors configuration variation, initiates to subscribe to upper level according to configuration information；In start completion
Afterwards, service registration is registered to registration center；S22: when receiving securities market data, read its stock code section and from
The middle securities market data chosen stock code and fall into specified interval range；S23: the securities data in process range is solved
Analysis processing, and next stage data subscription person is pushed to after additional unique identification after processing；S24: local memory can save one
Part latest data when receiving new subscription request, can push immediately and return to internal storage data.
It is further preferred that the step S3 is specifically included in above-mentioned securities data method for parallel processing: S31: service starting
When, configuration information is read in configuration center and monitors configuration variation, initiates to subscribe to upper level according to configuration information, and starting
Service registration is registered to registration center after the completion；S32: after receiving market data, periodically the index of configuration is counted
It calculates；S33: a latest data is saved after the completion of calculating every time in local memory, and Redis is written into calculated result, simultaneously
The calculated result is pushed into next stage data subscription person；S34: calling timed task after closing, the data in Redis are saved
To historical data base.
Compared with prior art, the present invention can be improved concurrency, reduces packet loss, supports the extension of service management.
Detailed description of the invention
Present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments:
Fig. 1 is the structural schematic diagram of embodiment 1；
Fig. 2 is the work flow diagram of basic processing module in embodiment 1；
Fig. 3 is the work flow diagram of 1 mesoplate index computing module of embodiment.
Each appended drawing reference and component names corresponding relationship are as follows:
1, format conversion module；2, basic processing module；3, plate index computing module；4, market AM access module；5, row
Feelings data source；6, financial database.
Specific embodiment
In order to illustrate more clearly of technical solution of the present invention, it is further described below in conjunction with each embodiment.
As shown in Figure 1 it is the embodiment of the present invention 1:
A kind of securities data parallel processing system (PPS): comprising: the format conversion module 1 sequentially connected, basic processing module
2, plate index computing module 3 and market AM access module 4；The format conversion module 1 is for connecting market source data, to market
Source data is converted into the market data of unified format after being decoded；
The basis processing module 2 is used for the market data that reading format conversion module 1 exports and by market data by card
The interval range of voucher code is split, is processed, the market data of the security in specified interval range to same
The market data of security add unique identification and form intermediate data.Generation disk mouth, timesharing line, minute line, day K line, moon K line, by
Pen, and the relevant index of single branch stock.
The plate index computing module 3 connection for reading intermediate data that basic processing module 2 exports, to specified
Index carries out calculation processing and generates final data；The market AM access module 4 is used to read the output of plate index computing module 3
Final data simultaneously feeds back to external request port.
In this example: the format conversion module 1, basic processing module 2, plate index computing module 3 and market access mould
Data transmission is realized by Netty Open Framework between block 4.The format conversion module 1, basic processing module 2, plate index
Computing module 3 and market AM access module 4 all pass through Google Protocol Buffers protocol realization internal data transfer.Institute
It states market AM access module and is based on Http agreement and WebSocket agreement to external request end mouthful to user feedback final data.
In practice, the course of work is as Figure 2-3:
S1: reading different market data sources 6 in real time, be decoded to market source data and is converted into the row of unified format
Feelings data；
S21: service starting reads configuration information to configuration center, and monitors configuration variation, according to configuration information upward one
Grade is initiated to subscribe to；After start completion, service registration is registered to registration center；
S22: it when receiving securities market data, reads its stock code section and therefrom selection stock code falls into finger
Determine the securities market data of interval range；
S23: carrying out parsing processing to the securities data in process range, formed Pan Kou, timesharing, K line, by data such as pens,
Then it adds unique identification and is pushed to next stage data subscription person, timestamp and exchange hand can be used in the unique identification；
S24: local memory can save a latest data, when receiving new subscription request, push can return to memory immediately
Data.
S31: when service starting, reading configuration information in configuration center and monitor configuration variation, upward according to configuration information
Level-one is initiated to subscribe to, and service registration is registered to registration center after start completion；
S32: after receiving market data, periodically calculating the index of configuration, as plate calculates, ranking list meter
Calculation and the calculating of other various indexs etc.；
S33: saving a latest data in local memory, and Redis be written in calculated result after the completion of calculating every time,
The calculated result is pushed into next stage data subscription person simultaneously；
S34: calling timed task after closing, the data in Redis are saved to historical data base.
S4: final data obtained by step S3 is fed back into external request port.
The above, only specific embodiments of the present invention, but scope of protection of the present invention is not limited thereto, it is any ripe
The technical staff of art technology is known in technical scope disclosed by the invention, any changes or substitutions that can be easily thought of, should all contain
Lid is within protection scope of the present invention.Protection scope of the present invention is subject to the scope of protection of the claims.
Claims (7)
1. a kind of securities data parallel processing system (PPS): it is characterised by comprising: the format conversion module (1) sequentially connected, basis
Processing module (2), plate index computing module (3) and market AM access module (4)；
The format conversion module (1) is for being converted into unified lattice after connecting market source data, being decoded to market source data
The market data of formula；
The basis processing module (2) is used for the market data of reading format conversion module (1) output and by market data by card
The code interval range of certificate is split, is processed to the market data of the security in specified interval range and right
The market data of same security add unique identification, form intermediate data；
The plate index computing module (3) is used to read the intermediate data of basic processing module (2) output, to specified index
It carries out calculation processing and generates final data；The market AM access module (4) is used to read plate index computing module (3) output
Final data simultaneously feeds back to external request port.
2. securities data parallel processing system (PPS) as described in claim 1, it is characterised in that: the format conversion module (1), basis
Processing module (2) realizes data by Netty Open Framework between plate index computing module (3) and market AM access module (4)
Transmission.
3. securities data parallel processing system (PPS) as described in claim 1, it is characterised in that: the format conversion module (1), basis
Processing module (2), plate index computing module (3) and market AM access module (4) all pass through Google Protocol Buffers
Protocol realization internal data transfer.
4. securities data parallel processing system (PPS) as described in claim 1, it is characterised in that: the market AM access module is based on Http
Agreement and WebSocket agreement are to external request end mouthful feedback final data.
5. a kind of securities data method for parallel processing, which comprises the steps of:
S1: reading different market data sources in real time, be decoded to market source data and is converted into the market number of unified format
According to；
S2: market data obtained by step S1 are split by stock code interval range and to the card in specified interval range
The market data of certificate are processed, add unique identification formation intermediate data to the market data of same security simultaneously；
S3: the specified index in the intermediate data exported obtained by step S2 is subjected to calculation processing and generates final data；
S4: final data obtained by step S3 is fed back into external request port.
6. securities data method for parallel processing as claimed in claim 5, which is characterized in that the step S2 includes:
S21: service starting reads configuration information to configuration center, and monitors configuration variation, is sent out according to configuration information to upper level
It rises and subscribes to；After start completion, service registration is registered to registration center；
S22: it when receiving securities market data, reads its stock code section and therefrom selection stock code falls into specified area
Between range securities market data；
S23: carrying out parsing processing to the securities data in process range, and is pushed to down after additional unique identification after processing
Level one data subscriber；
S24: local memory can save a latest data, when receiving new subscription request, push can return to internal storage data immediately.
7. securities data method for parallel processing as claimed in claim 5, which is characterized in that the step S3 includes:
S31: when service starting, reading configuration information in configuration center and monitor configuration variation, according to configuration information to upper level
It initiates to subscribe to, and service registration is registered to registration center after start completion；
S32: after receiving market data, periodically the index of configuration is calculated；
S33: a latest data is saved after the completion of calculating every time in local memory, and Redis is written into calculated result, simultaneously
The calculated result is pushed into next stage data subscription person；
S34: calling timed task after closing, the data in Redis are saved to historical data base.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910474388.1A CN110335148A (en) | 2019-05-31 | 2019-05-31 | Securities data parallel processing system (PPS) and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910474388.1A CN110335148A (en) | 2019-05-31 | 2019-05-31 | Securities data parallel processing system (PPS) and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110335148A true CN110335148A (en) | 2019-10-15 |
Family
ID=68140698
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910474388.1A Pending CN110335148A (en) | 2019-05-31 | 2019-05-31 | Securities data parallel processing system (PPS) and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110335148A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111104594A (en) * | 2019-12-12 | 2020-05-05 | 上海金纳信息科技有限公司 | Market quotation pushing method and device |
| CN114723565A (en) * | 2022-06-02 | 2022-07-08 | 高盈国际创新科技(深圳)有限公司 | Multi-source multi-language information processing method and system based on golang |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101124578A (en) * | 2005-01-14 | 2008-02-13 | 国际商业机器公司 | Sharable multi-tenant reference data utility and repository, including value enhancement and on-demand data delivery and methods of operation |
| CN101222682A (en) * | 2008-01-28 | 2008-07-16 | 么梦冬 | Communication platform, system and method facing mobile terminal stock exchange transaction |
| CN107133873A (en) * | 2017-06-12 | 2017-09-05 | 安徽简道科技有限公司 | A kind of security sequencing transaction system and method for commerce |
| CN107481140A (en) * | 2017-07-24 | 2017-12-15 | 湖南搜云网络科技股份有限公司 | Voucher identification code processing method and system, server and storage medium |
| CN107480858A (en) * | 2017-07-10 | 2017-12-15 | 武汉楚鼎信息技术有限公司 | A kind of Aided intelligent decision-making and method based on the analysis of stock big data |
| CN109544037A (en) * | 2018-12-19 | 2019-03-29 | 上海金融期货信息技术有限公司 | It is a kind of without intrusive securities futures trading air control gateway system in advance |
-
2019
- 2019-05-31 CN CN201910474388.1A patent/CN110335148A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101124578A (en) * | 2005-01-14 | 2008-02-13 | 国际商业机器公司 | Sharable multi-tenant reference data utility and repository, including value enhancement and on-demand data delivery and methods of operation |
| CN101222682A (en) * | 2008-01-28 | 2008-07-16 | 么梦冬 | Communication platform, system and method facing mobile terminal stock exchange transaction |
| CN107133873A (en) * | 2017-06-12 | 2017-09-05 | 安徽简道科技有限公司 | A kind of security sequencing transaction system and method for commerce |
| CN107480858A (en) * | 2017-07-10 | 2017-12-15 | 武汉楚鼎信息技术有限公司 | A kind of Aided intelligent decision-making and method based on the analysis of stock big data |
| CN107481140A (en) * | 2017-07-24 | 2017-12-15 | 湖南搜云网络科技股份有限公司 | Voucher identification code processing method and system, server and storage medium |
| CN109544037A (en) * | 2018-12-19 | 2019-03-29 | 上海金融期货信息技术有限公司 | It is a kind of without intrusive securities futures trading air control gateway system in advance |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111104594A (en) * | 2019-12-12 | 2020-05-05 | 上海金纳信息科技有限公司 | Market quotation pushing method and device |
| CN114723565A (en) * | 2022-06-02 | 2022-07-08 | 高盈国际创新科技(深圳)有限公司 | Multi-source multi-language information processing method and system based on golang |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200293549A1 (en) | Blockchain-based data processing method and device | |
| CN100531055C (en) | Data synchronous system and its method | |
| CN101355686B (en) | Method and system for statistic of audience rating | |
| CN109951463A (en) | A kind of Internet of Things big data analysis method stored based on stream calculation and novel column | |
| US9215288B2 (en) | Methods and apparatus to share online media impressions data | |
| CN106933724B (en) | Distributed information tracking system, information processing method and device | |
| CN110413599A (en) | Generating date and storage system and method | |
| US11747889B2 (en) | Customized Internet-of-Things data packaging and brokering | |
| CN106126641A (en) | A kind of real-time recommendation system and method based on Spark | |
| CN107634848A (en) | A kind of system and method for collection analysis network equipment information | |
| US9614794B2 (en) | Message consumer orchestration framework | |
| US20200159764A1 (en) | Method for Processing and Displaying Real-Time Social Data on Map | |
| CN110335148A (en) | Securities data parallel processing system (PPS) and method | |
| CN104486116A (en) | Multidimensional query method and multidimensional query system of flow data | |
| CN100452704C (en) | Method and method for issuing blog articles | |
| CN109783330B (en) | Log processing method, log display method, and related device and system | |
| CN103179211A (en) | Cloud-computing-based data processing method and system of comprehensive operation support system | |
| KR101736382B1 (en) | Ems server and log data management method thereof | |
| CN107526808B (en) | Real-time data processing method and device | |
| CN105592327A (en) | IPTV-system-based processing method and apparatus for stock information | |
| CN106713374A (en) | DNS-based traffic analysis and optimal traffic scheduling system | |
| CN113468246B (en) | Intelligent data statistics and subscription system and method based on OLTP | |
| CN103166819A (en) | Network structure based on business priorities and push method of network structure | |
| CN113268363B (en) | Global capability-based call tracking method, device, server and storage medium | |
| CN113032383B (en) | Method, device, system and storage medium for processing data in real time |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191015 |