CN108304443A - Data commission playback monitoring system based on flow data inverting - Google Patents
Data commission playback monitoring system based on flow data inverting Download PDFInfo
- Publication number
- CN108304443A CN108304443A CN201711230259.5A CN201711230259A CN108304443A CN 108304443 A CN108304443 A CN 108304443A CN 201711230259 A CN201711230259 A CN 201711230259A CN 108304443 A CN108304443 A CN 108304443A
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- unit
- commission
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
Abstract
A kind of data commission playback monitoring system based on flow data inverting, including：The server end being made of the data analysis layer, bussiness processing layer and the preposition access layer that are sequentially connected and the client being made of service display layer and application programming interface unit.The present invention can extract huge continuous data stream, filtering, the operations such as processing, realize the marketing behavior visual analyzing based on flow data, dynamic reduction commission scene, reappears whole order depth of whole market, and can play back the orders received book of any one time point, it supports to press clients filter simultaneously, the visualizing monitor of the commitment of client's dimension is realized in the trading activity of dynamic demonstration client.
Description
Technical field
The present invention relates to a kind of technology of field of information processing, specifically a kind of data committee based on flow data inverting
Support playback monitoring system.
Background technology
Traditional data process flow usually first collects data, first stores data into database or file, data
Success is read data again after landing from storage and is processed, and this mode similar to processed offline unavoidably exists
The consuming problem of delay can not meet need very well for the higher research and application scene of this requirement of real-time of transaction detection
It asks.In addition transaction scene intuitively can not be communicated to monitoring personnel by the data recurrence of simple backstage trading activity.
Invention content
The present invention is directed to deficiencies of the prior art, proposes a kind of data commission playback based on flow data inverting
Monitoring system can extract huge continuous data stream, filtering, and the operations such as processing realize that the market based on flow data is handed over
Easy is visual analyzing, and dynamic reduction commission scene reappears whole order depth of whole market, and can play back any one
The orders received book of time point, while supporting to press clients filter, the commitment of client's dimension is realized in the trading activity of dynamic demonstration client
Visualizing monitor..
The present invention is achieved by the following technical solutions：
The present invention includes：The server being made of the data analysis layer, bussiness processing layer and the preposition access layer that are sequentially connected
End and the client being made of service display layer and application programming interface unit, wherein：Data analysis layer receives transaction
Pipelined data simultaneously carries out preliminary filtering extraction, and bussiness processing layer access data analysis layer tentatively filters the data after extraction into line number
It is calculated according to management, commission stream calculation and orders received book；Preposition access layer provides access service for client, visual for business demand
The service display layer of change is by application programming interface unit Connection Service device bussiness processing layer request data and graphically opens up
Show.
The data analysis layer includes：Data receipt unit, data inversion unit and data processing unit.
It is equipped in the bussiness processing layer and passes through the transaction journal of data analysis layer and based in real time by receiving
Calculate the arithmetic element of flow data.
The preposition access layer includes：Flow controlling unit, session management unit, safe encryption unit, data are reliable
Control unit, heartbeat administrative unit and Data Management Unit, wherein：Session management unit and heartbeat administrative unit manage use jointly
Family log in and log in after with the session of system and realize disconnection reconnecting, Data Management Unit carries out information subscribing publication, and forwarding is used
Family request data simultaneously receives user's response result, and safe encryption unit logs in user and the data of information communication add every time
Close and certification, flow controlling unit to request flow limit and load balancing, number are realized in the connection of balanced user to system
According to reliable control unit for storing user's private data.
Technique effect
Compared with prior art, the data management for the high speed that the present invention is combined using most fast insertion and most fast inquiry
Data structure, the data communication of communication realization highly effective and safe between preposition access layer and client end AP I, using high-performance without lock team
Row frame, cross-platform C++ frameworks, Zero-Copy；Client application DevExpress Chart chart controls, using asynchronous API
Coordinate Dynamic Display trading activity with .NetTPL.
Description of the drawings
Fig. 1 is present system logic distribution schematic diagram；
Fig. 2 is embodiment framework flow diagram；
Fig. 3 is embodiment effect diagram.
Specific implementation mode
The present embodiment includes：C++ background servers, C#WinForm foreground clients.Required data include transaction flow number
According to., wherein：Server configuration requirement：Linux RedHat 7.3OS, 16 core CPU, 64G memories, 500G hard disks；Client：
Win7 64bit OS, 4 core CPU, 8G memories, 80G hard disks, DevExpress 17.1, Visual Studio2015.Server
It supports run within 7*24 hours, reaches the stream process rate of single day 30G, data capacity 20M per second, the premise that data weekend preloads
The lower inquiry service for providing one week.Client supports the time series data inquiry in second grade, supports to entrust queue to any time point
And orders received book displaying and contract transaction by a dynamic playback.
As shown in Figure 1, the present embodiment includes：By data analysis layer, bussiness processing layer and the preposition access layer being sequentially connected
The server end of composition and the client being made of service display layer and application programming interface unit, wherein：At data
It manages layer to receive transaction journal data and carry out tentatively filtering extraction, bussiness processing layer access data processing layer data, which calculate, to be added
Work；Preposition access layer provides access service for client, passes through application program for the visual service display layer of business demand
Programming interface unit Connection Service device bussiness processing layer request data and graphical representation.
As shown in Fig. 2, the server end uses lock-free queue and high-performance calculation frame, framework is supported in a distributed manner
Highly reliable 7*24 hours operation free of discontinuities, uses Docker+ micro services thoughts, component plug and play.
The bussiness processing layer is worked using SEDA high concurrent model whole process asynchronizations, and is carried out soon for business datum
Slow separation, raising system are responded and are handled up；Using paxos principal and subordinate's election algorithms, realize that the big data of system is extending transversely；In number
It is realized according to the access data structure that process layer and bussiness processing layer junction use XTP+ data compression protocols and use for reference SBE agreements
Multi-level buffer.
The preposition access layer uses publish-subscribe model and point-to-point messaging mode based on broadcast, realizes
Encrypted safety data transmission.
The service display layer uses C#Winform, is based on .NET Framework tasks in parallel library application server
The API of offer sends Asynchronous Request and subscribes to order flow, market stream, commission stream and stream calculation as a result, using DevExpress charts
Control exposure trading situation it is interactive to realize that user carries out with the essential element on figure, and update per number by thread dynamic
Strong point, linkage update fresh picture situation.
The displaying, including using grid Grid display commission queues, whole day market are shown using line chart, using heap
Folded figure realizes that contract trade order real-time update and the displaying of historical trading order, and its corresponding whole city orders received book overall picture are presented；
Efficiently playback：Second under big data quantity go out orders received book and contract transaction by a dynamic playback, and the synchronized update on market figure
Play line；Market link：Follow order inverse time point linkage market line highlighted；Filtering plays：Play back given client and client
The commission operation of group.
The data analysis layer includes：Data receipt unit, data inversion unit and data processing unit, realization
Target is exactly that data receiver accesses real-time or historical trading flowing water, and transfers to data inversion module reduction transaction scene, then docks
Data processing unit carries out task scheduling control and accomplishes that multithreading orderly restores.
It is equipped in the bussiness processing layer and passes through the transaction journal of data analysis layer and based in real time by receiving
Calculate the arithmetic element of flow data.Wherein data management (is inserted into fast, access with vector (random access is fast, is inserted into slow), deque
Need to be traversed for, data-storage, map used to carry out key value storage) the data management data structure of realizing high speed, according to data volume into
Row sort algorithm unrestricted choice uses quicksort when big data quantity, carries out segmentation recurrence, changes into after small data quantity using being suitble to
Insertion Sorting Algorithm.
The described real-time calculating refers to：Interval division data block computation partition time point is pressed according to commission playback business demand
Corresponding commission queue, orders received book and its offset.
The preposition access layer includes：Flow controlling unit, session management unit, safe encryption unit, data are reliable
Control unit, heartbeat administrative unit and Data Management Unit, wherein：Session management unit and heartbeat administrative unit manage use jointly
Family log in and log in after with the session of system and realize disconnection reconnecting, Data Management Unit is responsible for information subscribing publication, and forwarding is used
Family request data simultaneously receives user's response result, and safe encryption unit logs in user and the data of information communication add every time
Close and certification, flow controlling unit to request flow limit and load balancing, number are realized in the connection of balanced user to system
According to reliable control unit for storing user's private data.And transmission data is compressed using protocol stack in data communication
And transmission, the Efficient Compression of data is carried out in the data compression layer of protocol stack, is encrypted using efficient in the safe floor of protocol stack
Algorithm carries out data encryption.
The present embodiment is related to the control method of above system, by from server end import transaction flowing water, stream data
Trading activity carries out inverting and efficiently calculates in real time, supports the transaction of efficient time series data inquiry and quick-searching random time point
Orders received book situation；Client visualizes stream calculation result (market, commission queue, orders received book) by chart, and to each
Pen operation supports that Visual Dynamic effect realizes that trading activity plays on chart.
Above-mentioned specific implementation can by those skilled in the art under the premise of without departing substantially from the principle of the invention and objective with difference
Mode carry out local directed complete set to it, protection scope of the present invention is subject to claims and not by above-mentioned specific implementation institute
Limit, each implementation within its scope is by the constraint of the present invention.
Claims (8)
1. a kind of data commission playback monitoring system based on flow data inverting, which is characterized in that including：By the number being sequentially connected
It is connect according to the server end of process layer, bussiness processing layer and preposition access layer composition and by service display layer and application programming
The client of mouth unit composition, wherein：Data analysis layer receives transaction journal data and carries out preliminary filtering extraction, business processing
Layer access data processing layer data carries out calculating processing；Preposition access layer provides access service for client, is used for business demand
Visual service display layer passes through application programming interface unit Connection Service device bussiness processing layer request data and figure
Change displaying.
2. system according to claim 1, characterized in that the data analysis layer includes：Data receipt unit, data
The target of inverting unit and data processing unit, realization is exactly that data receiver accesses real-time or historical trading flowing water, and transfers to
Data inversion module reduction transaction scene, then Interworking Data processing unit carry out task scheduling control and accomplish multithreading orderly also
It is former.
3. system according to claim 1, characterized in that be equipped in the bussiness processing layer and pass through data for receiving
The transaction journal of process layer and arithmetic element for calculating flow data in real time, this is asynchronous using SEDA high concurrent model whole process
Chemical industry is made, and carries out speed separation for business datum, and raising system is responded and handled up；It is real using paxos principal and subordinate's election algorithms
The big data of existing system is extending transversely；It using XTP+ data compression protocols and is borrowed in data analysis layer and bussiness processing layer junction
The access data structure for SBE agreements of reflecting realizes multi-level buffer.
4. system according to claim 3, characterized in that the described real-time calculating refers to：According to commission playback business need
It asks by the corresponding commission queue of interval division data block computation partition time point, orders received book and its offset.
5. system according to claim 1, characterized in that the preposition access layer uses the distribution subscription based on broadcast
Pattern and point-to-point messaging mode, realize encrypted safety data transmission, which includes：Flow control list
Member, session management unit, safe encryption unit, the reliable control unit of data, heartbeat administrative unit and Data Management Unit,
In：Session management unit and heartbeat administrative unit manage the session after user logs in and logs in system and realize broken string weight jointly
Even, Data Management Unit carries out information subscribing publication, forwards user's request data and receives user's response result, and safety encryption is single
Member logs in user and the data of information communication are encrypted every time and certification, flow controlling unit limit request flow
And load balancing is realized in the connection of balanced user to system, the reliable control unit of data is for storing user's private data.
6. system according to claim 1, characterized in that the server end uses lock-free queue and high-performance calculation
Frame, in a distributed manner framework support highly reliable 7*24 hours operation free of discontinuities；Use SEDA high concurrent model whole process asynchronizations
Work, and carry out speed separation for business datum；Using paxos principal and subordinate's election algorithms, realize that the big data of system laterally expands
Exhibition；Using XTP+ data compression protocols and use for reference SBE agreements access data structure realize multi-level buffer；Publication based on broadcast
Subscribing mode and point-to-point messaging mode, realize encrypted safety data transmission；Using Docker+ micro services thoughts,
Component plug and play.
7. system according to claim 1, characterized in that the service display layer uses C#Winform, is based on
.NETFramework the API that tasks in parallel library application server provides sends Asynchronous Request and subscribes to order flow, market stream, commission
Stream and stream calculation using DevExpress chart controls as a result, show trading situation, realization user and the essential element on figure
Interaction is carried out, and each data point, linkage update fresh picture situation are updated by thread dynamic.
8. system according to claim 7, characterized in that the displaying, including using grid Grid display commissions team
Row show whole day market using line chart, are schemed using stacking, realize contract trade order real-time update and historical trading order exhibition
Show, and its corresponding whole city orders received book overall picture is presented；Efficiently playback：Second under big data quantity go out orders received book and contract transaction by
Dynamic playback, and synchronized update plays line on market figure；Market link：Follow order inverse time point linkage market line high
It is bright；Filtering plays：Play back the commission operation of given client and client's group.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711230259.5A CN108304443A (en) | 2017-11-29 | 2017-11-29 | Data commission playback monitoring system based on flow data inverting |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711230259.5A CN108304443A (en) | 2017-11-29 | 2017-11-29 | Data commission playback monitoring system based on flow data inverting |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108304443A true CN108304443A (en) | 2018-07-20 |
Family
ID=62869776
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711230259.5A Pending CN108304443A (en) | 2017-11-29 | 2017-11-29 | Data commission playback monitoring system based on flow data inverting |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108304443A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110209724A (en) * | 2019-04-28 | 2019-09-06 | 必成汇(成都)科技有限公司 | The transaction depth map generalization of digital encryption currency spot market and show method |
| CN110955636A (en) * | 2018-10-17 | 2020-04-03 | 郑州期米信息技术有限公司 | Three-dimensional panoramic historical market playback method |
| CN116506333A (en) * | 2023-06-27 | 2023-07-28 | 深圳华锐分布式技术股份有限公司 | Transaction system production inversion detection method and equipment |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103247003A (en) * | 2013-05-15 | 2013-08-14 | 上海大智慧股份有限公司 | Distributed program trading system oriented to event processing |
| CN103678339A (en) * | 2012-09-06 | 2014-03-26 | 阿里巴巴集团控股有限公司 | Data backflow method and system and data access method and system in relational database |
| US20140229514A1 (en) * | 2013-02-13 | 2014-08-14 | International Business Machines Corporation | Supporting big data in enterprise content management systems |
| CN105184647A (en) * | 2015-08-25 | 2015-12-23 | 王子瑜 | Transaction flow recovery method and system |
| CN105528728A (en) * | 2015-12-09 | 2016-04-27 | 江苏易销电子商务有限公司 | Mall e-commerce service platform based on cloud computing and method thereof |
| CN106651463A (en) * | 2016-12-30 | 2017-05-10 | 上海富聪金融信息服务有限公司 | Financial institution service access system and access method |
| CN106780027A (en) * | 2016-12-08 | 2017-05-31 | 北京金融资产交易所有限公司 | A kind of data handling system and method |
| CN106815062A (en) * | 2015-12-02 | 2017-06-09 | 阿里巴巴集团控股有限公司 | A kind of business pipelined data processing method and processing device |
| CN107123047A (en) * | 2017-04-14 | 2017-09-01 | 北京金融资产交易所有限公司 | The data collecting system and its collecting method merchandised based on bond |
-
2017
- 2017-11-29 CN CN201711230259.5A patent/CN108304443A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678339A (en) * | 2012-09-06 | 2014-03-26 | 阿里巴巴集团控股有限公司 | Data backflow method and system and data access method and system in relational database |
| US20140229514A1 (en) * | 2013-02-13 | 2014-08-14 | International Business Machines Corporation | Supporting big data in enterprise content management systems |
| CN103247003A (en) * | 2013-05-15 | 2013-08-14 | 上海大智慧股份有限公司 | Distributed program trading system oriented to event processing |
| CN105184647A (en) * | 2015-08-25 | 2015-12-23 | 王子瑜 | Transaction flow recovery method and system |
| CN106815062A (en) * | 2015-12-02 | 2017-06-09 | 阿里巴巴集团控股有限公司 | A kind of business pipelined data processing method and processing device |
| CN105528728A (en) * | 2015-12-09 | 2016-04-27 | 江苏易销电子商务有限公司 | Mall e-commerce service platform based on cloud computing and method thereof |
| CN106780027A (en) * | 2016-12-08 | 2017-05-31 | 北京金融资产交易所有限公司 | A kind of data handling system and method |
| CN106651463A (en) * | 2016-12-30 | 2017-05-10 | 上海富聪金融信息服务有限公司 | Financial institution service access system and access method |
| CN107123047A (en) * | 2017-04-14 | 2017-09-01 | 北京金融资产交易所有限公司 | The data collecting system and its collecting method merchandised based on bond |
Non-Patent Citations (1)
| Title |
|---|
| 陈云: "《金融大数据》", 31 January 2015 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110955636A (en) * | 2018-10-17 | 2020-04-03 | 郑州期米信息技术有限公司 | Three-dimensional panoramic historical market playback method |
| CN110955636B (en) * | 2018-10-17 | 2023-05-16 | 郑州期米信息技术有限公司 | Stereoscopic panoramic historical market playback method |
| CN110209724A (en) * | 2019-04-28 | 2019-09-06 | 必成汇(成都)科技有限公司 | The transaction depth map generalization of digital encryption currency spot market and show method |
| CN116506333A (en) * | 2023-06-27 | 2023-07-28 | 深圳华锐分布式技术股份有限公司 | Transaction system production inversion detection method and equipment |
| CN116506333B (en) * | 2023-06-27 | 2023-09-01 | 深圳华锐分布式技术股份有限公司 | Transaction system production inversion detection method and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11757981B2 (en) | Efficient and reliable host distribution of totally ordered global state | |
| CN108304443A (en) | Data commission playback monitoring system based on flow data inverting | |
| CN106453564A (en) | Elastic cloud distributed massive request processing method, device and system | |
| US8462369B2 (en) | Hybrid image processing system for a single field of view having a plurality of inspection threads | |
| Ekmefjord et al. | Scalable federated machine learning with fedn | |
| CN107391606A (en) | Log processing method and device based on Storm | |
| CN105007337A (en) | Cluster system load balancing method and system thereof | |
| Kuki et al. | A contribution to modeling two-way communication with retrial queueing systems | |
| CN110071965A (en) | A kind of data center management system based on cloud platform | |
| CN102404615A (en) | Video processing system based on cloud computing | |
| CN107612984B (en) | Big data platform based on internet | |
| CN109951320A (en) | A kind of expansible multi layer monitoing frame and its monitoring method of facing cloud platform | |
| CN109062697A (en) | It is a kind of that the method and apparatus of spatial analysis service are provided | |
| CN108984333A (en) | The method and device calculated in real time for big data | |
| CN109254846A (en) | The dynamic dispatching method and system of CPU and GPU cooperated computing based on two-level scheduler | |
| Lohitha et al. | Integrated publish/subscribe and push-pull method for cloud based IoT framework for real time data processing | |
| CN110232136A (en) | A kind of big data processing system based on cloud computing | |
| CN111932663A (en) | Parallel drawing method based on multi-level asymmetric communication management | |
| CN107085579A (en) | A kind of data acquisition distribution method and device | |
| CN107395694A (en) | A kind of big data management system | |
| US9853933B2 (en) | Message queue replication with message ownership migration | |
| CN103442034B (en) | A kind of stream media service method based on cloud computing technology and system | |
| Gerald‐Yamasaki | Cooperative visualization of computational fluid dynamics | |
| CN107329832A (en) | A kind of data receiver method and device | |
| CN107481154A (en) | The analysis method and device of social networks interpersonal relationships |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180720 |
|
| RJ01 | Rejection of invention patent application after publication |