CN106845926A - A kind of Third-party payment supervisory systems distributed data method for stream processing and system - Google Patents
A kind of Third-party payment supervisory systems distributed data method for stream processing and system Download PDFInfo
- Publication number
- CN106845926A CN106845926A CN201611226657.5A CN201611226657A CN106845926A CN 106845926 A CN106845926 A CN 106845926A CN 201611226657 A CN201611226657 A CN 201611226657A CN 106845926 A CN106845926 A CN 106845926A
- Authority
- CN
- China
- Prior art keywords
- data
- model
- processing
- task
- party payment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/103—Workflow collaboration or project management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
Landscapes
- Business, Economics & Management (AREA)
- Engineering & Computer Science (AREA)
- Strategic Management (AREA)
- Human Resources & Organizations (AREA)
- General Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Marketing (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- Physics & Mathematics (AREA)
- Finance (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Technology Law (AREA)
- Data Mining & Analysis (AREA)
- Development Economics (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a kind of Third-party payment supervisory systems distributed data method for stream processing, including：Parsing user's request, deposit pipe, report mode for Third-party payment supervisory systems set up Business Processing model respectively；Focus flow is gone out based on the Business Processing model combing set up, setting up data control model carries out focus flow processing；Carry out flow chart of data processing dependence task division；Based on the task creation subtask scheduling database model for dividing.The present invention can improve the utilization rate of system resource.The invention also discloses a kind of Third-party payment supervisory systems distributed data stream processing system.
Description
Technical field
The present invention relates to bank management technical field, more particularly to a kind of Third-party payment supervisory systems distributed traffic
Processing method and system.
Background technology
At present, when processing complex data stream process task, top-down whole process serial process side is used
Method.It is treatment request one user of application, sets up a set of thread, complicated treatment request is carried out top-down serial
Processing.The method for using in the prior art cannot meet mass data amount, the process demand of short time.System is simultaneously in face of multiplex
When family is asked, locked by resource, set up queuing mechanism, all clients are asked into serial process.May knot caused by so
It is really：Client's turn time is long, and all resources of system need to be estimated and prepare according to the maximum respond request of data volume, greatly
Huge waste is caused to system resource in idle state when most.Therefore, how system resource is farthest improved
Utilization rate is a problem demanding prompt solution.
The content of the invention
The invention provides a kind of Third-party payment supervisory systems distributed data method for stream processing, it is possible to increase system is provided
The utilization rate in source.
The invention provides a kind of Third-party payment supervisory systems distributed data method for stream processing, including：
Parsing user's request, deposit pipe, report mode for Third-party payment supervisory systems set up Business Processing mould respectively
Type；
Business Processing model combing based on the foundation goes out focus flow, and setting up data control model carries out focus flow
Treatment；
Carry out flow chart of data processing dependence task division；
Based on the task creation subtask scheduling database model for dividing.
Preferably, the parsing user's request, deposit pipe, report mode for Third-party payment supervisory systems are set up respectively
Business Processing model includes：
Merged by operation flow passage, obtain data processing basic procedure model, mark off data processing focus and
Bottleneck.
Preferably, it is described to be merged by operation flow passage, data processing basic procedure model is obtained, mark off at data
The focus and ring collar of reason include：
The detailed data of big data quantity is received and parsed through, preliminary treatment is carried out to detailed data；
For the report data received under pretreated detailed or report mode under detailed pattern carry out data hook check and
Parsing.
Preferably, the Business Processing model combing based on the foundation goes out focus flow, sets up data control model
Carrying out focus flow processing includes：
Cutting is carried out to different clients account by hash algorithm to customer information, and is set up to database point library storage
Mapping relations, set up data model.
Preferably, it is described that cutting is carried out to different clients account by hash algorithm to customer information, and data are thought in foundation
The mapping relations of storehouse point library storage, setting up data model includes：
The numbering of uniform rules is set up for client, by supervised entities' account carry under customer number；
It is detailed for the substantially even distribution of premise of account based on client, customer number and account are given birth to by hash algorithm
Into Hash result；
The isomorphism that the detailed data of customer accounting code is stored in after mole value database reference numeral is measured to database data
Divide in storehouse, complete database equally loaded.
A kind of Third-party payment supervisory systems distributed data stream processing system, including：
First sets up module, for parsing user's request, pipe, report mode point is deposited for Third-party payment supervisory systems
Business Processing model is not set up；
Second sets up module, and focus flow is gone out for the Business Processing model combing based on the foundation, sets up data control
Simulation carries out focus flow processing；
Data processing module, for carrying out flow chart of data processing dependence task division；
3rd sets up module, for based on the task creation subtask scheduling database model for dividing.
Preferably, described first set up module and include：
Division unit, for merging by operation flow passage, obtains data processing basic procedure model, marks off data
The focus and bottleneck for the treatment of.
Preferably, the division unit includes：
Preliminary treatment subelement, the detailed data for receiving and parsing through big data quantity carries out preparing place to detailed data
Reason；
Data processing subelement, for for the form received under pretreated detailed or report mode under detailed pattern
Data carry out data hook and check and parse.
Preferably, described second set up module and include：
Set up subelement, for carrying out cutting to different clients account by hash algorithm to customer information, and set up to
The mapping relations of database point library storage, set up data model.
Preferably, the subelement of setting up includes：
Carry subelement, the numbering for setting up uniform rules for client, by supervised entities' account carry in client
Under numbering；
Generation subelement, for detailed for the substantially even distribution of premise of account based on client, by hash algorithm pair
Customer number and account generation Hash result；
Be stored in subelement, for measuring mole value to database data after the detailed data of customer accounting code is stored in database
In the isomorphism of reference numeral point storehouse, database equally loaded is completed.
A kind of Third-party payment supervisory systems distributed data stream process side provided from such scheme, the present invention
Method, when needing to process Third-party payment supervisory systems distributed traffic, parses user's request, for the 3rd first
Deposit pipe, the report mode of Fang Zhifu supervisory systems set up Business Processing model respectively；It is then based on the Business Processing model set up
Combing goes out focus flow, and setting up data control model carries out focus flow processing；Then flow chart of data processing dependence task is carried out
Divide；It is then based on the task creation subtask scheduling database model for dividing.It is only complete set to be established for customer information
Vertical task model, realizes parallel processing between client, and pass is relied on by dividing task in a set of client treatment application
System sets up point storehouse mapping to database, carries out that task dependence task is serial, non-dependent tasks in parallel handling process, has saved place
The reason time, farthest make use of system resource.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
The accompanying drawing to be used needed for having technology description is briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with
Other accompanying drawings are obtained according to these accompanying drawings.
Fig. 1 is a kind of Third-party payment supervisory systems distributed data method for stream processing embodiment 1 disclosed by the invention
Flow chart；
Fig. 2 is a kind of Third-party payment supervisory systems distributed data method for stream processing embodiment 2 disclosed by the invention
Flow chart；
Fig. 3 is a kind of Third-party payment supervisory systems distributed data method for stream processing embodiment 3 disclosed by the invention
Flow chart；
Fig. 4 is a kind of Third-party payment supervisory systems distributed data method for stream processing embodiment 4 disclosed by the invention
Flow chart；
Fig. 5 is a kind of Third-party payment supervisory systems distributed data method for stream processing embodiment 5 disclosed by the invention
Flow chart；
Fig. 6 is a kind of Third-party payment supervisory systems distributed data stream processing system embodiment 1 disclosed by the invention
Structural representation；
Fig. 7 is a kind of Third-party payment supervisory systems distributed data stream processing system embodiment 2 disclosed by the invention
Structural representation；
Fig. 8 is a kind of Third-party payment supervisory systems distributed data stream processing system embodiment 3 disclosed by the invention
Structural representation；
Fig. 9 is a kind of Third-party payment supervisory systems distributed data stream processing system embodiment 4 disclosed by the invention
Structural representation；
Figure 10 is a kind of Third-party payment supervisory systems distributed data stream processing system embodiment 5 disclosed by the invention
Structural representation.
Specific embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation is described, it is clear that described embodiment is only a part of embodiment of the invention, rather than whole embodiments.It is based on
Embodiment in the present invention, it is every other that those of ordinary skill in the art are obtained under the premise of creative work is not made
Embodiment, belongs to the scope of protection of the invention.
As shown in figure 1, being a kind of Third-party payment supervisory systems distributed data method for stream processing reality disclosed by the invention
The flow chart of example 1 is applied, the method is comprised the following steps：
S101, parsing user's request, deposit pipe, report mode for Third-party payment supervisory systems are set up at business respectively
Reason model；
When needing to process Third-party payment supervisory systems distributed traffic, the demand first to user is carried out
Parsing, the demand according to user sets up Business Processing mould respectively for deposit pipe and the report mode of Third-party payment supervisory systems
Type.
S102, the Business Processing model combing based on foundation go out focus flow, and setting up data control model carries out focus stream
Journey treatment；
After the Business Processing model combing that has built up goes out focus flow, and then set up data control model and carry out heat
Point flow processing.
S103, carry out flow chart of data processing dependence task division；
Upon receipt of a service request, obtain file and decompress the pretreatment verification self-service task returning result of reservation, then
Self-service task is registered, starts basic data processing, table of being delivered newspaper in verification contract information parsing verifies form and cell legitimacy
Data loading, registers Third-party payment batch tasks, registers Third-party payment batch tasks data loading, registers Third-party payment
Batch tasks, poll plays task process data, by data loading, generates other forms, and generation hook checks check results.
S104, the task creation subtask scheduling database model based on division.
To task creation database model, each client sets up an identification information for unique binding customer number, root
Main task, and the carry subtask under main task are set up according to task partitioning model, is numbered by task instances and is stored in database, and
By autotask polling mechanism, parallel calling is carried out for the flow set up in above-mentioned data model and data processing model,
To reach, different clients are parallel, non-dependent tasks in parallel purpose.
In sum, in the above-described embodiments, when needing at Third-party payment supervisory systems distributed traffic
During reason, user's request is parsed first, deposit pipe, the report mode for Third-party payment supervisory systems set up Business Processing mould respectively
Type；The Business Processing model combing for being then based on setting up goes out focus flow, and setting up data control model carries out focus flow processing；
Then flow chart of data processing dependence task division is carried out；It is then based on the task creation subtask scheduling database model for dividing.
The task model of complete set independence is established for customer information, parallel processing is realized between client, in a set of client
Point storehouse mapping for arriving database is set up by dividing task dependence in treatment application, task dependence task is carried out serial, non-
Dependence task parallel processing flow, has saved process time, farthest make use of system resource.
As shown in Fig. 2 being a kind of Third-party payment supervisory systems distributed data method for stream processing reality disclosed by the invention
The flow chart of example 2 is applied, the method is comprised the following steps：
S201, merged by operation flow passage, obtain data processing basic procedure model, mark off the heat of data processing
Point and bottleneck；
When needing to process Third-party payment supervisory systems distributed traffic, the demand first to user is carried out
Parsing, the demand according to user sets up Business Processing mould respectively for deposit pipe and the report mode of Third-party payment supervisory systems
Type.Specifically, merging by operation flow passage, data processing basic procedure model is obtained, mark off the focus of data processing
And bottleneck.
S202, the Business Processing model combing based on foundation go out focus flow, and setting up data control model carries out focus stream
Journey treatment；
After the Business Processing model combing that has built up goes out focus flow, and then set up data control model and carry out heat
Point flow processing.
S203, carry out flow chart of data processing dependence task division；
Upon receipt of a service request, obtain file and decompress the pretreatment verification self-service task returning result of reservation, then
Self-service task is registered, starts basic data processing, table of being delivered newspaper in verification contract information parsing verifies form and cell legitimacy
Data loading, registers Third-party payment batch tasks, registers Third-party payment batch tasks data loading, registers Third-party payment
Batch tasks, poll plays task process data, by data loading, generates other forms, and generation hook checks check results.
S204, the task creation subtask scheduling database model based on division.
To task creation database model, each client sets up an identification information for unique binding customer number, root
Main task, and the carry subtask under main task are set up according to task partitioning model, is numbered by task instances and is stored in database, and
By autotask polling mechanism, parallel calling is carried out for the flow set up in above-mentioned data model and data processing model,
To reach, different clients are parallel, non-dependent tasks in parallel purpose.
In sum, in the above-described embodiments, when needing at Third-party payment supervisory systems distributed traffic
During reason, user's request is parsed first, deposit pipe, the report mode for Third-party payment supervisory systems set up Business Processing mould respectively
Type；The Business Processing model combing for being then based on setting up goes out focus flow, and setting up data control model carries out focus flow processing；
Then flow chart of data processing dependence task division is carried out；It is then based on the task creation subtask scheduling database model for dividing.
The task model of complete set independence is established for customer information, parallel processing is realized between client, in a set of client
Point storehouse mapping for arriving database is set up by dividing task dependence in treatment application, task dependence task is carried out serial, non-
Dependence task parallel processing flow, has saved process time, farthest make use of system resource.
As shown in figure 3, being a kind of Third-party payment supervisory systems distributed data method for stream processing reality disclosed by the invention
The flow chart of example 3 is applied, the method is comprised the following steps：
S301, the detailed data for receiving and parsing through big data quantity, preliminary treatment is carried out to detailed data；
S302, the report data for reception under pretreated detailed or report mode under detailed pattern carry out data hook
Check and parse；
When needing to process Third-party payment supervisory systems distributed traffic, the demand first to user is carried out
Parsing, the demand according to user sets up Business Processing mould respectively for deposit pipe and the report mode of Third-party payment supervisory systems
Type.Specifically, receiving and parsing through the detailed data of big data quantity, preliminary treatment is carried out to detailed data, under detailed pattern
The report data received under pretreated detailed or report mode carries out data hook and checks and parse.
S303, the Business Processing model combing based on foundation go out focus flow, and setting up data control model carries out focus stream
Journey treatment；
After the Business Processing model combing that has built up goes out focus flow, and then set up data control model and carry out heat
Point flow processing.
S304, carry out flow chart of data processing dependence task division；
Upon receipt of a service request, obtain file and decompress the pretreatment verification self-service task returning result of reservation, then
Self-service task is registered, starts basic data processing, table of being delivered newspaper in verification contract information parsing verifies form and cell legitimacy
Data loading, registers Third-party payment batch tasks, registers Third-party payment batch tasks data loading, registers Third-party payment
Batch tasks, poll plays task process data, by data loading, generates other forms, and generation hook checks check results.
S305, the task creation subtask scheduling database model based on division.
To task creation database model, each client sets up an identification information for unique binding customer number, root
Main task, and the carry subtask under main task are set up according to task partitioning model, is numbered by task instances and is stored in database, and
By autotask polling mechanism, parallel calling is carried out for the flow set up in above-mentioned data model and data processing model,
To reach, different clients are parallel, non-dependent tasks in parallel purpose.
In sum, in the above-described embodiments, when needing at Third-party payment supervisory systems distributed traffic
During reason, user's request is parsed first, deposit pipe, the report mode for Third-party payment supervisory systems set up Business Processing mould respectively
Type；The Business Processing model combing for being then based on setting up goes out focus flow, and setting up data control model carries out focus flow processing；
Then flow chart of data processing dependence task division is carried out；It is then based on the task creation subtask scheduling database model for dividing.
The task model of complete set independence is established for customer information, parallel processing is realized between client, in a set of client
Point storehouse mapping for arriving database is set up by dividing task dependence in treatment application, task dependence task is carried out serial, non-
Dependence task parallel processing flow, has saved process time, farthest make use of system resource.
As shown in figure 4, being a kind of Third-party payment supervisory systems distributed data method for stream processing reality disclosed by the invention
The flow chart of example 4 is applied, the method is comprised the following steps：
S401, the detailed data for receiving and parsing through big data quantity, preliminary treatment is carried out to detailed data；
S402, the report data for reception under pretreated detailed or report mode under detailed pattern carry out data hook
Check and parse；
When needing to process Third-party payment supervisory systems distributed traffic, the demand first to user is carried out
Parsing, the demand according to user sets up Business Processing mould respectively for deposit pipe and the report mode of Third-party payment supervisory systems
Type.Specifically, receiving and parsing through the detailed data of big data quantity, preliminary treatment is carried out to detailed data, under detailed pattern
The report data received under pretreated detailed or report mode carries out data hook and checks and parse.
S403, cutting is carried out to different clients account by hash algorithm to customer information, and set up to database point storehouse
The mapping relations of storage, set up data model；
After the Business Processing model combing that has built up goes out focus flow, and then set up data control model and carry out heat
Point flow processing.Specifically, carrying out cutting to different clients account by hash algorithm to customer information, and set up to database
Divide the mapping relations of library storage, set up data model
S404, carry out flow chart of data processing dependence task division；
Upon receipt of a service request, obtain file and decompress the pretreatment verification self-service task returning result of reservation, then
Self-service task is registered, starts basic data processing, table of being delivered newspaper in verification contract information parsing verifies form and cell legitimacy
Data loading, registers Third-party payment batch tasks, registers Third-party payment batch tasks data loading, registers Third-party payment
Batch tasks, poll plays task process data, by data loading, generates other forms, and generation hook checks check results.
S405, the task creation subtask scheduling database model based on division.
To task creation database model, each client sets up an identification information for unique binding customer number, root
Main task, and the carry subtask under main task are set up according to task partitioning model, is numbered by task instances and is stored in database, and
By autotask polling mechanism, parallel calling is carried out for the flow set up in above-mentioned data model and data processing model,
To reach, different clients are parallel, non-dependent tasks in parallel purpose.
In sum, in the above-described embodiments, when needing at Third-party payment supervisory systems distributed traffic
During reason, user's request is parsed first, deposit pipe, the report mode for Third-party payment supervisory systems set up Business Processing mould respectively
Type；The Business Processing model combing for being then based on setting up goes out focus flow, and setting up data control model carries out focus flow processing；
Then flow chart of data processing dependence task division is carried out；It is then based on the task creation subtask scheduling database model for dividing.
The task model of complete set independence is established for customer information, parallel processing is realized between client, in a set of client
Point storehouse mapping for arriving database is set up by dividing task dependence in treatment application, task dependence task is carried out serial, non-
Dependence task parallel processing flow, has saved process time, farthest make use of system resource.
As shown in figure 5, being a kind of Third-party payment supervisory systems distributed data method for stream processing reality disclosed by the invention
The flow chart of example 5 is applied, the method is comprised the following steps：
S501, the detailed data for receiving and parsing through big data quantity, preliminary treatment is carried out to detailed data；
S502, the report data for reception under pretreated detailed or report mode under detailed pattern carry out data hook
Check and parse；
When needing to process Third-party payment supervisory systems distributed traffic, the demand first to user is carried out
Parsing, the demand according to user sets up Business Processing mould respectively for deposit pipe and the report mode of Third-party payment supervisory systems
Type.Specifically, receiving and parsing through the detailed data of big data quantity, preliminary treatment is carried out to detailed data, under detailed pattern
The report data received under pretreated detailed or report mode carries out data hook and checks and parse.
S503, the numbering that uniform rules is set up for client, by supervised entities' account carry under customer number；
It is S504, detailed for the substantially even distribution of premise of account based on client, by hash algorithm to customer number and
Account generates Hash result；
S505, mole value is measured to database data after the detailed data of customer accounting code is stored in database reference numeral
In isomorphism point storehouse, database equally loaded is completed；
After the Business Processing model combing that has built up goes out focus flow, and then set up data control model and carry out heat
Point flow processing.Specifically, the numbering of uniform rules is set up for client, by supervised entities' account carry in customer number
Under；It is detailed for the substantially even distribution of premise of account based on client, customer number and account are generated by hash algorithm is breathed out
Uncommon result；The isomorphism that the detailed data of customer accounting code is stored in database reference numeral is divided after mole value is measured to database data
In storehouse, database equally loaded is completed.
S506, carry out flow chart of data processing dependence task division；
Upon receipt of a service request, obtain file and decompress the pretreatment verification self-service task returning result of reservation, then
Self-service task is registered, starts basic data processing, table of being delivered newspaper in verification contract information parsing verifies form and cell legitimacy
Data loading, registers Third-party payment batch tasks, registers Third-party payment batch tasks data loading, registers Third-party payment
Batch tasks, poll plays task process data, by data loading, generates other forms, and generation hook checks check results.
S507, the task creation subtask scheduling database model based on division.
To task creation database model, each client sets up an identification information for unique binding customer number, root
Main task, and the carry subtask under main task are set up according to task partitioning model, is numbered by task instances and is stored in database, and
By autotask polling mechanism, parallel calling is carried out for the flow set up in above-mentioned data model and data processing model,
To reach, different clients are parallel, non-dependent tasks in parallel purpose.
In sum, in the above-described embodiments, when needing at Third-party payment supervisory systems distributed traffic
During reason, user's request is parsed first, deposit pipe, the report mode for Third-party payment supervisory systems set up Business Processing mould respectively
Type；The Business Processing model combing for being then based on setting up goes out focus flow, and setting up data control model carries out focus flow processing；
Then flow chart of data processing dependence task division is carried out；It is then based on the task creation subtask scheduling database model for dividing.
The task model of complete set independence is established for customer information, parallel processing is realized between client, in a set of client
Point storehouse mapping for arriving database is set up by dividing task dependence in treatment application, task dependence task is carried out serial, non-
Dependence task parallel processing flow, has saved process time, farthest make use of system resource.
As shown in fig. 6, being a kind of Third-party payment supervisory systems distributed data stream processing system reality disclosed by the invention
The structural representation of example 1 is applied, the system includes：
First sets up module 601, for parsing user's request, pipe, form mould is deposited for Third-party payment supervisory systems
Formula sets up Business Processing model respectively；
When needing to process Third-party payment supervisory systems distributed traffic, the demand first to user is carried out
Parsing, the demand according to user sets up Business Processing mould respectively for deposit pipe and the report mode of Third-party payment supervisory systems
Type.
Second sets up module 602, for going out focus flow based on the Business Processing model combing set up, sets up data control
Model carries out focus flow processing；
After the Business Processing model combing that has built up goes out focus flow, and then set up data control model and carry out heat
Point flow processing.
Data processing module 603, for carrying out flow chart of data processing dependence task division；
Upon receipt of a service request, obtain file and decompress the pretreatment verification self-service task returning result of reservation, then
Self-service task is registered, starts basic data processing, table of being delivered newspaper in verification contract information parsing verifies form and cell legitimacy
Data loading, registers Third-party payment batch tasks, registers Third-party payment batch tasks data loading, registers Third-party payment
Batch tasks, poll plays task process data, by data loading, generates other forms, and generation hook checks check results.
3rd sets up module 604, for based on the task creation subtask scheduling database model for dividing.
To task creation database model, each client sets up an identification information for unique binding customer number, root
Main task, and the carry subtask under main task are set up according to task partitioning model, is numbered by task instances and is stored in database, and
By autotask polling mechanism, parallel calling is carried out for the flow set up in above-mentioned data model and data processing model,
To reach, different clients are parallel, non-dependent tasks in parallel purpose.
In sum, in the above-described embodiments, when needing at Third-party payment supervisory systems distributed traffic
During reason, user's request is parsed first, deposit pipe, the report mode for Third-party payment supervisory systems set up Business Processing mould respectively
Type；The Business Processing model combing for being then based on setting up goes out focus flow, and setting up data control model carries out focus flow processing；
Then flow chart of data processing dependence task division is carried out；It is then based on the task creation subtask scheduling database model for dividing.
The task model of complete set independence is established for customer information, parallel processing is realized between client, in a set of client
Point storehouse mapping for arriving database is set up by dividing task dependence in treatment application, task dependence task is carried out serial, non-
Dependence task parallel processing flow, has saved process time, farthest make use of system resource.
As shown in fig. 7, being a kind of Third-party payment supervisory systems distributed data stream processing system reality disclosed by the invention
The structural representation of example 2 is applied, the system includes：
Division unit 701, for merging by operation flow passage, obtains data processing basic procedure model, marks off
The focus and bottleneck of data processing；
When needing to process Third-party payment supervisory systems distributed traffic, the demand first to user is carried out
Parsing, the demand according to user sets up Business Processing mould respectively for deposit pipe and the report mode of Third-party payment supervisory systems
Type.Specifically, merging by operation flow passage, data processing basic procedure model is obtained, mark off the focus of data processing
And bottleneck.
Second sets up module 702, for going out focus flow based on the Business Processing model combing set up, sets up data control
Model carries out focus flow processing；
After the Business Processing model combing that has built up goes out focus flow, and then set up data control model and carry out heat
Point flow processing.
Data processing module 703, for carrying out flow chart of data processing dependence task division；
Upon receipt of a service request, obtain file and decompress the pretreatment verification self-service task returning result of reservation, then
Self-service task is registered, starts basic data processing, table of being delivered newspaper in verification contract information parsing verifies form and cell legitimacy
Data loading, registers Third-party payment batch tasks, registers Third-party payment batch tasks data loading, registers Third-party payment
Batch tasks, poll plays task process data, by data loading, generates other forms, and generation hook checks check results.
3rd sets up module 704, for based on the task creation subtask scheduling database model for dividing.
To task creation database model, each client sets up an identification information for unique binding customer number, root
Main task, and the carry subtask under main task are set up according to task partitioning model, is numbered by task instances and is stored in database, and
By autotask polling mechanism, parallel calling is carried out for the flow set up in above-mentioned data model and data processing model,
To reach, different clients are parallel, non-dependent tasks in parallel purpose.
In sum, in the above-described embodiments, when needing at Third-party payment supervisory systems distributed traffic
During reason, user's request is parsed first, deposit pipe, the report mode for Third-party payment supervisory systems set up Business Processing mould respectively
Type；The Business Processing model combing for being then based on setting up goes out focus flow, and setting up data control model carries out focus flow processing；
Then flow chart of data processing dependence task division is carried out；It is then based on the task creation subtask scheduling database model for dividing.
The task model of complete set independence is established for customer information, parallel processing is realized between client, in a set of client
Point storehouse mapping for arriving database is set up by dividing task dependence in treatment application, task dependence task is carried out serial, non-
Dependence task parallel processing flow, has saved process time, farthest make use of system resource.
As shown in figure 8, being a kind of Third-party payment supervisory systems distributed data stream processing system reality disclosed by the invention
The structural representation of example 3 is applied, the system includes：
Preliminary treatment subelement 801, the detailed data for receiving and parsing through big data quantity, to detailed data preparation
Treatment；
Data processing subelement 802, for for reception under pretreated detailed or report mode under detailed pattern
Report data carries out data hook and checks and parse；
When needing to process Third-party payment supervisory systems distributed traffic, the demand first to user is carried out
Parsing, the demand according to user sets up Business Processing mould respectively for deposit pipe and the report mode of Third-party payment supervisory systems
Type.Specifically, receiving and parsing through the detailed data of big data quantity, preliminary treatment is carried out to detailed data, under detailed pattern
The report data received under pretreated detailed or report mode carries out data hook and checks and parse.
Second sets up module 803, for going out focus flow based on the Business Processing model combing set up, sets up data control
Model carries out focus flow processing；
After the Business Processing model combing that has built up goes out focus flow, and then set up data control model and carry out heat
Point flow processing.
Data processing module 804, for carrying out flow chart of data processing dependence task division；
Upon receipt of a service request, obtain file and decompress the pretreatment verification self-service task returning result of reservation, then
Self-service task is registered, starts basic data processing, table of being delivered newspaper in verification contract information parsing verifies form and cell legitimacy
Data loading, registers Third-party payment batch tasks, registers Third-party payment batch tasks data loading, registers Third-party payment
Batch tasks, poll plays task process data, by data loading, generates other forms, and generation hook checks check results.
3rd sets up module 805, for based on the task creation subtask scheduling database model for dividing.
To task creation database model, each client sets up an identification information for unique binding customer number, root
Main task, and the carry subtask under main task are set up according to task partitioning model, is numbered by task instances and is stored in database, and
By autotask polling mechanism, parallel calling is carried out for the flow set up in above-mentioned data model and data processing model,
To reach, different clients are parallel, non-dependent tasks in parallel purpose.
In sum, in the above-described embodiments, when needing at Third-party payment supervisory systems distributed traffic
During reason, user's request is parsed first, deposit pipe, the report mode for Third-party payment supervisory systems set up Business Processing mould respectively
Type；The Business Processing model combing for being then based on setting up goes out focus flow, and setting up data control model carries out focus flow processing；
Then flow chart of data processing dependence task division is carried out；It is then based on the task creation subtask scheduling database model for dividing.
The task model of complete set independence is established for customer information, parallel processing is realized between client, in a set of client
Point storehouse mapping for arriving database is set up by dividing task dependence in treatment application, task dependence task is carried out serial, non-
Dependence task parallel processing flow, has saved process time, farthest make use of system resource.
As shown in figure 9, being a kind of Third-party payment supervisory systems distributed data stream processing system reality disclosed by the invention
The structural representation of example 4 is applied, the system includes：
Preliminary treatment subelement 901, the detailed data for receiving and parsing through big data quantity, to detailed data preparation
Treatment；
Data processing subelement 902, for for reception under pretreated detailed or report mode under detailed pattern
Report data carries out data hook and checks and parse；
When needing to process Third-party payment supervisory systems distributed traffic, the demand first to user is carried out
Parsing, the demand according to user sets up Business Processing mould respectively for deposit pipe and the report mode of Third-party payment supervisory systems
Type.Specifically, receiving and parsing through the detailed data of big data quantity, preliminary treatment is carried out to detailed data, under detailed pattern
The report data received under pretreated detailed or report mode carries out data hook and checks and parse.
Subelement 903 is set up, for carrying out cutting to different clients account by hash algorithm to customer information, and is set up
To the mapping relations of database point library storage, data model is set up；
After the Business Processing model combing that has built up goes out focus flow, and then set up data control model and carry out heat
Point flow processing.Specifically, carrying out cutting to different clients account by hash algorithm to customer information, and set up to database
Divide the mapping relations of library storage, set up data model
Data processing module 904, for carrying out flow chart of data processing dependence task division；
Upon receipt of a service request, obtain file and decompress the pretreatment verification self-service task returning result of reservation, then
Self-service task is registered, starts basic data processing, table of being delivered newspaper in verification contract information parsing verifies form and cell legitimacy
Data loading, registers Third-party payment batch tasks, registers Third-party payment batch tasks data loading, registers Third-party payment
Batch tasks, poll plays task process data, by data loading, generates other forms, and generation hook checks check results.
3rd sets up module 905, for based on the task creation subtask scheduling database model for dividing.
To task creation database model, each client sets up an identification information for unique binding customer number, root
Main task, and the carry subtask under main task are set up according to task partitioning model, is numbered by task instances and is stored in database, and
By autotask polling mechanism, parallel calling is carried out for the flow set up in above-mentioned data model and data processing model,
To reach, different clients are parallel, non-dependent tasks in parallel purpose.
In sum, in the above-described embodiments, when needing at Third-party payment supervisory systems distributed traffic
During reason, user's request is parsed first, deposit pipe, the report mode for Third-party payment supervisory systems set up Business Processing mould respectively
Type；The Business Processing model combing for being then based on setting up goes out focus flow, and setting up data control model carries out focus flow processing；
Then flow chart of data processing dependence task division is carried out；It is then based on the task creation subtask scheduling database model for dividing.
The task model of complete set independence is established for customer information, parallel processing is realized between client, in a set of client
Point storehouse mapping for arriving database is set up by dividing task dependence in treatment application, task dependence task is carried out serial, non-
Dependence task parallel processing flow, has saved process time, farthest make use of system resource.
As shown in Figure 10, it is a kind of Third-party payment supervisory systems distributed data stream processing system reality disclosed by the invention
The structural representation of example 5 is applied, the system is comprised the following steps：
Preliminary treatment subelement 1001, the detailed data for receiving and parsing through big data quantity is carried out pre- to detailed data
Standby treatment；
Data processing subelement 1002, for for reception under pretreated detailed or report mode under detailed pattern
Report data carries out data hook and checks and parse；
When needing to process Third-party payment supervisory systems distributed traffic, the demand first to user is carried out
Parsing, the demand according to user sets up Business Processing mould respectively for deposit pipe and the report mode of Third-party payment supervisory systems
Type.Specifically, receiving and parsing through the detailed data of big data quantity, preliminary treatment is carried out to detailed data, under detailed pattern
The report data received under pretreated detailed or report mode carries out data hook and checks and parse.
Carry subelement 1003, the numbering for setting up uniform rules for client exists supervised entities' account carry
Under customer number；
Generation subelement 1004, for detailed for the substantially even distribution of premise of account based on client, is calculated by Hash
Method is to customer number and account generation Hash result；
Be stored in subelement 1005, for measuring mole value to database data after the detailed data of customer accounting code is stored in number
According to the isomorphism point storehouse of storehouse reference numeral, database equally loaded is completed；
After the Business Processing model combing that has built up goes out focus flow, and then set up data control model and carry out heat
Point flow processing.Specifically, the numbering of uniform rules is set up for client, by supervised entities' account carry in customer number
Under；It is detailed for the substantially even distribution of premise of account based on client, customer number and account are generated by hash algorithm is breathed out
Uncommon result；The isomorphism that the detailed data of customer accounting code is stored in database reference numeral is divided after mole value is measured to database data
In storehouse, database equally loaded is completed.
Data processing module 1006, for carrying out flow chart of data processing dependence task division；
Upon receipt of a service request, obtain file and decompress the pretreatment verification self-service task returning result of reservation, then
Self-service task is registered, starts basic data processing, table of being delivered newspaper in verification contract information parsing verifies form and cell legitimacy
Data loading, registers Third-party payment batch tasks, registers Third-party payment batch tasks data loading, registers Third-party payment
Batch tasks, poll plays task process data, by data loading, generates other forms, and generation hook checks check results.
3rd sets up module 1007, for based on the task creation subtask scheduling database model for dividing.
To task creation database model, each client sets up an identification information for unique binding customer number, root
Main task, and the carry subtask under main task are set up according to task partitioning model, is numbered by task instances and is stored in database, and
By autotask polling mechanism, parallel calling is carried out for the flow set up in above-mentioned data model and data processing model,
To reach, different clients are parallel, non-dependent tasks in parallel purpose.
In sum, in the above-described embodiments, when needing at Third-party payment supervisory systems distributed traffic
During reason, user's request is parsed first, deposit pipe, the report mode for Third-party payment supervisory systems set up Business Processing mould respectively
Type；The Business Processing model combing for being then based on setting up goes out focus flow, and setting up data control model carries out focus flow processing；
Then flow chart of data processing dependence task division is carried out；It is then based on the task creation subtask scheduling database model for dividing.
The task model of complete set independence is established for customer information, parallel processing is realized between client, in a set of client
Point storehouse mapping for arriving database is set up by dividing task dependence in treatment application, task dependence task is carried out serial, non-
Dependence task parallel processing flow, has saved process time, farthest make use of system resource.
If function described in the present embodiment method is to realize in the form of SFU software functional unit and as independent product pin
When selling or using, can store in a computing device read/write memory medium.Based on such understanding, the embodiment of the present invention
The part contributed to prior art or the part of the technical scheme can be embodied in the form of software product, and this is soft
Part product is stored in a storage medium, including some instructions are used to so that computing device (can be personal computer,
Server, mobile computing device or network equipment etc.) perform all or part of step of each embodiment methods described of the invention
Suddenly.And foregoing storage medium includes：USB flash disk, mobile hard disk, read-only storage (ROM, Read-Only Memory), deposit at random
Access to memory (RAM, Random Access Memory), magnetic disc or CD etc. are various can be with the medium of store program codes.
Each embodiment is described by the way of progressive in this specification, and what each embodiment was stressed is and other
The difference of embodiment, between each embodiment same or similar part mutually referring to.
The foregoing description of the disclosed embodiments, enables professional and technical personnel in the field to realize or uses the present invention.
Various modifications to these embodiments will be apparent for those skilled in the art, as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the present invention.Therefore, the present invention
The embodiments shown herein is not intended to be limited to, and is to fit to and principles disclosed herein and features of novelty phase one
The scope most wide for causing.
Claims (10)
1. a kind of Third-party payment supervisory systems distributed data method for stream processing, it is characterised in that including：
Parsing user's request, deposit pipe, report mode for Third-party payment supervisory systems set up Business Processing model respectively；
Business Processing model combing based on the foundation goes out focus flow, and setting up data control model is carried out at focus flow
Reason；
Carry out flow chart of data processing dependence task division；
Based on the task creation subtask scheduling database model for dividing.
2. method according to claim 1, it is characterised in that the parsing user's request, for Third-party payment supervision
Deposit pipe, the report mode of system are set up Business Processing model and are included respectively：
Merged by operation flow passage, obtain data processing basic procedure model, mark off the focus and bottleneck of data processing
Link.
3. method according to claim 2, it is characterised in that described to be merged by operation flow passage, is obtained at data
Reason basic procedure model, the focus and ring collar for marking off data processing includes：
The detailed data of big data quantity is received and parsed through, preliminary treatment is carried out to detailed data；
Data are carried out for the report data received under pretreated detailed or report mode under detailed pattern and hook checking and parsing.
4. the method according to any one in claim 1-3, it is characterised in that at the business based on the foundation
Reason model combing goes out focus flow, and setting up data control model and carrying out focus flow processing includes：
Cutting is carried out to different clients account by hash algorithm to customer information, and sets up the mapping to database point library storage
Relation, sets up data model.
5. method according to claim 4, it is characterised in that it is described to customer information by hash algorithm to different clients
Account carries out cutting, and sets up the mapping relations for thinking database point library storage, and setting up data model includes：
The numbering of uniform rules is set up for client, by supervised entities' account carry under customer number；
It is detailed for the substantially even distribution of premise of account based on client, customer number and account are generated by hash algorithm is breathed out
Uncommon result；
The isomorphism point storehouse that the detailed data of customer accounting code is stored in after mole value database reference numeral is measured to database data
In, complete database equally loaded.
6. a kind of Third-party payment supervisory systems distributed data stream processing system, it is characterised in that including：
First sets up module, and for parsing user's request, deposit pipe, report mode for Third-party payment supervisory systems are built respectively
Vertical Business Processing model；
Second sets up module, and focus flow is gone out for the Business Processing model combing based on the foundation, sets up data control mould
Type carries out focus flow processing；
Data processing module, for carrying out flow chart of data processing dependence task division；
3rd sets up module, for based on the task creation subtask scheduling database model for dividing.
7. system according to claim 6, it is characterised in that described first sets up module includes：
Division unit, for merging by operation flow passage, obtains data processing basic procedure model, marks off data processing
Focus and bottleneck.
8. system according to claim 7, it is characterised in that the division unit includes：
Preliminary treatment subelement, the detailed data for receiving and parsing through big data quantity, preliminary treatment is carried out to detailed data；
Data processing subelement, for for the report data received under pretreated detailed or report mode under detailed pattern
Data hook is carried out to check and parse.
9. the system according to any one in claim 6-8, it is characterised in that described second sets up module includes：
Subelement is set up, for carrying out cutting to different clients account by hash algorithm to customer information, and is set up to data
The mapping relations of storehouse point library storage, set up data model.
10. system according to claim 9, it is characterised in that the subelement of setting up includes：
Carry subelement, the numbering for setting up uniform rules for client, by supervised entities' account carry in customer number
Under；
Generation subelement, for detailed for the substantially even distribution of premise of account based on client, by hash algorithm to client
Numbering and account generate Hash result；
Subelement is stored in, the detailed data of customer accounting code is stored in database correspondence after mole value for being measured to database data
In the isomorphism of numbering point storehouse, database equally loaded is completed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611226657.5A CN106845926B (en) | 2016-12-27 | 2016-12-27 | Distributed data stream processing method and system for third-party payment supervision system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611226657.5A CN106845926B (en) | 2016-12-27 | 2016-12-27 | Distributed data stream processing method and system for third-party payment supervision system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106845926A true CN106845926A (en) | 2017-06-13 |
| CN106845926B CN106845926B (en) | 2020-11-27 |
Family
ID=59136514
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611226657.5A Active CN106845926B (en) | 2016-12-27 | 2016-12-27 | Distributed data stream processing method and system for third-party payment supervision system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106845926B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109800965A (en) * | 2018-12-28 | 2019-05-24 | 广州亚信技术有限公司 | Traffic flow processing method, device, computer equipment and storage medium |
| CN114064231A (en) * | 2021-10-13 | 2022-02-18 | 北京市应急管理科学技术研究院 | Task configuration method, system, storage medium and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102508639A (en) * | 2011-10-10 | 2012-06-20 | 北京邮电大学 | Distributed parallel processing method based on satellite remote sensing data characteristics |
| US20120317579A1 (en) * | 2011-06-13 | 2012-12-13 | Huan Liu | System and method for performing distributed parallel processing tasks in a spot market |
| CN104156380A (en) * | 2014-03-04 | 2014-11-19 | 深圳信息职业技术学院 | Distributed memory Hash indexing method and system |
| CN104239141A (en) * | 2014-09-05 | 2014-12-24 | 北京邮电大学 | Task optimized-scheduling method in data center on basis of critical paths of workflow |
| US20150178126A1 (en) * | 2013-12-23 | 2015-06-25 | International Business Machines Corporation | Backfill scheduling for embarrassingly parallel jobs |
| CN105869048A (en) * | 2016-03-28 | 2016-08-17 | 中国建设银行股份有限公司 | Data processing method and system |
-
2016
- 2016-12-27 CN CN201611226657.5A patent/CN106845926B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120317579A1 (en) * | 2011-06-13 | 2012-12-13 | Huan Liu | System and method for performing distributed parallel processing tasks in a spot market |
| CN102508639A (en) * | 2011-10-10 | 2012-06-20 | 北京邮电大学 | Distributed parallel processing method based on satellite remote sensing data characteristics |
| US20150178126A1 (en) * | 2013-12-23 | 2015-06-25 | International Business Machines Corporation | Backfill scheduling for embarrassingly parallel jobs |
| CN104156380A (en) * | 2014-03-04 | 2014-11-19 | 深圳信息职业技术学院 | Distributed memory Hash indexing method and system |
| CN104239141A (en) * | 2014-09-05 | 2014-12-24 | 北京邮电大学 | Task optimized-scheduling method in data center on basis of critical paths of workflow |
| CN105869048A (en) * | 2016-03-28 | 2016-08-17 | 中国建设银行股份有限公司 | Data processing method and system |
Non-Patent Citations (1)
| Title |
|---|
| 童维勤，黄林鹏: "《数据密集型计算和模型》", 31 December 2015 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109800965A (en) * | 2018-12-28 | 2019-05-24 | 广州亚信技术有限公司 | Traffic flow processing method, device, computer equipment and storage medium |
| CN114064231A (en) * | 2021-10-13 | 2022-02-18 | 北京市应急管理科学技术研究院 | Task configuration method, system, storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106845926B (en) | 2020-11-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110163474A (en) | A kind of method and apparatus of task distribution | |
| US20140289286A1 (en) | System and method for performance tuning of garbage collection algorithms | |
| CN110838065A (en) | Transaction data processing method and device | |
| CN111400344B (en) | Electronic bill printing method, device, equipment and storage medium | |
| CN110602056B (en) | Service parameter transmission method and device | |
| CN112306719B (en) | Task scheduling method and device | |
| CN108228363A (en) | A kind of message method and device | |
| CN110191160A (en) | A kind of concurrency control method and device | |
| CN110417888A (en) | Flow control methods, volume control device and electronic equipment | |
| CN110221914B (en) | File processing method and device | |
| CN111562893A (en) | Reprinting method and device | |
| CN106845926A (en) | A kind of Third-party payment supervisory systems distributed data method for stream processing and system | |
| CN111681071A (en) | Sub-cost data generation system and method, storage medium, and electronic device | |
| CN109286617B (en) | Data processing method and related equipment | |
| CN116055052A (en) | Block chain-based data processing method, device, equipment and readable storage medium | |
| CN111899018B (en) | Order data processing method, device, server and storage medium | |
| CN109117295A (en) | A kind of overtime monitoring method of transaction and device | |
| CN110083501A (en) | Interface calls method of counting and device | |
| CN110858211B (en) | Data storage method, device and system and storage medium | |
| CN103605480B (en) | Web server and disk resource access control method thereof | |
| CN113630451B (en) | Computing service system based on block chain and spark | |
| CN111292142A (en) | Billing method and device, tax control equipment and storage medium | |
| CN112000657A (en) | Data management method, device, server and storage medium | |
| CN113132114B (en) | Method, device, medium and equipment for realizing multi-cloud-pipe unified interface gateway | |
| CN114422496A (en) | Mass traffic data processing system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |