CN106845926B - Distributed data stream processing method and system for third-party payment supervision system - Google Patents
Distributed data stream processing method and system for third-party payment supervision system Download PDFInfo
- Publication number
- CN106845926B CN106845926B CN201611226657.5A CN201611226657A CN106845926B CN 106845926 B CN106845926 B CN 106845926B CN 201611226657 A CN201611226657 A CN 201611226657A CN 106845926 B CN106845926 B CN 106845926B
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- establishing
- task
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/103—Workflow collaboration or project management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
Landscapes
- Business, Economics & Management (AREA)
- Engineering & Computer Science (AREA)
- Strategic Management (AREA)
- Human Resources & Organizations (AREA)
- Accounting & Taxation (AREA)
- General Physics & Mathematics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Finance (AREA)
- Entrepreneurship & Innovation (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Data Mining & Analysis (AREA)
- Operations Research (AREA)
- Development Economics (AREA)
- Technology Law (AREA)
- Computer And Data Communications (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a distributed data stream processing method of a third-party payment supervision system, which comprises the following steps: analyzing the user requirements, and respectively establishing a service processing model aiming at the storage and management and report mode of a third-party payment supervision system; combing out a hot flow based on the established business processing model, and establishing a data control model to perform hot flow processing; dividing a data processing flow dependent task; and establishing a subtask scheduling database model based on the divided tasks. The invention can improve the utilization rate of system resources. The invention also discloses a distributed data stream processing system of the third-party payment supervision system.
Description
Technical Field
The invention relates to the technical field of bank management, in particular to a distributed data stream processing method and system of a third-party payment supervision system.
Background
Currently, when complex data stream processing tasks are processed, a top-down full-flow serial processing method is used. A set of threads is established for applying for a user for a processing request, and the complex processing request is processed in series from top to bottom. The method adopted in the prior art cannot meet the processing requirements of a large amount of data and short time. When the system faces multi-user request, it builds queue mechanism by locking resource to process all client request in series. This leads to the possible results: the waiting time of the client is long, all the resources of the system need to be evaluated and prepared according to the response request with the maximum data volume, and the idle state is generated in most of the time, so that huge waste is caused to the system resources. Therefore, how to improve the utilization rate of system resources to the maximum extent is an urgent problem to be solved.
Disclosure of Invention
The invention provides a distributed data stream processing method of a third-party payment supervision system, which can improve the utilization rate of system resources.
The invention provides a distributed data stream processing method of a third-party payment supervision system, which comprises the following steps:
analyzing the user requirements, and respectively establishing a service processing model aiming at the storage and management and report mode of a third-party payment supervision system;
combing out a hot flow based on the established business processing model, and establishing a data control model to process the hot flow;
dividing a data processing flow dependent task;
and establishing a subtask scheduling database model based on the divided tasks.
Preferably, the analyzing the user requirement, and the respectively establishing a service processing model for the storage and management and reporting mode of the third-party payment supervision system includes:
and merging the service process channels to obtain a basic data processing process model, and dividing hot spots and bottleneck links of data processing.
Preferably, the obtaining of the basic flow model for data processing by merging the service flow channels, and the dividing of the hot spot and the bottleneck ring for data processing includes:
receiving and analyzing detailed data with large data volume, and performing preparation processing on the detailed data;
and performing data checking and analysis on the preprocessed detail in the detail mode or the report data received in the report mode.
Preferably, the step of combing out a hot spot flow based on the established business processing model, and the step of establishing a data control model to perform hot spot flow processing includes:
and segmenting the client information into different client accounts through a Hash algorithm, establishing a mapping relation stored in a database in a sub-database mode, and establishing a data model.
Preferably, the segmenting the customer information by the hash algorithm for different customer accounts, and establishing a mapping relation for database sublibrary storage, and the establishing a data model includes:
establishing a uniform rule number for a client system, and mounting a monitoring object account under the client number;
on the premise that the account numbers are basically and evenly distributed according to the customer details, generating a hash result for the customer numbers and the account numbers through a hash algorithm;
and after the mole value of the database data is measured, the detailed data of the client account is stored in the isomorphic sublibrary with the corresponding serial number of the database, and the balanced load of the database is completed.
A third party payment supervision system distributed data stream processing system, comprising:
the first establishing module is used for analyzing the user requirements and respectively establishing a service processing model aiming at the storage and management and report modes of the third-party payment supervision system;
the second establishing module is used for combing a hot spot flow based on the established business processing model and establishing a data control model for hot spot flow processing;
the data processing module is used for dividing the data processing flow dependent tasks;
and the third establishing module is used for establishing a subtask scheduling database model based on the divided tasks.
Preferably, the first establishing module comprises:
and the dividing unit is used for obtaining a basic data processing flow model through the merging of the service flow channels and dividing hot spots and bottleneck links of data processing.
Preferably, the dividing unit includes:
the preparation processing subunit is used for receiving and analyzing the detail data with large data volume and performing preparation processing on the detail data;
and the data processing subunit is used for performing data auditing and analysis on the preprocessed details in the detail mode or the report data received in the report mode.
Preferably, the second establishing module includes:
and the establishing subunit is used for segmenting the client information into different client accounts through a Hash algorithm, establishing a mapping relation stored in a database in a sub-database mode and establishing a data model.
Preferably, the establishing subunit comprises:
the mounting subunit is used for establishing a uniform rule number for the client system and mounting the account number of the monitoring object under the client number;
the generation subunit is used for generating a hash result for the customer number and the account number through a hash algorithm on the premise that the customer details are basically and evenly distributed to the account number;
and the storing subunit is used for storing the detailed data of the customer account into the isomorphic sublibraries with corresponding numbers of the database after the mole value of the database data is measured, so as to complete the balanced load of the database.
According to the scheme, when the distributed data stream of the third-party payment supervision system needs to be processed, the user requirements are analyzed, and a business processing model is respectively established according to the storage and management and report modes of the third-party payment supervision system; then combing out a hot flow based on the established business processing model, and establishing a data control model to carry out hot flow processing; then, dividing a data processing flow dependent task; and then establishing a subtask scheduling database model based on the divided tasks. A set of complete and independent task models are established for client information, parallel processing is achieved among clients, sub-library mapping from a database is established by dividing task dependency relationships in a set of client processing applications, and task dependent task serial and independent task parallel processing flows are performed, so that processing time is saved, and system resources are utilized to the maximum extent.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of an embodiment 1 of a distributed data stream processing method of a third-party payment supervision system according to the present invention;
FIG. 2 is a flowchart of an embodiment 2 of a distributed data stream processing method for a third-party payment supervision system according to the present invention;
FIG. 3 is a flowchart of an embodiment 3 of a distributed data stream processing method for a third-party payment supervision system according to the present invention;
FIG. 4 is a flowchart of an embodiment 4 of a distributed data stream processing method of a third-party payment supervision system according to the present invention;
FIG. 5 is a flowchart of an embodiment 5 of a distributed data stream processing method for a third-party payment supervision system according to the present invention;
fig. 6 is a schematic structural diagram of a distributed data stream processing system 1 of a third-party payment supervision system according to an embodiment of the present invention;
FIG. 7 is a schematic structural diagram of a distributed data stream processing system of a third-party payment supervision system according to an embodiment 2 of the present disclosure;
FIG. 8 is a schematic structural diagram of an embodiment 3 of a distributed data stream processing system of a third-party payment supervision system according to the present invention;
FIG. 9 is a schematic structural diagram of an embodiment 4 of a distributed data stream processing system of a third-party payment supervision system according to the present invention;
fig. 10 is a schematic structural diagram of a distributed data stream processing system of a third party payment supervision system according to an embodiment 5 of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1, the method is a flowchart of an embodiment 1 of a distributed data stream processing method of a third party payment supervision system, and the method includes the following steps:
s101, analyzing user requirements, and respectively establishing a service processing model aiming at a storage management mode and a report mode of a third-party payment supervision system;
when the distributed data stream of the third-party payment supervision system needs to be processed, firstly, the requirements of the user are analyzed, and a business processing model is respectively established according to the requirements of the user aiming at the storage and management and report modes of the third-party payment supervision system.
S102, combing out a hot spot flow based on the established business processing model, and establishing a data control model to process the hot spot flow;
and after the hot flow is combed out through the established business processing model, a data control model is further established to carry out hot flow processing.
S103, dividing a data processing flow dependent task;
after receiving a service request, acquiring a file, decompressing a preprocessing check reservation self-service task return result, then registering a self-service task, starting basic data processing, analyzing and uploading check contract information, checking a report form and warehousing unit cell validity data, registering a third party payment batch task data warehousing, registering a third party payment batch task, polling starting task processing data, warehousing the data, generating other reports, and generating a check result.
And S104, establishing a subtask scheduling database model based on the divided tasks.
Establishing a database model for the task, establishing identification information for each client to uniquely bind a client number, establishing a main task according to a task division model, mounting a subtask under the main task, storing the subtask into the database through the task instance number, and calling the flows established in the data model and the data processing model in parallel through an automatic task polling mechanism so as to achieve the aim of parallel and independent task parallel of different clients.
In summary, in the above embodiments, when the distributed data stream of the third party payment monitoring system needs to be processed, the user requirements are firstly analyzed, and a service processing model is respectively established for the storage and management and reporting modes of the third party payment monitoring system; then combing out a hot flow based on the established business processing model, and establishing a data control model to carry out hot flow processing; then, dividing a data processing flow dependent task; and then establishing a subtask scheduling database model based on the divided tasks. A set of complete and independent task models are established for client information, parallel processing is achieved among clients, sub-library mapping from a database is established by dividing task dependency relationships in a set of client processing applications, and task dependent task serial and independent task parallel processing flows are performed, so that processing time is saved, and system resources are utilized to the maximum extent.
As shown in fig. 2, which is a flowchart of an embodiment 2 of a distributed data stream processing method of a third party payment supervision system, the method includes the following steps:
s201, merging through service process channels to obtain a basic data processing process model, and dividing hot points and bottleneck links of data processing;
when the distributed data stream of the third-party payment supervision system needs to be processed, firstly, the requirements of the user are analyzed, and a business processing model is respectively established according to the requirements of the user aiming at the storage and management and report modes of the third-party payment supervision system. Specifically, a basic data processing flow model is obtained through service flow channel combination, and hot spots and bottleneck links of data processing are divided.
S202, combing out a hot spot flow based on the established business processing model, and establishing a data control model to process the hot spot flow;
and after the hot flow is combed out through the established business processing model, a data control model is further established to carry out hot flow processing.
S203, dividing the data processing flow dependent tasks;
after receiving a service request, acquiring a file, decompressing a preprocessing check reservation self-service task return result, then registering a self-service task, starting basic data processing, analyzing and uploading check contract information, checking a report form and warehousing unit cell validity data, registering a third party payment batch task data warehousing, registering a third party payment batch task, polling starting task processing data, warehousing the data, generating other reports, and generating a check result.
And S204, establishing a subtask scheduling database model based on the divided tasks.
Establishing a database model for the task, establishing identification information for each client to uniquely bind a client number, establishing a main task according to a task division model, mounting a subtask under the main task, storing the subtask into the database through the task instance number, and calling the flows established in the data model and the data processing model in parallel through an automatic task polling mechanism so as to achieve the aim of parallel and independent task parallel of different clients.
In summary, in the above embodiments, when the distributed data stream of the third party payment monitoring system needs to be processed, the user requirements are firstly analyzed, and a service processing model is respectively established for the storage and management and reporting modes of the third party payment monitoring system; then combing out a hot flow based on the established business processing model, and establishing a data control model to carry out hot flow processing; then, dividing a data processing flow dependent task; and then establishing a subtask scheduling database model based on the divided tasks. A set of complete and independent task models are established for client information, parallel processing is achieved among clients, sub-library mapping from a database is established by dividing task dependency relationships in a set of client processing applications, and task dependent task serial and independent task parallel processing flows are performed, so that processing time is saved, and system resources are utilized to the maximum extent.
As shown in fig. 3, which is a flowchart of embodiment 3 of a distributed data stream processing method of a third party payment supervision system, the method includes the following steps:
s301, receiving and analyzing detailed data with large data volume, and performing preparation processing on the detailed data;
s302, data auditing and analyzing are carried out on the preprocessed details in the detail mode or the report data received in the report mode;
when the distributed data stream of the third-party payment supervision system needs to be processed, firstly, the requirements of the user are analyzed, and a business processing model is respectively established according to the requirements of the user aiming at the storage and management and report modes of the third-party payment supervision system. Specifically, receiving and analyzing detailed data with large data volume, performing preliminary processing on the detailed data, and performing data check and analysis on the preprocessed detailed data in a detailed mode or the report data received in a report mode.
S303, combing out a hot spot flow based on the established business processing model, and establishing a data control model to process the hot spot flow;
and after the hot flow is combed out through the established business processing model, a data control model is further established to carry out hot flow processing.
S304, dividing the data processing flow dependent tasks;
after receiving a service request, acquiring a file, decompressing a preprocessing check reservation self-service task return result, then registering a self-service task, starting basic data processing, analyzing and uploading check contract information, checking a report form and warehousing unit cell validity data, registering a third party payment batch task data warehousing, registering a third party payment batch task, polling starting task processing data, warehousing the data, generating other reports, and generating a check result.
S305, establishing a subtask scheduling database model based on the divided tasks.
Establishing a database model for the task, establishing identification information for each client to uniquely bind a client number, establishing a main task according to a task division model, mounting a subtask under the main task, storing the subtask into the database through the task instance number, and calling the flows established in the data model and the data processing model in parallel through an automatic task polling mechanism so as to achieve the aim of parallel and independent task parallel of different clients.
In summary, in the above embodiments, when the distributed data stream of the third party payment monitoring system needs to be processed, the user requirements are firstly analyzed, and a service processing model is respectively established for the storage and management and reporting modes of the third party payment monitoring system; then combing out a hot flow based on the established business processing model, and establishing a data control model to carry out hot flow processing; then, dividing a data processing flow dependent task; and then establishing a subtask scheduling database model based on the divided tasks. A set of complete and independent task models are established for client information, parallel processing is achieved among clients, sub-library mapping from a database is established by dividing task dependency relationships in a set of client processing applications, and task dependent task serial and independent task parallel processing flows are performed, so that processing time is saved, and system resources are utilized to the maximum extent.
As shown in fig. 4, which is a flowchart of embodiment 4 of a distributed data stream processing method of a third party payment supervision system, the method includes the following steps:
s401, receiving and analyzing detailed data with large data volume, and performing preparation processing on the detailed data;
s402, performing data auditing and analysis on the preprocessed details in the detail mode or the report data received in the report mode;
when the distributed data stream of the third-party payment supervision system needs to be processed, firstly, the requirements of the user are analyzed, and a business processing model is respectively established according to the requirements of the user aiming at the storage and management and report modes of the third-party payment supervision system. Specifically, receiving and analyzing detailed data with large data volume, performing preliminary processing on the detailed data, and performing data check and analysis on the preprocessed detailed data in a detailed mode or the report data received in a report mode.
S403, segmenting the customer information into different customer accounts through a Hash algorithm, establishing a mapping relation stored in a database in a sub-database mode, and establishing a data model;
and after the hot flow is combed out through the established business processing model, a data control model is further established to carry out hot flow processing. Specifically, the client information is divided into different client accounts through a Hash algorithm, a mapping relation stored in a database in a sub-database mode is established, and a data model is established
S404, dividing the data processing flow dependent tasks;
after receiving a service request, acquiring a file, decompressing a preprocessing check reservation self-service task return result, then registering a self-service task, starting basic data processing, analyzing and uploading check contract information, checking a report form and warehousing unit cell validity data, registering a third party payment batch task data warehousing, registering a third party payment batch task, polling starting task processing data, warehousing the data, generating other reports, and generating a check result.
S405, establishing a subtask scheduling database model based on the divided tasks.
Establishing a database model for the task, establishing identification information for each client to uniquely bind a client number, establishing a main task according to a task division model, mounting a subtask under the main task, storing the subtask into the database through the task instance number, and calling the flows established in the data model and the data processing model in parallel through an automatic task polling mechanism so as to achieve the aim of parallel and independent task parallel of different clients.
In summary, in the above embodiments, when the distributed data stream of the third party payment monitoring system needs to be processed, the user requirements are firstly analyzed, and a service processing model is respectively established for the storage and management and reporting modes of the third party payment monitoring system; then combing out a hot flow based on the established business processing model, and establishing a data control model to carry out hot flow processing; then, dividing a data processing flow dependent task; and then establishing a subtask scheduling database model based on the divided tasks. A set of complete and independent task models are established for client information, parallel processing is achieved among clients, sub-library mapping from a database is established by dividing task dependency relationships in a set of client processing applications, and task dependent task serial and independent task parallel processing flows are performed, so that processing time is saved, and system resources are utilized to the maximum extent.
As shown in fig. 5, which is a flowchart of an embodiment 5 of a distributed data stream processing method of a third party payment supervision system, the method includes the following steps:
s501, receiving and analyzing detailed data with large data volume, and performing preparation processing on the detailed data;
s502, performing data auditing and analysis on the preprocessed details in the detail mode or the report data received in the report mode;
when the distributed data stream of the third-party payment supervision system needs to be processed, firstly, the requirements of the user are analyzed, and a business processing model is respectively established according to the requirements of the user aiming at the storage and management and report modes of the third-party payment supervision system. Specifically, receiving and analyzing detailed data with large data volume, performing preliminary processing on the detailed data, and performing data check and analysis on the preprocessed detailed data in a detailed mode or the report data received in a report mode.
S503, establishing a uniform rule number for the client system, and mounting the account number of the supervision object under the client number;
s504, on the premise that the account numbers are basically and evenly distributed according to the customer details, generating a hash result for the customer numbers and the account numbers through a hash algorithm;
s505, after the mole value of the database data is measured, the detail data of the customer account is stored in the isomorphic sublibrary with the corresponding number of the database, and the balanced load of the database is completed;
and after the hot flow is combed out through the established business processing model, a data control model is further established to carry out hot flow processing. Specifically, a unified rule number is established for a client system, and a monitoring object account is mounted under the client number; on the premise that the account numbers are basically and evenly distributed according to the customer details, generating a hash result for the customer numbers and the account numbers through a hash algorithm; and after the mole value of the database data is measured, the detailed data of the client account is stored in the isomorphic sublibrary with the corresponding serial number of the database, and the balanced load of the database is completed.
S506, dividing the data processing flow dependent tasks;
after receiving a service request, acquiring a file, decompressing a preprocessing check reservation self-service task return result, then registering a self-service task, starting basic data processing, analyzing and uploading check contract information, checking a report form and warehousing unit cell validity data, registering a third party payment batch task data warehousing, registering a third party payment batch task, polling starting task processing data, warehousing the data, generating other reports, and generating a check result.
And S507, establishing a subtask scheduling database model based on the divided tasks.
Establishing a database model for the task, establishing identification information for each client to uniquely bind a client number, establishing a main task according to a task division model, mounting a subtask under the main task, storing the subtask into the database through the task instance number, and calling the flows established in the data model and the data processing model in parallel through an automatic task polling mechanism so as to achieve the aim of parallel and independent task parallel of different clients.
In summary, in the above embodiments, when the distributed data stream of the third party payment monitoring system needs to be processed, the user requirements are firstly analyzed, and a service processing model is respectively established for the storage and management and reporting modes of the third party payment monitoring system; then combing out a hot flow based on the established business processing model, and establishing a data control model to carry out hot flow processing; then, dividing a data processing flow dependent task; and then establishing a subtask scheduling database model based on the divided tasks. A set of complete and independent task models are established for client information, parallel processing is achieved among clients, sub-library mapping from a database is established by dividing task dependency relationships in a set of client processing applications, and task dependent task serial and independent task parallel processing flows are performed, so that processing time is saved, and system resources are utilized to the maximum extent.
As shown in fig. 6, which is a schematic structural diagram of an embodiment 1 of a distributed data stream processing system of a third party payment supervision system disclosed in the present invention, the system includes:
the first establishing module 601 is used for analyzing user requirements and respectively establishing a service processing model aiming at a storage and management mode and a report mode of a third-party payment supervision system;
when the distributed data stream of the third-party payment supervision system needs to be processed, firstly, the requirements of the user are analyzed, and a business processing model is respectively established according to the requirements of the user aiming at the storage and management and report modes of the third-party payment supervision system.
A second establishing module 602, configured to comb out a hot spot flow based on the established service processing model, and establish a data control model to perform hot spot flow processing;
and after the hot flow is combed out through the established business processing model, a data control model is further established to carry out hot flow processing.
A data processing module 603, configured to perform data processing flow dependent task division;
after receiving a service request, acquiring a file, decompressing a preprocessing check reservation self-service task return result, then registering a self-service task, starting basic data processing, analyzing and uploading check contract information, checking a report form and warehousing unit cell validity data, registering a third party payment batch task data warehousing, registering a third party payment batch task, polling starting task processing data, warehousing the data, generating other reports, and generating a check result.
And a third establishing module 604, configured to establish a subtask scheduling database model based on the divided tasks.
Establishing a database model for the task, establishing identification information for each client to uniquely bind a client number, establishing a main task according to a task division model, mounting a subtask under the main task, storing the subtask into the database through the task instance number, and calling the flows established in the data model and the data processing model in parallel through an automatic task polling mechanism so as to achieve the aim of parallel and independent task parallel of different clients.
In summary, in the above embodiments, when the distributed data stream of the third party payment monitoring system needs to be processed, the user requirements are firstly analyzed, and a service processing model is respectively established for the storage and management and reporting modes of the third party payment monitoring system; then combing out a hot flow based on the established business processing model, and establishing a data control model to carry out hot flow processing; then, dividing a data processing flow dependent task; and then establishing a subtask scheduling database model based on the divided tasks. A set of complete and independent task models are established for client information, parallel processing is achieved among clients, sub-library mapping from a database is established by dividing task dependency relationships in a set of client processing applications, and task dependent task serial and independent task parallel processing flows are performed, so that processing time is saved, and system resources are utilized to the maximum extent.
As shown in fig. 7, which is a schematic structural diagram of an embodiment 2 of a distributed data stream processing system of a third party payment supervision system disclosed in the present invention, the system includes:
the dividing unit 701 is configured to obtain a basic data processing flow model by merging the service flow channels, and divide hot spots and bottleneck links of data processing;
when the distributed data stream of the third-party payment supervision system needs to be processed, firstly, the requirements of the user are analyzed, and a business processing model is respectively established according to the requirements of the user aiming at the storage and management and report modes of the third-party payment supervision system. Specifically, a basic data processing flow model is obtained through service flow channel combination, and hot spots and bottleneck links of data processing are divided.
A second establishing module 702, configured to comb out a hot spot flow based on the established service processing model, and establish a data control model to perform hot spot flow processing;
and after the hot flow is combed out through the established business processing model, a data control model is further established to carry out hot flow processing.
The data processing module 703 is configured to perform data processing flow dependent task division;
after receiving a service request, acquiring a file, decompressing a preprocessing check reservation self-service task return result, then registering a self-service task, starting basic data processing, analyzing and uploading check contract information, checking a report form and warehousing unit cell validity data, registering a third party payment batch task data warehousing, registering a third party payment batch task, polling starting task processing data, warehousing the data, generating other reports, and generating a check result.
And a third establishing module 704, configured to establish a subtask scheduling database model based on the divided tasks.
Establishing a database model for the task, establishing identification information for each client to uniquely bind a client number, establishing a main task according to a task division model, mounting a subtask under the main task, storing the subtask into the database through the task instance number, and calling the flows established in the data model and the data processing model in parallel through an automatic task polling mechanism so as to achieve the aim of parallel and independent task parallel of different clients.
In summary, in the above embodiments, when the distributed data stream of the third party payment monitoring system needs to be processed, the user requirements are firstly analyzed, and a service processing model is respectively established for the storage and management and reporting modes of the third party payment monitoring system; then combing out a hot flow based on the established business processing model, and establishing a data control model to carry out hot flow processing; then, dividing a data processing flow dependent task; and then establishing a subtask scheduling database model based on the divided tasks. A set of complete and independent task models are established for client information, parallel processing is achieved among clients, sub-library mapping from a database is established by dividing task dependency relationships in a set of client processing applications, and task dependent task serial and independent task parallel processing flows are performed, so that processing time is saved, and system resources are utilized to the maximum extent.
As shown in fig. 8, which is a schematic structural diagram of an embodiment 3 of a distributed data stream processing system of a third party payment supervision system disclosed in the present invention, the system includes:
a preparation processing subunit 801, configured to receive and analyze detail data with a large data size, and perform preparation processing on the detail data;
a data processing subunit 802, configured to perform data auditing and analysis on the preprocessed details in the details mode or the report data received in the report mode;
when the distributed data stream of the third-party payment supervision system needs to be processed, firstly, the requirements of the user are analyzed, and a business processing model is respectively established according to the requirements of the user aiming at the storage and management and report modes of the third-party payment supervision system. Specifically, receiving and analyzing detailed data with large data volume, performing preliminary processing on the detailed data, and performing data check and analysis on the preprocessed detailed data in a detailed mode or the report data received in a report mode.
A second establishing module 803, configured to comb out a hot flow based on the established service processing model, and establish a data control model to perform hot flow processing;
and after the hot flow is combed out through the established business processing model, a data control model is further established to carry out hot flow processing.
The data processing module 804 is used for dividing data processing flow dependent tasks;
after receiving a service request, acquiring a file, decompressing a preprocessing check reservation self-service task return result, then registering a self-service task, starting basic data processing, analyzing and uploading check contract information, checking a report form and warehousing unit cell validity data, registering a third party payment batch task data warehousing, registering a third party payment batch task, polling starting task processing data, warehousing the data, generating other reports, and generating a check result.
A third establishing module 805, configured to establish a subtask scheduling database model based on the divided tasks.
Establishing a database model for the task, establishing identification information for each client to uniquely bind a client number, establishing a main task according to a task division model, mounting a subtask under the main task, storing the subtask into the database through the task instance number, and calling the flows established in the data model and the data processing model in parallel through an automatic task polling mechanism so as to achieve the aim of parallel and independent task parallel of different clients.
In summary, in the above embodiments, when the distributed data stream of the third party payment monitoring system needs to be processed, the user requirements are firstly analyzed, and a service processing model is respectively established for the storage and management and reporting modes of the third party payment monitoring system; then combing out a hot flow based on the established business processing model, and establishing a data control model to carry out hot flow processing; then, dividing a data processing flow dependent task; and then establishing a subtask scheduling database model based on the divided tasks. A set of complete and independent task models are established for client information, parallel processing is achieved among clients, sub-library mapping from a database is established by dividing task dependency relationships in a set of client processing applications, and task dependent task serial and independent task parallel processing flows are performed, so that processing time is saved, and system resources are utilized to the maximum extent.
As shown in fig. 9, which is a schematic structural diagram of an embodiment 4 of a distributed data stream processing system of a third party payment supervision system disclosed in the present invention, the system includes:
a preparation processing subunit 901, configured to receive and analyze the detail data with a large data volume, and perform preparation processing on the detail data;
a data processing subunit 902, configured to perform data auditing and analysis on the preprocessed details in the details mode or the report data received in the report mode;
when the distributed data stream of the third-party payment supervision system needs to be processed, firstly, the requirements of the user are analyzed, and a business processing model is respectively established according to the requirements of the user aiming at the storage and management and report modes of the third-party payment supervision system. Specifically, receiving and analyzing detailed data with large data volume, performing preliminary processing on the detailed data, and performing data check and analysis on the preprocessed detailed data in a detailed mode or the report data received in a report mode.
The establishing subunit 903 is used for segmenting the client information into different client accounts through a hash algorithm, establishing a mapping relation stored in a database and establishing a data model;
and after the hot flow is combed out through the established business processing model, a data control model is further established to carry out hot flow processing. Specifically, the client information is divided into different client accounts through a Hash algorithm, a mapping relation stored in a database in a sub-database mode is established, and a data model is established
A data processing module 904, configured to perform data processing flow dependent task division;
after receiving a service request, acquiring a file, decompressing a preprocessing check reservation self-service task return result, then registering a self-service task, starting basic data processing, analyzing and uploading check contract information, checking a report form and warehousing unit cell validity data, registering a third party payment batch task data warehousing, registering a third party payment batch task, polling starting task processing data, warehousing the data, generating other reports, and generating a check result.
A third establishing module 905, configured to establish a subtask scheduling database model based on the divided tasks.
Establishing a database model for the task, establishing identification information for each client to uniquely bind a client number, establishing a main task according to a task division model, mounting a subtask under the main task, storing the subtask into the database through the task instance number, and calling the flows established in the data model and the data processing model in parallel through an automatic task polling mechanism so as to achieve the aim of parallel and independent task parallel of different clients.
In summary, in the above embodiments, when the distributed data stream of the third party payment monitoring system needs to be processed, the user requirements are firstly analyzed, and a service processing model is respectively established for the storage and management and reporting modes of the third party payment monitoring system; then combing out a hot flow based on the established business processing model, and establishing a data control model to carry out hot flow processing; then, dividing a data processing flow dependent task; and then establishing a subtask scheduling database model based on the divided tasks. A set of complete and independent task models are established for client information, parallel processing is achieved among clients, sub-library mapping from a database is established by dividing task dependency relationships in a set of client processing applications, and task dependent task serial and independent task parallel processing flows are performed, so that processing time is saved, and system resources are utilized to the maximum extent.
As shown in fig. 10, which is a schematic structural diagram of an embodiment 5 of a distributed data stream processing system of a third party payment supervision system disclosed in the present invention, the system includes the following steps:
a preparation processing subunit 1001 configured to receive and analyze the detail data with a large data size, and perform preparation processing on the detail data;
a data processing subunit 1002, configured to perform data auditing and analysis on the details preprocessed in the detail mode or the report data received in the report mode;
when the distributed data stream of the third-party payment supervision system needs to be processed, firstly, the requirements of the user are analyzed, and a business processing model is respectively established according to the requirements of the user aiming at the storage and management and report modes of the third-party payment supervision system. Specifically, receiving and analyzing detailed data with large data volume, performing preliminary processing on the detailed data, and performing data check and analysis on the preprocessed detailed data in a detailed mode or the report data received in a report mode.
The mounting sub-unit 1003 is used for establishing a number of a unified rule for the client system and mounting the account of the monitoring object under the client number;
a generating subunit 1004, configured to generate a hash result for the customer number and the account number through a hash algorithm based on a premise that the customer details are substantially evenly distributed for the account number;
a storing subunit 1005, configured to store the detail data of the customer account into the isomorphic sub-libraries with corresponding numbers of the database after taking the mole value of the database data, so as to complete the database load balancing;
and after the hot flow is combed out through the established business processing model, a data control model is further established to carry out hot flow processing. Specifically, a unified rule number is established for a client system, and a monitoring object account is mounted under the client number; on the premise that the account numbers are basically and evenly distributed according to the customer details, generating a hash result for the customer numbers and the account numbers through a hash algorithm; and after the mole value of the database data is measured, the detailed data of the client account is stored in the isomorphic sublibrary with the corresponding serial number of the database, and the balanced load of the database is completed.
A data processing module 1006, configured to perform data processing flow dependent task division;
after receiving a service request, acquiring a file, decompressing a preprocessing check reservation self-service task return result, then registering a self-service task, starting basic data processing, analyzing and uploading check contract information, checking a report form and warehousing unit cell validity data, registering a third party payment batch task data warehousing, registering a third party payment batch task, polling starting task processing data, warehousing the data, generating other reports, and generating a check result.
A third establishing module 1007, configured to establish a subtask scheduling database model based on the divided tasks.
Establishing a database model for the task, establishing identification information for each client to uniquely bind a client number, establishing a main task according to a task division model, mounting a subtask under the main task, storing the subtask into the database through the task instance number, and calling the flows established in the data model and the data processing model in parallel through an automatic task polling mechanism so as to achieve the aim of parallel and independent task parallel of different clients.
In summary, in the above embodiments, when the distributed data stream of the third party payment monitoring system needs to be processed, the user requirements are firstly analyzed, and a service processing model is respectively established for the storage and management and reporting modes of the third party payment monitoring system; then combing out a hot flow based on the established business processing model, and establishing a data control model to carry out hot flow processing; then, dividing a data processing flow dependent task; and then establishing a subtask scheduling database model based on the divided tasks. A set of complete and independent task models are established for client information, parallel processing is achieved among clients, sub-library mapping from a database is established by dividing task dependency relationships in a set of client processing applications, and task dependent task serial and independent task parallel processing flows are performed, so that processing time is saved, and system resources are utilized to the maximum extent.
The functions described in the method of the present embodiment, if implemented in the form of software functional units and sold or used as independent products, may be stored in a storage medium readable by a computing device. Based on such understanding, part of the contribution of the embodiments of the present invention to the prior art or part of the technical solution may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computing device (which may be a personal computer, a server, a mobile computing device, a network device, or the like) to execute all or part of the steps of the method described in the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A third party payment supervision system distributed data stream processing method is characterized by comprising the following steps:
analyzing the user requirements, and respectively establishing a service processing model aiming at the storage and management and report mode of a third-party payment supervision system;
combing out a hot flow based on the established business processing model, and establishing a data control model to process the hot flow;
dividing a data processing flow dependent task;
establishing a subtask scheduling database model based on the divided tasks;
performing parallel processing among clients based on the data processing flows established in the subtask scheduling database model and the data control model through an automatic task polling mechanism, and performing serial processing instead of parallel processing on tasks in dependence on the request processing process of one client;
wherein, the dividing of the data processing flow dependent task comprises:
after receiving the service request, acquiring a file and decompressing a preprocessing check reservation self-service task return result;
registering self-service tasks and processing basic data;
checking contract information analysis and uploading a report form, checking the report form and storing unit grid validity data, registering third party payment batch tasks, registering third party payment batch task data and storing the third party payment batch tasks;
polling task processing data, storing the data in a database, generating other reports, and generating a check result;
the division-based task establishment subtask scheduling database model comprises the following steps:
and establishing unique identification information binding the client number for each client, establishing a main task according to a task division model, mounting a subtask under the main task, and storing the subtask into a database through the task instance number to obtain a subtask scheduling database model.
2. The method of claim 1, wherein the analyzing the user requirements and the establishing the business processing model for the inventory and report mode of the third party payment supervision system respectively comprises:
and merging the service process channels to obtain a basic data processing process model, and dividing hot spots and bottleneck links of data processing.
3. The method of claim 2, wherein the merging through the business process channels to obtain the basic process model for data processing, and the dividing the hot spots and the neck rings of the data processing comprises:
receiving and analyzing detailed data with large data volume, and performing preparation processing on the detailed data;
and performing data checking and analysis on the preprocessed detail in the detail mode or the report data received in the report mode.
4. The method according to any one of claims 1 to 3, wherein the hot spot processes are combed out based on the established business process model, and establishing a data control model for hot spot process processing comprises:
and segmenting the client information into different client accounts through a Hash algorithm, establishing a mapping relation stored in a database in a sub-database mode, and establishing a data model.
5. The method according to claim 4, wherein the splitting of the customer information through a hash algorithm is performed on different customer accounts, and the establishing of the mapping relation of the database sub-base storage is performed, and the establishing of the data model comprises:
establishing a uniform rule number for a client system, and mounting a monitoring object account under the client number;
on the premise that the account numbers are basically and evenly distributed according to the customer details, generating a hash result for the customer numbers and the account numbers through a hash algorithm;
and after the mole value of the database data is measured, the detailed data of the client account is stored in the isomorphic sublibrary with the corresponding serial number of the database, and the balanced load of the database is completed.
6. A third party payment supervision system distributed data stream processing system, comprising:
the first establishing module is used for analyzing the user requirements and respectively establishing a service processing model aiming at the storage and management and report modes of the third-party payment supervision system;
the second establishing module is used for combing a hot spot flow based on the established business processing model and establishing a data control model for hot spot flow processing;
the data processing module is used for dividing the data processing flow dependent tasks;
the third establishing module is used for establishing a subtask scheduling database model based on the divided tasks;
performing parallel processing among clients based on the data processing flows established in the subtask scheduling database model and the data control model through an automatic task polling mechanism, and performing serial processing instead of parallel processing on tasks in dependence on the request processing process of one client;
wherein, the dividing of the data processing flow dependent task comprises:
after receiving the service request, acquiring a file and decompressing a preprocessing check reservation self-service task return result;
registering self-service tasks and processing basic data;
checking contract information analysis and uploading a report form, checking the report form and storing unit grid validity data, registering third party payment batch tasks, registering third party payment batch task data and storing the third party payment batch tasks;
polling task processing data, storing the data in a database, generating other reports, and generating a check result;
the division-based task establishment subtask scheduling database model comprises the following steps:
and establishing unique identification information binding the client number for each client, establishing a main task according to a task division model, mounting a subtask under the main task, and storing the subtask into a database through the task instance number to obtain a subtask scheduling database model.
7. The system of claim 6, wherein the first establishing module comprises:
and the dividing unit is used for obtaining a basic data processing flow model through the merging of the service flow channels and dividing hot spots and bottleneck links of data processing.
8. The system of claim 7, wherein the partitioning unit comprises:
the preparation processing subunit is used for receiving and analyzing the detail data with large data volume and performing preparation processing on the detail data;
and the data processing subunit is used for performing data auditing and analysis on the preprocessed details in the detail mode or the report data received in the report mode.
9. The system according to any one of claims 6-8, wherein the second establishing module comprises:
and the establishing subunit is used for segmenting the client information into different client accounts through a Hash algorithm, establishing a mapping relation stored in a database in a sub-database mode and establishing a data model.
10. The system of claim 9, wherein the establishing subunit comprises:
the mounting subunit is used for establishing a uniform rule number for the client system and mounting the account number of the monitoring object under the client number;
the generation subunit is used for generating a hash result for the customer number and the account number through a hash algorithm on the premise that the customer details are basically and evenly distributed to the account number;
and the storing subunit is used for storing the detailed data of the customer account into the isomorphic sublibraries with corresponding numbers of the database after the mole value of the database data is measured, so as to complete the balanced load of the database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611226657.5A CN106845926B (en) | 2016-12-27 | 2016-12-27 | Distributed data stream processing method and system for third-party payment supervision system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611226657.5A CN106845926B (en) | 2016-12-27 | 2016-12-27 | Distributed data stream processing method and system for third-party payment supervision system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106845926A CN106845926A (en) | 2017-06-13 |
| CN106845926B true CN106845926B (en) | 2020-11-27 |
Family
ID=59136514
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611226657.5A Active CN106845926B (en) | 2016-12-27 | 2016-12-27 | Distributed data stream processing method and system for third-party payment supervision system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106845926B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109800965B (en) * | 2018-12-28 | 2021-12-17 | 广州亚信技术有限公司 | Process business processing method and device, computer equipment and storage medium |
| CN114064231B (en) * | 2021-10-13 | 2022-07-05 | 北京市应急管理科学技术研究院 | Task configuration method, system, storage medium and electronic equipment |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9063790B2 (en) * | 2011-06-13 | 2015-06-23 | Accenture Global Services Limited | System and method for performing distributed parallel processing tasks in a spot market |
| CN102508639B (en) * | 2011-10-10 | 2014-04-02 | 北京邮电大学 | Distributed parallel processing method based on satellite remote sensing data characteristics |
| US9563470B2 (en) * | 2013-12-23 | 2017-02-07 | International Business Machines Corporation | Backfill scheduling for embarrassingly parallel jobs |
| CN104156380B (en) * | 2014-03-04 | 2019-03-26 | 深圳信息职业技术学院 | A kind of distributed memory hash indexing method and system |
| CN104239141B (en) * | 2014-09-05 | 2017-07-28 | 北京邮电大学 | Optimizing and scheduling task method based on workflow critical path in data center |
| CN105869048A (en) * | 2016-03-28 | 2016-08-17 | 中国建设银行股份有限公司 | Data processing method and system |
-
2016
- 2016-12-27 CN CN201611226657.5A patent/CN106845926B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN106845926A (en) | 2017-06-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10841155B2 (en) | Systems and methods for management of cloud computing resources for information systems | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| US10430332B2 (en) | System and method for performance tuning of garbage collection algorithms | |
| CN107229619B (en) | Method and device for counting and displaying calling condition of internet service link | |
| CN103581336B (en) | Service flow scheduling method and system based on cloud computing platform | |
| US11956330B2 (en) | Adaptive data fetching from network storage | |
| CN105338082A (en) | Load balancing method and load balancing device based on application proxy server | |
| CN111726266B (en) | Hot spot data barreling method and system and computer equipment | |
| CN111523849A (en) | Resource transaction auditing method and device and server | |
| CN106845926B (en) | Distributed data stream processing method and system for third-party payment supervision system | |
| CN110737425A (en) | billing platform system application program establishing method and device | |
| CN113312359B (en) | Distributed job progress calculation method and device and storage medium | |
| CN113297188B (en) | Data processing method and device | |
| CN107193749A (en) | Method of testing, device and equipment | |
| CN113553518A (en) | Resource identifier generation method, device, equipment and storage medium | |
| CN112953993A (en) | Resource scheduling method, device, network system and storage medium | |
| CN115562861A (en) | Method and apparatus for data processing for data skew | |
| CN112700292A (en) | Method and system for issuing electronic invoice through automatic scheduling of tax control equipment | |
| CN111221858B (en) | Data processing method, device and equipment | |
| CN113297358A (en) | Data processing method, device, server and computer readable storage medium | |
| CN114428723A (en) | Test system, system test method, related device and storage medium | |
| CN113055419B (en) | Information sending method and device | |
| CN116385102B (en) | Information recommendation method, device, computer equipment and storage medium | |
| CN113347665B (en) | Method and device for controlling real-time transaction amount in gateway system | |
| CN105765569B (en) | A kind of data distributing method, loading machine and storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |