CN111930760A - Transaction flow distributed batch processing method and system - Google Patents
Transaction flow distributed batch processing method and system Download PDFInfo
- Publication number
- CN111930760A CN111930760A CN202010702599.9A CN202010702599A CN111930760A CN 111930760 A CN111930760 A CN 111930760A CN 202010702599 A CN202010702599 A CN 202010702599A CN 111930760 A CN111930760 A CN 111930760A
- Authority
- CN
- China
- Prior art keywords
- clearing
- module
- transaction
- batch
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/06—Buying, selling or leasing transactions
- G06Q30/0601—Electronic shopping [e-shopping]
- G06Q30/0633—Lists, e.g. purchase orders, compilation or processing
- G06Q30/0635—Processing of requisition or of purchase orders
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Finance (AREA)
- Databases & Information Systems (AREA)
- Accounting & Taxation (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Economics (AREA)
- Development Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Computing Systems (AREA)
- Technology Law (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distributed batch processing method and a distributed batch processing system for transaction pipelining, which comprise a scheduling module, a transaction pipelining module, a clearing module and a moistening calculation module, the scheduling module, the transaction flow, the clearing module and the sub-lubricating calculation module are connected in sequence, the scheduling module is used for triggering the clearing module to execute clearing transaction, the transaction pipeline is synchronized to a transaction storage database in the clearing module in real time through a database of the transaction pipeline module, a plurality of types of order consumption data are searched at intervals, through cleaning, converting and filtering, the data is stored in the database of the clearing module after being cleared successfully, and the data is used for the dividend calculation module to extract and calculate the dividend fee, the invention can support the data volume of the service dimension to rise and clear, and the batch running execution time of other dimensions is not influenced, the data can be supported to be migrated quickly, and various business processes are facilitated.
Description
Technical Field
The invention relates to the technical field of network business transaction, in particular to a transaction flow distributed batch processing method and a system thereof.
Background
In the prior art, the system is calculated by service (such as customer) dimensions, and this way can cause that when the transaction amount of a certain service dimension is unevenly distributed, the transaction amount of an individual service is increased quickly, and the data is too large, the service can affect the calculation time of the scores of all the customers, and the system is slow to execute, thereby affecting the running time of the whole system.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides a method and a system for carrying out distributed batch processing on transaction flow.
The technical scheme adopted by the invention for solving the technical problems is as follows: in a transaction pipelining distributed batch processing method, the improvement comprising the steps of:
s1: the scheduling module guides the transaction flow in the transaction flow module into the clearing module at regular time;
s2: the database of the transaction pipelining module synchronizes transaction pipelining in the transaction storage database of the clearing module;
s3: the clearing module finds out the data of the transaction, the data is filtered by the cleaning conversion filter and then converted into clearing running water with clearing batch numbers to form clearing batch number data, and the clearing batch number data are stored in a database of the clearing module in a clearing batch table format.
S4: the dispatching module extracts the clearing batch table from the database of the clearing module to the diff calculating module to calculate the diff commission charge.
As an improvement of the above technical solution, in the step S2, the step of cleaning the conversion filter and storing the cleaned conversion filter in the database of the clearing module includes:
s21: the clearing module inquires out successful transaction running water;
s22: leading the successful transaction flow into a clearing module;
s23: the clearing module converts the successful clearing running water into required fields by associating with other transaction running water meters, and then forms the clearing running water with clearing batch word numbers or the clearing running water with the clearing batch word numbers being empty;
s24: and updating the clearing batch number information of each transaction to form a clearing batch number table, and storing the clearing batch number table in a database of the clearing module through the dispatching module.
As an improvement to the above solution, the transaction pipeline includes consumption, order, payment by agency, refund or other transaction type data.
As an improvement of the above technical solution, the clearing batch table includes a clearing date, a batch number, a group number, a clearing status, and a number of strokes.
The improvement of the transaction flow distributed batch processing system is that the system comprises a scheduling module, a transaction flow module, a clearing module and a sub-lubricating calculation module, wherein the scheduling module, the transaction flow module, the clearing module and the sub-lubricating calculation module are sequentially connected, the scheduling module is used for triggering the clearing module to execute clearing transaction, the transaction flow can be synchronized to a transaction storage database in the clearing module through a database of the transaction flow module in real time, multiple types of order consumption data are checked out at intervals, and after clearing is successful, the order consumption data are stored in the database of the clearing module through cleaning, converting and filtering, so that the sub-lubricating calculation module can extract and calculate sub-lubricating commission charge.
As an improvement of the above technical solution, the step of cleaning, converting, filtering and storing in the database of the liquidation module comprises:
s21: the clearing module inquires out successful transaction running water;
s22: leading the successful transaction flow into a clearing module;
s23: the clearing module converts the successful clearing running water into required fields by associating with other transaction running water meters, and then forms the clearing running water with clearing batch word numbers or the clearing running water with the clearing batch word numbers being empty;
s24: and updating the clearing batch number information of each transaction to form a clearing batch number table, and storing the clearing batch number table in a database of the clearing module.
As an improvement of the above technical solution, the transaction flow cleared by the clearing module comprises consumption, order or other transaction type data.
As an improvement of the above technical solution, the clearing batch table includes a clearing date, a batch number, a group number, a clearing status, and a number of strokes.
In a computer device including a memory, a processor, and a computer program stored on the memory and executable on the processor, the improvement wherein the processor implements any of the above methods when executing the computer program.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the method of any one of the above.
The invention has the beneficial effects that: the invention can support the data volume of the service dimension to rise and clear, does not influence the batch execution time of other dimensions, can support the rapid data migration, and is convenient for various service processing.
Drawings
The invention is further illustrated with reference to the following figures and examples.
FIG. 1 is a system flow diagram of the present invention.
FIG. 2 is a system flow diagram of the present invention.
FIG. 3 is a clearing transaction flow diagram of the present invention.
Detailed Description
The conception, the specific structure, and the technical effects produced by the present invention will be clearly and completely described below in conjunction with the embodiments and the accompanying drawings to fully understand the objects, the features, and the effects of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments, and those skilled in the art can obtain other embodiments without inventive effort based on the embodiments of the present invention, and all embodiments are within the protection scope of the present invention. In addition, all the connection/connection relations referred to in the patent do not mean that the components are directly connected, but mean that a better connection structure can be formed by adding or reducing connection auxiliary components according to specific implementation conditions. All technical characteristics in the invention can be interactively combined on the premise of not conflicting with each other.
Referring to fig. 1 and 2, the present invention discloses a transaction pipelining distributed batch processing method, which comprises the following steps:
s1: the scheduling module 1 guides the transaction flow in the transaction flow module 2 into the clearing module 3 at regular time;
s2: the database of the transaction pipelining module 2 synchronizes transaction pipelining in the transaction storage database of the clearing module 3;
s3: the clearing module 3 checks the data of the transaction, converts the data into clearing running water with clearing batch numbers after being filtered by a cleaning conversion filter to form clearing batch number data, and the clearing batch number data are stored in a database of the clearing module 3 in a clearing batch table format.
S4: the dispatching module 1 extracts the clearing batch table from the database of the clearing module 3 to the diff calculating module 4 for calculating the diff commission charge.
In step S2, the step of cleaning the conversion filter and storing it in the database of the liquidation module 3 includes:
s21: the clearing module 3 inquires out successful transaction running water;
s22: the successful transaction flow is led into a clearing module 3;
s23: the clearing module 3 converts the successful clearing running water into required fields by associating with other transaction running water meters, and then forms the clearing running water with clearing batch font numbers or the clearing running water with the clearing batch font numbers being empty;
s24: and updating the clearing batch number information of each transaction to form a clearing batch number table, and storing the clearing batch number table in a database of the clearing module 3 through the dispatching module 1.
In the above embodiment, referring to fig. 3, the transaction flow may be synchronized to the transaction storage database of the clearing module 3 through the database of the transaction flow module 2 in real time to form a clearing transaction table, the clearing transaction table may be triggered to be executed through the scheduling module 1, a batch of data of clearing transaction flow of different types of order consumption and the like may be searched at intervals, a successful transaction flow may be searched through cleaning, converting and filtering, and after being converted into a required field by associating with other transaction flow water meters, the required field is converted into a clearing flow, and the clearing flow is stored in the database.
Further, the transaction pipeline includes consumption, order, payment, refund, or other transaction type data, supporting a plurality of clearing transaction types.
Still further, the clearing batch table includes a clearing date, a batch number, a group number, a clearing status, and a number of strokes. The clearing date, the batch number, the group number, the clearing state and the stroke number of the transaction are comprehensively integrated, so that the transaction flow data is comprehensively transparent, and the transaction flow data is convenient to clear.
When the dispatching module 1 of the invention extracts data from the transaction pipeline module 2 to the clearing module 3 at regular time, each clearing pipeline is added with a clearing batch number field, after the transaction pipeline is imported into the clearing module 3, every N clearing pipelines are set with a clearing batch number, the group number is determined by the number of machine nodes, the number X of the machine nodes is obtained by a configuration file and can be dynamically modified, and the group number of the execution X is distributed according to the number of the nodes of the machine X, assuming that the number of the nodes is X, namely 1 to X, each group number is distributed with N transaction pipelines, and the dispatching module 1 can send execution commands to a plurality of servers in the distributed computing module 4 according to the clearing date, the batch number, the group number, the clearing state and the number of strokes when the dispatching module 1 executes a clearing batch task, as shown in figure 1, each server will select a machine according to the command sent by the dispatching module 1, and the server will obtain the transaction flow related to the group number to clear the calculation flow, thereby achieving the purpose of multi-service distributed batch processing.
A transaction flow distributed batch processing system comprises a scheduling module 1, a transaction flow module 2, a clearing module 3 and a sub-lubricating calculation module 4, the dispatching module 1, the transaction flow module 2, the clearing module 3 and the sub-lubricating calculating module 4 are connected in sequence, the scheduling module 1 is used for triggering the clearing module 3 to execute clearing transaction, the transaction pipeline is synchronized to a transaction storage database in the clearing module 3 through the database of the transaction pipeline module 2 in real time, various types of order consumption data are searched out at intervals, and through cleaning, converting and filtering, the clearing is stored in the database of the clearing module 3 after being successfully cleared, the diffluence calculating module 4 extracts and calculates the diffluence commission charge, when the data volume of the service dimension is increased, the batch running execution time of other dimensions is reduced, the data is supported to be migrated quickly, and various business processes are facilitated.
The steps of cleaning the conversion filter and storing in the database of the clearing module include:
s21: the clearing module 3 inquires out successful transaction running water;
s22: the successful transaction flow is led into a clearing module 3;
s23: the clearing module 3 converts the successful clearing running water into required fields by associating with other transaction running water meters, and then forms the clearing running water with clearing batch font numbers or the clearing running water with the clearing batch font numbers being empty;
s24: and updating the clearing batch number information of each transaction to form a clearing batch number table, and storing the clearing batch number table in a database of the clearing module 3.
In the above embodiment, as shown in fig. 3, the transaction flow may be synchronized to the transaction storage database of the clearing module 3 through the database in real time to form a clearing transaction table, the clearing transaction table may be triggered to be executed through the scheduling module 1, a batch of data of clearing transaction flow of different types of order consumption and the like may be found out at intervals, a success status that needs to be imported into the clearing flow is found out through cleaning conversion filtering, and after being converted into a required field by associating with other transaction flow water meters, the required field is converted into the clearing flow and stored in the database of the clearing module 3.
Further, the transaction pipeline includes consumption, order, payment by agency, refund, or other transaction type data. Multiple clearing transaction types are supported.
Still further, the clearing batch table includes a clearing date, a batch number, a group number, a clearing status, and a number of strokes. The clearing date, the batch number, the group number, the clearing state and the stroke number of the transaction are comprehensively integrated, so that the transaction flow data is comprehensively transparent, and the transaction flow data is convenient to clear.
When the dispatching module 1 of the invention extracts data from the transaction pipeline module 2 to the clearing module 3 at regular time, each clearing pipeline is added with a clearing batch number field, after the transaction pipeline is imported into the clearing module 3, every N clearing pipelines are provided with a clearing batch number, the group number is determined by the number of machine nodes, the number X of the machine nodes is obtained by a configuration file and can be dynamically modified, and the executing X group number is distributed according to the number of the X nodes of the machine, the number of the nodes is assumed to be X and is 1 to X, each group number is distributed with N transaction pipelines, and the dispatching module 1 sends executing commands to a plurality of servers in the branch calculating module 4 according to the clearing date, the batch number, the group number, the clearing state and the number of strokes which are stored as a clearing batch table, when the dispatching module 1 executes a clearing batch running task, the dispatching module 1 can execute the executing commands according to the number X of the machine nodes distributed by the clearing batch table, each server selects a machine according to the command sent by the scheduling module, and the server obtains the transaction flow related to the group number to divide the calculation flow, so that the aim of multi-service distributed batch processing is fulfilled.
The invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of the above when executing the computer program.
The invention also provides a computer readable storage medium having stored thereon a computer program for executing the method of any of the above.
The invention has the beneficial effects that: the invention can support the data volume of the service dimension to rise and clear, does not influence the batch execution time of other dimensions, can support the rapid data migration, and is convenient for various service processing.
While the preferred embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A transaction pipelining distributed batch processing method, comprising the steps of:
s1: the scheduling module guides the transaction flow in the transaction flow module into the clearing module at regular time;
s2: the database of the transaction pipelining module synchronizes transaction pipelining in the transaction storage database of the clearing module;
s3: the clearing module finds out the data of the transaction, the data is filtered by the cleaning conversion filter and then converted into clearing running water with clearing batch numbers to form clearing batch number data, and the clearing batch number data are stored in a database of the clearing module in a clearing batch table format.
S4: the dispatching module extracts the clearing batch table from the database of the clearing module to the diff calculating module to calculate the diff commission charge.
2. The distributed batch processing method for transaction pipelining according to claim 1, wherein in step S2, the step of cleaning the conversion filter and storing it in the database of the clearing module comprises:
s21: the clearing module inquires out successful transaction running water;
s22: leading the successful transaction flow into a clearing module;
s23: the clearing module converts the successful clearing running water into required fields by associating with other transaction running water meters, and then forms the clearing running water with clearing batch word numbers or the clearing running water with the clearing batch word numbers being empty;
s24: and updating the clearing batch number information of each transaction to form a clearing batch number table, and storing the clearing batch number table in a database of the clearing module through the dispatching module.
3. A transaction flow distributed batch processing method according to claim 1 in which the transaction flow comprises consumption, order, payment by agency, refund or other transaction type data.
4. The transaction pipelining distributed batch processing method of claim 1, wherein the clearing batch table includes a clearing date, a batch number, a group number, a clearing status, and a number of strokes.
5. A transaction flow distributed batch processing system is characterized by comprising a scheduling module, a transaction flow module, a clearing module and a sub-lubricating calculation module, wherein the scheduling module, the transaction flow module, the clearing module and the sub-lubricating calculation module are sequentially connected, the scheduling module is used for triggering the clearing module to execute clearing transaction, the transaction flow can be synchronized to a transaction storage database in the clearing module through a database of the transaction flow module in real time, multiple types of order consumption data are checked out at intervals, and after clearing is successful, the order consumption data are stored in the database of the clearing module through cleaning, converting and filtering, and the sub-lubricating calculation module can extract and calculate sub-lubricating commission charges.
6. The transaction pipelining distributed batch processing system of claim 5, wherein the step of cleansing the translation filter and storing in a database of a clearing module comprises:
s21: the clearing module inquires out successful transaction running water;
s22: leading the successful transaction flow into a clearing module;
s23: the clearing module converts the successful clearing running water into required fields by associating with other transaction running water meters, and then forms the clearing running water with clearing batch word numbers or the clearing running water with the clearing batch word numbers being empty;
s24: and updating the clearing batch number information of each transaction to form a clearing batch number table, and storing the clearing batch number table in a database of the clearing module.
7. The transaction flow distributed batch processing system of claim 5, wherein the transaction flow includes consumption, order, payment by agency, refund, or other transaction type data.
8. The distributed batch processing system as recited in claim 5 wherein said clearing batch table includes a clearing date, a batch number, a group number, a clearing status, and a number of strokes.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1 to 4 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010702599.9A CN111930760A (en) | 2020-07-21 | 2020-07-21 | Transaction flow distributed batch processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010702599.9A CN111930760A (en) | 2020-07-21 | 2020-07-21 | Transaction flow distributed batch processing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111930760A true CN111930760A (en) | 2020-11-13 |
Family
ID=73313673
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010702599.9A Pending CN111930760A (en) | 2020-07-21 | 2020-07-21 | Transaction flow distributed batch processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111930760A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112650568A (en) * | 2020-12-22 | 2021-04-13 | 银盛支付服务股份有限公司 | Merchant netting difference settlement method, system, computer equipment and storage medium |
| CN112860745A (en) * | 2021-01-29 | 2021-05-28 | 拉卡拉支付股份有限公司 | Separate settlement method, server and storage medium |
-
2020
- 2020-07-21 CN CN202010702599.9A patent/CN111930760A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112650568A (en) * | 2020-12-22 | 2021-04-13 | 银盛支付服务股份有限公司 | Merchant netting difference settlement method, system, computer equipment and storage medium |
| CN112860745A (en) * | 2021-01-29 | 2021-05-28 | 拉卡拉支付股份有限公司 | Separate settlement method, server and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111930760A (en) | Transaction flow distributed batch processing method and system | |
| CN102508709A (en) | Distributed-cache-based acquisition task scheduling method in purchase, supply and selling integrated electric energy acquiring and monitoring system | |
| CN102521712A (en) | Process instance data processing method and device | |
| CN109739919A (en) | A kind of front end processor and acquisition system for electric system | |
| CN105787058B (en) | A kind of user tag system and the data delivery system based on user tag system | |
| CN104699757A (en) | Distributed network information acquisition method in cloud environment | |
| CN110471945A (en) | Processing method, system, computer equipment and the storage medium of alive data | |
| CN114691658A (en) | Data backtracking method and device, electronic equipment and storage medium | |
| CN110175206A (en) | Intellectual analysis operational approach, system and medium for multiple database separation | |
| CN110135967A (en) | The processing method and system of collection data | |
| CN111339108A (en) | Transaction parallel execution method, device and storage medium | |
| CN109815387A (en) | A kind of web retrieval method of servicing and system based on script | |
| CN110324211A (en) | A kind of data capture method and device | |
| CN112579691B (en) | Data processing method and device supporting large-screen display | |
| CN110096339A (en) | A kind of scalable appearance configuration recommendation system and method realized based on system load | |
| CN111752541B (en) | Payment routing method based on Rete algorithm | |
| Masouleh et al. | Optimization of ETL process in data warehouse through a combination of parallelization and shared cache memory | |
| CN112907336A (en) | Method and server for realizing order state synchronization based on mybatis interceptor | |
| CN111666144A (en) | Batch processing task execution method and system and machine room deployment system | |
| CN107291872A (en) | A kind of platform blog search rendering method based on mass data | |
| US8027996B2 (en) | Commitment control for less than an entire record in an in-memory database in a parallel computer system | |
| CN107678856B (en) | Method and device for processing incremental information in business entity | |
| CN115794744A (en) | Log display method, device, equipment and storage medium | |
| CN115080607A (en) | Method, device, equipment and storage medium for optimizing structured query statement | |
| CN109785099B (en) | Method and system for automatically processing service data information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |