CN114547113A - Mass data account checking system, method, network equipment and storage medium - Google Patents
Mass data account checking system, method, network equipment and storage medium Download PDFInfo
- Publication number
- CN114547113A CN114547113A CN202210441004.8A CN202210441004A CN114547113A CN 114547113 A CN114547113 A CN 114547113A CN 202210441004 A CN202210441004 A CN 202210441004A CN 114547113 A CN114547113 A CN 114547113A
- Authority
- CN
- China
- Prior art keywords
- reconciliation
- data
- account
- account checking
- tie
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/12—Accounting
- G06Q40/125—Finance or payroll
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- General Business, Economics & Management (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a mass data account checking system, a mass data account checking method, network equipment and a storage medium. The reconciliation system comprises: the data access module comprises an external/internal reconciliation data access module; the account checking module comprises a real-time/off-line account checking module; the real-time account checking module is used for carrying out real-time account checking on the internal account checking model data to obtain a real-time account checking result; the offline reconciliation module is used for performing offline reconciliation on the external/internal reconciliation model data according to a preset period to obtain an offline reconciliation result; the tie-account module comprises an unordered set processor and a tie-account rule engine; the unordered set processor is used for generating account checking difference data and account checking data according to the real-time account checking result and/or the off-line account checking result; the account balancing rule engine is used for matching the corresponding account balancing rule for account balancing according to the account balancing difference data and storing account balancing failure data into an account balancing difference database; the tie-out database is used for storing all tie-out data; the problems of low account checking efficiency of mass data, complex data access operation and the like are solved.
Description
Technical Field
The invention relates to the technical field of finance, in particular to a mass data account checking system, a mass data account checking method, network equipment and a storage medium.
Background
The traditional account checking system usually reads data in a memory by pulling data upstream and downstream of a service, compares the data with each other to find differences, then executes account checking rules to try to check account, and for data pulling, such as multiple protocols SFTP and HTTP, and different data types XML, JSON, CSV, TXT and BINLOG, hard coding is required to be realized one by one to complete data access; and reading data comparison at T +1 during account checking, when the matching is inconsistent, trying to level account one by one through a level account rule, and performing account checking and level account checking in a mode that data is written into MySQL after the level account is finished.
Above-mentioned prior art all needs hard coding and the operation of going on the line when reconciliation data access, very influences reconciliation efficiency to the condition that needs adjust reconciliation rule to and along with the increasing of reconciliation data, can't realize real-time reconciliation, also increased the degree of difficulty for the fortune dimension of reconciliation system.
Disclosure of Invention
The invention provides a mass data account checking system, a mass data account checking method, network equipment and a storage medium, and aims to solve the problems that account checking efficiency is low, account checking data access operation is complex and the like in the prior art.
In order to solve the technical problem, the invention is realized as follows:
in a first aspect, an embodiment of the present invention provides a mass data reconciliation system, where the system includes:
the data access module comprises an external reconciliation data access module and an internal reconciliation data access module; the external reconciliation data access module is used for receiving external reconciliation data and converting the external reconciliation data into external reconciliation model data; the internal reconciliation data access module is used for receiving internal reconciliation data and converting the internal reconciliation data into internal reconciliation model data;
the account checking module comprises a real-time account checking module and an offline account checking module; the real-time reconciliation module is used for carrying out real-time reconciliation on the internal reconciliation model data and acquiring a real-time reconciliation result; the offline reconciliation module is used for performing offline reconciliation on the internal reconciliation model data in the preset period and the external reconciliation model data in the preset period according to a preset period, and acquiring an offline reconciliation result;
the tie account module comprises an unordered set processor and a tie account rule engine;
the unordered set processor is used for generating account checking difference data and account checking data according to the real-time account checking result and/or the off-line account checking result;
the account reconciliation rule engine is used for matching the corresponding account reconciliation rule to perform account reconciliation according to the account reconciliation difference data, and storing the account reconciliation data with failed account reconciliation into an account reconciliation difference database if the account reconciliation fails;
and the tie account database is used for storing all tie account data.
Optionally, the system further comprises;
the account checking difference database is used for storing account checking difference data which cannot be generated by the account checking rule engine through account checking and sending the account checking difference data to the manual account checking;
and the warning module is used for warning the target account checking difference data if the target account checking difference data still cannot complete account checking after manual account checking.
Optionally, the offline reconciliation module is further configured to:
and when the offline reconciliation of the internal reconciliation model data in the preset period and the external reconciliation model data in the preset period cannot be completed in time, storing the internal and external model data which are not subjected to the reconciliation into the next period, and continuing the offline reconciliation in the next period.
Optionally, the tie rule engine is further configured to:
and when account checking is started for each account checking batch, loading a tie-account rule according to the type of the account checking batch, if the configuration of the tie-account rule is changed, reloading the updated tie-account rule, and performing account checking on the corresponding account checking batch by using the updated tie-account rule.
Optionally, the system further includes:
the account checking model database is used for storing the external account checking model data and the internal account checking model data;
the cloud storage module is used for regularly backing up tie-account data in the tie-account database and regularly backing up the external tie-account model data and the internal tie-account model data in the tie-account model database;
and the query module is used for providing corresponding data in the reconciliation model database and/or the cloud storage module based on a manually triggered query condition.
Optionally, the out-of-order set processor is further configured to:
carrying out real-time account checking includes: generating a reconciliation model data set of each internal business system according to the internal reconciliation model data, and comparing the reconciliation model data sets of each internal business system to generate a reconciliation difference data set;
generating reconciliation model data sets of the internal business system and the external system according to the external reconciliation model data, and comparing the reconciliation model data sets of the internal business system and the external system to generate a reconciliation difference data set.
In a second aspect, an embodiment of the present invention provides a mass data reconciliation method, where the method includes:
receiving reconciliation data, including receiving external reconciliation data and internal reconciliation data;
respectively converting the received external reconciliation data and internal reconciliation data into external reconciliation model data and internal reconciliation model data;
performing real-time account checking on the internal account checking model data to obtain a real-time account checking result;
performing offline reconciliation on the internal reconciliation model data in the preset period and the external reconciliation model data in the preset period according to a preset period, and acquiring an offline reconciliation result;
generating account checking difference data and account checking data according to the real-time account checking result and/or the off-line account checking result;
matching corresponding tie-in rules for tie-in according to the tie-in difference data, and storing the tie-in data with the failure of tie-in into a tie-in difference database if the tie-in fails;
and storing all the account checking data which are successfully checked and the data which are successfully checked into the account checking database.
Optionally, the method further includes:
and when the offline reconciliation of the internal reconciliation model data in the preset period and the external reconciliation model data in the preset period cannot be completed in time, storing the internal and external model data which are not subjected to the reconciliation into the next period, and continuing the offline reconciliation in the next period.
Optionally, the method further includes:
storing account checking difference data which cannot be generated by matching the corresponding account checking rule to finish the account checking, and sending the account checking difference data to the manual account checking;
and if the target account checking difference data still cannot complete account checking after manual account checking, alarming the target account checking difference data.
Optionally, the matching a corresponding tie-out rule for tie-out according to the tie-out difference data includes:
and when account checking is started for each account checking batch, loading a tie-account rule according to the type of the account checking batch, if the configuration of the tie-account rule is changed, reloading the updated tie-account rule, and performing account checking on the corresponding account checking batch by using the updated tie-account rule.
Optionally, the method further includes:
storing the external reconciliation model data and the internal reconciliation model data;
regularly backing up tie-account data in the tie-account database, and regularly backing up the external tie-account model data and the internal tie-account model data in the tie-account model database;
and providing corresponding data in the reconciliation model database and/or the cloud storage module based on the manually triggered query condition.
In a third aspect, an embodiment of the present invention provides a network device, including: a processor, a memory and a program stored on the memory and executable on the processor, the program implementing the steps of the mass data reconciliation method of the second aspect described above when executed by the processor.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the mass data reconciliation method in the second aspect are implemented.
The technical scheme provided by the embodiment of the invention has the beneficial effects that at least a mass data reconciliation system is provided, which comprises the following steps: the data access module comprises an external reconciliation data access module and an internal reconciliation data access module; the account checking module comprises a real-time account checking module and an offline account checking module; the real-time reconciliation module is used for carrying out real-time reconciliation on the internal reconciliation model data and acquiring a real-time reconciliation result; the offline reconciliation module is used for performing offline reconciliation on the internal reconciliation model data in the preset period and the external reconciliation model data in the preset period according to a preset period to obtain an offline reconciliation result; the tie account module comprises an unordered set processor and a tie account rule engine; the unordered set processor is used for generating account checking difference data and account checking data according to the real-time account checking result and/or the off-line account checking result; the tie-account rule engine is used for matching a corresponding tie-account rule to carry out tie-account according to the tie-account difference data, and storing tie-account failed tie-account data into a tie-account difference database if the tie-account fails; the tie-out database is used for storing all tie-out data; through the mass data account checking system, the problems of low account checking efficiency, complex account checking data access operation and the like in the prior art are solved.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1 is a schematic structural diagram of a mass data reconciliation system according to an embodiment of the present invention;
fig. 2 is a schematic diagram of another mass data reconciliation system architecture according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a massive data reconciliation system hierarchy according to an embodiment of the present invention;
fig. 4 is a flowchart of a mass data reconciliation method according to an embodiment of the present invention;
fig. 5 is an interaction diagram of a mass data reconciliation system according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a network device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, a schematic structural diagram of a mass data reconciliation system 100 according to an embodiment of the present invention is shown, where the system includes:
the data access module 10 comprises an external reconciliation data access module 11 and an internal reconciliation data access module 12;
the external reconciliation data access module 11 is configured to receive external reconciliation data and convert the external reconciliation data into external reconciliation model data;
the internal reconciliation data access module 12 is configured to receive internal reconciliation data and convert the internal reconciliation data into internal reconciliation model data;
the account checking module 20 comprises a real-time account checking module 22 and an offline account checking module 21;
the real-time account checking module 22 is configured to perform real-time account checking on the internal account checking model data to obtain a real-time account checking result;
the offline reconciliation module 21 is configured to perform offline reconciliation on the internal reconciliation model data in the preset period and the external reconciliation model data in the preset period according to a preset period, and obtain an offline reconciliation result;
a tie-out module 30 including an unordered collection processor 31 and a tie-out rule engine 32;
the unordered set processor 31 is configured to generate reconciliation difference data and reconciliation data according to the real-time reconciliation result and/or the offline reconciliation result;
the tie-out rule engine 32 is configured to match a corresponding tie-out rule for tie-out according to the tie-out difference data, and if the tie-out fails, store the tie-out data of which the tie-out fails in the tie-out difference database 40;
and the tie account database 50 is used for storing all tie account data.
Specifically, the mass data reconciliation system 100 receives and converts internal and external reconciliation data into corresponding reconciliation model data through the data access module 10, and finishes sorting, converting and collecting the reconciliation data according to preset requirements; the account checking module 20 realizes two functions of real-time account checking and offline account checking, provides real-time account checking for internal account checking data, and provides offline account checking data for internal and external data, thereby ensuring that both account checking and account checking can be obtained for external data; the reconciliation module 30 obtains reconciliation difference data and reconciliation data from the reconciliation model data through the unordered set processor 31, and then calls a reconciliation rule through the reconciliation rule engine 32 to reconcile the reconciliation difference data; and finally, the data which cannot be leveled through the leveling rule engine 32 is stored in the leveling difference database 40, and all account checking data which are leveled are stored in the leveling database 50, so that account checking of internal and external data can be completed without manual operation, account checking of the account checking difference data is automatically performed, and account checking results are classified and stored, and the problems that the account checking efficiency is low, the account checking data access operation is complex and the like in the prior art are solved.
Optionally, the unordered set processor 31 is further configured to:
carrying out real-time account checking includes: generating a reconciliation model data set of each internal business system according to the internal reconciliation model data, and comparing the reconciliation model data sets of each internal business system to generate a reconciliation difference data set;
generating reconciliation model data sets of the internal business system and the external system according to the external reconciliation model data, and comparing the reconciliation model data sets of the internal business system and the external system to generate a reconciliation difference data set.
Specifically, referring to fig. 3, an architecture diagram of a mass data reconciliation system provided by the embodiment of the present invention is shown, fig. 5 is an interaction diagram of the mass data reconciliation system provided by the embodiment of the present invention, and the mass data reconciliation system 100 is further described below according to fig. 3 and fig. 5.
The mass data reconciliation system 100 can be divided into three layers: the system comprises a data access layer, a reconciliation layer and a data layer.
The data access layer is responsible for converting original data into model data to be checked, wherein the original data comprises data in various formats such as binlog, txt, csv, json, xml and the like; the data access layer reads the data cleaning configuration, cleans the data according to the corresponding configuration, and stores the data in a database, such as a reconciliation model database; illustratively, a TiDB database may be employed for storage.
Specifically, the data accessed by the data access layer includes internal reconciliation data and external reconciliation data, and in combination with the schematic structural diagram of the mass data reconciliation system 100 shown in fig. 1-2, the external reconciliation data can be received by the external reconciliation data access module 11 and converted into external reconciliation model data; internal reconciliation data is received and converted into internal reconciliation model data by the internal reconciliation data access module 12.
Referring to fig. 5, the external reconciliation data is mainly a bank reconciliation file, or other external data may be obtained, generally obtained according to a network protocol (FTP/HTTP), decrypted and cleaned uniformly to be external reconciliation model data, and re-encrypted for sensitive fields to ensure security, the internal reconciliation data is derived from multiple service systems inside the system, and the internal reconciliation data can be resolved into internal reconciliation model data through a binary log binlog of a kafka channel (see below in the principle of open source components).
Illustratively, the internal reconciliation data with reference to FIG. 5 may be internal reconciliation data generated separately by a business system such as A, B; the external reconciliation data is bank D docked outside the system, as well as other external data.
The reconciliation layer mainly relates to five functions, and is completed through a reconciliation module 20, a tie account module 30 and subordinate modules thereof:
the real-time account checking function: the method is mainly used for reconciliation of internal data, and can be directly written into a database after binlog data is analyzed, such as a TiDB database. Specifically, the real time reconciliation is to check the order numbers and the states on both sides, so that only the order numbers need to be maintained, the order numbers are not long, the space pressure caused by a large number of keywords is not high, the set can be divided according to the states of the orders, then the unordered set processor 31 is used for reconciling the order data in real time, the efficiency can be improved by making a difference set on the order data and generating a set to be reconciled, and the time complexity is O (N). Illustratively, the difference set can be made by an unordered set processor using aset.difference (BSet) and generating a to-be-tie set CSet, where CSet is the difference set data, and the difference set data can be consumed for processing by setting difference set data processing cset.pop ().
It is worth noting that the performance bottleneck of reconciliation may need to repeatedly inquire a plurality of business parties to try to tie account mainly when the reconciliation is performed, and in the case of very large data volume, the embodiment of the invention utilizes a distributed computing horizontal expansion machine to accelerate the verification and cancellation of reconciliation difference data to generate a reconciliation data set. After the internal reconciliation model data are compared to obtain differences, the reconciliation difference data are processed by a plurality of reconciliation rule engines 32, for example, each batch of CSet.PopN (100) processes 100 pieces of reconciliation difference data, optionally, when a mass data scene is carried out, the data are obtained through the CSet.PopN (100) and then are sent to a message queue, the multi-consumer consumption message queue can realize transverse expansion, the quantity of consumer machines is transversely expanded by utilizing distributed calculation to accelerate the reconciliation of the reconciliation difference data to generate the reconciliation data, and when the reconciliation consumer processes, the rule engines can load the rule engine configuration stored in the data layer to try to reconcile the reconciliation difference data according to the configuration rules. And if the automatic account balancing fails, storing account checking difference data of the account balancing failure into an account checking difference data set, giving an alarm, and informing other business personnel of manual intervention processing.
Offline reconciliation function: the method is mainly used for reconciliation of internal reconciliation data and external reconciliation data, a plurality of upstream and downstream system service data in the system can also be reconciled, an offline object can be started according to a preset period, if the offline object is started on T +1 day, the data of the previous day is loaded into an order data set of an unordered set processor, then the unordered set processor 31 is called, a reconciliation difference data set is generated, and a to-be-reconciled data set is generated.
It should be noted that, unlike real-time reconciliation, there are many reconciliation difference data generated in the offline reconciliation part, and at this time, the reconciliation difference data is consumed and processed by calling the tie-account rule engine similarly to real-time reconciliation but the data is consumed from the difference set by using the processing capacity of multiple consumers by using the distributed tie-account rule engine, so as to solve the problem of the reconciliation and tie-account efficiency of mass data. The reconciliation operator will check the reconciliation difference data stored in the reconciliation difference database 40 according to the received alarm, try manual reconciliation, and if the reconciliation is successful, change the status of the reconciliation difference data successfully leveled into the reconciliation data and store the reconciliation data into the reconciliation database 50.
Daily cutting processing function: for the offline reconciliation function, the offline reconciliation task on the current day cannot be completed due to too large quantity, for example, a large amount of transaction data is usually generated when the quantity is close to early morning every night, the reconciliation data may be too late to be reconciled, because an external system, such as a bank, usually switches the accounting date in advance, the data after the time node is regarded as the data on the next day and cannot be recorded into the reconciliation data on the current day, and therefore, the daily reconciliation rule can be read according to the daily reconciliation habit of different banks; the data after the daily tangent point can be regarded as the data to be checked and tie-up on the next day, is not directly marked as difference data, and the data of the daily tangent point can be checked again on the next day T + 2.
The account leveling function: the tie-out rule is realized by a tie-out rule engine, and the tie-out processing rules are called by the tie-out rule engine 31, so that various tie-out rules can be matched for different types of tie-out data batches to try tie-out, for example: and trying to find whether the system A has a single number or not, trying to find the system B if the system A fails, then failing to judge whether the type is offline operation or result type or not, and calling a plurality of tie-down rules stored in the database to try to tie down continuously. If the balance can not be finished, the balance requirement is considered not to be met, and at the moment, a corresponding preset function is called for processing (data is usually written in, and daily cutting judgment can also be carried out); if the account balancing can not be finished finally, the automatic account balancing fails, and account checking difference data of the account balancing failure is stored in the account checking difference database 40 to wait for manual account balancing.
Data encryption and decryption functions: all the transactions with account checking data as ciphertexts need to be decrypted and configured to the mass data account checking system 100, and the account checking system 100 loads the secret key on a data access layer and analyzes plaintext data; after the plaintext account checking data are analyzed, according to the type of the account checking data, the secret key of the account checking system 100 is used for encrypting the plaintext account checking data into a ciphertext, and user sensitive data are protected. It should be noted that the mass data reconciliation system 100 stores the key by itself because of the convenience and efficiency of operation and the simplicity of interaction, and may also call the ciphertext of the external service interface to encrypt and decrypt the sensitive data in order to improve the security.
Specifically, the data layer is mainly used for storing various types of data, and may include one or more databases, such as: a tie-out model database 70, a tie-out difference database 40, a cloud storage module 80, and a tie-out database 50.
The reconciliation model database 70 is used for storing internal and external reconciliation model data, the reconciliation model data are extracted according to fields required by reconciliation, such as single number, amount, time key words and the like, different business order data and reconciliation data provided by external systems such as banks have a plurality of business attribute fields, but only partial key fields are required for reconciliation.
The reconciliation difference database 40 is used for storing reconciliation difference data, namely the reconciliation difference data of reconciliation failure after the reconciliation failure is passed through the reconciliation rule engine 50, and after the reconciliation failure data are stored in the reconciliation difference database 40, manual processing and investigation can be carried out at a reconciliation operator, the data are generally data generated due to imperfect reconciliation rules and possibly data which does not exist in an offline generation system, and after the manual reconciliation is carried out on the part, the part can be marked as the completion of the reconciliation and then is transferred into the reconciliation database 50.
The tie-out database 50 is used for storing all tie-out data which are used for finishing tie-out, including parameter information such as tie-out process of the data.
And the cloud storage module 80 can be used for storing the configuration information of the tie-out rule engine, the data cleaning configuration, the internal tie-out model data, the external tie-out model data, the real-time tie-out result and the offline tie-out result.
Specifically, the rule engine configuration, that is, the rule for how to level the account checking lot of the specified type, configures how to search according to the account checking lot type configuration, and configures the rule according to the leveling rule such as the search priority.
The data cleaning configuration is used for a data access layer, describing a conversion rule, generally considering that the accessed data is a two-dimensional array, calling various functions to process and clean the data into model data after the position of the two-dimensional data is appointed to be taken out, and describing the conversion rule for the data access layer.
It should be noted that the storage database of the mass data reconciliation system 100 adopts a cold and hot combination mode, and stores various generated reconciliation data, query data, reconciliation rules and other data through three types of databases, so as to meet the requirement for mass data reconciliation:
the first type is a TiDB database, external/internal reconciliation model data, reconciliation difference data, tie account data and the like exist in TiDB types, the TiDB type database is used as a tie account data storage database, the data support magnitude is improved, tens of millions of levels of data are supported, the TiDB is an infinite database, PB level data can be supported by expanding a cluster machine, and referring to fig. 5, the reconciliation model database, the tie account database and the difference database can be the TiDB database.
The second type is a distributed file system which is mainly used for exporting cold data with the storage time more than the preset storage time in the TiDB type database to a file to be stored in the distributed file system, and part of the data is basically not used but needs to be stored and cannot be lost, so that the services of historical data query and the like are facilitated, the data volume of the TiDB is reduced by transferring the data to the distributed file system, and the cost can be saved. Specifically, the original reconciliation document, the original data representing the external business system such as the bank, and the original data of each internal business system all need to be permanently stored.
The third type is a MySQL database which mainly stores tie account rule engine configuration, key configuration for data encryption and decryption, administrator information authority information and the like.
Optionally, the system 100 further comprises;
the account checking difference database 40 is used for storing account checking difference data which cannot be generated by the account checking rule engine 32 through account checking and sending the account checking difference data to the manual account checking;
and the warning module 60 is configured to, if the target reconciliation difference data cannot be leveled after manual reconciliation, warn the target reconciliation difference data.
Through the warning module 60, an administrator of the mass data reconciliation system can conveniently find the data of the reconciliation failure in time and carry out manual reconciliation treatment in time, so that the reconciliation efficiency and the comprehensiveness and the integrity of the reconciliation data treatment are improved.
Optionally, the offline reconciliation module 22 is further configured to:
and when the offline reconciliation of the internal reconciliation model data in the preset period and the external reconciliation model data in the preset period cannot be completed in time, storing the internal and external model data which are not subjected to the reconciliation into the next period, and continuing the offline reconciliation in the next period.
Specifically, if offline reconciliation is performed in a period of 24 hours, if there are unfinished reconciliation model data in the period, the reconciliation model data are stored in the next period, and are used as the data to be reconciled and reconciled in the next period, and offline reconciliation is continuously performed in the next period.
Optionally, the tie-out rules engine 32 is further configured to:
and when account checking is started for each account checking batch, loading a tie-account rule according to the type of the account checking batch, reloading the updated tie-account rule if the configuration of the tie-account rule is changed, and checking the corresponding account checking batch by using the updated tie-account rule.
Specifically, the tie-out rule engine 32 loads the tie-out rules when the tie-out is started for each tie-out batch, the tie-out rule engine 32 reads the latest configuration from the database (such as MySQL database) in which the tie-out rules are stored, and if the rules are changed, the latest configuration is updated into the MySQL configuration table, and at this time, the change can be made in real time by re-checking, and the code does not need to be changed and the tie-out efficiency can be greatly improved.
Furthermore, for providing a visual interface, a worker can also manually drag to introduce a new tie-out rule, or modify configuration information, and the like.
Optionally, referring to fig. 2, the system 100 further includes:
a reconciliation model database 70 for storing the external reconciliation model data and the internal reconciliation model data;
the cloud storage module 80 is used for regularly backing up tie-account data in the tie-account database, and regularly backing up the external tie-account model data and the internal tie-account model data in the tie-account model database;
and the query module 90 is configured to provide corresponding data in the tie-out model database and/or the cloud storage module based on a manually triggered query condition.
Specifically, the reconciliation model database 70 may adopt a TiDB database, and the original reconciliation data can be queried through a query module, and the reconciliation data of each link includes: the system comprises internal account checking data, internal account checking model data, account checking difference data, account balancing data, external account checking model data, data processed in an artificial account balancing link, specific difference marks of the account checking difference data, an account balancing mode and account balancing rules adopted by the account balancing data, and the whole process of account checking and account balancing can be completely displayed, so that the system has reliability.
In particular, with regard to the unordered set processor 31, this may be implemented by:
the map based on the go language realizes the unordered set processor, supports various types including characters, integer and the like, defines and realizes the set as the following codes, and also internally provides functions such as Add () for adding elements to the unordered set, comparison function contexts () for two unordered sets, whether a subset IsSubset () is provided, diff two sets are combined and return a Difference set Difference (), and functions such as Pop () for returning and deleting one element are provided, so that the unordered set processor is realized.
Illustratively, the difference data set is generated by:
func (set *threadUnsafeSet) Difference(other Set) Set {
_ = other.(*threadUnsafeSet)
difference := newThreadUnsafeSet()
for elem := range *set {
if !other.Contains(elem) {
difference.Add(elem)
}
}
return &difference
}
and setting a comparison function of the unordered set as A B two sets, and through traversing the set A (time complexity O (N)), judging whether the sets exist in the set B (time complexity O (1)) one by one, if not, the sets are regarded as difference data sets and put into the difference set C, returning the difference set C after comparison is completed, and at the moment, performing processing such as consumption Pop () on the difference set C.
It should be noted that the unordered set processor 31 is not limited to the above-described embodiment, and the unordered set processor based on the comparison function of the unordered set may be configured by a plurality of language tools. Generating a reconciliation model data set according to the key words by converting the reconciliation data into the reconciliation model data, and then comparing the internal reconciliation model data set and the external reconciliation model data set of different internal business systems, thereby obtaining a reconciliation difference data set.
For example, the internal business system A, B, C and the external bank D respectively generate an internal reconciliation model data set a, an internal reconciliation model data set B, an internal reconciliation model data set C and an external reconciliation model data set D, and respectively compare data sets AB, AC, AD, BC, BD and CD pairwise according to the unordered set comparison function to generate a reconciliation difference data set, and also can design different unordered set comparison functions to simultaneously realize synchronous comparison of a plurality of data sets.
The complete implementation does not require the collection of processor reference code:
// non-thread Security set
type threadUnsafeSet map[interface{}]struct{}
// An OrderedPair represents a 2-tuple of values.
type OrderedPair struct {
First interface{}
Second interface{}
}
func newThreadUnsafeSet() threadUnsafeSet {
return make(threadUnsafeSet)
}
// Equal says whether two 2-tuples contain the same values in the same order.
func (pair *OrderedPair) Equal(other OrderedPair) bool {
if pair.First == other.First &&
pair.Second == other.Second {
return true
}
return false
}
func (set *threadUnsafeSet) Add(i interface{}) bool {
_, found := (*set)[i]
if found {
return false //False if it existed already
}
(*set)[i] = struct{}{}
return true
}
func (set *threadUnsafeSet) Contains(i ...interface{}) bool {
for _, val := range i {
if _, ok := (*set)[val]; !ok {
return false
}
}
return true
}
func (set *threadUnsafeSet) IsSubset(other Set) bool {
_ = other.(*threadUnsafeSet)
if set.Cardinality() > other.Cardinality() {
return false
}
for elem := range *set {
if !other.Contains(elem) {
return false
}
}
return true
}
func (set *threadUnsafeSet) IsProperSubset(other Set) bool {
return set.IsSubset(other) && !set.Equal(other)
}
func (set *threadUnsafeSet) IsSuperset(other Set) bool {
return other.IsSubset(set)
}
func (set *threadUnsafeSet) IsProperSuperset(other Set) bool {
return set.IsSuperset(other) && !set.Equal(other)
}
func (set *threadUnsafeSet) Union(other Set) Set {
o := other.(*threadUnsafeSet)
unionedSet := newThreadUnsafeSet()
for elem := range *set {
unionedSet.Add(elem)
}
for elem := range *o {
unionedSet.Add(elem)
}
return &unionedSet
}
func (set *threadUnsafeSet) Intersect(other Set) Set {
o := other.(*threadUnsafeSet)
intersection := newThreadUnsafeSet()
// loop over smaller set
if set.Cardinality() < other.Cardinality() {
for elem := range *set {
if other.Contains(elem) {
intersection.Add(elem)
}
}
} else {
for elem := range *o {
if set.Contains(elem) {
intersection.Add(elem)
}
}
}
return &intersection
}
func (set *threadUnsafeSet) Difference(other Set) Set {
_ = other.(*threadUnsafeSet)
difference := newThreadUnsafeSet()
for elem := range *set {
if !other.Contains(elem) {
difference.Add(elem)
}
}
return &difference
}
func (set *threadUnsafeSet) SymmetricDifference(other Set) Set {
_ = other.(*threadUnsafeSet)
aDiff := set.Difference(other)
bDiff := other.Difference(set)
return aDiff.Union(bDiff)
}
func (set *threadUnsafeSet) Clear() {
*set = newThreadUnsafeSet()
}
func (set *threadUnsafeSet) Remove(i interface{}) {
delete(*set, i)
}
func (set *threadUnsafeSet) Cardinality() int {
return len(*set)
}
func (set *threadUnsafeSet) Each(cb func(interface{}) bool) {
for elem := range *set {
if cb(elem) {
break
}
}
}
func (set *threadUnsafeSet) Iter() <-chan interface{} {
ch := make(chan interface{})
go func() {
for elem := range *set {
ch <- elem
}
close(ch)
}()
return ch
}
func (set *threadUnsafeSet) Iterator() *Iterator {
iterator, ch, stopCh := newIterator()
go func() {
L:
for elem := range *set {
select {
case <-stopCh:
break L
case ch <- elem:
}
}
close(ch)
}()
return iterator
}
func (set *threadUnsafeSet) Equal(other Set) bool {
_ = other.(*threadUnsafeSet)
if set.Cardinality() != other.Cardinality() {
return false
}
for elem := range *set {
if !other.Contains(elem) {
return false
}
}
return true
}
func (set *threadUnsafeSet) Clone() Set {
clonedSet := newThreadUnsafeSet()
for elem := range *set {
clonedSet.Add(elem)
}
return &clonedSet
}
func (set *threadUnsafeSet) String() string {
items := make([]string, 0, len(*set))
for elem := range *set {
items = append(items, fmt.Sprintf("%v", elem))
}
return fmt.Sprintf("Set{%s}", strings.Join(items, ", "))
}
// String outputs a 2-tuple in the form "(A, B)".
func (pair OrderedPair) String() string {
return fmt.Sprintf("(%v, %v)", pair.First, pair.Second)
}
// pop a item
func (set *threadUnsafeSet) Pop() interface{} {
for item := range *set {
delete(*set, item)
return item
}
return nil
}
func (set *threadUnsafeSet) CartesianProduct(other Set) Set {
o := other.(*threadUnsafeSet)
cartProduct := NewThreadUnsafeSet()
for i := range *set {
for j := range *o {
elem := OrderedPair{First: i, Second: j}
cartProduct.Add(elem)
}
}
return cartProduct
}
func (set *threadUnsafeSet) ToSlice() []interface{} {
keys := make([]interface{}, 0, set.Cardinality())
for elem := range *set {
keys = append(keys, elem)
}
return keys
}
// MarshalJSON creates a JSON array from the set, it marshals all elements
func (set *threadUnsafeSet) MarshalJSON() ([]byte, error) {
items := make([]string, 0, set.Cardinality())
for elem := range *set {
b, err := json.Marshal(elem)
if err != nil {
return nil, err
}
items = append(items, string(b))
}
return []byte(fmt.Sprintf("[%s]", strings.Join(items, ","))), nil
}
// UnmarshalJSON recreates a set from a JSON array, it only decodes
// primitive types. Numbers are decoded as json.Number.
func (set *threadUnsafeSet) UnmarshalJSON(b []byte) error {
var i []interface{}
d := json.NewDecoder(bytes.NewReader(b))
d.UseNumber()
err := d.Decode(&i)
if err != nil {
return err
}
for _, v := range i {
switch t := v.(type) {
case []interface{}, map[string]interface{}:
continue
default:
set.Add(t)
}
}
return nil
}
Further, based on the non-thread security set, the thread security set is realized through a sync.RWMetex read-write lock of golang
type threadSafeSet struct {
s threadUnsafeSet
sync.RWMutex
}
func newThreadSafeSet() threadSafeSet {
return threadSafeSet{s: newThreadUnsafeSet()}
}
func (set *threadSafeSet) Add(i interface{}) bool {
set.Lock()
ret := set.s.Add(i)
set.Unlock()
return ret
}
func (set *threadSafeSet) Contains(i ...interface{}) bool {
set.RLock()
ret := set.s.Contains(i...)
set.RUnlock()
return ret
}
func (set *threadSafeSet) IsSubset(other Set) bool {
o := other.(*threadSafeSet)
set.RLock()
o.RLock()
ret := set.s.IsSubset(&o.s)
set.RUnlock()
o.RUnlock()
return ret
}
func (set *threadSafeSet) IsProperSubset(other Set) bool {
o := other.(*threadSafeSet)
set.RLock()
defer set.RUnlock()
o.RLock()
defer o.RUnlock()
return set.s.IsProperSubset(&o.s)
}
func (set *threadSafeSet) IsSuperset(other Set) bool {
return other.IsSubset(set)
}
func (set *threadSafeSet) IsProperSuperset(other Set) bool {
return other.IsProperSubset(set)
}
func (set *threadSafeSet) Union(other Set) Set {
o := other.(*threadSafeSet)
set.RLock()
o.RLock()
unsafeUnion := set.s.Union(&o.s).(*threadUnsafeSet)
ret := &threadSafeSet{s: *unsafeUnion}
set.RUnlock()
o.RUnlock()
return ret
}
func (set *threadSafeSet) Intersect(other Set) Set {
o := other.(*threadSafeSet)
set.RLock()
o.RLock()
unsafeIntersection := set.s.Intersect(&o.s).(*threadUnsafeSet)
ret := &threadSafeSet{s: *unsafeIntersection}
set.RUnlock()
o.RUnlock()
return ret
}
func (set *threadSafeSet) Difference(other Set) Set {
o := other.(*threadSafeSet)
set.RLock()
o.RLock()
unsafeDifference := set.s.Difference(&o.s).(*threadUnsafeSet)
ret := &threadSafeSet{s: *unsafeDifference}
set.RUnlock()
o.RUnlock()
return ret
}
func (set *threadSafeSet) SymmetricDifference(other Set) Set {
o := other.(*threadSafeSet)
set.RLock()
o.RLock()
unsafeDifference := set.s.SymmetricDifference(&o.s).(*threadUnsafeSet)
ret := &threadSafeSet{s: *unsafeDifference}
set.RUnlock()
o.RUnlock()
return ret
}
func (set *threadSafeSet) Clear() {
set.Lock()
set.s = newThreadUnsafeSet()
set.Unlock()
}
func (set *threadSafeSet) Remove(i interface{}) {
set.Lock()
delete(set.s, i)
set.Unlock()
}
func (set *threadSafeSet) Cardinality() int {
set.RLock()
defer set.RUnlock()
return len(set.s)
}
func (set *threadSafeSet) Each(cb func(interface{}) bool) {
set.RLock()
for elem := range set.s {
if cb(elem) {
break
}
}
set.RUnlock()
}
func (set *threadSafeSet) Iter() <-chan interface{} {
ch := make(chan interface{})
go func() {
set.RLock()
for elem := range set.s {
ch <- elem
}
close(ch)
set.RUnlock()
}()
return ch
}
func (set *threadSafeSet) Iterator() *Iterator {
iterator, ch, stopCh := newIterator()
go func() {
set.RLock()
L:
for elem := range set.s {
select {
case <-stopCh:
break L
case ch <- elem:
}
}
close(ch)
set.RUnlock()
}()
return iterator
}
func (set *threadSafeSet) Equal(other Set) bool {
o := other.(*threadSafeSet)
set.RLock()
o.RLock()
ret := set.s.Equal(&o.s)
set.RUnlock()
o.RUnlock()
return ret
}
func (set *threadSafeSet) Clone() Set {
set.RLock()
unsafeClone := set.s.Clone().(*threadUnsafeSet)
ret := &threadSafeSet{s: *unsafeClone}
set.RUnlock()
return ret
}
func (set *threadSafeSet) String() string {
set.RLock()
ret := set.s.String()
set.RUnlock()
return ret
}
func (set *threadSafeSet) Pop() interface{} {
set.Lock()
defer set.Unlock()
return set.s.Pop()
}
func (set *threadSafeSet) CartesianProduct(other Set) Set {
o := other.(*threadSafeSet)
set.RLock()
o.RLock()
unsafeCartProduct := set.s.CartesianProduct(&o.s).(*threadUnsafeSet)
ret := &threadSafeSet{s: *unsafeCartProduct}
set.RUnlock()
o.RUnlock()
return ret
}
func (set *threadSafeSet) ToSlice() []interface{} {
keys := make([]interface{}, 0, set.Cardinality())
set.RLock()
for elem := range set.s {
keys = append(keys, elem)
}
set.RUnlock()
return keys
}
func (set *threadSafeSet) MarshalJSON() ([]byte, error) {
set.RLock()
b, err := set.s.MarshalJSON()
set.RUnlock()
return b, err
}
func (set *threadSafeSet) UnmarshalJSON(p []byte) error {
set.RLock()
err := set.s.UnmarshalJSON(p)
set.RUnlock()
return err
}
Based on the implementation case method, an unordered set processor can be deployed on any machine, data is taken out and then put into a message queue through a method that messages are not consumed directly, the cluster consumption is expanded through the method, and the method comprises the following steps:
machine deploying unordered set processors- > message queues (which may be kafka or other message queues) > multiple consumers participate in consumption to enable support for mass data reconciliation.
In summary, the mass data reconciliation system provided by the embodiment of the invention can achieve the following technical effects:
the real-time account checking is realized, the condition that data in the distributed transaction system are inconsistent is found after second-level delay (about 10 seconds), the mode of generating a difference set by comparing account checking model data and combining the account checking model data which are converted into corresponding account checking data by an internal and external account checking data and an unordered set processor can be realized, account checking difference data are rapidly compared, and the account checking efficiency is greatly improved.
The real-time account checking method provided by the invention realizes the high requirements of financial transaction data on financial safety and reliability, service response speed and convenience, the financial transaction scene is different from the traditional data account checking, the urgent need is provided for solving the problem of preventing capital loss when the problem is found in the second level, and the problem of speed reduction caused by large data volume is well solved.
The problem that account checking data access needs hard coding and online is solved, a preposed data processor is used as a general processor according to two layers of protocols and data types, and data processing of multiple data types such as SFTP, HTTP multiprotocol, XML, JSON, CSV, TXT, BINLOG and the like is supported.
The problem that the account reconciliation rule needs to be online again when the account reconciliation rule is accessed into a new rule and is changed is solved, the account reconciliation rule engine is introduced, the account reconciliation rule is converted from a traditional hard coding mode to be configured through the account reconciliation rule engine, the rule hot update is supported, the new account reconciliation rule can be configured when being generated, a page configuration tool is provided, and the account reconciliation efficiency is improved.
The method supports the increase of data magnitude, supports the reconciliation of mass data, introduces a TiDB database, and can support PB-level data by expanding a cluster machine.
The problem that a large amount of cold data does not need to be accessed normally but the storage cost is high is solved, and the data are archived from the TiDB to the distributed file system in a timing mode so as to save the storage cost and guarantee the integrity of the data.
Referring to fig. 4, a flow chart of a mass data reconciliation method provided by the present invention includes:
and step 47, storing all reconciliation data which are successfully reconciled and the reconciliation successful data into a reconciliation database.
Optionally, the method further includes:
and when the offline reconciliation of the internal reconciliation model data in the preset period and the external reconciliation model data in the preset period cannot be completed in time, storing the internal and external model data which are not subjected to the reconciliation into the next period, and continuing the offline reconciliation in the next period.
Optionally, the method further includes:
storing account checking difference data which cannot be generated by matching the corresponding account checking rule to finish the account checking, and sending the account checking difference data to the manual account checking;
and if the target account checking difference data still cannot complete account checking after manual account checking, alarming the target account checking difference data.
Optionally, the matching a corresponding tie-out rule for tie-out according to the tie-out difference data includes:
and when account checking is started for each account checking batch, loading a tie-account rule according to the type of the account checking batch, if the configuration of the tie-account rule is changed, reloading the updated tie-account rule, and performing account checking on the corresponding account checking batch by using the updated tie-account rule.
Optionally, the method further includes:
storing the external reconciliation model data and the internal reconciliation model data;
regularly backing up tie-account data in the tie-account database, and regularly backing up the external tie-account model data and the internal tie-account model data in the tie-account model database;
and providing corresponding data in the reconciliation model database and/or the cloud storage module based on the manually triggered query condition.
Optionally, the performing real-time reconciliation includes: generating a reconciliation model data set of each internal business system according to the internal reconciliation model data, and comparing the reconciliation model data sets of each internal business system to generate a reconciliation difference data set;
the performing offline reconciliation comprises: generating reconciliation model data sets of the internal business system and the external system according to the external reconciliation model data, and comparing the reconciliation model data sets of the internal business system and the external system to generate a reconciliation difference data set.
Referring to fig. 6, an embodiment of the present invention further provides a network device 600, which includes a processor 61, a memory 62, and a computer program that is stored in the memory 62 and is executable on the processor 61, and when the computer program is executed by the processor 61, the computer program implements each process of the foregoing embodiment of the mass data reconciliation method, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements each process of the above-mentioned embodiment of the mass data reconciliation method, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here. The computer-readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (12)
1. A mass data reconciliation system, the system comprising:
the data access module comprises an external reconciliation data access module and an internal reconciliation data access module; the external reconciliation data access module is used for receiving external reconciliation data and converting the external reconciliation data into external reconciliation model data; the internal reconciliation data access module is used for receiving internal reconciliation data and converting the internal reconciliation data into internal reconciliation model data;
the account checking module comprises a real-time account checking module and an offline account checking module; the real-time reconciliation module is used for carrying out real-time reconciliation on the internal reconciliation model data and acquiring a real-time reconciliation result; the offline reconciliation module is used for performing offline reconciliation on the internal reconciliation model data in the preset period and the external reconciliation model data in the preset period according to a preset period to obtain an offline reconciliation result;
the tie account module comprises an unordered set processor and a tie account rule engine;
the unordered set processor is used for generating account checking difference data and account checking data according to the real-time account checking result and/or the off-line account checking result;
the account reconciliation rule engine is used for matching the corresponding account reconciliation rule to perform account reconciliation according to the account reconciliation difference data, and storing the account reconciliation data with failed account reconciliation into an account reconciliation difference database if the account reconciliation fails;
and the tie account database is used for storing all tie account data.
2. A mass data reconciliation system according to claim 1 further comprising;
the account checking difference database is used for storing account checking difference data which cannot be generated by the account checking rule engine through account checking and sending the account checking difference data to the manual account checking;
and the warning module is used for warning the target account checking difference data if the target account checking difference data still cannot complete account checking after manual account checking.
3. The mass data reconciliation system of claim 1, wherein the offline reconciliation module is further configured to:
and when the offline reconciliation of the internal reconciliation model data in the preset period and the external reconciliation model data in the preset period cannot be completed in time, storing the internal and external model data which are not subjected to the reconciliation into the next period, and continuing the offline reconciliation in the next period.
4. The mass data reconciliation system of claim 1, wherein the tie rule engine is further configured to:
and when account checking is started for each account checking batch, loading a tie-account rule according to the type of the account checking batch, if the configuration of the tie-account rule is changed, reloading the updated tie-account rule, and performing account checking on the corresponding account checking batch by using the updated tie-account rule.
5. The mass data reconciliation system of claim 1, wherein the system further comprises:
the account checking model database is used for storing the external account checking model data and the internal account checking model data;
the cloud storage module is used for regularly backing up tie-account data in the tie-account database and regularly backing up the external tie-account model data and the internal tie-account model data in the tie-account model database;
and the query module is used for providing corresponding data in the reconciliation model database and/or the cloud storage module based on a manually triggered query condition.
6. A mass data reconciliation system according to claim 1 wherein the unordered set processor is further configured to:
generating a reconciliation model data set of each internal business system according to the internal reconciliation model data, and comparing the reconciliation model data sets of each internal business system to generate a reconciliation difference data set;
generating reconciliation model data sets of the internal business system and the external system according to the external reconciliation model data, and comparing the reconciliation model data sets of the internal business system and the external system to generate a reconciliation difference data set.
7. A mass data reconciliation method is characterized by comprising the following steps:
receiving reconciliation data, including receiving external reconciliation data and internal reconciliation data;
respectively converting the received external reconciliation data and internal reconciliation data into external reconciliation model data and internal reconciliation model data;
performing real-time account checking on the internal account checking model data to obtain a real-time account checking result;
performing offline reconciliation on the internal reconciliation model data in the preset period and the external reconciliation model data in the preset period according to a preset period, and acquiring an offline reconciliation result;
generating account checking difference data and account checking data according to the real-time account checking result and/or the off-line account checking result;
matching corresponding tie-in rules for tie-in according to the tie-in difference data, and storing the tie-in data with the failure of tie-in into a tie-in difference database if the tie-in fails;
and storing all the account checking data which are successfully checked and the data which are successfully checked into the account checking database.
8. The mass data reconciliation method of claim 7, wherein the method further comprises:
and when the offline reconciliation of the internal reconciliation model data in the preset period and the external reconciliation model data in the preset period cannot be completed in time, storing the internal and external model data which are not subjected to the reconciliation into the next period, and continuing the offline reconciliation in the next period.
9. The mass data reconciliation method of claim 7, wherein the method further comprises:
storing account checking difference data which cannot be generated by matching the corresponding account checking rule to finish the account checking, and sending the account checking difference data to the manual account checking;
and if the target account checking difference data still cannot complete account checking after manual account checking, alarming the target account checking difference data.
10. The mass data reconciliation method of claim 7, wherein the matching of the corresponding reconciliation rule for reconciliation according to the reconciliation difference data comprises:
and when account checking is started for each account checking batch, loading a tie-account rule according to the type of the account checking batch, if the configuration of the tie-account rule is changed, reloading the updated tie-account rule, and performing account checking on the corresponding account checking batch by using the updated tie-account rule.
11. A network device, comprising: processor, memory and program stored on the memory and executable on the processor, which when executed by the processor implements the steps of the mass data reconciliation method of any of claims 7 to 10.
12. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the mass data reconciliation method according to one of the claims 7 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210441004.8A CN114547113A (en) | 2022-04-26 | 2022-04-26 | Mass data account checking system, method, network equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210441004.8A CN114547113A (en) | 2022-04-26 | 2022-04-26 | Mass data account checking system, method, network equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114547113A true CN114547113A (en) | 2022-05-27 |
Family
ID=81666755
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210441004.8A Pending CN114547113A (en) | 2022-04-26 | 2022-04-26 | Mass data account checking system, method, network equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114547113A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115689793A (en) * | 2023-01-05 | 2023-02-03 | 翌飞锐特电子商务(北京)有限公司 | Interactive account checking method based on calculation model |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030149656A1 (en) * | 2002-02-07 | 2003-08-07 | Brian Magruder | Home asset management account |
| CN105243528A (en) * | 2015-09-15 | 2016-01-13 | 普元信息技术股份有限公司 | Financial IT system graphical centralized reconciliation system and method under big data environment |
| CN109685634A (en) * | 2018-09-11 | 2019-04-26 | 深圳平安财富宝投资咨询有限公司 | Pay account checking method, equipment, storage medium and device |
| CN112634006A (en) * | 2020-12-31 | 2021-04-09 | 湖南快乐阳光互动娱乐传媒有限公司 | Account checking processing method and device, electronic equipment and storage medium |
| CN112862578A (en) * | 2020-12-14 | 2021-05-28 | 苏宁消费金融有限公司 | Account checking system and device |
-
2022
- 2022-04-26 CN CN202210441004.8A patent/CN114547113A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030149656A1 (en) * | 2002-02-07 | 2003-08-07 | Brian Magruder | Home asset management account |
| CN105243528A (en) * | 2015-09-15 | 2016-01-13 | 普元信息技术股份有限公司 | Financial IT system graphical centralized reconciliation system and method under big data environment |
| CN109685634A (en) * | 2018-09-11 | 2019-04-26 | 深圳平安财富宝投资咨询有限公司 | Pay account checking method, equipment, storage medium and device |
| CN112862578A (en) * | 2020-12-14 | 2021-05-28 | 苏宁消费金融有限公司 | Account checking system and device |
| CN112634006A (en) * | 2020-12-31 | 2021-04-09 | 湖南快乐阳光互动娱乐传媒有限公司 | Account checking processing method and device, electronic equipment and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115689793A (en) * | 2023-01-05 | 2023-02-03 | 翌飞锐特电子商务(北京)有限公司 | Interactive account checking method based on calculation model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109739929A (en) | Method of data synchronization, apparatus and system | |
| CN106656536A (en) | Method and device for processing service invocation information | |
| CN111563101B (en) | Execution plan optimization method, device, equipment and storage medium | |
| CN107103064B (en) | Data statistical method and device | |
| CN108062243A (en) | Generation method, task executing method and the device of executive plan | |
| CN112488714A (en) | Method for linking remark field based on block chain transaction process | |
| CN103235811A (en) | Data storage method and device | |
| CN114547113A (en) | Mass data account checking system, method, network equipment and storage medium | |
| CN107870982B (en) | Data processing method, system and computer readable storage medium | |
| US20110071933A1 (en) | System For Surveillance Of Financial Data | |
| CN110297822A (en) | Key management method, device, equipment and storage medium towards block chain | |
| CN109146672B (en) | Matching method and device based on production mode | |
| CN115757642A (en) | Data synchronization method and device based on filing log file | |
| CN113672776B (en) | Fault analysis method and device | |
| CN115564429A (en) | System for prepaid transaction monitoring and related methods and blockchains | |
| CN114138760A (en) | Data management method, device, equipment and storage medium | |
| CN116975117A (en) | Log data management method and device | |
| CN110634073B (en) | Transaction freezing method and system | |
| CN112101894A (en) | Coal dressing intelligent system | |
| CN117692877B (en) | Charging C++ application-oriented distributed message distribution method and system | |
| CN117708219B (en) | Processing method, processing device and storage medium for data of Internet of things | |
| CN115577985B (en) | Multi-equipment consensus method for power block chain | |
| US10936571B1 (en) | Undo based logical rewind in a multi-tenant system | |
| CN107426027A (en) | A kind of guard method of big data resource | |
| Blanchot | Solving large-scale stochastic optimization programs: application to investment problems for power systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |