CN110910238B - Data processing method and device for bank system - Google Patents
Data processing method and device for bank system Download PDFInfo
- Publication number
- CN110910238B CN110910238B CN201911149205.5A CN201911149205A CN110910238B CN 110910238 B CN110910238 B CN 110910238B CN 201911149205 A CN201911149205 A CN 201911149205A CN 110910238 B CN110910238 B CN 110910238B
- Authority
- CN
- China
- Prior art keywords
- data
- subsystem
- batch
- day
- transaction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
Landscapes
- Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Engineering & Computer Science (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data processing method and a data processing device of a bank system, wherein the data processing method comprises the following steps: after the accounting date is switched, exporting data to be processed in the online subsystem to the batch subsystem; the batch subsystem extracts and obtains T-day data from the data to be processed; the batch subsystem performs batch processing on the T-day data to obtain processing result data; and the batch subsystem writes the processing result data back to the online subsystem. The data processing method and the data processing device of the bank system can acquire static data of the day before the day of the switching under the condition of uninterrupted online transaction for 7x24 hours for batch processing after the accounting day is switched, and write back processing result data to the online subsystem after the batch processing.
Description
Technical Field
The invention relates to a data processing technology, in particular to a data processing method and device of a bank system.
Background
Making an online transaction on the bank's core system, it is very likely that a large amount of background database table information will need to be updated, and if every online transaction updates all the tables involved in the transaction, it is impractical with the existing computer capabilities while maintaining high responsiveness to the customer. Therefore, the difficult problem is solved in a core bank system by means of on-line transaction and batch processing, wherein only the balance of the transaction and the states of some transactions are recorded during on-line transaction, and account billing, tax accounting, lifting, expense, accounting and the like are processed during batch processing.
Typical batch processing of banks is divided into three parts of night cutting, end-of-day batch and day cutting. However, during batch processing, the batch needs a static T-day final data for processing, and if the data is updated by online transaction, the batch cannot acquire the accurate T-day data. For example, while a batch requires a balance at the end of day T for interest processing, a customer may perform a deposit and withdrawal transaction (online transaction) in the morning, which may cause a change in the balance of the account, but the change should be reflected in day T +1, and the batch cannot perform data processing for day T with the changed data, that is, the batch requires data that is pure for day T and is not contaminated by transactions for day T + 1.
In order to ensure accurate data of T days, in the prior art, a database-dividing and table-dividing mode exists, namely system data is divided into a day database and a night database, the day database is operated in an online manner in the daytime, the day database is processed in a batch manner at night, the night database is operated in an online manner, and after batch processing is completed, online transaction of the night database is performed in the day database for repasting. Wherein, for a 7x 24-hour system in a database-based and table-based mode, the batch subsystem processes the daily schedule and the online subsystem processes the night schedule during the batch period. After the batch subsystem completes the batch, the operation of the day system of the day table is synchronized back to the online subsystem's transaction in the night table.
For the database and table dividing structure, transactions coming at T +1 night need to be redone once in the daytime through replication so as to ensure that transaction data sent by the night table is synchronized to the day table. However, on special days, such as the days promoted by business activities, such as twenty-one, night transactions will increase sharply, the pressure of the whole system in the repasting stage will also increase sharply, the batch time of the whole repasting will also be greatly prolonged, and the time of day-cutting operation may be stressed.
Disclosure of Invention
In view of this, the present invention provides a data processing method and device for a bank system, so as to overcome the problem that in the prior art, a database-based and table-based architecture adopts replication to perform data synchronization, which brings time-to-day pressure to the system.
In order to achieve the purpose, the invention provides the following technical scheme:
a data processing method of a banking system comprises the following steps:
after the accounting date is switched, exporting data to be processed in the online subsystem to the batch subsystem;
the batch subsystem extracts and obtains T day data from the data to be processed;
the batch subsystem performs batch processing on the T-day data to obtain processing result data;
and the batch subsystem writes the processing result data back to the online subsystem.
Optionally, exporting the data to be processed in the online subsystem to the batch subsystem includes:
after the accounting date is switched, the process of exporting the data to be processed in the online subsystem to the batch subsystem is a date switching window period, and in the date switching window period, if transaction data are obtained, data related to the transaction data on day T are copied to a shadow table, and then the transaction data are updated to a transaction main table;
and exporting the transaction main table and the shadow table in the online subsystem to a batch subsystem.
Optionally, in the date switching window period, if transaction data is acquired, first copying data related to the transaction data on the T date to a shadow table, and then updating the transaction data to a transaction main table, where the method includes:
in the date cutting window period, if the content of the first table relates to modification, checking whether the last modification date is T +1 day;
if not, after copying the content of the first table into a shadow table, modifying the first table.
Optionally, the obtaining, by the batch subsystem, T-day data from the data to be processed includes:
the batch subsystem covers the data in the shadow table on the corresponding data in the transaction main table to obtain the transaction main table containing static data of T days;
and determining the T-day static data as T-day data.
Optionally, the structures of the transaction main table and the shadow table are the same.
Optionally, the exporting the data to be processed in the online subsystem to the batch subsystem includes:
and acquiring full data from the online subsystem in a dirty reading mode through a heterogeneous data source offline synchronization tool DataX and synchronizing the full data to the batch subsystem.
Optionally, the batch subsystem writes the processing result data back to the contact subsystem through log transaction.
Optionally, the writing back, by the batch subsystem, the processing result data to the online subsystem includes:
and the batch subsystem directly writes the processing result data operation back to the online subsystem.
Optionally, after the batch subsystem writes the processing result data back to the online subsystem, the method further includes:
and stopping batch processing, and cleaning the data of the T day.
A data processing apparatus of a banking system, comprising:
the data export module is used for exporting the data to be processed in the online subsystem to the batch subsystem after the accounting date is switched;
the data acquisition module is used for enabling the batch subsystem to extract and obtain T-day data from the data to be processed;
the data processing module is used for enabling the batch subsystem to carry out batch processing on the T-day data to obtain processing result data;
and the data write-back module is used for enabling the batch subsystem to write back the processing result data to the online subsystem.
Compared with the prior art, the embodiment of the invention discloses a data processing method and a data processing device of a bank system, which comprise the following steps: after the accounting date is switched, exporting data to be processed in the online subsystem to the batch subsystem; the batch subsystem extracts and obtains T-day data from the data to be processed; the batch subsystem performs batch processing on the T-day data to obtain processing result data; and the batch subsystem writes the processing result data back to the online subsystem. The data processing method and the data processing device of the bank system can acquire static data of the day before the day of the switching under the condition of uninterrupted online transaction for 7x24 hours for batch processing after the accounting day is switched, and write back processing result data to the online subsystem after the batch processing.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a data processing method of a banking system according to an embodiment of the present invention;
FIG. 2 is a flowchart of exporting data to be processed according to an embodiment of the present disclosure;
FIG. 3 is a flowchart of obtaining data for day T according to an embodiment of the present invention;
fig. 4 is a flowchart of a specific implementation of a data processing method of a banking system disclosed in an embodiment of the present invention;
FIG. 5 is a logic diagram illustrating a modification of the main table of the window period of the date cutting according to the embodiment of the present invention;
fig. 6 is a schematic structural diagram of a data processing device of a banking system according to an embodiment of the present invention;
FIG. 7 is a schematic structural diagram of a data export module according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a data acquisition module according to an embodiment of the present invention.
Detailed Description
For reference and clarity, the descriptions, abbreviations or abbreviations of the technical terms used hereinafter are summarized as follows:
7x24 hours: with the rapid development of technology and business, the business processing is no longer limited by time and space, and the core banking system is required to provide all-weather online service, so that the core banking system is required to provide 7 × 24 hours online transaction service.
Reproduction: for a 7x 24-hour system in a warehouse and table dividing mode, during the batch period, the batch subsystem processes the daily schedule, and the online subsystem processes the night schedule. After the batch subsystem completes the batch, the operation of the daytime system to synchronize back to the day table for transactions in the night table by the online subsystem is called replication.
DJ: the Day Journal (DJ for short) is an internal batch accounting processing subsystem, and is mainly used for processing the internal accounting processing of the system. Day Journal can run in the beginning of the Day processing, or be initiated by hand in the daytime system, and finish the processing of the data by calling the corresponding online transaction module in the background. Transactions that were not processed successfully are reprocessed at the end of the day and a process status report is provided.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
Fig. 1 is a flowchart of a data processing method of a banking system disclosed in an embodiment of the present invention, and referring to fig. 1, the data processing method of the banking system may include:
step 101: and after the accounting date is switched, exporting the data to be processed in the online subsystem to the batch subsystem.
The accounting date is generally switched in the morning of each day, and after the accounting date is switched every day, for example, after T- > T +1 switching day, since the new service data is divided into data of T +1 day, the service data of T day is determined and will not change any more, at this time, the determined data of T day (i.e., the data to be processed) may be exported from the online subsystem to the batch processing subsystem, so that the batch processing subsystem performs corresponding batch processing on the data of T day according to a preset processing flow.
Step 102: and the batch subsystem extracts and obtains T-day data from the data to be processed.
The data to be processed may include useful data and useless or inaccurate data, and therefore, after all the data to be processed is acquired from the online subsystem, some processing needs to be performed on the data to be processed first to acquire accurate T-day data, and the accurate processed data can be acquired by performing batch processing based on the accurate T-day data.
The specific processing in the process of performing some processing on the data to be processed to obtain accurate T-day data may be configured correspondingly according to the data content and the data content type included in the data to be processed, and in the following embodiments, specific implementation of obtaining accurate T-day data will be described in detail.
Step 103: and the batch subsystem performs batch processing on the T-day data to obtain processing result data.
The batch processing of the batch subsystem on the T-day data is a process of processing specific data according to actual banking business needs to obtain target data, and because the banking business data volume is huge and the processing flows of some data are the same, the processing of the data is batch processing.
Step 104: and the batch subsystem writes the processing result data back to the online subsystem.
After the batch processing of the data of T days is completed, the processed result data can be updated to the online subsystem, so that the subsequent related business data can be processed on the basis of the result data, and the continuous and smooth development of banking business is ensured.
In this embodiment, the data processing method of the bank system can acquire static data of day before and after switching accounting days under 7 × 24 hours of uninterrupted online transaction for batch processing, and write back processing result data to the online subsystem after batch processing.
In the foregoing embodiment, reference may be made to fig. 2 for a specific process of exporting to-be-processed data in an online subsystem to a batch subsystem, where fig. 2 is a flowchart of exporting to-be-processed data disclosed in an embodiment of the present invention, and as shown in fig. 2, the specific process may include:
step 201: in the date cutting window period, if transaction data are acquired, firstly copying data related to the transaction data on the T date to a shadow table, and then updating the transaction data to a transaction main table.
And after the accounting date is switched, the process of exporting the data to be processed in the online subsystem to the batch subsystem is the date switching window period.
The process of copying the data related to the transaction data in the T day to the shadow table and updating the transaction data to the transaction main table may include: in the date cutting window period, if the content of the first table relates to modification, checking whether the last modification date is T +1 day; if not, after copying the content of the first table into a shadow table, modifying the first table; if yes, the first table is directly modified. In the time period of the day-cutting time window, when first online transaction data after the day-cutting is generated, firstly copying complete and accurate data of T days before the day-cutting into a shadow table, recording accurate data of the data related to the first online transaction data in the T days, and then updating the first online transaction data to a transaction main table.
Step 202: and exporting the transaction main table and the shadow table in the online subsystem to a batch subsystem.
In the date cutting window period, not all the service data of the users are updated, so only a small part of the T-day data of the users are copied in the shadow table, and therefore, when all the T-day data need to be processed in batch, the transaction master table and the shadow table in the online subsystem need to be exported to the batch subsystem.
Based on the above, referring to fig. 3, a specific process of the batch subsystem obtaining the T-day data from the data to be processed may be shown, where fig. 3 is a flowchart of obtaining the T-day data disclosed in the embodiment of the present invention, and as shown in fig. 3, the specific process may include:
step 301: and the batch subsystem covers the data in the shadow table to the corresponding data in the transaction main table to obtain the transaction main table containing the static data of the T day.
Since the data in the shadow table is some data in the end of the T day, and the data related to the data in the shadow table in the transaction master table is updated after T +1 day, when the T-day data needs to be processed in batch, all the data in the accurate end of the T day needs to be acquired first, and therefore, the data in the shadow table needs to be overlaid on the corresponding data in the transaction master table. For example, the transaction master table includes 1 ten thousand table data, and the shadow table includes 100 table data, so that 100 table data in the shadow table needs to replace 100 tables corresponding to 100 tables in the transaction master table in the process of obtaining the transaction master table including the static data of the T day.
To simplify operation and processing, the transaction master table and the shadow table may be configured as the same data structure.
Step 302: and determining the T-day static data as T-day data.
The above embodiment introduces the detailed process of exporting the data to be processed and extracting the data of T day from the data to be processed in detail, which is beneficial to better understand the data processing method for implementing the bank system, and realizes 7 × 24 hours of uninterrupted online transaction on the premise of not bringing the time pressure of day cutting to the system.
In the above embodiment, the exporting of the data to be processed in the online subsystem to the batch subsystem may include: and acquiring full data from the online subsystem in a dirty reading mode through a heterogeneous data source offline synchronization tool DataX and synchronizing the full data to the batch subsystem. The dirty read mode is to read data even if the database locks the table, and the read data may be old data. The dirty reading mode is used in the data synchronization scheme of the transaction main table and the shadow table, so that online transaction is not affected even when data are synchronized, online transaction can continue to operate an online subsystem or an online database, and meanwhile, the batch subsystem can acquire required T day end data for batch processing.
In the above embodiment, the writing back of the processing result data to the online subsystem by the batch subsystem may include: the batch subsystem directly writes the processing result data operation back to the online subsystem; or the batch subsystem writes the processing result data back to the contact subsystem through log transaction.
In other embodiments, the data processing method of the banking system may further include the step of stopping batch processing and cleaning the data for T day after the batch subsystem writes back the processing result data to the online subsystem. After the processing result data is written back to the online subsystem, the T-day data in the batch subsystem is used up, and the T-day data can be cleaned up to release system resources under the condition of no other use.
Fig. 4 is a flowchart of a specific implementation of the data processing method of the banking system according to the embodiment of the present invention, where the batch transaction subsystem is a batch subsystem, and the online transaction subsystem is an online subsystem. The following can be understood in conjunction with fig. 4.
In window time after T- > T +1 cutting day, if the content of one table relates to modification, checking the last modification date, if the last modification date is not T +1 day, copying the content before modification to the corresponding shadow table, and then modifying the current main table.
The present embodiment can realize 7 × 24 hours uninterrupted online service, and may include the following three stages:
1. the system is in a normal mode of T-day online, incoming read-only query transaction accesses an online read database at the time, financial transaction needs access of an update database table to an online write database, and transaction flow records correspond to records of T days.
2. And after the T day, the system directly switches the accounting day to the next day. At this time, for the online transaction, the next day (T + 1) is entered, the transaction still follows the above mode, the read-only query transaction accesses the online reading database, the financial transaction accesses the online writing database, but the transaction running record corresponds to the running record of the T +1 day, but the first transaction occurring at T +1 needs to write the last T day data into the shadow table first, and then update the data of the main table. For the complete data of T days of batch processing, the data of the shadow table and the data of the T days of the main table can be obtained.
This phase is already for day T +1 online, and the transactions that occur are considered to be day T +1 transactions. But for batch processing, the processed data is still data for day T.
3. And after the batch processing of the T day is finished, the system state is updated to an online normal mode, the batch also starts to enter the next day, and partial data initiates updating of the data table through the DJ.
After the day is cut, the data table is acquired in a dirty reading mode through DataX, and the acquisition of the data table needs to be carried out in the sequence of a main table and a shadow table. The date window switching period can be ended after the data synchronization is completed, so that the window time is short, the estimation cannot exceed half an hour, and the data in the shadow table is few. The shadow tables are firstly merged into the main table in batches, and the subsequent main batch only needs to operate the main table (newly added data of T +1 needs to be filtered), so that the subsequent operation is simplified.
After the data are synchronized to the batch database from the online read-only database, the batch time window (namely the day-cutting window period) of T day is entered at this time, the batch is processed in the batch subsystem database in batch, and after the processing is finished, the batch processed data table is written back to the online database.
In order to relieve the pressure of DJ and avoid that all data are updated to an online data writing database in a DJ mode, the following six conditions of classification processing are carried out on the specific write-back condition.
1. The data columns are only updated in batch, and are not updated online, so that the online written database table is directly updated after batch business processing is finished;
2. the data fields are only used online and read only in batches, but need to be updated in batch time windows, such as accounting dates, system states and the like;
3. when the data fields are online, the batch is updated, but the final data is updated based on the update of the T +1 online transaction, in the case of batch updating, the judgment needs to be made according to the last modification date of the data, and if the date is more than T days, the last result of the batch is not updated to the online write database;
4. the data fields are online, the batch is updated, but the final data is subject to the result of batch processing for T days, and the online written database is directly updated after the batch service processing is finished;
5. if the online T +1 transaction can automatically complete the required part of data processing, the online data processing of the account is firstly carried out before the T +1 transaction is carried out, then the online transaction is carried out, and marking is carried out firstly on corresponding records when a data table is processed, and the data are not written back when the batch write-back is carried out, so that the result calculated at the moment of online is used as the standard;
6. the online and batch transaction data are not in sequence, and are updated according to time priority, and the batch data are written back to an online database in a DJ mode, such as deposit batch settlement, account conversion and the like.
When the main table with the shadow table is modified in the batch window time (the window period of the cutting day) after the cutting day, the modification is processed according to the logic shown in FIG. 5.
The online system continuously operates according to the service working date and is divided into an online transaction stage and a date switching stage (or a system maintenance stage). The online system is in an online transaction stage for most of time, and responds to transactions initiated by different channels, which is a main working stage of the system; the date switching stage is a data static point of the system, after the counting date is switched to the next day, the system starts to synchronize the data of the online system into the batch system to provide the batch data for the batch system, and at the moment, the online system needs to synchronously consider the processing of the shadow table when updating the master table with the shadow table.
The batch system is divided into a batch preparation stage, a batch processing stage and a batch data output stage. The batch preparation stage prepares for batch processing, cleans logs, temporary files and the like generated in batches in the last working day, and finishes the cleaning before the cutting of the online system; the batch processing stage is a batch processing stage of the system and is carried out after the on-line system is cut day by day; the batch data output stage is a stage of outputting data generated by batch processing to a corresponding system, and is performed after the batch processing is completed.
While, for purposes of simplicity of explanation, the foregoing method embodiments have been described as a series of acts or combination of acts, it will be appreciated by those skilled in the art that the present invention is not limited by the illustrated ordering of acts, as some steps may occur in other orders or concurrently with other steps in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.
The method is described in detail in the embodiments disclosed above, and the method of the present invention can be implemented by various types of apparatuses, so that the present invention also discloses an apparatus, and the following detailed description is given of specific embodiments.
Fig. 6 is a schematic structural diagram of a data processing device of a banking system according to an embodiment of the present invention, and referring to fig. 6, a data processing device 60 of the banking system may include:
and the data export module 601 is configured to export the to-be-processed data in the online subsystem to the batch subsystem after the accounting date is switched.
The accounting date is generally switched in the early morning of each day, and after the accounting date is switched every day, for example, after T- > T +1 switching day, because the new service data is divided into data of T +1 day, the service data of T day is determined and will not change, at this time, the determined data of T day (i.e., the data to be processed) may be exported from the online subsystem to the batch processing subsystem, so that the batch processing subsystem performs corresponding batch processing on the data of T day according to a preset processing flow.
And a data obtaining module 602, configured to enable the batch subsystem to extract the data for T day from the data to be processed.
The data to be processed may include useful data and useless or inaccurate data, and therefore, after all the data to be processed is acquired from the online subsystem, some processing needs to be performed on the data to be processed first to acquire accurate T-day data, and batch processing is performed based on the accurate T-day data, so that accurate processed data can be acquired.
And the data processing module 603 is configured to enable the batch subsystem to perform batch processing on the T-day data to obtain processing result data.
The batch processing of the batch subsystem on the T-day data is a process of processing specific data according to actual banking business needs to obtain target data, and the processing of the data is batch processing due to the fact that the banking business data volume is huge and the processing flows of some data are the same.
And a data write-back module 604, configured to enable the batch subsystem to write back the processing result data to the online subsystem.
After the batch processing of the data of T days is completed, the processed result data can be updated to the online subsystem, so that the subsequent related business data can be processed on the basis of the result data, and the continuous and smooth development of banking business is ensured.
In this embodiment, the data processing device of the banking system can acquire static data of the day before the switching day for batch processing under the uninterrupted online transaction of 7 × 24 hours after the switching of the accounting day, and the processing result data is written back to the online subsystem after batch processing, the device cancels a replication stage required by a database-by-table scheme, reduces the time pressure of the replication stage on the system, and has good practical application significance.
Fig. 7 is a schematic structural diagram of a data export module disclosed in the embodiment of the present invention, and as shown in fig. 7, the data export module 601 may include:
the data copying module 701 is configured to copy, in the date cutting window period, data related to the transaction data on the T date to a shadow table if the transaction data is acquired, and then update the transaction data to a transaction main table.
And after the accounting date is switched, the process of exporting the data to be processed in the online subsystem to the batch subsystem is the date switching window period.
The process of copying the data related to the transaction data in the T day to the shadow table and updating the transaction data to the transaction main table may include: in the date cutting window period, if the content of the first table relates to modification, checking whether the last modification date is T +1 day; if not, after copying the content of the first table into a shadow table, modifying the first table; if yes, the first table is directly modified. In the time period of the day-cutting time window, when first online transaction data after the day-cutting is generated, firstly copying complete and accurate data of T days before the day-cutting into a shadow table, recording accurate data of the data related to the first online transaction data in the T days, and then updating the first online transaction data to a transaction main table.
And the data export submodule 702 is configured to export the transaction master table and the shadow table in the online subsystem to the batch subsystem.
In the date cutting window period, not all the service data of the users are updated, so only a small part of the T-day data of the users are copied in the shadow table, and therefore, when all the T-day data need to be processed in batch, the transaction master table and the shadow table in the online subsystem need to be exported to the batch subsystem.
Based on the above, the specific structure of the data obtaining module can be seen in fig. 8, and as shown in fig. 8, the data obtaining module 602 may include:
and the data merging module 801 is configured to control the batch subsystem to overlay the data in the shadow table onto the corresponding data in the transaction master table to obtain the transaction master table including the static data of day T.
Since the data in the shadow table is some data in the end of the T day, and the data related to the data in the shadow table in the transaction master table is updated after T +1 day, when the T-day data needs to be processed in batch, all the data in the accurate end of the T day needs to be acquired first, and therefore, the data in the shadow table needs to be overlaid on the corresponding data in the transaction master table.
To simplify operation and processing, the transaction master table and the shadow table may be configured as the same data structure.
A data determining module 802, configured to determine the static data for T day as data for T day.
The above embodiment introduces the detailed process of exporting the data to be processed and extracting the data of T day from the data to be processed in detail, which is beneficial to better understand the data processing method for implementing the bank system, and realizes 7 × 24 hours of uninterrupted online transaction on the premise of not bringing the time pressure of day cutting to the system.
In the above embodiment, the data deriving module may be specifically configured to: and acquiring full data from the online subsystem in a dirty reading mode through a heterogeneous data source offline synchronization tool DataX and synchronizing the full data to the batch subsystem. The data synchronization scheme of the transaction main table and the shadow table uses a dirty reading mode, so that online transaction is not influenced even when data are synchronized, online transaction can continue to operate an online subsystem or an online database, and meanwhile, the batch subsystem can acquire required T day end data for batch processing.
In the above embodiment, the data write-back module 604 may be specifically configured to: controlling the batch subsystem to directly write back the processing result data operation to the online subsystem; or the batch subsystem writes the processing result data back to the contact subsystem through log transaction.
In other embodiments, the data processing apparatus of the banking system may further include a data cleaning module, configured to control the step of stopping batch processing and cleaning the data for T day after the data write-back module causes the batch subsystem to write back the processing result data to the online subsystem. After the processing result data is written back to the online subsystem, the T-day data in the batch subsystem is used up, and the T-day data can be cleaned up to release system resources under the condition of no other use.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (7)
1. A data processing method of a bank system is characterized by comprising the following steps:
after switching accounting dates, exporting data to be processed in the online subsystem to the batch subsystem, wherein the exporting process comprises the following steps: after the accounting date is switched, taking the process of exporting data to be processed in the online subsystem to the batch subsystem as a date switching window period, copying data related to the transaction data on the T date to a shadow table if the transaction data is acquired in the date switching window period, and then updating the transaction data to a transaction main table; exporting the transaction main table and the shadow table in the online subsystem to a batch subsystem;
the batch subsystem covers the data in the shadow table to the corresponding data in the transaction main table to obtain the transaction main table containing the static data of the T day;
determining the static data of the T day as data of the T day;
the batch subsystem performs batch processing on the T-day data to obtain processing result data;
the batch subsystem writes back the processing result data to the online subsystem, and the method comprises the following steps: and the batch subsystem writes the processing result data back to the online subsystem through log transaction.
2. The data processing method of the banking system according to claim 1, wherein in the date switching window period, if transaction data is acquired, first copying data related to the transaction data on the T-day to a shadow table, and then updating the transaction data to a transaction master table, includes:
in the date cutting window period, if the content of the transaction main table relates to modification, checking whether the last modification date is T +1 day;
if not, the content of the transaction main table is copied into the shadow table, and then the transaction main table is modified.
3. The data processing method of a banking system according to claim 1, wherein the transaction main table and the shadow table are identical in structure.
4. The data processing method of the banking system as claimed in claim 1, wherein the exporting of the data to be processed in the online subsystem to the batch subsystem further comprises:
and acquiring full data from the online subsystem in a dirty reading mode through a heterogeneous data source offline synchronization tool DataX and synchronizing the full data to the batch subsystem.
5. The data processing method of the banking system as claimed in claim 1, wherein the batching subsystem writes back the processing result data to the online subsystem, further comprising:
and the batch subsystem directly writes the processing result data operation back to the online subsystem.
6. The data processing method of the banking system as claimed in claim 1, wherein after the batch subsystem writes the processing result data back to the online subsystem, the method further comprises:
and stopping batch processing, and cleaning the data of the T day.
7. A data processing apparatus of a banking system, comprising:
the data export module is used for exporting the data to be processed in the online subsystem to the batch subsystem after switching the accounting date, and comprises: after the accounting date is switched, taking the process of exporting data to be processed in the online subsystem to the batch subsystem as a date switching window period, copying data related to the transaction data on the T date to a shadow table if the transaction data is acquired in the date switching window period, and then updating the transaction data to a transaction main table; exporting the transaction main table and the shadow table in the online subsystem to a batch subsystem;
the data acquisition module is used for enabling the batch subsystem to cover the data in the shadow table on the corresponding data in the transaction main table to obtain the transaction main table containing the static data of the T day; determining the static data of the T day as data of the T day;
the data processing module is used for enabling the batch subsystem to carry out batch processing on the T-day data to obtain processing result data;
the data write-back module is used for enabling the batch subsystem to write back the processing result data to the online subsystem, and comprises: and the batch subsystem writes the processing result data back to the subsystem through log transaction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911149205.5A CN110910238B (en) | 2019-11-21 | 2019-11-21 | Data processing method and device for bank system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911149205.5A CN110910238B (en) | 2019-11-21 | 2019-11-21 | Data processing method and device for bank system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110910238A CN110910238A (en) | 2020-03-24 |
| CN110910238B true CN110910238B (en) | 2022-10-18 |
Family
ID=69818396
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911149205.5A Active CN110910238B (en) | 2019-11-21 | 2019-11-21 | Data processing method and device for bank system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110910238B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111444281B (en) * | 2020-04-16 | 2023-05-02 | 中国银行股份有限公司 | Database parameter synchronization method and system |
| CN111767260A (en) * | 2020-06-15 | 2020-10-13 | 中国建设银行股份有限公司 | Method and system for data synchronization |
| CN112905635A (en) * | 2021-03-11 | 2021-06-04 | 深圳市分期乐网络科技有限公司 | Service processing method, device, equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1811838A (en) * | 2005-12-23 | 2006-08-02 | 中国工商银行股份有限公司 | Data processing method and system for realizing continuous service |
| CN106649796A (en) * | 2016-12-28 | 2017-05-10 | 中国建设银行股份有限公司 | Data processing method and apparatus |
| CN110276614A (en) * | 2019-04-30 | 2019-09-24 | 阿里巴巴集团控股有限公司 | The update method and device of ledger |
-
2019
- 2019-11-21 CN CN201911149205.5A patent/CN110910238B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1811838A (en) * | 2005-12-23 | 2006-08-02 | 中国工商银行股份有限公司 | Data processing method and system for realizing continuous service |
| CN106649796A (en) * | 2016-12-28 | 2017-05-10 | 中国建设银行股份有限公司 | Data processing method and apparatus |
| CN110276614A (en) * | 2019-04-30 | 2019-09-24 | 阿里巴巴集团控股有限公司 | The update method and device of ledger |
Non-Patent Citations (1)
| Title |
|---|
| 核心银行系统 之五 7X24小时 不间断运行的核心系统设计;CS1026;《https://www.jianshu.com/p/c0c20144027b》;20180521;第1-9页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110910238A (en) | 2020-03-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110910238B (en) | Data processing method and device for bank system | |
| US6442533B1 (en) | Multi-processing financial transaction processing system | |
| CN101233497B (en) | Revenue management system and method | |
| CN106649796B (en) | Data processing method and device | |
| CN105989059B (en) | Data record checking method and device | |
| CN110321339B (en) | Data migration method, device, equipment and storage medium | |
| CN107657049A (en) | A kind of data processing method based on data warehouse | |
| CN109063066B (en) | Data query method and device and data management system | |
| CN110795287A (en) | Data recovery method, system, electronic device and computer storage medium | |
| CN115145697B (en) | Database transaction processing method and device and electronic equipment | |
| CN107729448A (en) | A kind of data handling system based on data warehouse | |
| CN106991606B (en) | Transaction data processing method and device | |
| CN114202334A (en) | Method and system suitable for asynchronously updating balance of hotspot account in batch | |
| CN113626527A (en) | Financial data processing method and system | |
| CN117033205A (en) | Flow playback comparison test method and device | |
| CN115658391A (en) | Backup recovery method of WAL mechanism based on QianBase MPP database | |
| EP4348441A1 (en) | Systems and methods for ensuring quality of search system data | |
| CN115408200A (en) | Data backup method and device for multiple storage engines, electronic equipment and storage medium | |
| CN114722045A (en) | Time series data storage method and device | |
| JPH0667944A (en) | Database management system | |
| CN100424687C (en) | On-line processing system and method based on network | |
| CN110597669B (en) | Bank historical data parameterized backup recovery method and device | |
| CN110389988A (en) | A kind of the user data processing method and system of real-time high-efficiency | |
| CN101101646A (en) | Personal income tax declaration management system | |
| CN108335185B (en) | Method and device for accessing latest overdraft limit in night online transaction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |