CN113468150A - Horizontal segmentation capacity expansion and migration method for payment subscription data - Google Patents
Horizontal segmentation capacity expansion and migration method for payment subscription data Download PDFInfo
- Publication number
- CN113468150A CN113468150A CN202111029012.3A CN202111029012A CN113468150A CN 113468150 A CN113468150 A CN 113468150A CN 202111029012 A CN202111029012 A CN 202111029012A CN 113468150 A CN113468150 A CN 113468150A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- new
- migration
- old
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/38—Payment protocols; Details thereof
Abstract
The invention relates to the field of payment systems, in particular to a horizontal segmentation, expansion and migration method of payment subscription data. According to the invention, whether a new database switch configuration item is written, whether a signed data query switch configuration item is written or not and whether an old database switch configuration item is written or not are set, so that external service of application service for 7 × 24 hours can be ensured in the expansion and migration processes of the distributed database, the data in migration cannot be locked, and the old database in the migration process can still be accessed to the outside. If a fault exists after capacity expansion or migration, fast rollback can be performed by modifying whether to write a new data switch configuration item, inquiring a switch configuration item through subscription data, and writing an old database switch configuration item.
Description
Technical Field
The invention relates to the field of payment systems, in particular to a horizontal segmentation, expansion and migration method of payment subscription data.
Background
The agreement payment means a payment service that the payee entrusts the collection agency to deduct the account fund of the payer to the payee according to the agreed conditions of frequency, amount and the like, and the payer account agency does not perform transaction confirmation with the payer one by one any more. The agreement payment service is suitable for a specific scene that the payee is fixed, the conditions such as payment frequency or amount are agreed in advance and are relatively fixed. The payer account opening authority should obtain the payer's authorization in advance or when the payer is transacted with the first agreement payment service. The authorization should specify data such as payee name, payment purpose, payment account number, payment period or condition, authorization deadline, etc. Therefore, the account opening mechanism should complete the client authorization and signing processes before the first agreement payment transaction is carried out, and completely store the payment signing data.
Because a single payer account can be authorized to deduct money from multiple merchants, multiple pieces of payment signing data may exist in the single payment account, so that a great amount of payment signing data are inevitably accumulated gradually in the process of gradual business development, and the storage and access of the great amount of payment signing data become a bottleneck problem in system design and use. For processing mass data, data stored in the same database can be segmented and stored on a plurality of databases (hosts) in a dispersed manner under a certain specific condition, so that the effect of dispersing the load of a single device is achieved.
The data segmentation can be divided into two segmentation modes according to the type of the segmentation rule. One is to divide the data into different databases (hosts) according to tables corresponding to different services, and the division can be called as vertical (longitudinal) division of the data; in another method, data in the same table is split to multiple databases (hosts) according to a certain condition according to the logical relationship of the data in the table, and the splitting is called horizontal (horizontal) splitting of the data. The vertical segmentation is relatively simple, but after the vertical segmentation, the payment signing data stored in the single-node database are gradually accumulated, and the single-table query and write-in performance is influenced. And the payment signing data is stored in a plurality of node databases in a dispersed manner according to the card number hash or other field segmentation rules through horizontal segmentation. The payment signing data are stored in the distributed database in a horizontal segmentation mode, so that the requirements of high-efficiency storage and high-concurrency access of mass payment signing data are met.
When the number of the distributed database nodes designed in advance is not enough to meet the business development and a single node accumulates a large amount of data, the capacity expansion and data migration of the distributed database storing the payment signing data are needed. By expanding and migrating the distributed database, the payment signing data can be migrated to more data nodes, the data volume of a single node is reduced, and the writing and query concurrency capability of the single-machine node database is improved.
The existing capacity expansion and migration scheme mainly has the following problems:
1. most of the existing schemes perform data migration by using a database semi-synchronization mechanism, for example, a mysql semi-synchronization function is used to synchronize data in a library a to a1, and then logic for querying the library a is modified to query a1 library. Because the database semi-synchronization function has the problem of synchronization delay of the main library and the standby library, in the process of switching the query A library to the query A1 library in the query logic of the external service within 7 × 24 hours, the error report of the query failure can occur.
2. In the migration and capacity expansion process, if an exception occurs, the existing capacity expansion and migration scheme has no quick rollback capability. In an application providing service for 7 × 24 hours, all major changes such as program change, data capacity expansion, data migration and the like need to ensure the capability of fast rollback. The existing scheme can not enable the system to quickly return to the state before migration and capacity expansion at the minute level.
3. In the existing scheme, a database lock mechanism is used in the expansion and migration processes, partial data is locked, the locked partial data cannot be provided with services, and partial service interruption is caused.
Disclosure of Invention
The invention provides a horizontal segmentation capacity expansion and migration method of payment signing data, which aims to solve the technical problems that switching among databases causes error report of data query failure, capacity expansion of the databases and no quick rollback capability of a migration party, and capacity expansion and migration processes of the databases lock partial data, so that the partial data cannot provide external services and the service part is interrupted.
A horizontal segmentation capacity expansion and migration method of payment subscription data comprises the following steps:
step 1: before capacity expansion and migration, the payment signing service accesses a plurality of old databases through the database middleware; the old database middleware uniformly writes the payment signing data into all the old databases according to the database dividing rule; the payment signing service provides a signing data query function, and the query function is accurately positioned to an old database to execute query by taking a database splitting rule set in the middleware as an entry parameter;
step 2: establishing a plurality of new databases, and configuring an access rule in the middleware of the new databases, wherein the access rule is based on all new databases signed by the sub-database rule for reading and writing payment;
and step 3: increasing whether to write a new database switch configuration item, including configurations in which new database configuration is not written and incremental subscription data is written in a new database;
and 4, step 4: adding a signing data query switch configuration item, including the configuration of querying an old database and the configuration of querying a new database;
and 5: adding configuration items of whether to write the old database or not, wherein the configuration items comprise the configuration of the old database and the configuration of the old database which is not written any more;
step 6: changing the new data switch configuration item modification into the configuration of writing the incremental signing data into a new database; at the moment, the payment signing service writes the incremental signing data into the old database according to the database partitioning rule, writes the incremental signing data into all new databases according to the access rule, and finally returns the success of the payment signing application;
and 7: inquiring data in all old databases according to the database partitioning rule, and writing all stock data into a new database according to the access rule;
and 8: in the data checking stage, an independent data migration check service is newly added, a new database and an old database are inquired, data comparison and check are carried out, whether the data in the old database is consistent with the data in the new database or not is checked, and if the data in the old database is consistent with the data in the new database, the step 9 is executed;
and step 9: modifying the configuration item of the signing data query switch into the configuration of querying a new database, querying all the new databases according to the access rule, and not querying an old database; detecting whether the online transaction is abnormal or not, if so, modifying the configuration item of the signing data query switch into the configuration for querying the old database for fast rollback; if no abnormity or complaint exists, executing step 10;
step 10: modifying the configuration item of whether to write the old database into the configuration of not writing the old database any more, and writing the online incremental signing data into the new database only according to the access rule at the moment;
step 11: the payment signing service accesses all the new databases through the database middleware, the database middleware uniformly writes the payment signing data into the new databases according to the access rules, the payment signing service simultaneously provides the inquiry function of the signing data, the inquiry function is accurately positioned into the new databases through the access rules of the middleware to execute inquiry, and the expansion and the migration are completed.
Preferably, the middleware takes the signed card number field as a database dividing rule;
preferably, the middleware takes a signed card number field as an access rule;
further, the step 7 comprises the following steps:
step 7.1: increasing stock data migration service, wherein the service is used for configuring data migration starting time, data migration ending time and time range of single data query; the time range of the single query is minutes;
step 7.2: querying historical data in an old database according to the configured data migration starting time, data migration ending time, the time range of single data query and a database partitioning rule, and putting the data of the single query into a list A;
step 7.3: circularly traversing the list A, and submitting and inserting the data in the list A into a new database in batches according to the access rule of the middleware;
step 7.4: and (4) increasing the query time period according to the time range of the single query, performing next cycle query, putting the queried data into the list A again, executing the batch insertion logic again, inserting the data in the list A into the new database, and repeating the above cycle until all the data in the range from the data migration starting time to the data migration ending time are migrated.
Further, the step 8 comprises the following steps:
step 8.1: adding a detail comparison switch, a stroke comparison switch, a comparison starting time and a comparison ending time of the data migration check service configuration;
step 8.2: the detail comparison switch is turned on, and the data migration check service circularly queries the database in the old database according to the database division rule, the comparison starting time, the comparison ending time and the mysql paging query parameter to obtain a list B;
step 8.3: traversing the data in the list B, inquiring the new database according to the access rule, judging whether the new database has the data of the corresponding card number, if not, reporting an error, and failing to compare; if yes, continuing the next data comparison and continuing the cyclic query of the step 8.2 until the paging query is finished and the checking is successful;
step 8.4: starting a stroke number comparison switch, and inquiring detailed stroke numbers in an old database by the data migration check service according to the database dividing rule, the comparison starting time and the comparison ending time to obtain a stroke number value A; inquiring the detailed stroke number in the new database according to the access rule, the comparison starting time and the comparison ending time to obtain a stroke number value B; comparing whether the stroke value A and the stroke value B are equal or not, and if not, reporting an error; if the two are equal, the checking is successful;
step 8.5: modifying the values of the comparison starting time and the comparison ending time, restarting the data migration check service, repeating the steps from 8.2 to 8.4, and checking the data in other time periods in a spot check mode;
further, if the configured comparison start time is equal to the data migration start time and the comparison end time is equal to the data migration end time, the step 8.2 to the step 8.4 are full check; and if the migration time end is not equal to the checking time period, checking for spot check.
Further, when the volume expansion and migration data volume is larger than a preset threshold, the step 8.2 to the step 8.4 are spot check; if the volume expansion and migration data volume is less than or equal to the preset threshold, the step 8.2 to the step 8.4 are full check.
Compared with the prior art, the invention has the beneficial effects that:
1. according to the invention, whether a new database switch configuration item is written, whether a signed data query switch configuration item is written or not and whether an old database switch configuration item is written or not are set, so that external service of application service for 7 × 24 hours can be ensured in the expansion and migration processes of the distributed database, the data in migration cannot be locked, and the old database in the migration process can still be accessed to the outside. If a fault exists after capacity expansion or migration, fast rollback can be performed by modifying whether to write a new data switch configuration item, inquiring a switch configuration item through subscription data, and writing an old database switch configuration item.
2. Through the double writing of the new database and the old database in the step 6, the old database is written firstly in the incremental signing transaction, then the new database is written, and finally the historical data migration is carried out, so that the incremental signing data and the historical signing data are ensured to be written into the new database. And when the subscription data query logic is switched to query the new database, the subscription data in the new database is complete. Therefore, the problem of time delay between the main database and the standby database in the database semi-synchronous copy migration scheme is solved.
3. Establishing independent stock data migration service, wherein the data volume can be flexibly configured according to the time range through single cycle query; data acquired by single query use batch submission of insertion statements, and data migration efficiency is improved. And an independent data migration check service is created, and the check granularity supports check modes such as detail comparison in a specified time range and stroke comparison in a specified time range of a new database and an old database, so that the consistency of data before migration and after migration is ensured.
4. The method and the device have the advantages that the cost is low, the implementation scheme is simple, the operation is convenient, the capacity expansion and the migration can be completed only by modifying the read-write logic of the existing application program aiming at the subscription data and adding the stock data migration function. And the main database and the semi-synchronous standby database are switched for many times without professional DBA intervention.
5. The problem of data delay of a main database and a standby database is solved: the double-write old-new database scheme is characterized in that the old database is written firstly in the incremental signing transaction, then the new database is written, and finally historical data migration is carried out. This ensures that both incremental subscription data and historical subscription data have been written into the new database. When the subscription data query logic is switched to query the new database, the subscription data in the new database is complete. Therefore, the problem of time delay between the main database and the standby database in the database semi-synchronous copy migration scheme is solved.
6. The migration speed is high: in the stock data migration stage, an independent stock data migration service is established, and the quantity of old databases queried in a single cycle can be flexibly configured according to the time range; data acquired by single query are written into a new database by using a batch submission and insertion sql statement, so that the data migration efficiency is improved.
Drawings
FIG. 1 is a schematic diagram of an architecture of an old database before expansion and migration.
Fig. 2 is a schematic diagram of the architecture of the incremental payment signing data double-writing to the new and old databases and searching the old database according to the present invention.
FIG. 3 is a diagram of an inventory data migration architecture according to the present invention.
FIG. 4 is a schematic diagram of the architecture of the present invention for maintaining double-writing of new and old databases, query switching, and modification to check new databases.
FIG. 5 is a schematic diagram of the architecture after capacity expansion and migration according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1 to 5, a method for expanding and migrating horizontal split of payment subscription data includes the following steps:
step 1: before capacity expansion and migration, the payment signing service accesses a plurality of old databases through the database middleware; the old database middleware uniformly writes the payment signing data into all old databases according to the signing card number field as a database splitting rule; the payment signing service provides a signing data query function, and the query function takes the card number as an entry parameter through a database splitting rule set in the middleware and accurately positions the old database to execute query;
step 2: establishing a plurality of new databases, configuring an access rule in the middleware of the new databases, and reading and writing all new databases signed with payment according to the access rule by taking a signed card number field as a database splitting rule;
and step 3: increasing whether to write a new database switch configuration item, including configurations in which new database configuration is not written and incremental subscription data is written in a new database;
and 4, step 4: adding a signing data query switch configuration item, including the configuration of querying an old database and the configuration of querying a new database;
and 5: adding configuration items of whether to write the old database or not, wherein the configuration items comprise the configuration of the old database and the configuration of the old database which is not written any more;
step 6: changing the new data switch configuration item modification into the configuration of writing the incremental signing data into a new database; at the moment, the payment signing service writes the incremental signing data into the old database according to the database partitioning rule, writes the incremental signing data into all new databases according to the access rule, and finally returns the success of the payment signing application; therefore, the incremental payment signing data is synchronously written into the new database and the old database before and after capacity expansion.
And 7: inquiring data in all old databases according to the database partitioning rule, and writing all stock data into a new database according to the access rule;
the step 7 comprises the following steps:
step 7.1: increasing stock data migration service, wherein the service is used for configuring data migration starting time, data migration ending time and time range of single data query; the time range of the single query is minutes;
step 7.2: querying historical data in an old database according to the configured data migration starting time, data migration ending time, the time range of single data query and a database partitioning rule, and putting the data of the single query into a list A;
step 7.3: circularly traversing the list A, and submitting and inserting the data in the list A into a new database in batches according to the access rule of the middleware;
step 7.4: and (4) increasing the query time period according to the time range of the single query, performing next cycle query, putting the queried data into the list A again, executing the batch insertion logic again, inserting the data in the list A into the new database, and repeating the above cycle until all the data in the range from the data migration starting time to the data migration ending time are migrated.
And 8: in the data checking stage, an independent data migration check service is newly added, a new database and an old database are inquired, data comparison and check are carried out, whether the data in the old database is consistent with the data in the new database or not is checked, and if the data in the old database is consistent with the data in the new database, the step 9 is executed;
the step 8 comprises the following steps:
step 8.1: adding a detail comparison switch, a stroke comparison switch, a comparison starting time and a comparison ending time of the data migration check service configuration;
step 8.2: the detail comparison switch is turned on, and the data migration check service circularly queries the database in the old database according to the database division rule, the comparison starting time, the comparison ending time and the mysql paging query parameter to obtain a list B;
step 8.3: traversing the data in the list B, inquiring the new database according to the access rule, judging whether the new database has the data of the corresponding card number, if not, reporting an error, and failing to compare; if yes, continuing the next data comparison and continuing the cyclic query of the step 8.2 until the paging query is finished and the checking is successful;
step 8.4: starting a stroke number comparison switch, and inquiring detailed stroke numbers in an old database by the data migration check service according to the database dividing rule, the comparison starting time and the comparison ending time to obtain a stroke number value A; inquiring the detailed stroke number in the new database according to the access rule, the comparison starting time and the comparison ending time to obtain a stroke number value B; comparing whether the stroke value A and the stroke value B are equal or not, and if not, reporting an error; if the two are equal, the checking is successful;
step 8.5: and modifying the values of the comparison starting time and the comparison ending time, restarting the data migration check service, repeating the steps from 8.2 to 8.4, and checking the data in other time periods in a spot check mode.
Step 8.6: if the configured comparison starting time is equal to the data migration starting time and the comparison ending time is equal to the data migration ending time, the step 8.2 to the step 8.4 are total amount checking; and if the migration time period is not equal to the checking time period, checking for spot check.
If the volume expansion and migration data volume is larger than the preset threshold value, the step 8.2 to the step 8.4 are spot check; if the volume expansion and migration data volume is less than or equal to the preset threshold, the step 8.2 to the step 8.4 are full check.
And step 9: modifying the configuration item of the signing data query switch into the configuration of querying a new database, querying all the new databases according to the access rule, and not querying an old database; detecting whether the online transaction is abnormal or not, if so, modifying the configuration item of the signing data query switch into the configuration for querying the old database for fast rollback; if no abnormity or complaint exists, executing step 10;
step 10: modifying the configuration item of whether to write the old database into the configuration item of the old database, wherein the old database is not written any more, and the online incremental signing data is written into the new database only according to the access rule;
step 11: the payment signing service accesses all the new databases through the database middleware, the database middleware uniformly writes the payment signing data into the new databases according to the access rules, the payment signing service simultaneously provides the inquiry function of the signing data, the inquiry function is accurately positioned into the new databases through the access rules of the middleware to execute inquiry, and the expansion and the migration are completed.
Referring to fig. 1 to 5, the following describes in detail a method for horizontally splitting, expanding, and migrating payment contract data according to the present invention, including the following steps:
corresponding to the step 1: before capacity expansion and migration, the payment signing service accesses the payment signing databases DB1 and DB2 through the database middleware Mycat, the database middleware Mycat takes the signing card number field as a library division rule _1, and the payment signing data are uniformly written into the DB1 and the DB 2; the payment signing service provides a signing data query function, the query function takes the card number as a reference through a library division rule _1 of a middleware Mycat, and the card number is accurately positioned to the database DB1 or the database DB2 to execute query;
corresponding to the step 2: newly building a database New _ DB1, New _ DB2, New _ DB3 and New _ DB4, configuring an access rule _2 in the middleware Mycat, wherein the rule _2 reads and writes the payment signing New databases New _ DB1, New _ DB2, New _ DB3 and New _ DB4 according to the signing card number field as a library dividing rule.
Corresponding to the step 3: setting whether to write the switch configuration item isWriteNewDB of the new database: the initial value is set to 0, when isWriteNewDB is 0, it indicates that the new database is not written: new _ DB1, New _ DB2, New _ DB3, New _ DB 4; when isWriteNewDB is modified to 1, it indicates that incremental subscription data will be written to the new database according to rule _ 2.
Corresponding to the step 4: setting a switch configuration item queryFlag for signing data query: the initial value is set to 0, when queryFlag is 0, it indicates that the old database is queried according to rule _ 1: DB1, DB 2; when queryFlag is modified to 1, it indicates that the new database will be queried according to rule _ 2: new _ DB1, New _ DB2, New _ DB3, New _ DB 4.
Corresponding to the step 5: setting whether to write the switch configuration item isWriteOldDB of the old database: the initial value is set to 0, when isWriteOldDB is 0, indicating that the old database is written according to rule _ 1: DB1, DB 2; when isWriteOldDB is modified to 1, it indicates that no more writes will be needed to the old database: DB1, DB 2.
Corresponding to the step 6: in the incremental signing data double-writing stage, the configuration of isWriteNewDB is modified to 1, at this time, the payment signing service firstly writes the incremental signing data into an old database (DB 1 and DB 2) according to rule _1, then writes the signing data into a New database New _ DB1, New _ DB2, New _ DB3 and New _ DB4 according to rule _2, and finally returns that the payment signing application is successful. Therefore, the incremental payment signing data is synchronously written into the old and new databases before and after capacity expansion.
Corresponding to the step 7: and in the stock data migration stage, an independent stock data migration service is added, the service inquires the stock data in the old databases DB1 and DB2 according to rule _1 and writes the stock data into the New databases New _ DB1, New _ DB2, New _ DB3 and New _ DB4 according to rule _ 2.
Corresponding to the step 7.1: adding stock data migration service, the service supports configuration: data migration start time begin _ time, data migration end _ time, time range time _ frame for a single data query. Where the time _ frame is in minutes. For example, begin _ time is configured to be 2021-05-2417: 00:01, end _ time is configured to be 2021-05-2417: 30:00, and time _ frame is configured to be 10, which means that data of the transitions 2021-05-2417: 00:01 to 2021-05-2417: 30:00 are migrated, data of 10 minutes per query is migrated, and all the transitions can be completed by cycling 3 queries together. The 3 query time periods are respectively as follows: 2021-05-2417: 00:01 to 2021-05-2417: 10: 00; 2021-05-2417: 10:01 to 2021-05-2417: 20: 00; 2021-05-2417: 20:01 to 2021-05-2417: 30: 00.
Corresponding to the step 7.2: according to the data migration start time begin _ time, the data migration end _ time, the time range time _ frame of the single data query and the rule _1, circularly querying the historical data in the DB1 and the DB2, and putting the data of the single query into a List List _1;
corresponding to the step 7.3: and circularly traversing the List _1, and inserting the data in the List _1 into New databases New _ DB1, New _ DB2, New _ DB3 and New _ DB4 in batch submission according to the Mycat library rule _ 2. The number of the single-time submitted data inserted into the new database is 5000, namely 5000 data are circularly assembled one by one into a batch inserted statement mode supported by Mycat: insert _ table (xxx) values (value1), (value 2). After the traversal of the List _1 is completed, all the data in the List _1 List acquired by the query at this time are inserted into the new database. The number of strokes of inserting the new database in the single submission is a configurable item, namely the number of strokes of inserting the new database in the single submission can be set through configuration;
for example: if the number of detailed entries in the List _1 List is 30000, the number of single submission is configured to be 5000, 5000 sql statements are submitted to mysql at one time to perform insertion, and the insertion of the data in List _1 into the new database can be completed after 6 cycles. Clearing List _1 after circulation is completed, and carrying out the next step;
corresponding to the step 7.4: and (4) according to the time _ frame increment query time period, carrying out next cycle query, putting the queried data into List _1 again, executing the batch insertion logic again, and inserting the data in List _1 into the new database. The above loop is repeated until all data in the range from begin _ time to end _ time is migrated.
Corresponding to the step 8: in the data checking stage, an independent data migration checking service is added, the service inquires the data in the old databases DB1 and DB2 according to rule _1, and inquires the data in the New databases New _ DB1, New _ DB2, New _ DB3 and New _ DB4 according to rule _2 for checking. If the check is matched, step 8 is performed.
Corresponding to the step 8.1: newly adding a data migration check service, wherein the service needs to be configured with details comparison switch details _ check _ flag, stroke comparison switch count _ check _ flag, comparison start time begin _ check _ time, comparison end time end _ check _ time and the like. Wherein the frame _ details _ check is configured to be 1, which indicates that detail comparison is required, and the frame _ details _ check is configured to be 0, which indicates that detail comparison is not required; the count _ check _ flag is configured to be 1, which indicates that the number of strokes needs to be compared, and the count _ check _ flag is configured to be 0, which indicates that the number of strokes does not need to be compared.
Corresponding to the step 8.2: the details _ check _ flag is configured to be 1, the data migration check service circularly queries data in the old databases DB1 and DB2 according to rule _1, start time begin _ check _ time, end _ check _ time, and limit, offset and other mysql paging query parameters (5000 single-page queries, the number of the queries is configurable), and obtains the List _ 2.
And traversing the data in the List _2 corresponding to the step 8.3, inquiring the new database according to the card number and rule _2 rules, and judging whether the new database has the data. If not, reporting an error, and failing to compare; if yes, continuing the next data comparison; and continuing the cyclic query of the 8.2 steps until the paging query is finished and the check is successful.
Corresponding to the step 8.4: configuring count _ check _ flag to be 1, inquiring the number of detail in an old database by the data migration check service according to a sub-library rule _1, a comparison start time begin _ check _ time and a comparison end time end _ check _ time to obtain a number value total _ count _1, inquiring the number of detail in a new database according to a sub-library rule _2, a comparison start time begin _ check _ time and a comparison end time end _ check _ time to obtain a number value total _ count _ 2; comparing whether the total _ count _1 and the total _ count _2 are equal or not, and if not, reporting an error; if they are equal, the check is successful.
Corresponding to the step 8.5: modifying the values of begin _ check _ time and end _ check _ time, restarting the data migration check service, repeating the check steps from 8.2 to 8.4, and checking the data in other time periods.
Corresponding to the step 8.6: if the comparison start time begin _ check _ time is configured to be equal to the data migration start time begin _ time and the comparison end time end _ check _ time is configured to be equal to the data migration end _ time, the above 8.2 to 8.4 steps are full-size checking; and if the migration time period is not equal to the check time period, performing spot check.
If the volume expansion and migration data volume is larger than the preset threshold value, the step 8.2 to the step 8.4 are spot check; if the volume expansion and migration data volume is less than or equal to the preset threshold, the step 8.2 to the step 8.4 are full check.
Corresponding to the step 9: and in the stage of inquiring the New database, the configuration of a signing data inquiry switch configuration item queryFlag is modified to be 1, which means that when inquiring and paying the signing, the New databases New _ DB1, New _ DB2, New _ DB3 and New _ DB4 are inquired according to rule _ 2. Old databases DB1, DB2 are not queried anymore. Observing whether the online transaction is abnormal or not, and if the online transaction is abnormal or complained, modifying the queryFlag configuration to be 0 and quickly returning; if there is no abnormality or complaint, step 10 is performed.
Corresponding to the step 10: the no longer write old database phase, modifies the isWriteOldDB configuration to 1, indicating that no more writes to the old databases DB1, DB2 are required. At this time, the online incremental subscription data will be written into the New databases New _ DB1, New _ DB2, New _ DB3, and New _ DB4 only according to rule _ 2.
Corresponding to the step 11: after capacity expansion and migration, the payment signing service accesses a payment signing database New _ DB1, New _ DB2, New _ DB3 and New _ DB4 through a database middleware Mycat, and the database middleware Mycat uniformly writes the payment signing data into New _ DB1, New _ DB2, New _ DB3 and New _ DB4 according to rule _ 2; the payment signing service provides a signing data query function, and the query function also performs query by using a Mycat rule _2 and accurately positioning to databases New _ DB1, New _ DB2, New _ DB3 and New _ DB4 according to a card number as an entry parameter. At this point, the expansion and migration are completed.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
Furthermore, it should be understood that although the present description refers to embodiments, not every embodiment may contain only a single embodiment, and such description is for clarity only, and those skilled in the art should integrate the description, and the embodiments may be combined as appropriate to form other embodiments understood by those skilled in the art.
Claims (7)
1. A horizontal segmentation capacity expansion and migration method of payment subscription data is characterized by comprising the following steps:
step 1: before capacity expansion and migration, the payment signing service accesses a plurality of old databases through the database middleware; the old database middleware uniformly writes the payment signing data into all the old databases according to the database dividing rule; the payment signing service provides a signing data query function, and the query function is accurately positioned to an old database to execute query by taking a database splitting rule set in the middleware as an entry parameter;
step 2: establishing a plurality of new databases, configuring an access rule in the middleware of the new databases, and reading and writing all new databases signed with payment based on the access rule;
and step 3: increasing whether to write a new database switch configuration item, including configurations in which new database configuration is not written and incremental subscription data is written in a new database;
and 4, step 4: adding a signing data query switch configuration item, including the configuration of querying an old database and the configuration of querying a new database;
and 5: adding configuration items of whether to write the old database or not, wherein the configuration items comprise the configuration of the old database and the configuration of the old database which is not written any more;
step 6: changing the new data switch configuration item modification into the configuration of writing the incremental signing data into a new database; at the moment, the payment signing service writes the incremental signing data into the old database according to the database partitioning rule, writes the incremental signing data into all new databases according to the access rule, and finally returns the success of the payment signing application;
and 7: inquiring data in all old databases according to the database partitioning rule, and writing all stock data into a new database according to the access rule;
and 8: inquiring data in all old databases according to the database partitioning rule, inquiring data in all new databases according to the access rule, checking whether the data in the old databases are consistent with the data in the new databases or not, and executing the step 9 if the data in the old databases are consistent with the data in the new databases;
and step 9: modifying the configuration item of the signing data query switch into the configuration of querying a new database, querying all the new databases according to the access rule, and not querying an old database; detecting whether the online transaction is abnormal or not, if so, modifying the configuration item of the signing data query switch into the configuration for querying the old database for fast rollback; if no abnormity or complaint exists, executing step 10;
step 10: modifying the configuration item of whether to write the old database into the configuration of not writing the old database any more, and writing the online incremental signing data into the new database only according to the access rule at the moment;
step 11: the payment signing service accesses all the new databases through the database middleware, the database middleware uniformly writes the payment signing data into the new databases according to the access rules, the payment signing service simultaneously provides the inquiry function of the signing data, the inquiry function is accurately positioned into the new databases through the access rules of the middleware to execute inquiry, and the expansion and the migration are completed.
2. The method as claimed in claim 1, wherein the middleware uses the signed card number field as a banking rule.
3. The method as claimed in claim 1, wherein the middleware uses a signed card number field as an access rule.
4. The method for horizontally splitting, expanding and migrating the payment contract data according to any one of claims 1 to 3, wherein the step 7 comprises the following steps:
step 7.1: increasing stock data migration service, wherein the service is used for configuring data migration starting time, data migration ending time and time range of single data query; the time range of the single query is minutes;
step 7.2: querying historical data in an old database according to the configured data migration starting time, data migration ending time, the time range of single data query and a database partitioning rule, and putting the data of the single query into a list A;
step 7.3: circularly traversing the list A, and submitting and inserting the data in the list A into a new database in batches according to the access rule of the middleware;
step 7.4: and (4) increasing the query time period according to the time range of the single query, performing next cycle query, putting the queried data into the list A again, executing the batch insertion logic again, inserting the data in the list A into the new database, and repeating the above cycle until all the data in the range from the data migration starting time to the data migration ending time are migrated.
5. The method of claim 4, wherein the step 8 comprises the steps of:
step 8.1: adding a detail comparison switch, a stroke comparison switch, a comparison starting time and a comparison ending time of the data migration check service configuration;
step 8.2: the detail comparison switch is turned on, and the data migration check service circularly queries the database in the old database according to the database division rule, the comparison starting time, the comparison ending time and the mysql paging query parameter to obtain a list B;
step 8.3: traversing the data in the list B, inquiring the new database according to the access rule, judging whether the new database has the data of the corresponding card number, if not, reporting an error, and failing to compare; if yes, continuing the next data comparison and continuing the cyclic query of the step 8.2 until the paging query is finished and the checking is successful;
step 8.4: starting a stroke number comparison switch, and inquiring detailed stroke numbers in an old database by the data migration check service according to the database dividing rule, the comparison starting time and the comparison ending time to obtain a stroke number value A; inquiring the detailed stroke number in the new database according to the access rule, the comparison starting time and the comparison ending time to obtain a stroke number value B; comparing whether the stroke value A and the stroke value B are equal or not, and if not, reporting an error; if the two are equal, the checking is successful;
step 8.5: and modifying the values of the comparison starting time and the comparison ending time, restarting the data migration check service, repeating the steps from 8.2 to 8.4, and checking the data in other time periods in a spot check mode.
6. The method for horizontally splitting, expanding and migrating the payment contract data according to claim 5, wherein if the configured comparison start time is equal to the data migration start time and the comparison end time is equal to the data migration deadline, the step 8.2 to the step 8.4 are full amount checks; and if the migration time end is not equal to the checking time period, checking for spot check.
7. The method for horizontally splitting, expanding and migrating the payment signing data according to claim 5, wherein when the data volume of expansion and migration is larger than a preset threshold, the step 8.2 to the step 8.4 are spot check checks; if the volume expansion and the migrated data amount are equal to or larger than the preset threshold, the step 8.2 to the step 8.4 are the full check.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111029012.3A CN113468150A (en) | 2021-09-03 | 2021-09-03 | Horizontal segmentation capacity expansion and migration method for payment subscription data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111029012.3A CN113468150A (en) | 2021-09-03 | 2021-09-03 | Horizontal segmentation capacity expansion and migration method for payment subscription data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113468150A true CN113468150A (en) | 2021-10-01 |
Family
ID=77867265
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111029012.3A Pending CN113468150A (en) | 2021-09-03 | 2021-09-03 | Horizontal segmentation capacity expansion and migration method for payment subscription data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113468150A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115860734A (en) * | 2022-12-12 | 2023-03-28 | 上海汇付支付有限公司 | Unitized payment method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160292427A1 (en) * | 2015-03-30 | 2016-10-06 | Airbnb, Inc. | Database Encryption to Provide Write Protection |
| CN108874828A (en) * | 2017-05-12 | 2018-11-23 | 北京京东尚科信息技术有限公司 | The method and apparatus of Data Migration |
| CN110019133A (en) * | 2017-12-21 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Online data moving method and device |
| CN110825813A (en) * | 2019-11-14 | 2020-02-21 | 中国民航信息网络股份有限公司 | Data migration method and device |
| CN112256675A (en) * | 2020-11-05 | 2021-01-22 | 深圳壹账通智能科技有限公司 | Data migration method and device, terminal equipment and storage medium |
| CN112783859A (en) * | 2021-01-08 | 2021-05-11 | 河北志晟信息技术股份有限公司 | Lightweight concurrent migration method for database |
| CN112860670A (en) * | 2021-03-29 | 2021-05-28 | 中信银行股份有限公司 | Data migration method and device, electronic equipment and computer storage medium |
-
2021
- 2021-09-03 CN CN202111029012.3A patent/CN113468150A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160292427A1 (en) * | 2015-03-30 | 2016-10-06 | Airbnb, Inc. | Database Encryption to Provide Write Protection |
| CN108874828A (en) * | 2017-05-12 | 2018-11-23 | 北京京东尚科信息技术有限公司 | The method and apparatus of Data Migration |
| CN110019133A (en) * | 2017-12-21 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Online data moving method and device |
| CN110825813A (en) * | 2019-11-14 | 2020-02-21 | 中国民航信息网络股份有限公司 | Data migration method and device |
| CN112256675A (en) * | 2020-11-05 | 2021-01-22 | 深圳壹账通智能科技有限公司 | Data migration method and device, terminal equipment and storage medium |
| CN112783859A (en) * | 2021-01-08 | 2021-05-11 | 河北志晟信息技术股份有限公司 | Lightweight concurrent migration method for database |
| CN112860670A (en) * | 2021-03-29 | 2021-05-28 | 中信银行股份有限公司 | Data migration method and device, electronic equipment and computer storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115860734A (en) * | 2022-12-12 | 2023-03-28 | 上海汇付支付有限公司 | Unitized payment method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111143389B (en) | Transaction execution method and device, computer equipment and storage medium | |
| EP2797013B1 (en) | Database update execution according to power management schemes | |
| CN111597015B (en) | Transaction processing method and device, computer equipment and storage medium | |
| CN111338766A (en) | Transaction processing method and device, computer equipment and storage medium | |
| US10621156B1 (en) | Application schemas for journal-based databases | |
| CN111159252A (en) | Transaction execution method and device, computer equipment and storage medium | |
| US20090063807A1 (en) | Data redistribution in shared nothing architecture | |
| CN108961029A (en) | A kind of distribution reconciliation processing method, system and terminal device | |
| US20100169289A1 (en) | Two Phase Commit With Grid Elements | |
| CN111522631A (en) | Distributed transaction processing method, device, server and medium | |
| CN112035528A (en) | Data query method and device | |
| US20230137119A1 (en) | Method for replaying log on data node, data node, and system | |
| CN111367994A (en) | Method and system for synchronously backing up incremental data of database | |
| CN110597835A (en) | Transaction data deleting method and device based on block chain | |
| CN115599807A (en) | Data access method, device, application server and storage medium | |
| CN115712670A (en) | Data source management system | |
| CN107704603A (en) | A kind of method and device for realizing read and write abruption | |
| CN113468150A (en) | Horizontal segmentation capacity expansion and migration method for payment subscription data | |
| US20070174329A1 (en) | Presenting a reason why a secondary data structure associated with a database needs rebuilding | |
| US20200279309A1 (en) | Blockchain-based electronic bill cancellation method, apparatus, and electronic device | |
| CN112131305A (en) | Account processing system | |
| CN115469810A (en) | Data acquisition method, device, equipment and storage medium | |
| CN115114289A (en) | Data query method and device and electronic equipment | |
| CN111382207B (en) | Data processing method, device, system and storage medium | |
| CN103685350B (en) | The synchronous method of storage system and relevant equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211001 |