CN112948359A - Non-stop client data migration method based on database sub-base sub-table - Google Patents
Non-stop client data migration method based on database sub-base sub-table Download PDFInfo
- Publication number
- CN112948359A CN112948359A CN202110300776.5A CN202110300776A CN112948359A CN 112948359 A CN112948359 A CN 112948359A CN 202110300776 A CN202110300776 A CN 202110300776A CN 112948359 A CN112948359 A CN 112948359A
- Authority
- CN
- China
- Prior art keywords
- data
- migration
- old
- database
- new
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of data migration, and provides a non-stop client data migration method based on database sub-table. The main scheme includes that a stage of data migration is carried out after the time set by the T day, and the migration data range is all customer information data in an old system database before the T day; opening a business transaction switch of a new system and an old system to carry out customer business verification; performing two-stage data migration, wherein the data migrated in the two stages is newly added data after the T day 0 point of the old system; the old and new systems are parallel, and data check is carried out daily in the period.
Description
Technical Field
The technology is applied to data migration between two systems, wherein the two systems are an old system and a new system respectively, the old system adopts a single database to store data, the new system adopts a database and a table to store data, the coupling between database tables can be reduced through the technology, and higher data consistency is realized on the premise of no shutdown.
Background
With the rapid development of internet technology, since an old system adopts a single database mode to store data during initial design, and a single database gradually fails to meet the requirements of high-speed increasing user data scale and increasingly complex customer information acquisition, a new data storage mode is urgently needed, and sub-database and sub-table operation is needed. The new system carries out modeling again on the customer information, and stores the customer information into a plurality of libraries according to certain rules in a mode of separating libraries and tables.
When data migration is performed according to a new data model, in order to ensure data consistency, service is usually stopped before data migration, if the data migration is performed after the service is stopped, the new data model needs to be started immediately after data processing is completed, the coupling degree in the whole process is high, the implementation process is complex, and if the risk of failure and rollback are high, unpredictable loss may be brought to business.
Disclosure of Invention
When a new system is on line, the migration work of the client data of the old system exists, and the migration work is divided into two stages: first-stage migration and second-stage migration; the scheme explains that the migration of the client data of the old system to the new system is completed under the condition that the old system does not shut down. And synchronously migrating the customer data on the day of online commissioning, and opening a service switch of a new and old system to verify the service transaction after the data migration is finished. After the new system is put into operation, the client information is subject to the data of the new system; and the new system and the old system are in parallel for a period of time until the new system stably operates.
In order to achieve the purpose, the invention adopts the following technical scheme:
a non-stop client data migration method based on database sub-base sub-table comprises the following steps:
s1., performing a stage of data migration after the time set by the T day, wherein the migration data range is all the customer information data in the old system database before the T day;
s2, after step s1, opening a new and old system service migration switch for customer service verification;
s3. after step s2, performing two-stage data migration, where the data migrated in the two stages is newly added data after the old system T day 0 point;
s4. after step s3, the old and new systems are parallelized, during which data checks are performed daily.
2. The non-stop client data migration method based on database sub-base and sub-table as claimed in claim 1, wherein: the step S1 includes the steps of:
step 1.1: placing the inspection script, the statistical script, the dirty data cleaning script and the migration program at the designated position of the migration server, wherein the migration policy of the migration program is as follows:
a stage of data migration: because the new system has a small amount of customer data, the customer data of the new system stock is taken as the standard in the migration process, a temporary table is created in a new system database during migration, all data in an old system database is migrated into the temporary table, a customer number of a corresponding customer in the new system is searched in a corresponding database of the new system according to a certificate number of each record, replacement is carried out, the replaced data is compared with the data of the new system stock, whether the new system has the record or not is judged according to the certificate number and the certificate type, if yes, the record is not migrated, otherwise, the record is migrated;
two-stage data migration: the two-stage migration range is T daily increment data, and the rest steps are the same as the one-stage migration range;
step 1.2: starting at the time set on day T-1, using the checking script in step 1.1 to check and modify data, specifically checking whether the old system database has abnormal data:
if the date field is the date which does not exist, or illegal characters exist in the telephone field; if the data appear, corresponding modification is needed;
step 1.3: after the time set by the day T, the migration program in the step 1.1 is used for starting the first migration, the migration data range is all the customer information data in the old system database before the day T, and the migration strategy is the strategy in the step 1.1;
step 1.4: after step 1.3, the dirty data cleaning script in step 1.1 is used to clean the invalid data records in the database of the new system, and the invalid data judgment criteria are as follows: if the key field of a certain record such as the ID card number does not accord with the rule, abandoning the record;
step 1.5: after step 1.4, the statistical script in step 1 is used for data checking, and the total number of the data in the old database and the new database before T days is compared.
In the above technical solution, the step S2 includes the following steps:
step 2.1: opening a business transaction switch of the new and old systems, verifying customer business, forwarding each request to the new system by the old system in real time, enabling the new and old systems to be parallel, if one business runs normally in the old system and the new system fails, failing verification, and if not, directly performing two-stage data migration;
step 2.2: if the client data is verified to have problems or performance problems, the verification fails, the service switch of the new and old system is closed, then problem troubleshooting and repairing are carried out, and in the case, other time needs to be scheduled for re-online.
In the above technical solution, the step S3 includes the following steps:
step 3.1: after the data migration in a phase of T days is completed and the verification is successful, incremental data migration is started, and data are newly added after the migration range is 0 point of T days of the old system;
step 3.2: after step 3.1 is completed, the data cleaning script is run to clean invalid data in the new system, and the invalid data judgment standard is as follows: if the key field of a certain record such as the ID card number does not accord with the rule, abandoning the record;
step 3.3: and after the step 3.2 is completed, running a quantity query script to perform data checking and comparing T day data total quantity, and completing two-stage data migration if verification is successful.
In the above technical solution, the step S4 includes the following steps:
step 4.1: after the two-stage data migration is completed, manually executing a T-day rechecking script to check the data of the T day every day so as to ensure the consistency of the T-day data in the new and old systems;
step 4.2: during the parallel period of the new system and the old system, the data on two sides may be inconsistent, and if the inconsistent data appears, the data of the old system is used as the standard.
Because the invention adopts the technical scheme, the invention has the following beneficial effects:
the database partitioning and table partitioning are beneficial to reducing the pressure of a single database, improving the stability of the database in a high-concurrency and high-throughput access scene and improving the risk resistance of a system;
the data migration without shutdown can be compatible with the use of new and old data, the transition period is smooth, the online service is not affected in the data migration process, and the risk of service interruption is avoided;
drawings
FIG. 1 is a flow chart of step 1 of a non-stop client data migration method based on database sub-base sub-table according to the present invention;
FIG. 2 is a flow chart of step 2 of a non-stop client data migration method based on database sub-base sub-table according to the present invention;
FIG. 3 is a flow chart of step 3 of a non-stop client data migration method based on database sub-base sub-table according to the present invention;
fig. 4 is a flowchart of step 4 of a non-stop client data migration method based on database sub-base sub-table according to the present invention.
Detailed Description
The following detailed description of embodiments of the present invention is provided in connection with the accompanying drawings and examples. The following examples are intended to illustrate the invention but are not intended to limit the scope of the invention.
As shown in fig. 1, fig. 2, fig. 3, and fig. 4, which are flow charts of steps of a non-stop client data migration method based on database sub-base sub-table according to the present invention, respectively, includes the following steps:
fig. 1 comprises the following steps:
step 1: placing the inspection script, the statistical script, the dirty data cleaning script and the migration program at the designated position of the migration server, wherein the migration policy of the migration program is as follows:
a stage of data migration: because the new system has a small amount of customer data, the customer data of the new system stock is taken as the standard in the migration process, a temporary table is created in a new system database during migration, all data in an old system database is migrated into the temporary table, a customer number of a corresponding customer in the new system is searched in a corresponding database of the new system according to a certificate number of each record, replacement is carried out, the replaced data is compared with the data of the new system stock, whether the new system has the record or not is judged according to the certificate number and the certificate type, if yes, the record is not migrated, otherwise, the record is migrated;
two-stage data migration: the two-stage migration range is T daily increment data, and the rest steps are the same as the one-stage migration range;
step 2: starting at the time set by the day T-1, using the checking script in the step 1 to check and modify data, and specifically checking whether the old system database has abnormal data:
if the date field is the date which does not exist, or illegal characters exist in the telephone field; if the data appear, corresponding modification is needed;
and step 3: after the time set by the T day, starting the first migration by using the migration program in the step 1, wherein the migration data range is all the customer information data in the old system database before the T day, and the migration strategy is the strategy in the step 1;
and 4, step 4: after step 3, using the dirty data cleaning script in step 1 to clean invalid data records in the database of the new system, wherein the invalid data judgment criteria are as follows: if the key field of a certain record such as the ID card number does not accord with the rule, abandoning the record;
and 5: after step 4, using the statistical script in step 1 to perform data check, and comparing the total number of the data in the old database and the new database before T days.
Fig. 2 comprises the following steps:
step 1: opening a business transaction switch of the new and old systems, verifying customer business, forwarding each request to the new system by the old system in real time, enabling the new and old systems to be parallel, if one business runs normally in the old system and the new system fails, failing verification, and if not, directly performing two-stage data migration;
step 2: if the client data is verified to have problems or performance problems, the verification fails, the service switch of the new and old system is closed, then problem troubleshooting and repairing are carried out, and in the case, other time needs to be scheduled for re-online.
Fig. 3 comprises the following steps:
step 1: after the data migration in a phase of T days is completed and the verification is successful, incremental data migration is started, and data are newly added after the migration range is 0 point of T days of the old system;
step 2: after step 1 is completed, the data cleaning script is operated to clean invalid data in the new system, and the invalid data judgment standard is as follows: if the key field of a certain record such as the ID card number does not accord with the rule, abandoning the record;
and step 3: and after the step 2 is completed, running a quantity query script to perform data checking and comparing T day data total quantity, and completing two-stage data migration if verification is successful.
Fig. 4 comprises the following steps:
step 1: after the two-stage data migration is completed, manually executing a T-day rechecking script to check the data of the T day every day so as to ensure the consistency of the T-day data in the new and old systems;
step 2: during the parallel period of the new system and the old system, the data on two sides may be inconsistent, and if the inconsistent data appears, the data of the old system is used as the standard.
Claims (5)
1. A non-stop client data migration method based on database sub-base sub-table is characterized by comprising the following steps:
s1., performing a stage of data migration after the time set by the T day, wherein the migration data range is all the customer information data in the old system database before the T day;
s2, after step s1, opening a new and old system service migration switch for customer service verification;
s3. after step s2, performing two-stage data migration, where the data migrated in the two stages is newly added data after the old system T day 0 point;
s4. after step s3, the old and new systems are parallelized, during which data checks are performed daily.
2. The non-stop client data migration method based on database sub-base and sub-table as claimed in claim 1, wherein: the step S1 includes the steps of:
step 1.1: placing the inspection script, the statistical script, the dirty data cleaning script and the migration program at the designated position of the migration server, wherein the migration policy of the migration program is as follows:
a stage of data migration: because the new system has a small amount of customer data, the customer data of the new system stock is taken as the standard in the migration process, a temporary table is created in a new system database during migration, all data in an old system database is migrated into the temporary table, a customer number of a corresponding customer in the new system is searched in a corresponding database of the new system according to a certificate number of each record, replacement is carried out, the replaced data is compared with the data of the new system stock, whether the new system has the record or not is judged according to the certificate number and the certificate type, if yes, the record is not migrated, otherwise, the record is migrated;
two-stage data migration: the two-stage migration range is T daily increment data, and the rest steps are the same as the one-stage migration range;
step 1.2: starting at the time set on day T-1, using the checking script in step 1.1 to check and modify data, specifically checking whether the old system database has abnormal data:
if the date field is the date which does not exist, or illegal characters exist in the telephone field; if the data appear, corresponding modification is needed;
step 1.3: after the time set by the day T, the migration program in the step 1.1 is used for starting the first migration, the migration data range is all the customer information data in the old system database before the day T, and the migration strategy is the strategy in the step 1.1;
step 1.4: after step 1.3, the dirty data cleaning script in step 1.1 is used to clean the invalid data records in the database of the new system, and the invalid data judgment criteria are as follows: if the key field of a certain record such as the ID card number does not accord with the rule, abandoning the record;
step 1.5: after step 1.4, the statistical script in step 1 is used for data checking, and the total number of the data in the old database and the new database before T days is compared.
3. The non-stop client data migration method based on database sub-base and sub-table as claimed in claim 1, wherein: the step S2 includes the steps of:
step 2.1: opening a business transaction switch of the new and old systems, verifying customer business, forwarding each request to the new system by the old system in real time, enabling the new and old systems to be parallel, if one business runs normally in the old system and the new system fails, failing verification, and if not, directly performing two-stage data migration;
step 2.2: if the client data is verified to have problems or performance problems, the verification fails, the service switch of the new and old system is closed, then problem troubleshooting and repairing are carried out, and in the case, other time needs to be scheduled for re-online.
4. The non-stop client data migration method based on database sub-base and sub-table as claimed in claim 1, wherein: the step S3 includes the steps of:
step 3.1: after the data migration in a phase of T days is completed and the verification is successful, incremental data migration is started, and data are newly added after the migration range is 0 point of T days of the old system;
step 3.2: after step 3.1 is completed, the data cleaning script is run to clean invalid data in the new system, and the invalid data judgment standard is as follows: if the key field of a certain record such as the ID card number does not accord with the rule, abandoning the record;
step 3.3: and after the step 3.2 is completed, running a quantity query script to perform data checking and comparing T day data total quantity, and completing two-stage data migration if verification is successful.
5. The non-stop client data migration method based on database sub-base and sub-table as claimed in claim 1, wherein: the step S4 includes the steps of:
step 4.1: after the two-stage data migration is completed, manually executing a T-day rechecking script to check the data of the T day every day so as to ensure the consistency of the T-day data in the new and old systems;
step 4.2: during the parallel period of the new system and the old system, the data on two sides may be inconsistent, and if the inconsistent data appears, the data of the old system is used as the standard.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110300776.5A CN112948359A (en) | 2021-03-22 | 2021-03-22 | Non-stop client data migration method based on database sub-base sub-table |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110300776.5A CN112948359A (en) | 2021-03-22 | 2021-03-22 | Non-stop client data migration method based on database sub-base sub-table |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112948359A true CN112948359A (en) | 2021-06-11 |
Family
ID=76227842
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110300776.5A Withdrawn CN112948359A (en) | 2021-03-22 | 2021-03-22 | Non-stop client data migration method based on database sub-base sub-table |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112948359A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114138744A (en) * | 2021-11-24 | 2022-03-04 | 中国建设银行股份有限公司 | Client data migration method and system during operation |
| CN115080251A (en) * | 2022-08-23 | 2022-09-20 | 广州市宏方网络科技有限公司 | Archive data security management method and system based on cloud migration platform |
| CN117555883A (en) * | 2024-01-11 | 2024-02-13 | 梅州客商银行股份有限公司 | Bank system data database separation method and device, memory and electronic equipment |
-
2021
- 2021-03-22 CN CN202110300776.5A patent/CN112948359A/en not_active Withdrawn
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114138744A (en) * | 2021-11-24 | 2022-03-04 | 中国建设银行股份有限公司 | Client data migration method and system during operation |
| CN115080251A (en) * | 2022-08-23 | 2022-09-20 | 广州市宏方网络科技有限公司 | Archive data security management method and system based on cloud migration platform |
| CN117555883A (en) * | 2024-01-11 | 2024-02-13 | 梅州客商银行股份有限公司 | Bank system data database separation method and device, memory and electronic equipment |
| CN117555883B (en) * | 2024-01-11 | 2024-04-05 | 梅州客商银行股份有限公司 | Bank system data database separation method and device, memory and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112948359A (en) | Non-stop client data migration method based on database sub-base sub-table | |
| CN108959400B (en) | Bank system historical data cleaning method and device | |
| CN102236672B (en) | A kind of data lead-in method and device | |
| CN100375038C (en) | Finally agent optimization method and system of source in reorder two-stage refering | |
| US7043504B1 (en) | System and method for parallel primary and secondary backup reading in recovery of multiple shared database data sets | |
| CN106845781B (en) | Scene and flow generation system and method for business test | |
| US6594676B1 (en) | System and method for recovery of multiple shared database data sets using multiple change accumulation data sets as inputs | |
| CN105787058A (en) | User label system and data pushing system based on same | |
| CN111367994A (en) | Method and system for synchronously backing up incremental data of database | |
| CN100392651C (en) | Universal constraining realizing method for real-time internal memory database | |
| CN1976320B (en) | Data access control method and system | |
| CN111126970A (en) | Operation management system and method based on data consistency | |
| CN102156744A (en) | Method for eliminating repetition of memory dialog list | |
| CN106155838A (en) | A kind of database back-up data restoration methods and device | |
| US8639657B2 (en) | Reorganizing table-based data objects | |
| CN108984337A (en) | A kind of restorative procedure of data synchronous abnormality, prosthetic device, medium and calculate equipment | |
| CN111400390B (en) | Data processing method and device | |
| CN109426576B (en) | Fault-tolerant processing method and fault-tolerant assembly | |
| US20050066235A1 (en) | Automated fault finding in repository management program code | |
| CN116089394A (en) | Data rollback method, storage medium and device of database | |
| CN113987075A (en) | Data synchronization method, device, equipment and readable storage medium | |
| CN110675283B (en) | Power distribution equipment synchronous management system and method | |
| CN112380186A (en) | Domestic database migration method based on active adaptation | |
| CN115292343B (en) | Data processing method and device | |
| CN115048439B (en) | Data conversion method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210611 |