CN115757345A - Mass data smooth migration method and system - Google Patents
Mass data smooth migration method and system Download PDFInfo
- Publication number
- CN115757345A CN115757345A CN202211461666.8A CN202211461666A CN115757345A CN 115757345 A CN115757345 A CN 115757345A CN 202211461666 A CN202211461666 A CN 202211461666A CN 115757345 A CN115757345 A CN 115757345A
- Authority
- CN
- China
- Prior art keywords
- data
- user account
- user
- fingerprint
- migration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005012 migration Effects 0.000 title claims abstract description 117
- 238000013508 migration Methods 0.000 title claims abstract description 117
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000004364 calculation method Methods 0.000 claims description 8
- 230000007547 defect Effects 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 6
- 238000012913 prioritisation Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 18
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a system for smoothly migrating mass data, which are used for formulating the priority of a user account according to historical behavior data of the user account, wherein the user account with high priority preferentially migrates data, incremental data preferentially enters a new database, and is inserted, modified and locked in the migration process, so that the migration speed is greatly accelerated while the data consistency is ensured.
Description
Technical Field
The invention relates to the technical field of databases, in particular to a method and a system for smoothly migrating mass data.
Background
In the gradual evolution process of the system architecture, the database upgrading iteration cannot be avoided, such as the conversion from the ORACLE database to the MYSQL database and the conversion from the MYSQL database to the NOSQL database. The data volume of a large distributed database is usually huge, and can reach the level of billions or even billions. After the database is replaced, smooth migration and data integrity of a large amount of data become difficult problems, and particularly in the migration process, user data incremental data is written, the migration speed is slow, migration errors are likely to be caused in the migration process, and the consistency and the integrity of the migrated data are difficult to guarantee.
Disclosure of Invention
The invention provides a mass data smooth migration method and system, which are used for solving the technical problems that the existing mass data migration mode is low in migration speed and consistency and integrity of migration data are difficult to guarantee.
In view of this, a first aspect of the present invention provides a method for smoothly migrating mass data, including:
the method comprises the following steps of S1, obtaining historical behavior data of all user accounts, and carrying out priority division on each user account according to the historical behavior data of each user account, wherein the priority comprises a high level, a middle level and a low level;
s2, sequentially migrating high-level user account data, medium-level account data and low-level account data in batches according to the priority, locking data writing and modifying permission of a user account when migrating the user account data, and opening the data writing and modifying permission of the user account after migrating the user account data;
s3, after the data migration of each user account is completed, calculating a new data fingerprint and an old data fingerprint of the user account according to a preset data fingerprint generation algorithm, wherein the new data fingerprint is the data fingerprint of the user account migrated into a new database, and the old data fingerprint is the data fingerprint of the user account in an old database of migrated data;
s4, comparing the new data fingerprint and the old data fingerprint of the user account;
s5, if the new data fingerprint and the old data fingerprint of the user account are consistent, judging that the user account data are completely migrated, and finishing the migration of the user account data;
and S6, if the new data fingerprint and the old data fingerprint of the user account are inconsistent, carrying out error checking and defect filling on the data migrated by the user account, and returning to the step S4.
Optionally, step S1 specifically includes:
the method comprises the steps of obtaining historical behavior data of all user accounts, carrying out priority division on each user account according to core active operation frequency in the historical behavior data of the user accounts, dividing the user accounts with core active operation in a first preset time period into high-level user accounts, dividing the user accounts with core active operation in a second preset time period into middle-level user accounts, and dividing the user accounts without core active operation in a third preset time period into low-level user accounts, wherein the core active operation comprises login verification, file uploading, file downloading and file sharing, the first preset time period is earlier than the second preset time period, and the second preset time period is earlier than the third preset time period.
Optionally, the first preset period is 3 months, the second preset period is 4-6 months, and the third preset period is more than 6 months.
Optionally, step S5 further includes:
and making a migration mark for the user account with the migrated data in the old database.
Optionally, step S5 is followed by:
s7, detecting whether the user account has operation or not, and if so, judging whether the user account has a migration mark in an old database or not;
s8, if the user account has a migration mark in the old database, responding to the operation of the user account through the new database;
and S9, if the user account does not have the migration mark in the old database, responding to the operation of the user account through the old database.
Optionally, step S3 specifically includes:
s31, counting data information of each user account in an old database, wherein the data information comprises an account name, the number of migration tables, the number of data lines and the size of total migration data;
s32, separating each piece of data information of the user account by commas in parentheses, and generating a first data statistical information character string;
s33, calculating the first data statistical information character string by applying a hash function SHA256 to obtain an old data fingerprint of the user account;
and S34, counting the data information of each user account in the new database, separating each data information of the user account in parentheses by commas, generating a second data statistical information character string, and calculating the second data statistical information character string by applying a hash function SHA256 to obtain a new data fingerprint of the user account.
The second aspect of the present invention provides a mass data smooth migration system, including:
the system comprises a user priority dividing module, a priority calculating module and a priority calculating module, wherein the user priority dividing module is used for acquiring historical behavior data of all user accounts and performing priority division on each user account according to the historical behavior data of each user account, and the priority comprises a high level, a middle level and a low level;
the user data migration module is used for sequentially migrating high-level user account data, medium-level account data and low-level account data in batches according to priority, locking data writing and modifying authority of a user account when migrating the user account data, and opening the data writing and modifying authority of the user account after migrating the user account data;
the data fingerprint calculation module is used for calculating a new data fingerprint and an old data fingerprint of each user account according to a preset data fingerprint generation algorithm after data migration of each user account is completed, wherein the new data fingerprint is the data fingerprint of the user account migrated into a new database, and the old data fingerprint is the data fingerprint of the user account in an old database of migrated data;
the data fingerprint comparison module is used for comparing the new data fingerprint and the old data fingerprint of the user account;
the first result module is used for judging that the user account data are completely migrated if the new data fingerprint and the old data fingerprint of the user account are consistent, and finishing the migration of the user account data;
and the second result module is used for carrying out error checking and defect filling on the data migrated by the user account and returning to the execution data fingerprint comparison module if the new data fingerprint and the old data fingerprint of the user account are inconsistent.
Optionally, the user prioritization module is specifically configured to:
the method comprises the steps of obtaining historical behavior data of all user accounts, carrying out priority division on each user account according to core active operation frequency in the historical behavior data of the user accounts, dividing the user accounts with core active operation in a first preset time period into high-level user accounts, dividing the user accounts with core active operation in a second preset time period into middle-level user accounts, and dividing the user accounts without core active operation in a third preset time period into low-level user accounts, wherein the core active operation comprises login verification, file uploading, file downloading and file sharing, the first preset time period is earlier than the second preset time period, and the second preset time period is earlier than the third preset time period.
Optionally, the first preset time period is 3 months, the second preset time period is 4-6 months, and the third preset time period is more than 6 months.
Optionally, the data fingerprint calculation module is specifically configured to:
counting data information of each user account in an old database, wherein the data information comprises an account name, the number of migration tables, the number of data lines and the size of total migration data;
separating each piece of data information of the user account by commas in parentheses to generate a first data statistical information character string;
calculating the first data statistical information character string by using a hash function SHA256 to obtain an old data fingerprint of the user account;
and counting the data information of each user account in the new database, separating each data information of the user account in parentheses by commas, generating a second data statistical information character string, and calculating the second data statistical information character string by applying a hash function SHA256 to obtain a new data fingerprint of the user account.
According to the technical scheme, the mass data smooth migration method and the mass data smooth migration system have the following advantages:
according to the method for smoothly migrating the mass data, the priority of the user account is set according to the historical behavior data of the user account, the user account with high priority preferentially migrates data, incremental data preferentially enters a new database, modification locking is inserted in the migration process, the migration speed is greatly accelerated while the data consistency is ensured, meanwhile, after the data migration of each user account is completed, fingerprint data calculation is carried out on the data information of the user account to obtain new and old data fingerprints with uniqueness, the new and old data fingerprints are compared, the integrity of the user data before and after migration is ensured, the consistency of the user data before and after migration is further ensured, and the technical problems that the migration speed is low and the consistency and the integrity of the migrated data are difficult to ensure in the existing mass data migration mode are solved.
Meanwhile, the mass data smooth migration method provided by the invention also develops user shunting microservice, migration marks are carried out on the user account with migrated data in the old database, when the user account has operation, whether the user account has the migration marks in the old database is judged, if the user account has the migration marks, the operation of the user account responds through the new database, and if the user account has not migrated, the user account responds through the old database. The process is completely transparent to the user, the user does not perceive, and the user experience is not influenced. With the continuous progress of the migration operation, more and more users are migrated to the new database, after all the users are migrated, the old database cluster can be offline, and the new database takes over all the user data, so that the database cluster switching is realized.
The principle and the obtained technical effect of the mass data smooth migration system provided by the invention are the same as those of the mass data smooth migration method provided by the invention, and the details are not repeated herein.
Drawings
In order to clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other relevant drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a mass data smooth migration method provided in the present invention;
FIG. 2 is another schematic flow chart of a mass data smooth migration method provided in the present invention;
fig. 3 is a schematic structural diagram of a mass data smooth migration system provided in the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
For easy understanding, please refer to fig. 1, an embodiment of a mass data smooth migration method provided in the present invention includes:
It should be noted that, in the embodiment of the present invention, priority level operation is performed on a user account first, and the user account is divided into a high-level user account, a medium-level user account, and a low-level user account according to historical behavior data of the user account. Specifically, each user account is subjected to priority division according to core active operation frequency in historical behavior data of the user accounts, the user accounts with core active operation in a first preset time period are divided into high-level user accounts, the user accounts with core active operation in a second preset time period are divided into middle-level user accounts, and the user accounts without core active operation in a third preset time period are divided into low-level user accounts, wherein the core active operation comprises login verification, file uploading, file downloading and file sharing, the first preset time period is earlier than the second preset time period, and the second preset time period is earlier than the third preset time period. Preferably, the first preset period is 3 months, the second preset period is 4-6 months, and the third preset period is more than 6 months.
And 102, sequentially migrating high-level user account data, medium-level account data and low-level account data in batches according to the priority, locking data writing and modifying authority of the user account when migrating the user account data, and opening the data writing and modifying authority of the user account after migrating the user account data.
It should be noted that, the user account data is migrated in batches according to the priority, the user account data with the high priority is migrated first, and theoretically, the number of the newly added files of the user account with the high priority is large, so that the incremental file can directly enter the new database, the process that the incremental data is written into the old database and then migrated to the new database is avoided, and the migration speed is greatly increased. And then migrating the medium-priority user account data, and finally migrating the low-priority user account data, wherein the low-priority user account can be regarded as a silent user account, most of the low-priority user account can be regarded as a history file, and finally, all data migration is completed. The time consumption of the data migration process of a single user account is short, the writing and modifying authority of the user account data is temporarily locked in the migration process, and the writing and modifying authority is released after the migration is completed, so that the consistency of the user account data can be ensured, and the user experience cannot be basically influenced.
And 103, after the data migration of each user account is completed, calculating a new data fingerprint and an old data fingerprint of the user account according to a preset data fingerprint generation algorithm, wherein the new data fingerprint is the data fingerprint of the user account migrated into a new database, and the old data fingerprint is the data fingerprint of the user account in an old database of the migrated data.
It should be noted that after the migration of the user account data is completed, the data fingerprint is calculated by using a preset data fingerprint generation algorithm according to the number, size, core data and the like of the user account data in the new and old databases.
Specifically, the execution process of step 103 is:
counting data information of each user account in an old database, wherein the data information comprises an account name, the number of migration tables, the number of data lines and the size of total migration data;
separating each piece of data information of the user account by commas in parentheses to generate a first data statistical information character string;
calculating the first data statistical information character string by using a hash function SHA256 to obtain an old data fingerprint of the user account;
and counting the data information of each user account in the new database, separating each data information of the user account in brackets by commas, generating a second data statistical information character string, and calculating the second data statistical information character string by applying a hash function SHA256 to obtain a new data fingerprint of the user account.
And 105, if the new data fingerprint and the old data fingerprint of the user account are consistent, judging that the user account data are completely migrated, and finishing the migration of the user account data.
For example, the user account is user1@189.Cn, 10 tables are migrated to the user account, 36000 rows of data are provided, the total migration data size is 1M, that is, 1048576 bytes, and then the data statistics information string user1_ str _ old is:
[user1@189.cn,10,36000,1048576]
calculating the user1_ str _ old by using a hash function SHA256 to obtain the value of the data fingerprint of the user as user1_ zw _ old; similarly, the data of the user1 in the new database is counted to obtain a data information character string user1_ str _ new, the hash function SHA256 is applied to calculate the user1_ str _ new, and the value of the data fingerprint of the user is obtained as user1_ zw _ new. And comparing whether the fingerprint values of the two data of the user1_ zw _ old and the user1_ zw _ new are the same or not, and if the fingerprint values are the same, indicating that the data before and after the data migration of the user account user1@189.Cn are completely consistent.
And step 106, if the new data fingerprint and the old data fingerprint of the user account are inconsistent, performing error checking and defect filling on the data migrated by the user account, and returning to the step 104.
It should be noted that, if the new data fingerprint of the user account is inconsistent with the old data fingerprint, it indicates that there is data error or data missing in the data migration process of the user account, the user account migration process needs to be checked, the data error or data missing in the migration process is found, the data error or data missing in the migration process continues to be added or deleted, and the step 104 is returned to compare the data fingerprints again.
In an embodiment, in step 105, after the migration of the user account data is completed, a migration flag is made on the user account with the migrated data in the old database, so that on one hand, whether the user account data has migrated or not can be distinguished, and a situation of repeated migration or incomplete migration is avoided. That is, as shown in fig. 2, after step 105, the following steps may also be performed:
and 107, detecting whether the user account has operation, and if so, judging whether the user account has a migration mark in the old database.
and step 109, responding to the operation of the user account through the old database if the user account has no migration mark in the old database.
Compared with the prior art, the mass data smooth migration method provided by the invention has the following advantages:
according to the method for smoothly migrating the mass data, the priority of the user account is established according to the historical behavior data of the user account, the user account with high priority preferentially migrates data, incremental data preferentially enters a new database, modification locking is inserted in the migration process, the migration speed is greatly accelerated while the data consistency is ensured, meanwhile, after the data migration of each user account is completed, fingerprint data calculation is carried out on the data information of the user account to obtain unique new and old data fingerprints, the new and old data fingerprints are compared, the integrity of the user data before and after migration is ensured, the consistency of the user data before and after migration is further ensured, and the technical problems that the migration speed is low in the existing mass data migration mode and the consistency and the integrity of the migrated data are difficult to ensure are solved.
Meanwhile, the mass data smooth migration method provided by the invention also develops user shunting microservice, migration marks are carried out on the user account with migrated data in the old database, when the user account has operation, whether the user account has the migration marks in the old database is judged, if the user account has the migration marks, the operation of the user account responds through the new database, and if the user account has not migrated, the user account responds through the old database. The process is completely transparent to the user, the user does not perceive, and the user experience is not influenced. With the continuous progress of the migration operation, more and more users are migrated to the new database, after all the users are migrated, the old database cluster can be offline, and the new database takes over all the user data, so that the database cluster switching is realized.
For easy understanding, please refer to fig. 3, an embodiment of a mass data smooth migration system provided in the present invention includes:
the system comprises a user priority dividing module, a priority calculating module and a priority calculating module, wherein the user priority dividing module is used for acquiring historical behavior data of all user accounts and performing priority division on each user account according to the historical behavior data of each user account, and the priority comprises a high level, a middle level and a low level;
the user data migration module is used for sequentially migrating high-level user account data, medium-level account data and low-level account data in batches according to priority, locking data writing and modifying authority of a user account when migrating the user account data, and opening the data writing and modifying authority of the user account after migrating the user account data;
the data fingerprint calculation module is used for calculating a new data fingerprint and an old data fingerprint of each user account according to a preset data fingerprint generation algorithm after data migration of each user account is completed, wherein the new data fingerprint is the data fingerprint of the user account migrated into a new database, and the old data fingerprint is the data fingerprint of the user account in an old database of migrated data;
the data fingerprint comparison module is used for comparing the new data fingerprint and the old data fingerprint of the user account;
the first result module is used for judging that all the user account data are completely migrated if the new data fingerprint and the old data fingerprint of the user account are consistent, and ending the migration of the user account data;
and the second result module is used for carrying out error checking and defect filling on the data migrated by the user account and returning to the execution data fingerprint comparison module if the new data fingerprint and the old data fingerprint of the user account are inconsistent.
The user prioritization module is specifically configured to:
the method comprises the steps of obtaining historical behavior data of all user accounts, carrying out priority division on each user account according to core active operation frequency in the historical behavior data of the user accounts, dividing the user accounts with core active operation in a first preset time period into high-level user accounts, dividing the user accounts with core active operation in a second preset time period into middle-level user accounts, and dividing the user accounts without core active operation in a third preset time period into low-level user accounts, wherein the core active operation comprises login verification, file uploading, file downloading and file sharing, the first preset time period is earlier than the second preset time period, and the second preset time period is earlier than the third preset time period.
The first preset time interval is 3 months, the second preset time interval is 4-6 months, and the third preset time interval is more than 6 months.
The data fingerprint calculation module is specifically configured to:
counting data information of each user account in an old database, wherein the data information comprises an account name, the number of migration tables, the number of data lines and the size of total migration data;
separating each piece of data information of the user account by commas in parentheses to generate a first data statistical information character string;
calculating the first data statistical information character string by using a hash function SHA256 to obtain an old data fingerprint of the user account;
and counting the data information of each user account in the new database, separating each data information of the user account in brackets by commas, generating a second data statistical information character string, and calculating the second data statistical information character string by applying a hash function SHA256 to obtain a new data fingerprint of the user account.
The first results module is further to:
and making a migration mark for the user account with the migrated data in the old database.
Further comprising a response module for:
detecting whether the user account has operation, if so, judging whether the user account has a migration mark in an old database, if so, responding to the operation of the user account through a new database, and if not, responding to the operation of the user account through the old database.
The principle and the obtained technical effects of the mass data smooth migration system provided by the invention for executing the mass data smooth migration method provided by the invention are the same as those of the mass data smooth migration method provided by the invention, and are not described again.
The terms "first," "second," "third," "fourth," and the like in the description of the invention, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A mass data smooth migration method is characterized by comprising the following steps:
the method comprises the following steps of S1, obtaining historical behavior data of all user accounts, and carrying out priority division on each user account according to the historical behavior data of each user account, wherein the priority comprises a high level, a middle level and a low level;
s2, sequentially migrating high-level user account data, medium-level account data and low-level account data in batches according to the priority, locking data writing and modifying permission of a user account when migrating the user account data, and opening the data writing and modifying permission of the user account after migrating the user account data;
s3, after the data migration of each user account is completed, calculating a new data fingerprint and an old data fingerprint of the user account according to a preset data fingerprint generation algorithm, wherein the new data fingerprint is the data fingerprint of the user account migrated into a new database, and the old data fingerprint is the data fingerprint of the user account in an old database of migrated data;
s4, comparing the new data fingerprint with the old data fingerprint of the user account;
s5, if the new data fingerprint and the old data fingerprint of the user account are consistent, judging that the user account data are completely migrated, and finishing the migration of the user account data;
and S6, if the new data fingerprint and the old data fingerprint of the user account are inconsistent, carrying out error checking and defect filling on the data migrated by the user account, and returning to the step S4.
2. The mass data smooth migration method according to claim 1, wherein step S1 specifically includes:
the method comprises the steps of obtaining historical behavior data of all user accounts, conducting priority division on each user account according to core active operation frequency in the historical behavior data of the user accounts, dividing the user accounts with core active operation in a first preset time interval into high-level user accounts, dividing the user accounts with core active operation in a second preset time interval into middle-level user accounts, and dividing the user accounts without core active operation in a third preset time interval into low-level user accounts, wherein the core active operation comprises login verification, file uploading, file downloading and file sharing, the first preset time interval is earlier than the second preset time interval, and the second preset time interval is earlier than the third preset time interval.
3. The mass data smooth migration method according to claim 2, wherein the first preset time period is 3 months, the second preset time period is 4-6 months, and the third preset time period is more than 6 months.
4. The mass data smooth migration method according to claim 2, wherein step S5 further comprises:
and making a migration mark for the user account with the migrated data in the old database.
5. The mass data smooth migration method according to claim 4, further comprising, after step S5:
s7, detecting whether the user account has operation or not, and if so, judging whether the user account has a migration mark in an old database or not;
s8, if the user account has a migration mark in the old database, responding to the operation of the user account through the new database;
and S9, if the user account does not have the migration mark in the old database, responding to the operation of the user account through the old database.
6. The mass data smooth migration method according to claim 1, wherein step S3 specifically includes:
s31, counting data information of each user account in an old database, wherein the data information comprises an account name, the number of migration tables, the number of data lines and the size of total migration data;
s32, separating each piece of data information of the user account by commas in parentheses, and generating a first data statistical information character string;
s33, calculating the first data statistical information character string by using a hash function SHA256 to obtain an old data fingerprint of the user account;
s34, counting the data information of each user account in the new database, separating each data information of the user account in brackets by commas, generating a second data statistical information character string, and calculating the second data statistical information character string by applying a hash function SHA256 to obtain a new data fingerprint of the user account.
7. A mass data smooth migration system, comprising:
the system comprises a user priority dividing module, a priority calculating module and a priority calculating module, wherein the user priority dividing module is used for acquiring historical behavior data of all user accounts and performing priority division on each user account according to the historical behavior data of each user account, and the priority comprises a high level, a middle level and a low level;
the user data migration module is used for sequentially migrating high-level user account data, medium-level account data and low-level account data in batches according to priority, locking data writing and modifying authority of a user account when migrating the user account data, and opening the data writing and modifying authority of the user account after migrating the user account data;
the data fingerprint calculation module is used for calculating a new data fingerprint and an old data fingerprint of each user account according to a preset data fingerprint generation algorithm after data migration of each user account is completed, wherein the new data fingerprint is the data fingerprint of the user account migrated into a new database, and the old data fingerprint is the data fingerprint of the user account in an old database of migrated data;
the data fingerprint comparison module is used for comparing the new data fingerprint and the old data fingerprint of the user account;
the first result module is used for judging that the user account data are completely migrated if the new data fingerprint and the old data fingerprint of the user account are consistent, and finishing the migration of the user account data;
and the second result module is used for carrying out error checking and defect filling on the data migrated by the user account and returning to the execution data fingerprint comparison module if the new data fingerprint and the old data fingerprint of the user account are inconsistent.
8. The mass data smooth migration system according to claim 7, wherein the user prioritization module is specifically configured to:
the method comprises the steps of obtaining historical behavior data of all user accounts, conducting priority division on each user account according to core active operation frequency in the historical behavior data of the user accounts, dividing the user accounts with core active operation in a first preset time interval into high-level user accounts, dividing the user accounts with core active operation in a second preset time interval into middle-level user accounts, and dividing the user accounts without core active operation in a third preset time interval into low-level user accounts, wherein the core active operation comprises login verification, file uploading, file downloading and file sharing, the first preset time interval is earlier than the second preset time interval, and the second preset time interval is earlier than the third preset time interval.
9. The mass data smooth migration system according to claim 8, wherein the first preset period of time is 3 months, the second preset period of time is 4-6 months, and the third preset period of time is more than 6 months.
10. The mass data smooth migration system according to claim 8, wherein the data fingerprint calculation module is specifically configured to:
counting data information of each user account in an old database, wherein the data information comprises an account name, the number of migration tables, the number of data lines and the size of total migration data;
separating each piece of data information of the user account by commas in parentheses to generate a first data statistical information character string;
calculating the first data statistical information character string by using a hash function SHA256 to obtain an old data fingerprint of the user account;
and counting the data information of each user account in the new database, separating each data information of the user account in brackets by commas, generating a second data statistical information character string, and calculating the second data statistical information character string by applying a hash function SHA256 to obtain a new data fingerprint of the user account.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211461666.8A CN115757345B (en) | 2022-11-17 | 2022-11-17 | Smooth migration method and system for mass data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211461666.8A CN115757345B (en) | 2022-11-17 | 2022-11-17 | Smooth migration method and system for mass data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115757345A true CN115757345A (en) | 2023-03-07 |
| CN115757345B CN115757345B (en) | 2024-06-18 |
Family
ID=85334442
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211461666.8A Active CN115757345B (en) | 2022-11-17 | 2022-11-17 | Smooth migration method and system for mass data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115757345B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040064487A1 (en) * | 2002-10-01 | 2004-04-01 | Nguyen Liem M. | Method and apparatus to migrate a database |
| CN103514164A (en) * | 2012-06-15 | 2014-01-15 | 阿里巴巴集团控股有限公司 | Data migration method and system |
| US20150142660A1 (en) * | 2013-11-15 | 2015-05-21 | The Fusion Network LLC | Centralized financial account migration system |
| CN106855871A (en) * | 2015-12-09 | 2017-06-16 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of Data Migration |
| CN113836112A (en) * | 2021-09-24 | 2021-12-24 | 济南浪潮数据技术有限公司 | Data migration method, system, device and medium |
-
2022
- 2022-11-17 CN CN202211461666.8A patent/CN115757345B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040064487A1 (en) * | 2002-10-01 | 2004-04-01 | Nguyen Liem M. | Method and apparatus to migrate a database |
| CN103514164A (en) * | 2012-06-15 | 2014-01-15 | 阿里巴巴集团控股有限公司 | Data migration method and system |
| US20150142660A1 (en) * | 2013-11-15 | 2015-05-21 | The Fusion Network LLC | Centralized financial account migration system |
| CN106855871A (en) * | 2015-12-09 | 2017-06-16 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of Data Migration |
| CN113836112A (en) * | 2021-09-24 | 2021-12-24 | 济南浪潮数据技术有限公司 | Data migration method, system, device and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115757345B (en) | 2024-06-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109271321B (en) | Method and device for counting contribution code number | |
| US20210049715A1 (en) | Blockchain-based data procesing method, apparatus, and electronic device | |
| CN108021509B (en) | Test case dynamic sequencing method based on program behavior network aggregation | |
| CN110442473A (en) | A kind of non-volatile data storage method, apparatus, electronic equipment and medium | |
| CN109783459A (en) | The method, apparatus and computer readable storage medium of data are extracted from log | |
| CN107016016A (en) | A kind of method and device of data processing | |
| CN107944063A (en) | It is a kind of that method and system are recommended based on the news of topic model and groups of users | |
| CN110851307A (en) | Method, system, terminal and storage medium for testing performance of hard disk in vibration environment | |
| CN110442474B (en) | Data consistency verification method, system and data verification terminal | |
| CN112711633B (en) | Block chain-based BIM drawing multi-person collaborative creation method and system | |
| CN115757345A (en) | Mass data smooth migration method and system | |
| CN104506636B (en) | A kind of method of data synchronization and device | |
| CN111143463B (en) | Construction method and device of bank data warehouse based on topic model | |
| CN117495544A (en) | Sandbox-based wind control evaluation method, sandbox-based wind control evaluation system, sandbox-based wind control evaluation terminal and storage medium | |
| CN103685471B (en) | Method and system for updating software client sides in monopoly mode | |
| CN113190531A (en) | Database migration method, device, equipment and storage medium | |
| CN116610336A (en) | Firmware upgrading method, system, device and readable storage medium | |
| CN110673507A (en) | Data simulation method and device for Internet of things equipment | |
| CN115729990A (en) | Distributed data storage method and device, electronic equipment and storage medium | |
| CN112486957B (en) | Database migration detection method, device, equipment and storage medium | |
| CN111209597B (en) | Data verification method and application system | |
| CN113255287A (en) | OTP register verification method based on RAL | |
| CN109241205A (en) | A method of can be used for high-volume production Methods type analogue data | |
| CN110262856A (en) | A kind of application data acquisition method, device, terminal and storage medium | |
| CN110677494A (en) | Access response method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |