WO2021082104A1

WO2021082104A1 - Method and apparatus for data migration, computer device, and storage medium

Info

Publication number: WO2021082104A1
Application number: PCT/CN2019/119566
Authority: WO
Inventors: 罗国强
Original assignee: 平安科技（深圳）有限公司
Priority date: 2019-10-29
Filing date: 2019-11-20
Publication date: 2021-05-06
Also published as: CN111008188B; CN111008188A

Abstract

A method and apparatus for data migration, a computer device, and a storage medium, the method comprising: acquiring a first time corresponding to first item of data in data to be migrated in a source database and a second time corresponding to a last item of data in the data to be migrated; according to a first preset rule, splitting the time period between the first time and the second time to obtain a plurality of time slices; according to a second preset rule, determining a specified migration quantity corresponding to a specified time slice; according to the successive sequence of the times at which each item of data in the data to be migrated are entered into the source database, sequentially querying from the data to be migrated the same quantity of a plurality of items of specified data as the specified migration quantity; and migrating the specified data to a target database by means of a preset quantity of specified migration servers, and continuing until all of the data to be migrated in the source database is migrated to the target database. Thus, the rapid migration of data to be migrated may be achieved and data migration costs are reduced.

Description

Data migration method, device, computer equipment and storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on October 29, 2019, the application number is 2019110380724, and the invention title is "data migration method, device, computer equipment and storage medium", the entire content of which is incorporated by reference In this application.

Technical field

This application relates to the field of data processing technology, and in particular to a data migration method, device, computer equipment, and storage medium.

Background technique

With the rapid development of modern technology, people’s lives are becoming more and more inseparable from data, and the various storage systems we use daily also rely on data to perform various functions. Database management is more and more widely used, and In the process of database, due to system switching and statistics and filtering of data in different systems, data in the database often needs to be migrated.

At present, the common method of data migration includes the following steps: 1) Set the target database for receiving the data to be migrated in the target server; 2) Read the information of all the objects to be migrated in the source database, and analyze all the objects to be migrated 3) According to the order of the dependency relationship, perform object creation and data loading in the target database.

For the above data migration method, this method needs to read out the information of all the objects to be migrated in the source database before performing object creation and data loading in the target database. In the data migration process, when performing a read operation to read the information of all the objects to be migrated, if the objects to be migrated are massive data, there will be a lot of content to be queried, which may easily lead to time-consuming read operations. . Moreover, as the number of objects to be migrated increases, the speed of reading information will become slower and slower, resulting in subsequent object creation and data loading unable to start for a long time, which seriously affects the efficiency of data migration and the migration experience of users.

technical problem

The main purpose of this application is to provide a data migration method, device, computer equipment, and storage medium. It aims to solve the problem that the existing data migration method needs to read all the information of the object to be migrated in the source database before the target The creation of execution objects and the loading of data in the database seriously affect the efficiency of data migration and the technical problems of the user's migration experience.

Technical solutions

In order to achieve the purpose of the above application, this application proposes a data migration method, which includes the steps:

Obtain the first time corresponding to the first piece of data of the data to be migrated in the source database, and the second time corresponding to the last piece of data of the data to be migrated, where the first time is the time when the first piece of data is entered The time in the source database, and the second time is the time when the last piece of data is entered into the source database;

Split the time period between the first time and the second time according to a first preset rule to obtain multiple corresponding time segments;

Determine the designated migration quantity corresponding to the designated time segment according to the second preset rule, where the designated time segment is any one of the multiple time segments;

According to the sequence of the time when each piece of data in the data to be migrated is entered into the source database, the data to be migrated are sequentially queried out of the data to be migrated in the same quantity as the specified number of designated data to be migrated. ；

The designated data is migrated to the target database through a preset number of designated migration servers, and so on, until all the data to be migrated in the source database is migrated to the target database.

This application also provides a data migration device, including:

The first acquisition module is configured to acquire the first time corresponding to the first piece of data in the source database to be migrated, and the second time corresponding to the last piece of data in the data to be migrated, where the first time is The time when the first piece of data was entered into the source database, and the second time is the time when the last piece of data was entered into the source database;

A splitting module, configured to split the time period between the first time and the second time according to a first preset rule to obtain multiple corresponding time segments;

The determining module is configured to determine the designated migration quantity corresponding to the designated time segment according to the second preset rule, wherein the designated time segment is any one of the multiple time segments;

The query module is used to query the data to be migrated according to the sequence of the time when each piece of data in the data to be migrated is respectively entered into the source database, and query the data to be migrated in turn to find out that the quantity is the same as the specified number of migrations Of multiple specified data;

The migration module is configured to migrate the designated data to the target database through a preset number of designated migration servers, and so on, until all the data to be migrated in the source database is migrated to the target database.

The present application provides a computer device including a memory and a processor. The memory stores computer-readable instructions, and the processor implements the steps of the above method when the computer-readable instructions are executed by the processor.

The present application also provides a computer-readable storage medium on which computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, the steps of the foregoing method are implemented.

Beneficial effect

In the data migration method, device, computer equipment, and storage medium provided in this application, when the data to be migrated in the source database needs to be migrated, multiple time segments are set for the data to be migrated in the temporary database, and each time segment is executed. When migrating data in two time segments, each time segment can be intelligently allocated a specified number of designated data for migration for migration, and then a preset number of designated migration servers can be called to achieve rapid migration of designated data in the data to be migrated , Which effectively reduces the time spent in the data migration process, reduces the cost of data migration, and improves the efficiency of the migration server.

Description of the drawings

FIG. 1 is a schematic flowchart of a data migration method according to an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a data migration device according to an embodiment of the present application;

Fig. 3 is a schematic structural diagram of a computer device according to an embodiment of the present application.

The best mode of the present invention

It should be understood that the specific embodiments described here are only used to explain the present application, and are not used to limit the present application.

Referring to FIG. 1, a data migration method according to an embodiment of the present application includes:

S1: Obtain the first time corresponding to the first piece of data of the data to be migrated in the source database, and the second time corresponding to the last piece of data of the data to be migrated, where the first time is the first piece of data The time entered into the source database, and the second time is the time when the last piece of data was entered into the source database;

S2: Split the time period between the first time and the second time according to the first preset rule to obtain multiple corresponding time segments;

S3: Determine the designated migration quantity corresponding to the designated time segment according to the second preset rule, where the designated time segment is any one of the multiple time segments;

S4: According to the sequence of the time when each piece of data in the data to be migrated is separately entered into the source database, sequentially query from the data to be migrated the same number as the specified number of migrations Specify data;

S5: Migrate the designated data to the target database through a preset number of designated migration servers, and so on, until all the data to be migrated in the source database is migrated to the target database.

As described in the above steps S1 to S5, the execution subject of the embodiment of the present invention is a data migration device, by which the data to be migrated can be quickly migrated, effectively reducing the time spent in the data migration process. Specifically, first, the original data to be migrated from the source database can be synchronized to the temporary database according to the time sequence of the original data to be migrated, so as to store the data to be migrated corresponding to the original data to be migrated through the temporary database. The temporary database has the same structure type as the aforementioned source database. For example, if the source database is a mysql database, the temporary database is also a mysql database. By synchronizing the data to be migrated from the source database to the temporary database for storage, it is helpful to reduce the pressure on the source database, avoid the high concurrency caused by data migration directly through the source database, and effectively reduce the cost of the source database. The impact of normal online business. After completing the synchronization of the original data to be migrated, the first time corresponding to the first piece of data in the source database and the second time corresponding to the last piece of data in the data to be migrated are acquired. Wherein, when the data to be migrated is entered into the source database, the corresponding entry time is recorded in the source database. The first time is the time when the first piece of data is entered into the source database, and the second time is the last The time when a piece of data was entered into the source database. Then, the time period between the first time and the second time is split according to the first preset rule to obtain multiple corresponding time segments. Wherein, the above-mentioned first preset rule may include: averagely split the time period between the first time and the second time according to the preset number of divisions to obtain multiple time segments with the same number of divisions. The above-mentioned number of divisions can be set according to actual conditions, for example, it can be input by the user according to personal needs, or it can be set by the data migration device itself. In addition, the aforementioned time segment only refers to one time interval, and the time segment has no direct relationship with the time when the data to be migrated is entered into the source data. For example, a certain time segment is 30 minutes, such as 09:00-09:30. After the split time segment is obtained, the designated migration quantity corresponding to the designated time segment is determined according to the second preset rule, where the designated time segment is any one of the multiple time segments, The second preset rule may include: acquiring the total number of data to be migrated and the number of time segments, calculating the first quotient between the total number of data and the number of time segments, and calculating the first quotient The value is determined as the number of migrations specified above. Finally, according to the order of the time when each piece of data in the data to be migrated was entered into the source database, the temporary database was queried in turn from the temporary database with the same number of specified data as the specified number of migrations, and the specified data was obtained. After that, the designated data is migrated to the target database through a preset number of designated migration servers, and so on, until all the data to be migrated in the temporary database is transferred to the target database, where the target server is set to be used on the target server. The database that receives the data to be migrated can be a database with a different data structure type from the source database. In this embodiment, when the data to be migrated in the source database needs to be migrated, the original data to be migrated in the source database is first synchronized to a temporary database, and then multiple time segments are set for the data to be migrated in the temporary database. When performing data migration for each time segment, each time segment can be intelligently allocated a designated number of designated data for migration to be migrated, and a preset number of designated migration servers can be called to realize data migration for the designated data in the data to be migrated. Fast migration effectively reduces the time spent in the data migration process, reduces the cost of data migration, and improves the efficiency of the migration server. In another embodiment, in order to avoid data migration directly through the source database, resulting in excessively high concurrency, which will have a greater impact on the normal online business of the source database, the data to be migrated in the source database can be first transferred. Synchronize to a temporary database, where the temporary database has the same structure type as the source database. For example, if the source database is a mysql database, the temporary database is also a mysql database. After the synchronization of the data to be migrated is completed in the temporary database, the data to be migrated stored in the temporary database is migrated to the target database, so as to realize the migration process of the data to be migrated. In this embodiment, by synchronizing the data to be migrated from the source database to the temporary database for storage, it is beneficial to reduce the pressure on the source database, avoid the situation of excessively high concurrency caused by data migration directly through the source database, and effectively reduce the The influence of the online normal business of the source database.

Further, in an embodiment of the present application, the above step S2 includes:

S200: Use the first time as the left endpoint value of the first time segment, and determine whether the first time is within a preset time range, where the first time segment is based on the previous The first time segment divided in the order after arrival;

S201: If the first time is within the preset time range, calculate the first sum of the first time and the first preset time value to obtain the first specified time, and compare the first time with the The time period between the first designated times is marked as the first time segment;

S202: If the first time is not within the preset time range, calculate the second sum of the first time and the second preset time value to obtain the first specified time, and compare the first time with all The time period between the first designated times is marked as the first time segment;

S202: Use the first designated time as the left endpoint value of the second time segment, and determine whether the first designated time is within a preset time range, where the second time segment is the same as the first time segment. The next time segment adjacent to the segment;

S204: If the first specified time is within the preset time range, calculate the third sum of the first specified time and the first preset time value to obtain the second specified time, and add the first specified time The time period between and the second designated time is marked as the second time segment;

S205: If the first specified time is not within the preset time range, calculate the fourth sum of the first specified time and the second preset time value to obtain the second specified time, and compare the first specified time The time period between the time and the second designated time is marked as the second time segment, and so on, until the time period between the first time and the second time is split.

As described in the foregoing steps S200 to S205, this embodiment can implement the division of time segments through the first preset rule. The above step of splitting the time period between the first time and the second time according to the first preset rule to obtain multiple corresponding time segments may specifically include: first taking the first time as the first time segment The left endpoint value, and determine whether the first time is within the preset time range, where the first time segment is the first time segment that is divided in order from front to back among all the time segments. The time range is not specifically limited, for example, it can be set to a range from 9 o'clock to 21 o'clock. If the first time is within the preset time range, calculate the first sum value of the first time and the first preset time value to obtain the first designated time, and calculate the difference between the first time and the first designated time The time period of is marked as the above-mentioned first time segment, where the specific value of the first preset time value is not limited, and can be set according to actual needs, for example, it can be set to 5 minutes. If the first time is not within the preset time range, the second sum of the first time and the second preset time value is calculated to obtain the first designated time, and the first time and the first designated time are combined The time period between is marked as the aforementioned first time segment; wherein, the aforementioned second preset time value is not specifically limited, and can be set according to actual needs, for example, it can be set to 30 minutes. After the first designated time is obtained, the first designated time is then used as the left end value of the second time segment, and it is determined whether the first designated time is within the preset time range, where the second time segment is The next time segment adjacent to the above-mentioned first time segment. If the first specified time is within the preset time range, calculate the third sum of the first specified time and the first preset time value to obtain the second specified time, and combine the first specified time with the second specified time The time period between is marked as the second time segment described above. If the first specified time is not within the preset time range, calculate the fourth sum of the first specified time and the second preset time value to obtain the second specified time, and compare the first specified time with the second specified time. The time period between the designated times is marked as the above-mentioned second time segment, and so on, until the time period between the above-mentioned first time and the above-mentioned second time is split.

Further, in an embodiment of the present application, the above step S3 includes:

S300: When the designated time segment is the first time segment, determine the preset first initial query quantity as the designated migration quantity;

S301: After completing the migration of multiple pieces of first data with the same quantity as the first initial query in the data to be migrated in the source database, calculate the first average of each piece of the first data in the migration process Response time, and determine whether the first average response time is less than a preset first response time threshold;

S302: If yes, calculate the fifth sum of the first initial query quantity and the first preset quantity value to obtain a second initial query quantity, and determine the second initial query quantity to be the same as the second time segment The corresponding designated migration quantity;

S303: After completing the migration of multiple pieces of second data with the same quantity as the second initial query in the data to be migrated in the source database, calculate the second average of each piece of the second data in the migration process Response time, and determine whether the second average response time is less than the first response time threshold;

S303: If yes, calculate the sixth sum of the second initial query quantity and the first preset quantity value to obtain a third initial query quantity, and determine the third initial query quantity to be the same as the third time segment Corresponding designated migration quantity, wherein the third time segment is the next time segment adjacent to the second time segment;

S305: By analogy, in the process of data migration for other time segments, when the designated initial query quantity corresponding to the current time segment is equal to the preset query quantity threshold, after completing the data to be migrated to the source database After the migration of multiple pieces of third data with the same amount of the specified initial query, calculate the specified average response time of each piece of the third data during the migration process, and determine whether the specified average response time is equal to the preset first 2. Response time threshold;

S306: If it is equal to the preset second response time threshold, determine whether the target database is in a full load state;

S307: If it is in a full load state, determine the query amount threshold as the designated migration amount corresponding to all remaining time segments, respectively.

As described in the foregoing steps S300 to S307, in this embodiment, the second preset rule can be used to determine the specified number of migrations. The above step of determining the designated migration quantity corresponding to the designated time segment according to the second preset rule specifically includes: when the designated time segment is the first time segment, determining the preset first initial query quantity as the designated migration quantity, Among them, the first initial query quantity is the result of the stress test of the data migration work under the normal state in the test environment, and can be specifically set to 200. After completing the migration of multiple pieces of first data with the same number of first initial queries in the data to be migrated in the source database, calculate the first average response time of each piece of first data during the migration process, and determine the first average Whether the response time is less than the preset first response time threshold, where the first response time threshold can be set according to actual conditions, for example, it can be set to 200ms. If the first average response time is less than the first response time threshold, the fifth sum of the first initial query quantity and the first preset quantity value is calculated to obtain the second initial query quantity, and the second initial query quantity is determined as the first The designated migration quantity corresponding to the second time segment, wherein the first preset quantity value is not specifically limited, and can be set according to the actual situation, for example, it can be set to 30. After completing the migration of multiple pieces of second data with the same number of second initial queries in the data to be migrated in the source database, calculate the second average response time of each piece of second data during the migration process, and determine the second Whether the average response time is less than the preset first response time threshold. If the second average response time is less than the first response time threshold, the sixth sum of the second initial query quantity and the first preset quantity value is calculated to obtain the third initial query quantity, and the third initial query quantity is determined as the first The specified number of migrations corresponding to the three time segments, where the third time segment is the next time segment adjacent to the second time segment. By analogy, in the process of data migration for other time segments, when the specified initial query number corresponding to the current time segment is equal to the preset query volume threshold, the data to be migrated from the source database is completed with the specified After the migration of multiple pieces of third data with the same initial query amount, calculate the specified average response time of each piece of third data during the migration process, and determine whether the specified average response time is equal to the preset second response time threshold. Wherein, the above-mentioned second response time threshold is greater than the first response time threshold, and can be specifically set to 300 ms according to actual conditions. If the specified average response time is equal to the second response time threshold, it is further judged whether the target database is at full load. Among them, the CPU data and memory data of the target database can be obtained, and then the target database can be analyzed based on the CPU data and memory data The current load situation. If the target database is in a full load state, the query volume threshold is determined as the designated migration number corresponding to all remaining time segments. In this embodiment, when the data migration of each time segment is performed, the designated data of the designated migration quantity can be intelligently allocated for migration according to the second preset rule, which effectively realizes the rapid migration of the data to be migrated. In another embodiment, if after completing the data migration process corresponding to all the remaining time segments, the migration of all the data to be migrated has not been completed, the data migration device will continue to transfer data in the source database. The remaining un-migrated data is re-migrated, so as to realize the complete migration of all the data to be migrated. Specifically, after the data migration process corresponding to all the remaining time segments is completed, the number of data corresponding to the un-migrated data currently in the data to be migrated is obtained; the second quotient of the total amount of data and the query volume threshold is calculated Value; according to the second quotient, set multiple specific time segments with the same number as the second quotient; according to the chronological order of each of the above-mentioned unmigrated data, in each of the above specific time segments, call a certain number of migration servers to Multiple pieces of specific data that are the same as the query volume threshold are migrated to the target server until the migration of all the non-migrated data is completed, where the specific data is any piece of non-migrated data of the non-migrated data. In this embodiment, by performing re-migration processing on the remaining un-migrated data in the source database, the complete migration of all the data to be migrated in the source database is effectively realized, and the integrity of the data migration is ensured.

Further, in an embodiment of the present application, before the foregoing step S5, the method includes:

S500: Obtain the migration server IP input by the user, and configure the migration server IP to implement the invocation of the migration server;

S501: Obtain the number of migration server IPs, and determine whether the number of migration server IPs is greater than a preset configuration number threshold;

S502: If it is greater than the preset configuration number threshold, filter out the first migration server with the same number as the configuration number threshold from the migration server IP, and determine the first migration server as the designated migration server ；

S503: If it is not greater than a preset configuration number threshold, determine all second migration servers corresponding to the migration server IP as the designated migration server.

As described in the above steps S500 to S503, before the step of migrating the designated data to the target database through a preset number of designated migration servers, a process of determining a designated migration server for data migration is also included. Specifically, first obtain the migration server IP (Internet Protocol, Internet Protocol address) input by the user, and configure the migration server IP to implement the invocation of the migration server. Then obtain the number of migration server IPs, and determine whether the number of migration server IPs is greater than a preset configuration number threshold, where the configuration number threshold can be set according to actual conditions, for example, it can be set to 4. If the number of migration server IPs is greater than the preset configuration number threshold, the first migration server with the same number as the configuration number threshold is selected from the migration server IP, and the first migration server is determined as the designated migration server. For example, if the user configures five migration server IPs, four first migration servers with the same number as the configured number threshold will be randomly selected from the six migration server IPs as the designated migration server. If the number of migration server IPs is not greater than the preset configured number threshold, all second migration servers corresponding to the migration server IPs are determined as the designated migration servers. For example, if the user only configures three migration server IPs, three second migration servers corresponding to the three migration server IPs will be directly used as the designated migration server. In this embodiment, when performing data migration for each time segment, it will intelligently allocate a designated number of designated data to be migrated for each time segment for migration, and migrate the designated data through a corresponding number of designated migration servers, thereby effectively migrating the designated data. The rapid migration of the data to be migrated is realized, and the use efficiency of the migration server is improved.

Further, in an embodiment of the present application, the above step S5 includes:

S510: Acquire the number of designated migration servers;

S511: Divide all the designated data into multiple data sets with the same number as the designated migration server;

S512: Bind each designated migration server to a data set one by one, and migrate each data set to the target database one by one through each designated migration server.

As described in the above steps S510 to S512, the step of migrating the designated data to the target database through a preset number of designated migration servers may specifically include: first obtaining the number of designated migration servers, and then according to the number of designated migration servers , Divide all the above specified data into multiple data sets with the same number of the above specified migration servers. For example, if the number of designated migration servers is 4, the designated data can be divided into 4 data sets correspondingly. There is no specific limitation on the amount of data that can be accommodated in each data set. The amount of data that can be accommodated is set to an equal value. If the amount of specified data mentioned above is 200, the amount of data that can be accommodated in each data set is 50. After the data set is divided, each designated migration server is bound to a data set one by one, and each of the above-mentioned data sets is migrated to the target database through each of the above-mentioned designated migration servers. Among them, the binding of the designated migration server and the data set is not specifically limited, and the two can be randomly bound one by one. In this embodiment, multiple pieces of designated data are divided into multiple data sets with the same number of designated migration servers, and then each designated migration server bound to each data set is used to complete the migration of the designated data, which improves the migration. The use efficiency of the server effectively reduces the time spent in the data migration process and reduces the cost of data migration.

In an embodiment of the present application, after the above step S5, the method includes:

S520: After completing the migration of the data to be migrated, obtain a migration record of the data to be migrated;

S521: Determine, according to the migration record, whether there is migration failure data in the data to be migrated;

S522: If yes, perform a preset number of retry repair processing on the failed data;

S523: Determine whether there is any successful migration of the failed data after a preset number of retry repair processing.

S524: If not, transfer the failed data to manual repair processing.

As described in the above steps S520 to S524, during the process of data migration for the data to be migrated, there may be some failed data in the data to be migrated, and the failed data needs to be repaired. Specifically, after the migration of the data to be migrated is completed, the migration record of the data to be migrated is obtained. Among them, when the data to be migrated has not started to be migrated, the data to be migrated in the temporary database will be recorded in the first state mark that has not been migrated; during the migration of the data to be migrated, the data to be migrated in the temporary database will be recorded There is a second state mark in the migration; after the migration of the data to be migrated is completed, the successfully migrated data in the data to be migrated will be recorded with the third state mark of successful migration; and the unsuccessful in the data to be migrated The migrated data will be recorded with a fourth status mark that the migration failed. After the migration record is obtained, it is determined whether there is migration failure data in the data to be migrated according to the migration record. If there is failed data that fails to migrate, the failed data is retryed and repaired a preset number of times, where the failed data can be repaired by calling a pre-stored automatic repair program. In addition, the preset number of times can be based on The actual request is set, for example, it can be set to 3 times. Then it is determined whether there is any successful migration of the above-mentioned failed data after a preset number of retry repair processing. If it does not exist, it means that the failed data has a problem and cannot be repaired through a normal automatic repair procedure, and a reminder message will be sent to transfer the failed data to manual repair processing. After the failed data is successfully repaired manually, the failed data is re-migrated. In this embodiment, after the data to be migrated has failed data that fails to migrate, it will intelligently perform a preset number of retry repair processing on the failed data, so as to effectively repair the failed data that failed to migrate due to network fluctuations or database abnormalities. . In addition, if the above-mentioned multiple retries of the repair processing are not effective, the failed data will be manually repaired to ensure that the failed data can be effectively repaired, which effectively guarantees the data to be migrated during the migration process. Data integrity.

Further, in an embodiment of the present application, after the above step S5, the method includes:

S530: After completing the migration of the data to be migrated, obtain the first header information and the first data size value of the first migration data, where the first migration data is among all the migration data in the target database Any piece of migration data in, and the migration data and the data to be migrated are in a corresponding relationship;

S531: Determine whether the first header information and the first data size value of the first migration data are the same as the prestored second header information and the second data size value of the corresponding first data to be migrated;

S532: If the same, mark the first migration data as the reconciliation completed state, otherwise mark the first migration data as the reconciliation abnormal state, and so on, until the reconciliation process for all the migration data is completed.

As described in the above steps S530 to S532, the current complete data migration process usually includes two parts: migration and reconciliation. After completing the migration process of the data to be migrated, further reconciliation of the data to be migrated is required to Ensure the smooth and complete completion of the migration process for the data to be migrated. Specifically, after completing the migration process of the above-mentioned data to be migrated, first obtain the first header information and the first data size value of the first migration data, where the above-mentioned first migration data is any of all the migration data in the target database. One piece of migration data. In addition, the migration data is in a corresponding relationship with the data to be migrated, that is, the migration data is data obtained after the data to be migrated is migrated. In addition, the above-mentioned first header information includes at least the MD5 value and file format of the first migration data. After the first header information and the first data size value are obtained, it is then judged whether the first header information and the first data size value are the same as the second header information and the second data size value of the pre-stored first data to be migrated. the same. If they are the same, it indicates that the first migration data and the first data to be migrated are the same data, and the first migration data is marked as the reconciliation completed state. If they are not the same, it indicates that the first migration data and the first data to be migrated are not the same data. For example, there may be data missing during the migration of the first data to be migrated. The data is marked as a reconciliation abnormal state. And so on, until the reconciliation of all the above migration data is completed. Further, when the first header information and the first size value of the first migration data are different from the second header information and the second size value of the corresponding first data to be migrated in advance, the After the above-mentioned first migration data is abnormally marked, the first migration data with the abnormally marked first migration data can be subsequently transferred to manual abnormality processing, so that the corresponding maintenance personnel can perform abnormality detection and subsequent abnormality repair processing on the first migration data. .

2, an embodiment of the present application also provides a data migration device, including:

The first acquisition module 1 is configured to acquire the first time corresponding to the first piece of data in the data to be migrated in the source database, and the second time corresponding to the last piece of data in the data to be migrated, where the first time Is the time when the first piece of data is entered into the source database, and the second time is the time when the last piece of data is entered into the source database;

The splitting module 2 is configured to split the time period between the first time and the second time according to a first preset rule to obtain multiple corresponding time segments;

The first determining module 3 is configured to determine a designated migration quantity corresponding to a designated time segment according to a second preset rule, wherein the designated time segment is any one of the multiple time segments;

The query module 4 is configured to query the data to be migrated in the order of the time when each piece of data in the data to be migrated is separately entered into the source database to query the number of data to be migrated that corresponds to the specified number of migrations. The same multiple specified data;

The migration module 5 is configured to migrate the designated data to the target database through a preset number of designated migration servers, and so on, until all the data to be migrated in the source database is migrated to the target database.

Referring to FIG. 3, an embodiment of the present application also provides a computer device, and the computer device may be a server. The computer equipment includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor designed for the computer equipment is used to provide calculation and control capabilities. The memory of the computer device includes a volatile or non-volatile storage medium and internal memory. The volatile or non-volatile storage medium stores an operating system, computer readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the volatile or non-volatile storage medium. The database of the computer equipment is used to store data such as time segments and designated migration quantities. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer-readable instructions are executed by the processor to implement the data migration method shown in any of the above exemplary embodiments. Those skilled in the art can understand that the structure shown in FIG. 3 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the devices and computer equipment to which the solution of the present application is applied.

An embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium may be volatile or non-volatile, and computer-readable instructions are stored thereon. When executed by one or more processors, the steps in the foregoing data migration method embodiment are realized when executed by one or more processors.

The above are only the preferred embodiments of this application, and do not limit the scope of this application. Any equivalent structure or equivalent process transformation made using the content of the specification and drawings of this application, or directly or indirectly applied to other related The technical field is equally included in the scope of patent protection of this application.

Claims

A data migration method is characterized in that it includes:

Obtain the first time corresponding to the first piece of data of the data to be migrated in the source database, and the second time corresponding to the last piece of data of the data to be migrated, where the first time is the time when the first piece of data is entered The time in the source database, and the second time is the time when the last piece of data is entered into the source database;

Split the time period between the first time and the second time according to a first preset rule to obtain multiple corresponding time segments;

Determine the designated migration quantity corresponding to the designated time segment according to the second preset rule, where the designated time segment is any one of the multiple time segments;

According to the sequence of the time when each piece of data in the data to be migrated is entered into the source database, the data to be migrated are sequentially queried out of the data to be migrated in the same quantity as the specified number of designated data to be migrated. ；

The designated data is migrated to the target database through a preset number of designated migration servers, and so on, until all the data to be migrated in the source database is migrated to the target database.
The data migration method according to claim 1, wherein the time period between the first time and the second time is split according to a first preset rule to obtain corresponding multiple times The steps of the fragment include:

The first time is taken as the left endpoint value of the first time segment, and it is judged whether the first time is within a preset time range, wherein the first time segment is the order from front to back among all the time segments. The first time segment divided in the order of

If the first time is within the preset time range, calculate the first sum of the first time and the first preset time value to obtain the first designated time, and compare the first time with the first The time period between designated times is marked as the first time segment;

If the first time is not within the preset time range, calculate the second sum of the first time and the second preset time value to obtain the first designated time, and compare the first time with the first time A time period between designated times is marked as the first time segment;

Use the first designated time as the left endpoint value of the second time segment, and determine whether the first designated time is within a preset time range, wherein the second time segment is the same as the first time segment Next time segment of neighbor;

If the first specified time is within the preset time range, calculate the third sum of the first specified time and the first preset time value to obtain the second specified time, and combine the first specified time with all The time period between the second designated times is marked as the second time segment;

If the first specified time is not within the preset time range, calculate the fourth sum of the first specified time and the second preset time value to obtain the second specified time, and combine the first specified time with The time period between the second designated time is marked as the second time segment, and so on, until the time period between the first time and the second time is split.
The data migration method according to claim 2, wherein the step of determining a designated migration quantity corresponding to a designated time segment according to a second preset rule comprises:

When the designated time segment is the first time segment, determining the preset first initial query quantity as the designated migration quantity;

After completing the migration of multiple pieces of first data with the same quantity as the first initial query in the data to be migrated in the source database, calculate the first average response time of each piece of the first data in the migration process , And determine whether the first average response time is less than a preset first response time threshold;

If yes, calculate the fifth sum of the first initial query quantity and the first preset quantity value to obtain the second initial query quantity, and determine the second initial query quantity as corresponding to the second time segment Specify the number of migrations;

After completing the migration of multiple pieces of second data with the same quantity as the second initial query in the data to be migrated in the source database, calculate the second average response time of each piece of the second data in the migration process , And determine whether the second average response time is less than the first response time threshold;

If yes, calculate the sixth sum of the second initial query quantity and the first preset quantity value to obtain the third initial query quantity, and determine the third initial query quantity as corresponding to the third time segment Specify the number of migrations, where the third time segment is the next time segment adjacent to the second time segment;

By analogy, in the process of data migration for other time segments, when the designated initial query quantity corresponding to the current time segment is equal to the preset query quantity threshold, the data to be migrated from the source database is completed with all the data to be migrated. After the migration of the multiple pieces of third data with the same specified initial query amount, calculate the specified average response time of each piece of the third data during the migration process, and determine whether the specified average response time is equal to the preset second response Time threshold

If it is equal to the preset second response time threshold, determine whether the target database is in a full load state;

If it is in a full load state, the query amount threshold is determined as the designated migration amount corresponding to all the remaining time segments, respectively.
The data migration method according to claim 1, wherein before the step of migrating the designated data to the target database through a preset number of designated migration servers, the method comprises:

Acquire the migration server IP input by the user, and configure the migration server IP to implement the invocation of the migration server;

Acquiring the number of migration server IPs, and determining whether the number of migration server IPs is greater than a preset configuration number threshold;

If it is greater than the preset configuration number threshold, filter out the first migration server with the same number as the configuration number threshold from the migration server IP, and determine the first migration server as the designated migration server;

If it is not greater than the preset configuration number threshold, all second migration servers corresponding to the migration server IP are determined as the designated migration server.
The data migration method according to claim 1, wherein the step of migrating the designated data to the target database through a preset number of designated migration servers comprises:

Obtaining the number of the designated migration server;

Dividing all the designated data into multiple data sets with the same number as the designated migration server;

Each designated migration server is bound to a data set one by one, and each data set is migrated to the target database one by one through each designated migration server.
The data migration method according to claim 1, wherein the designated data is migrated to the target database through a preset number of designated migration servers, and so on, until all the data to be migrated in the source database are migrated. After the step of data migration to the target database, it includes:

After completing the migration of the data to be migrated, obtain the migration record of the data to be migrated;

According to the migration record, determine whether there is migration failure data in the data to be migrated;

If yes, perform a preset number of retry repair processing on the failed data;

Judging whether there is any successful migration of the failed data after a preset number of retry repair processing;

If not, transfer the failed data to manual repair processing.
The data migration method according to claim 1, wherein the designated data is migrated to the target database through a preset number of designated migration servers, and so on, until all the data to be migrated in the source database are migrated. After the step of data migration to the target database, it includes:

After completing the migration of the data to be migrated, obtain the first header information and the first data size value of the first migration data, where the first migration data is any of all the migration data in the target database One piece of migration data, and the migration data and the data to be migrated are in a corresponding relationship;

Judging whether the first header information and the first data size value of the first migration data are the same as the second header information and the second data size value of the pre-stored corresponding first data to be migrated;

If they are the same, mark the first migration data as the reconciliation completed state, otherwise mark the first migration data as the reconciliation abnormal state, and so on, until the reconciliation process for all the data to be migrated is completed.
A data migration device is characterized in that it comprises:

The first acquisition module is configured to acquire the first time corresponding to the first piece of data in the source database to be migrated, and the second time corresponding to the last piece of data in the data to be migrated, where the first time is The time when the first piece of data was entered into the source database, and the second time is the time when the last piece of data was entered into the source database;

A splitting module, configured to split the time period between the first time and the second time according to a first preset rule to obtain multiple corresponding time segments;

The determining module is configured to determine the designated migration quantity corresponding to the designated time segment according to the second preset rule, wherein the designated time segment is any one of the multiple time segments;

The query module is used to query the data to be migrated according to the sequence of the time when each piece of data in the data to be migrated is respectively entered into the source database, and query the data to be migrated in turn to find out that the quantity is the same as the specified number of migrations Of multiple specified data;

The migration module is configured to migrate the designated data to the target database through a preset number of designated migration servers, and so on, until all the data to be migrated in the source database is migrated to the target database.
The data migration device according to claim 8, wherein the splitting module comprises:

The first determining unit is configured to use the first time as the left endpoint value of the first time segment, and determine whether the first time is within a preset time range, wherein the first time segment is all the The first time segment in the time segment divided from front to back;

The first calculation unit is configured to calculate the first sum of the first time and the first preset time value if the first time is within the preset time range, to obtain the first specified time, and to compare the first time The time period between a time and the first designated time is marked as the first time segment;

The second calculation unit is configured to calculate the second sum of the first time and the second preset time value if the first time is not within the preset time range to obtain the first specified time, and compare the The time period between the first time and the first designated time is marked as the first time segment;

The second determining unit is configured to use the first designated time as the left endpoint value of the second time segment, and determine whether the first designated time is within a preset time range, wherein the second time segment is and The next time segment adjacent to the first time segment;

The third calculation unit is configured to calculate the third sum of the first specified time and the first preset time value if the first specified time is within the preset time range, to obtain the second specified time, and compare the The time period between the first designated time and the second designated time is marked as the second time segment;

The fourth calculation unit is configured to calculate the fourth sum of the first specified time and the second preset time value if the first specified time is not within the preset time range to obtain the second specified time, and The time period between the first designated time and the second designated time is marked as the second time segment, and so on, until the time period between the first time and the second time is completed Split.
The data migration device according to claim 9, wherein the first determining module comprises:

A first determining unit, configured to determine a preset first initial query quantity as the designated migration quantity when the designated time segment is the first time segment;

The third judgment unit is used to calculate the migration process of each piece of the first data after completing the migration of the multiple pieces of first data with the same quantity as the first initial query in the data to be migrated in the source database And determine whether the first average response time is less than a preset first response time threshold;

The second determining unit is configured to, if yes, calculate the fifth sum of the first initial query quantity and the first preset quantity value to obtain a second initial query quantity, and determine the second initial query quantity to be The designated migration quantity corresponding to the second time segment;

The fourth judgment unit is used to calculate the migration process of each piece of the second data after completing the migration of the multiple pieces of second data with the same quantity as the second initial query in the data to be migrated in the source database And determine whether the second average response time is less than the first response time threshold;

The third determining unit is configured to, if yes, calculate the sixth sum of the second initial query quantity and the first preset quantity value to obtain a third initial query quantity, and determine the third initial query quantity as A specified number of migrations corresponding to a third time segment, wherein the third time segment is the next time segment adjacent to the second time segment;

The fifth judging unit is used for analogy. In the process of data migration for other time segments, when the specified initial query quantity corresponding to the current time segment is equal to the preset query volume threshold, the source database is completed After the migration of multiple pieces of third data with the same amount of the specified initial query in the data to be migrated, calculate the specified average response time of each piece of the third data during the migration process, and determine whether the specified average response time Equal to the preset second response time threshold;

The sixth determining unit is configured to determine whether the target database is in a full load state if it is equal to a preset second response time threshold;

The fourth determining unit is configured to determine the query volume threshold as the designated migration number corresponding to all remaining time segments if it is in a full load state.
The data migration device according to claim 8, wherein the device comprises:

The second obtaining module is configured to obtain the migration server IP input by the user, and configure the migration server IP to implement the call to the migration server;

The first judgment module is configured to obtain the number of migration server IPs, and determine whether the number of migration server IPs is greater than a preset configuration number threshold;

The screening module is configured to, if it is greater than the preset configuration number threshold, filter out the first migration server with the same number as the configuration number threshold from the migration server IP, and determine the first migration server as the Specify the migration server;

The second determining module is configured to determine all second migration servers corresponding to the migration server IP as the designated migration server if it is not greater than a preset configuration number threshold.
The data migration device according to claim 8, wherein the migration module comprises:

An obtaining unit for obtaining the number of the designated migration server;

A dividing unit, configured to divide all the designated data into multiple data sets with the same quantity as the designated migration server;

The binding unit is used to bind each designated migration server to a data set one by one, and migrate each of the data sets to the target database one by one through each designated migration server.
The data migration device according to claim 8, wherein the device comprises:

The third acquiring module is configured to acquire the migration record of the data to be migrated after completing the migration of the data to be migrated;

The second judgment module is configured to judge whether there is migration failure data in the data to be migrated according to the migration record;

The first processing module is configured to, if yes, perform a preset number of retry repair processing on the failed data;

The third judgment module is used for judging whether there is any successful migration of the failed data after a preset number of retry repair processing;

The second processing module is configured to, if not, transfer the failed data to manual repair processing.
The data migration device according to claim 8, wherein the device comprises:

The fourth acquiring module is configured to acquire the first header information and the first data size value of the first migration data after the migration of the data to be migrated is completed, wherein the first migration data is in the target database Any piece of migration data among all migration data in, and the migration data and the data to be migrated are in a corresponding relationship;

The fourth judging module is used to judge whether the first header information and the first data size value of the first migration data are the same as the second header information and the second data size value of the pre-stored corresponding first data to be migrated ；

The third processing module is configured to mark the first migration data as the reconciliation completed state if they are the same, otherwise mark the first migration data as the reconciliation abnormal state, and so on, until the completion of all the migration data The reconciliation process.
A computer device includes a memory and a processor, the memory stores computer-readable instructions, and is characterized in that when the processor executes the computer-readable instructions, a data migration method is implemented, and the data migration method includes:

Obtain the first time corresponding to the first piece of data of the data to be migrated in the source database, and the second time corresponding to the last piece of data of the data to be migrated, where the first time is the time when the first piece of data is entered The time in the source database, and the second time is the time when the last piece of data is entered into the source database;

Split the time period between the first time and the second time according to a first preset rule to obtain multiple corresponding time segments;

Determine the designated migration quantity corresponding to the designated time segment according to the second preset rule, where the designated time segment is any one of the multiple time segments;

According to the sequence of the time when each piece of data in the data to be migrated is entered into the source database, the data to be migrated are sequentially queried out of the data to be migrated in the same quantity as the specified number of designated data to be migrated. ；

The designated data is migrated to the target database through a preset number of designated migration servers, and so on, until all the data to be migrated in the source database is migrated to the target database.
The computer device according to claim 15, wherein before the step of migrating the designated data to the target database through a preset number of designated migration servers, the method comprises:

Acquire the migration server IP input by the user, and configure the migration server IP to implement the invocation of the migration server;

Acquiring the number of migration server IPs, and determining whether the number of migration server IPs is greater than a preset configuration number threshold;

If it is greater than the preset configuration number threshold, filter out the first migration server with the same number as the configuration number threshold from the migration server IP, and determine the first migration server as the designated migration server;

If it is not greater than the preset configuration number threshold, all second migration servers corresponding to the migration server IP are determined as the designated migration server.
15. The computer device according to claim 15, wherein the step of migrating the designated data to the target database through a preset number of designated migration servers comprises:

Obtaining the number of the designated migration server;

Dividing all the designated data into multiple data sets with the same number as the designated migration server;

Each designated migration server is bound to a data set one by one, and each data set is migrated to the target database one by one through each designated migration server.
A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions are executed by a processor to implement a data migration method, and the data migration method includes:

Obtain the first time corresponding to the first piece of data of the data to be migrated in the source database, and the second time corresponding to the last piece of data of the data to be migrated, where the first time is the time when the first piece of data is entered The time in the source database, and the second time is the time when the last piece of data is entered into the source database;

Split the time period between the first time and the second time according to a first preset rule to obtain multiple corresponding time segments;

Determine the designated migration quantity corresponding to the designated time segment according to the second preset rule, where the designated time segment is any one of the multiple time segments;

According to the sequence of the time when each piece of data in the data to be migrated is entered into the source database, the data to be migrated are sequentially queried out of the data to be migrated in the same quantity as the specified number of designated data to be migrated. ；

The designated data is migrated to the target database through a preset number of designated migration servers, and so on, until all the data to be migrated in the source database is migrated to the target database.
18. The computer-readable storage medium according to claim 18, wherein before the step of migrating the designated data to the target database through a preset number of designated migration servers, the method comprises:

Acquire the migration server IP input by the user, and configure the migration server IP to implement the invocation of the migration server;

Acquiring the number of migration server IPs, and determining whether the number of migration server IPs is greater than a preset configuration number threshold;

If it is greater than the preset configuration number threshold, filter out the first migration server with the same number as the configuration number threshold from the migration server IP, and determine the first migration server as the designated migration server;

If it is not greater than the preset configuration number threshold, all second migration servers corresponding to the migration server IP are determined as the designated migration server.
18. The computer-readable storage medium according to claim 18, wherein the step of migrating the designated data to the target database through a preset number of designated migration servers comprises:

Obtaining the number of the designated migration server;

Dividing all the designated data into multiple data sets with the same number as the designated migration server;

Each designated migration server is bound to a data set one by one, and each data set is migrated to the target database one by one through each designated migration server.