WO2021017269A1

WO2021017269A1 - Data migration method and apparatus, computer device, and storage medium

Info

Publication number: WO2021017269A1
Application number: PCT/CN2019/116706
Authority: WO
Inventors: 包晓华
Original assignee: 平安科技（深圳）有限公司
Priority date: 2019-07-30
Filing date: 2019-11-08
Publication date: 2021-02-04
Also published as: CN110580246A; CN110580246B

Abstract

A data migration method and apparatus, a computer device, and a storage medium. The data migration method comprises: acquiring a service attribute of a source database, wherein the source database stores data to be migrated and the source database comprises a first index table (S1); dividing, according to a partition of the first index table and the service attribute, the data in the source database into a specified number of data slices by using a preconfigured division method (S2); acquiring a correspondence between each data slice and a data structure in a target database (S3); and migrating, according to the correspondence, each data slice from the source database to the target database by using a preconfigured migration method (S4). By using the data slice migration method, data is migrated between databases of different types, for example, from a relational database to a non-relational database. A correspondence between data stored in two databases is established according to a mapping relationship between data structures.

Description

Method, device, computer equipment and storage medium for migrating data

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on July 30, 2019, the application number is 201910696304.9, and the invention title is "methods, devices, computer equipment and storage media for migration data", the entire contents of which are incorporated by reference Incorporated in this application.

Technical field

This application relates to the field of computers, in particular to methods, devices, computer equipment and storage media for migrating data.

Background technique

Due to the rapid increase in the amount of data called in the system, the original relational database has been difficult to meet business needs in terms of read and write efficiency and storage software and hardware costs. Therefore, a new type of non-relational database is selected, which requires data on the two databases. Switch. However, it is necessary to ensure that there is no loss of valid data and no perception by platform users in production. In the industry, there is more data migration between databases of the same type, and there is no data migration program between databases with large data structure differences, such as from oracle database to cassandra database. The data migration from oracle to cassandra needs to be completed continuously at one time, which leads to the migration of old and new systems with a large number of network IOs for a long period of time, which will interfere with the normal business of the system. If only a small amount of data is used for shard migration, that is, it does not carry the business attributes of the data, but simply performs the same amount of data migration, it does not consider the different importance of data to the business, nor does it take into account the difference in the format of the new and old databases , Will cause a lot of new database data to be randomly written, and old database data to be randomly read, which has a great impact on the performance of migration data. It is necessary to design a safe and stable data source switching scheme to meet the one-time continuous data migration, which is transactional in operation and continuous in time, and does not interfere with the normal business of the system, and needs to avoid transactional nature in the process of data migration The operation caused a global failure because of a minor failure in the middle.

technical problem

The main purpose of this application is to provide a method of migration data, aiming to solve the existing technical problem that the stable migration of data from the oracle database to the cassandra database cannot be realized.

Technical solutions

This application proposes a method of migration data, including:

Acquiring business attributes of a source database, where the source database is a database storing data to be migrated, and the source database includes a first index table;

According to the partition of the first index table and the business attribute of the source database, dividing the data of the source database into a specified number of fragmented data according to a preset dividing manner;

Acquiring the corresponding relationship between each of the fragmented data and the data structure in the target database, wherein the target database is a database storing the migration data after migration;

According to the corresponding relationship, each piece of data is migrated from the source database to the target database in a preset migration manner.

This application also provides a device for migration data, including:

The first obtaining module is configured to obtain the business attributes of a source database, where the source database is a database storing data to be migrated, and the source database includes a first index table;

A dividing module, configured to divide the data of the source database into a specified number of fragmented data according to a preset dividing manner according to the partition of the first index table and the business attributes of the source database;

The second acquisition module is configured to acquire the corresponding relationship between each of the fragmented data and the data structure in the target database, wherein the target database is a database storing the migration data after migration;

The migration module is configured to migrate each piece of data from the source database to the target database according to a preset migration mode according to the corresponding relationship.

The present application also provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when the computer program is executed.

The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are realized.

Beneficial effect

This application implements data transfer between databases of different database types through the method of sharding data transfer, such as from a relational database to a non-relational database, and realizes the correspondence between the data stored in the two databases according to the mapping relationship of the data structure . The data division standard of the first index table is selected based on the business attribute priority sorting. For example, the computer room corresponding to the source database has the highest priority, and the computer room corresponding to the source database corresponds to three partitions, and the computer room corresponds to the three partition pairs. The source database is fragmented, and the data of the source database is divided into three fragmented data. By collecting a specified number of data sets from the source database, to evaluate the division ability coefficient of each business attribute, so as to realize the optimization of fragmented data. The index item attribute in the J=N/K attachment is preferred; the value of J is the same, and further analysis of the division ability through Manhattan distance needs to be considered.

Description of the drawings

FIG. 1 is a schematic flowchart of a migration data method according to an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a migration data device according to an embodiment of the present application;

Fig. 3 is a schematic diagram of the internal structure of a computer device according to an embodiment of the present application.

The best mode of the invention

1, a data migration method according to an embodiment of the present application includes:

S1: Obtain business attributes of a source database, where the source database is a database storing data to be migrated, and the source database includes a first index table.

The business attributes of this embodiment include, but are not limited to: the computer room, network environment, link role of the service corresponding to the source database, professional company to which the service belongs, system to which the service belongs, service registration code, service domain name, etc. The source database is a database storing data to be migrated, such as an oracle relational database. The index table of the database includes index item attributes, information catalogs and address links, which is convenient for querying data in the database. The data is obtained by first traversing the index table, and then according to the address link in the index table.

S2: According to the partition of the first index table and the business attribute of the source database, the data of the source database is divided into a specified number of fragmented data according to a preset division manner.

In this embodiment, the data in the source database is divided into pieces according to the partitions and business attributes of the first index table, and each piece of data is marked by the business attribute, so as to manage and distinguish each piece of data. The above-mentioned preset division method includes forming each sub-index by partitioning the first index table, and realizing the data in the source database to be divided into pieces by sub-index to form the piece data.

S3: Obtain the corresponding relationship between each of the fragmented data and the data structure in the target database, where the target database is a database storing the migration data after migration.

The target database in this embodiment is a database storing the migration data after migration, such as a non-relational database cassandra, so as to meet the requirements of data migration from an oracle database to a cassandra database. This embodiment divides the data in the source database into fragmented data, and realizes the data transfer from the source database to the target database by separately transferring the fragmented data, so as to avoid continuous data migration. It hinders the normal business in the system, and avoids the failure of data transfer in a local time period caused by the interference of accidental factors, which causes the entire data transfer transaction to fail. The above-mentioned corresponding relationship is formed according to a preset mapping relationship, and includes an association relationship between each piece of data of the source database and the storage structure position in the target database.

S4: According to the corresponding relationship, migrate each piece of data from the source database to the target database in a preset migration manner.

This embodiment implements data transfer between databases of different database types by sequentially transferring fragmented data, such as from a relational database to a non-relational database, and realizes storage in the two databases according to the mapping relationship of the data structure One-to-one correspondence of data.

Further, the step S2 of dividing the data of the source database into a specified number of pieces of data according to a preset dividing manner according to the partition of the first index table and the business attributes of the source database includes:

S21: Acquire priority rankings corresponding to the business attributes of all the source databases, where the priority ranking is a ranking of priority levels from high to low.

S22: Select each partition corresponding to the first business attribute before the designated serial number from the priority ranking as the data division standard of the first index table, wherein the first business attribute is included in all the source databases In the business attributes of, the first business attribute includes index item attributes.

S23: According to the data division standard of the first index table, divide the first index table into sub-indexes corresponding to each partition.

S24: Determine whether the data volume of the source database corresponding to each sub-index is within a preset single transmission volume.

S25: If yes, divide the data of the source database into a first specified number of first fragmented data according to each of the sub-indexes, where the first specified number is each partition corresponding to the first business attribute quantity.

In this embodiment, the data division criteria are selected based on the priority ranking of business attributes. For example, the computer room where the service corresponding to the source database has the highest priority, and the computer room where the service corresponding to the source database is located corresponds to three partitions, and the computer room where the service is located corresponds to the three partition pairs The source database is fragmented, and the data of the source database is divided into three fragmented data according to the three partitions corresponding to the service room. The above-mentioned first business attribute can include multiple at the same time, such as business attribute A and business attribute B. Business attribute A has three partitions A1, A2, and A3, and business attribute B has two partitions B1 and B2, so the corresponding first index The number of partitions of the table's data division standard is 6, and they are sorted by priority as A1B1, A1B2, A2B1, A2B2, A3B1, and A3B2. The above business attributes include the index item attributes and other attributes in the first index table, and the index item attributes are preferred for partitioning. Because the principle of index establishment is to divide the data in a balanced manner, the index item attributes are used to fragment the migration data. The obtained fragmented data is more balanced. In this embodiment, the index table is first divided into sub-indexes, and then the corresponding data is led by the sub-indexes to become fragmented data.

Further, after the step S24 of determining whether the data volume of the source database corresponding to each of the sub-indexes is within a preset single transmission volume, the method includes:

S241: If the data volume corresponding to each sub-index is not within the preset single transmission volume, add the partition corresponding to the second service attribute, where the second service attribute is the service attribute of all the source databases The second business attribute includes at least one attribute other than the attribute of the index item.

S242: According to each sub-index and the partition corresponding to the second business attribute, divide the data of the source database into a second specified number of second fragmented data, so that the data of each second fragmented data The amount is within a preset single transmission amount, wherein the second specified number is a product of the number of partitions of the first service attribute and the number of partitions of the second service attribute.

In this embodiment, when the data is fragmented by index item attributes, the data volume of the fragmented data is not within the single transmission volume, and the effect of a single fast transfer is not achieved. It can be added by adding other than the index item attributes Attributes, together with the attributes of index items, complete the slicing of the data in the source database, so that the data volume of the sharded data is within the preset single transmission volume, realizing rapid data transfer without affecting the normal operation of the system in the business .

Further, the step S21 of obtaining the respective priority rankings corresponding to the business attributes of all the source databases includes:

S211: According to a preset collection rule, collect a first number of data sets from the source database.

S212: Obtain the data volume of the single migration data of the service system.

S213: Divide the first amount by the data amount of the single migration data to obtain a division ability coefficient.

S214: Calculate the closeness of the aggregation result set corresponding to the business attributes of each source database to the division capability coefficient, where the aggregation result set is the aggregation classification result of the data set, and the number of aggregation result sets is equal to The number of partitions of the business attributes of each source database;

S215: Determine the priority sorting according to each of the proximity degrees, wherein the higher the proximity degree, the higher the priority corresponding to the aggregation result set.

In this embodiment, a specified number of data sets are collected from the source database to evaluate the division capability coefficient of each business attribute, so as to realize the optimization of the fragmented data. The foregoing preset collection rules include, for example, collecting a data set every specified time period, so that the collected data set is more analytically representative. For example, a total of 100 data sets are collected, 100 data sets are used as samples, and the number of aggregated result sets is equal to the number of data types divided by shards, such as the above 100 The data set distribution belongs to three professional companies, such as technology, property insurance, and life insurance. The number of aggregated result sets is three. Among the above-mentioned three professional companies, the corresponding quantity sets of science and technology, property insurance, and life insurance are: 30, 10, and 60. The above division ability coefficient is expressed as an evenly divided result set. The number of data sets in the evenly divided result set is equal to the total number of collected data sets divided by the number of aggregated result sets, which is equal to 100 divided by 3 equals 33.3333, that is, the evenly divided result set data set The number is 33.3333. First look at the total amount of the data set. For example, the total amount is N and the number of single migrations that the system can withstand is K. Then the number of aggregated result sets is preferably in the index item attribute of the J=N/K attachment, and the number of aggregated result sets The index item attribute in the J=N/K attachment has the ability to divide the fragmented data into a more balanced and reasonable division, and try to meet the needs of a single migration of data.

Further, the division ability coefficient is expressed as an evenly divided result set, and the evenly divided result set includes a data set obtained by dividing the total number of data sets collected according to a preset collection rule interval by the number of aggregate result sets, and said The step S215 of determining the priority ranking by the proximity degree includes:

S2151: Determine whether there is a third service attribute and a fourth service attribute that have the same degree of proximity to the division capability coefficient, where the third service attribute and the fourth service attribute are included in all the service attributes of the source database.

S2152: If it exists, obtain the first Manhattan distance in which the third business attribute corresponds to the number of data sets in the evenly divided result set, and the fourth business attribute corresponds to the second Manhattan distance of the number of data sets in the evenly divided result set. distance.

S2153: Determine whether the first Manhattan distance is greater than the second Manhattan distance.

S2154: If yes, arrange the priority order of the fourth service attribute corresponding to the second Manhattan distance before the third service attribute corresponding to the first Manhattan distance.

In this embodiment, when the J values corresponding to multiple service attributes are the same, it is necessary to consider further analysis of the division capability through Manhattan distance. For the same number of aggregated result sets, the smaller the Manhattan distance, the better the division ability. For example, the 100 data sets in the above example belong to 3 professional companies according to the index column index1, namely technology, property insurance, and life insurance; and the corresponding quantity sets of science and technology, property insurance, and life insurance in the 3 professional companies are: 30, 10 and 60. The Manhattan distance corresponding to the index column index1 is: Manhattan distance is expressed as D, then D=|30-33.3|+|10-33.3|+|60-33.3|=53.3, the above 33.3 is the number of data sets in the result set, Manhattan distance is an absolute distance.

Further, the data structure in the target database is a multi-level data nesting structure, and the step S3 of obtaining the corresponding relationship between each of the fragmented data and the data structure in the target database includes:

S31: Obtain the designated service attributes in the priority ranking that are invoked when the source database is divided into fragments, where the designated service attributes are included in all business attributes of the source database.

S32: Establish a one-to-one mapping relationship between the priority order corresponding to each of the designated business attributes and the multi-layer data nesting structure in the target database, wherein the designated business attribute with the highest priority corresponds to the multi-layer data nesting structure. The outermost layer of the sleeve structure.

The target database in this embodiment includes non-relational databases, such as the cassandra database. The data structure of the cassandra database is the primary key construction structure. In this embodiment, the data fragments determined in the data to be migrated are used as the primary key reference of the cassandra database. The primary key is the structure of the data organization. The data in cassandra is stored nested in the order of the primary key. For example, the primary key is: K1, K2, K3..., which can be understood as K1 data nested K2 data, K2 data nested K3 data. According to the priority order of business attributes, this embodiment is mapped to the data primary key sequence in cassandra. For example, the shard data corresponding to the business attributes with high priority is mapped to the data of the outermost nesting layer in the data in cassandra. In order to achieve a clear and reasonable correspondence between data migration between different databases, the file data is written in order, which improves the efficiency of data writing and facilitates the calling and management of data.

Further, the step S4 of migrating each piece of data from the source database to the target database according to a preset migration mode according to the corresponding relationship includes:

S41: Determine whether the service flow corresponding to the current moment is within a preset threshold.

S42: If yes, start a preset migration thread, and search for data to be migrated from the source database.

S43: Store the data to be migrated in the cache server, and convert the data format.

S44: Run the preset migration thread according to the preset thread mode, and sequentially inject the data to be migrated into the target database according to the fragmented data mode.

This embodiment implements data migration in batches and orderly by slicing the data. Each batch can be performed independently without mutual dependence, and the migration volume of each batch matches the single transmission volume supported by the system. , If a single migration fails, you only need to repeat the current failed single transmission again without affecting the overall data migration effect. And through the migration in batches, the fragmented time of the system running other businesses can be used to complete the data migration, and the efficiency of the system in processing transactions is improved. This embodiment judges whether the current business is at a low period or a peak period by identifying the operating load status of the system. For example, the business flow is within a preset threshold, indicating that it is in the low business period, and the data migration thread is started to perform data migration; the business flow is not at the preset threshold Inside, it means that in the peak period of business, the data migration thread is suspended to terminate the data migration. It not only supports the control of migration tasks according to business attributes, but also ensures the complete migration of business data by category, and avoids business peaks. After the fragmented data is formed in this embodiment, threads can be started to start data migration. During the migration process, the data is queried and cut out from the oracle database according to the fragmentation information, and the cut data results are stored in the cache server, and then formatted, and then injected into the cassandra database; in this way, one fragment of data is divided into one. The slice data is processed in stages through the same thread. Different fragmented data can also be allocated to different threads for parallel processing to improve the efficiency of fragment migration, but the number of threads running in parallel needs to be reasonably controlled to avoid overloading the databases at both ends.

Referring to FIG. 2, a data migration device according to an embodiment of the present application includes:

The first obtaining module 1 is configured to obtain the business attributes of a source database, where the source database is a database storing data to be migrated, and the source database includes a first index table.

The dividing module 2 is configured to divide the data of the source database into a specified number of pieces of data according to a preset dividing manner according to the partition of the first index table and the business attributes of the source database.

The second acquiring module 3 is configured to acquire the corresponding relationship between each of the fragmented data and the data structure in the target database, wherein the target database is a database storing the migration data after migration.

The migration module 4 is configured to migrate each piece of data from the source database to the target database according to a preset migration mode according to the corresponding relationship.

Further, the dividing module 2 includes:

The first obtaining sub-module is configured to obtain the priority rankings corresponding to the business attributes of all the source databases, wherein the priority ranking is a ranking of priority levels from high to low.

The selection sub-module is used to select each partition corresponding to the first business attribute before the designated sequence number from the priority ranking as the data division standard of the first index table, wherein the first business attribute is included in all In the business attributes of the source database, the first business attributes include index item attributes.

The first division submodule is configured to divide the first index table into sub-indexes corresponding to each partition according to the data division standard of the first index table.

The first judging sub-module is used to judge whether the data volume of the source database corresponding to each of the sub-indexes is within a preset single transmission volume.

The second division sub-module is configured to, if yes, divide the data of the source database into a first specified number of first fragmented data according to each of the sub-indexes, where the first specified number is the first The number of each partition corresponding to the business attribute.

Further, the division module 2 includes:

Adding a sub-module for adding a partition corresponding to a second service attribute if the data volume corresponding to each sub-index is not within the preset single transmission volume, where the second service attribute is all the sources Among the business attributes of the database other than the attribute of the index item, the second business attribute includes at least one attribute.

The third division submodule is configured to divide the data of the source database into a second specified number of second fragmented data according to each of the sub-indexes and the partition corresponding to the second business attribute, so that each of the first The data volume of the two-slice data is within a preset single transmission volume, wherein the second specified number is a product of the number of partitions of the first service attribute and the number of partitions of the second service attribute.

Further, the obtaining sub-module includes:

The collection unit is configured to collect a first number of data sets from the source database according to a preset collection rule.

The acquisition unit is used to acquire the data volume of a single migration data of the service system.

The obtaining unit is configured to obtain the division capability coefficient according to the data amount of the single migration data divided by the first amount.

The calculation unit is configured to calculate the closeness of the aggregation result set corresponding to the business attributes of each source database to the division capability coefficient, wherein the aggregation result set is the aggregation classification result of the data set, and the aggregation result set The number of is equal to the number of partitions of the business attributes of each source database;

The determining unit is configured to determine the priority ranking according to the respective degrees of proximity, wherein the higher the degree of proximity, the higher the priority corresponding to the aggregation result set.

Further, the division ability coefficient is expressed as an evenly divided result set, and the evenly divided result set includes a data set obtained by dividing the total amount of data sets collected at a preset collection regular interval by the number of aggregate result sets, and the determining unit, include:

The first judgment subunit is used to judge whether there is a third service attribute and a fourth service attribute that have the same degree of closeness as the division capability coefficient, wherein the third service attribute and the fourth service attribute are included in all the source databases Business properties.

The acquiring subunit is configured to, if it exists, acquire the first Manhattan distance in which the third business attribute corresponds to the number of data sets in the evenly divided result set, and the fourth business attribute corresponds to the number of data sets in the evenly divided result set Second Manhattan distance.

The second judgment subunit is used to judge whether the first Manhattan distance is greater than the second Manhattan distance.

The sorting subunit is configured to, if yes, arrange the priority order of the fourth service attribute corresponding to the second Manhattan distance before the third service attribute corresponding to the first Manhattan distance.

In this embodiment, when the J values corresponding to multiple service attributes are the same, it is necessary to consider further analysis of the division capability based on the Manhattan distance. For the same number of aggregated result sets, the smaller the Manhattan distance, the better the division ability. For example, the 100 data sets in the above example belong to 3 professional companies according to the index column index1, namely technology, property insurance, and life insurance; and the corresponding quantity sets of science and technology, property insurance, and life insurance in the 3 professional companies are: 30, 10 and 60. The Manhattan distance corresponding to the index column index1 is: Manhattan distance is expressed as D, then D=|30-33.3|+|10-33.3|+|60-33.3|=53.3, the above 33.3 is the number of data sets in the result set, Manhattan distance is an absolute distance.

Further, the data structure in the target database is a multi-level data nesting structure, and the second acquisition module 3 includes:

The second acquisition sub-module is used to acquire the designated service attributes in the priority ranking that are invoked when the source database is divided into slices, wherein the designated service attributes are included in all services of the source database Properties.

The mapping submodule is used to establish a one-to-one mapping relationship between the priority order corresponding to each of the designated business attributes and the multi-level data nesting structure in the target database, wherein the designated business attribute with the highest priority corresponds to the The outermost layer of the multi-level data nesting structure.

The target database in this embodiment includes non-relational databases, such as cassandra database. The data structure of the cassandra database is the primary key construction structure. In this embodiment, the data fragmentation point determined in the data to be migrated is used as the primary key reference of the cassandra database. The primary key in is the structure of data organization. The data in cassandra is stored in the order of the primary key. For example, the primary key is: K1, K2, K3..., which can be understood as K1 data nested K2 data, K2 data nested again K3 data. According to the priority order of business attributes, this embodiment is mapped to the data primary key sequence in cassandra. For example, the shard data corresponding to the business attributes with high priority is mapped to the data of the outermost nesting layer in the data in cassandra. In order to achieve a clear and reasonable correspondence between data migration between different databases, the file data is written in order, which improves the efficiency of data writing and facilitates the calling and management of data.

Further, the migration module 4 includes:

The second judgment sub-module is used to judge whether the corresponding service flow at the current moment is within a preset threshold.

The start sub-module is used to start a preset migration thread if it is, and search for data to be migrated from the source database.

The storage sub-module is used to store the data to be migrated in the cache server and convert the data format.

The running sub-module is used to run the preset migration thread according to the preset thread mode, and sequentially inject the data to be migrated into the target database according to the mode of fragmented data.

This embodiment implements data migration in batches and orderly by slicing the data. Each batch can be performed independently without mutual dependence, and the migration volume of each batch matches the single transmission volume supported by the system. , If a single migration fails, you only need to repeat the current failed single transmission again without affecting the overall data migration effect. And through the migration in batches, the fragmented time of the system running other businesses can be used to complete the data migration, and the efficiency of the system in processing transactions is improved. This embodiment judges whether the current business is at a low period or a peak period by identifying the operating load status of the system. For example, the business flow is within a preset threshold, indicating that it is in the low business period, and the data migration thread is started to perform data migration; the business flow is not at the preset threshold Inside, it means that in the peak period of business, the data migration thread is suspended to terminate data migration. It not only supports the control of migration tasks according to business attributes, but also ensures the complete migration of business data by category, and avoids business peak periods. After the fragmented data is formed in this embodiment, threads can be started to start data migration. During the migration process, the data is queried and cut out from the oracle database according to the fragmentation information, and the cut data results are stored in the cache server, and then formatted, and then injected into the cassandra database; in this way, one fragment of data is divided into one. The slice data is processed in stages through the same thread. Different fragmented data can also be allocated to different threads for parallel processing to improve the efficiency of fragment migration, but the number of threads running in parallel needs to be reasonably controlled to avoid overloading the databases at both ends.

3, an embodiment of the present application also provides a computer device. The computer device may be a server, and its internal structure may be as shown in FIG. 3. The computer equipment includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the computer designed processor is used to provide calculation and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, computer readable instructions, and a database. The above-mentioned readable storage medium includes a non-volatile readable storage medium and a volatile readable storage medium. The memory provides an environment for the operation of the operating system and computer readable instructions in the non-volatile storage medium. The database of the computer equipment is used to store data such as migration data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instruction is executed, it executes the process of the above-mentioned method embodiment. Those skilled in the art can understand that the structure shown in FIG. 3 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied.

An embodiment of the present application further provides a computer-readable storage medium on which computer-readable instructions are stored. When the computer-readable instructions are executed, the processes of the foregoing method embodiments are executed. The above-mentioned readable storage medium includes non-volatile readable storage medium and volatile readable storage medium. The above are only the preferred embodiments of this application, and do not limit the scope of this application. Any equivalent structure or equivalent process transformation made using the content of this application description and drawings, or directly or indirectly applied to other related The technical field is equally included in the scope of patent protection of this application.

Claims

A method for data migration, which is characterized in that it includes:

Acquiring business attributes of a source database, where the source database is a database storing data to be migrated, and the source database includes a first index table;

According to the partition of the first index table and the business attribute of the source database, dividing the data of the source database into a specified number of fragmented data according to a preset dividing manner;

Acquiring the corresponding relationship between each of the fragmented data and the data structure in the target database, wherein the target database is a database storing the migration data after migration;

According to the corresponding relationship, each piece of data is migrated from the source database to the target database in a preset migration manner.
The method for migrating data according to claim 1, wherein the data of the source database is divided into a predetermined division method according to the partition of the first index table and the business attribute of the source database. The steps to specify the number of pieces of data include:

Acquiring the priority rankings corresponding to the business attributes of all the source databases, where the priority ranking is a ranking of priority levels from high to low;

Select each partition corresponding to the first business attribute before the designated serial number from the priority ranking as the data division standard of the first index table, wherein the first business attribute includes all business attributes of the source database Wherein, the first business attribute includes an index item attribute;

Dividing the first index table into sub-indexes corresponding to each partition according to the data division standard of the first index table;

Judging whether the data volume of the source database corresponding to each sub-index is within a preset single transmission volume;

If yes, divide the data of the source database into a first specified number of first fragmented data according to each of the sub-indexes, where the first specified number is the number of each partition corresponding to the first business attribute .
The method for migrating data according to claim 2, wherein after the step of determining whether the data volume of the source database corresponding to each of the sub-indexes is within a preset single transmission volume, the method comprises:

If the data volume corresponding to each of the sub-indexes is not within the preset single transmission volume, then the partition corresponding to the second service attribute is added, where the second service attribute is the service attribute of all the source databases divided by Attributes other than index item attributes, the second business attribute includes at least one;

According to each of the sub-indexes and the partitions corresponding to the second business attributes, the data of the source database is divided into a second specified amount of second piece of data, so that the data volume of each of the second piece of data is Within the preset single transmission amount, wherein the second specified number is a product of the number of partitions of the first service attribute and the number of partitions of the second service attribute.
The method of migration data according to claim 2, wherein the step of obtaining the priority rankings corresponding to the business attributes of all the source databases comprises:

Collecting a first number of data sets from the source database according to a preset collection rule;

Obtain the data volume of a single migration data of the service system;

Divide the first number by the data amount of the single migration data to obtain the division ability coefficient;

Calculate the closeness of the aggregation result set corresponding to the business attributes of each source database to the division capability coefficient, where the aggregation result set is the aggregation classification result of the data set, and the number of aggregation result sets is equal to State the number of partitions of the business attribute of the source database;

The priority ranking is determined according to the respective degrees of proximity, wherein the higher the degree of proximity, the higher the priority corresponding to the aggregation result set.
The method for migration data according to claim 4, wherein the division ability coefficient is expressed as an evenly divided result set, and the evenly divided result set includes the total amount of data sets collected at a preset collection rule interval divided by the aggregate For a data set of the number of result sets, the step of determining the priority ranking according to each of the proximity degrees includes:

Judging whether there is a third service attribute and a fourth service attribute that have the same degree of closeness as the division capability coefficient;

If so, obtain the first Manhattan distance corresponding to the number of data sets in the equalized result set by the third business attribute, and obtain the first Manhattan distance corresponding to the number of data sets in the equalized result set by the fourth business attribute. Two Manhattan distance;

Determine whether the first Manhattan distance is greater than the second Manhattan distance;

If yes, the priority order of the fourth service attribute corresponding to the second Manhattan distance is arranged before the third service attribute corresponding to the first Manhattan distance.
The method of migrating data according to claim 2, wherein the data structure in the target database is a multi-layer data nesting structure, and the acquisition of each piece of data is different from the data structure in the target database. The steps of the corresponding relationship include:

Acquiring all the designated business attributes in the priority ranking that are invoked when the source database is divided into pieces of data, wherein the designated business attributes are included in all business attributes of the source database;

Establish a one-to-one mapping relationship between the priority order corresponding to each of the designated business attributes and the multi-level data nesting structure in the target database, wherein the designated business attribute with the highest priority corresponds to the multi-level data nesting structure The outermost layer.
The method for migrating data according to claim 1, wherein the step of migrating each piece of data from the source database to the target database according to a preset migration mode according to the corresponding relationship, include:

Determine whether the corresponding business traffic at the current moment is within a preset threshold;

If yes, start a preset migration thread, and search for the data to be migrated from the source database;

Store the data to be migrated in a cache server, and convert the data format;

Run the preset migration thread according to the preset thread mode, and sequentially inject the data to be migrated into the target database according to the mode of fragmented data.
A device for migrating data is characterized in that it comprises:

The first obtaining module is configured to obtain the business attributes of a source database, where the source database is a database storing data to be migrated, and the source database includes a first index table;

A dividing module, configured to divide the data of the source database into a specified number of fragmented data according to a preset dividing manner according to the partition of the first index table and the business attributes of the source database;

The second acquisition module is configured to acquire the corresponding relationship between each of the fragmented data and the data structure in the target database, wherein the target database is a database storing the migration data after migration;

The migration module is configured to migrate each piece of data from the source database to the target database according to a preset migration mode according to the corresponding relationship.
The device for migration data according to claim 8, wherein the division module comprises:

The first obtaining sub-module is configured to obtain the priority rankings corresponding to the business attributes of all the source databases, wherein the priority ranking is a ranking of priority levels from high to low;

The selection sub-module is used to select each partition corresponding to the first business attribute before the designated serial number from the priority ranking as the data division standard of the first index table, wherein the first business attribute is included in all In the business attributes of the source database, the first business attributes include index item attributes;

The first division sub-module is configured to divide the first index table into sub-indexes corresponding to each partition according to the data division standard of the first index table;

The first judging submodule is used to judge whether the data volume of the source database corresponding to each of the sub-indexes is within a preset single transmission volume;

The second division sub-module is configured to, if yes, divide the data of the source database into a first specified number of first fragmented data according to each of the sub-indexes, where the first specified number is the first The number of each partition corresponding to the business attribute.
The device for migration data according to claim 9, wherein the division module comprises:

Adding a sub-module for adding a partition corresponding to a second service attribute if the data volume corresponding to each sub-index is not within the preset single transmission volume, where the second service attribute is all the sources Attributes other than index item attributes in the business attributes of the database, the second business attribute includes at least one;

The third division submodule is configured to divide the data of the source database into a second specified number of second fragmented data according to each of the sub-indexes and the partition corresponding to the second business attribute, so that each of the first The data volume of the two-slice data is within a preset single transmission volume, wherein the second specified number is a product of the number of partitions of the first service attribute and the number of partitions of the second service attribute.
The device for migration data according to claim 9, wherein the acquiring sub-module comprises:

A collection unit, configured to collect a first number of data sets from the source database according to a preset collection rule;

The acquisition unit is used to acquire the data volume of a single migration data of the service system;

The obtaining unit is configured to obtain the division capability coefficient according to the data amount of the single migration data divided by the first amount;

The calculation unit is configured to calculate the closeness of the aggregation result set corresponding to the business attributes of each source database to the division capability coefficient, wherein the aggregation result set is the aggregation classification result of the data set, and the aggregation result set The number of is equal to the number of partitions of the business attributes of each source database;

The determining unit is configured to determine the priority ranking according to each of the proximity degrees, wherein the higher the proximity degree, the higher the priority corresponding to the aggregation result set.
The migration data device according to claim 11, wherein the division capability coefficient is expressed as an evenly divided result set, and the evenly divided result set includes the total amount of data sets collected at a preset collection rule interval divided by the aggregate A data set of the number of result sets, and the determining unit includes:

The first judging subunit is used to judge whether there are third service attributes and fourth service attributes that have the same degree of closeness as the division capability coefficient;

The acquiring subunit is configured to, if it exists, acquire the first Manhattan distance in which the third business attribute corresponds to the number of data sets in the evenly divided result set, and the fourth business attribute corresponds to the evenly divided result set The second Manhattan distance of the number of data sets;

The second judgment subunit is used to judge whether the first Manhattan distance is greater than the second Manhattan distance;

The sorting subunit is configured to, if yes, arrange the priority order of the fourth service attribute corresponding to the second Manhattan distance before the third service attribute corresponding to the first Manhattan distance.
The device for migrating data according to claim 9, wherein the data structure in the target database is a multi-level data nesting structure, and the second acquiring module comprises:

The second acquisition sub-module is used to acquire all the designated business attributes in the priority ranking that are invoked when the source database is divided into fragments, wherein the designated business attributes are included in all the source database Business attributes;

The mapping submodule is used to establish a one-to-one mapping relationship between the priority order corresponding to each of the designated business attributes and the multi-level data nesting structure in the target database, wherein the designated business attribute with the highest priority corresponds to the The outermost layer of the multi-level data nesting structure.
The device for migration data according to claim 8, wherein the migration module comprises:

The second judging sub-module is used to judge whether the corresponding service flow at the current moment is at a preset threshold;

The start sub-module is used to start a preset migration thread if it is, and search for the data to be migrated from the source database;

The storage sub-module is used to store the data to be migrated in the cache server and convert the data format;

The running sub-module is configured to run the preset migration thread according to the preset thread mode, and sequentially inject the data to be migrated into the target database according to the mode of fragmented data.
A computer device includes a memory and a processor, the memory stores a computer program, and is characterized in that a method for migrating data when the processor executes the computer program, and a method for migrating data includes:

Acquiring business attributes of a source database, where the source database is a database storing data to be migrated, and the source database includes a first index table;

According to the partition of the first index table and the business attribute of the source database, dividing the data of the source database into a specified number of fragmented data according to a preset dividing manner;

Acquiring the corresponding relationship between each of the fragmented data and the data structure in the target database, wherein the target database is a database storing the migration data after migration;

According to the corresponding relationship, each piece of data is migrated from the source database to the target database in a preset migration manner.
15. The computer device according to claim 15, wherein the data in the source database is divided into a specified number according to the partition of the first index table and the business attributes of the source database according to a preset division method The steps of sharding data include:

Acquiring the priority rankings corresponding to the business attributes of all the source databases, where the priority ranking is a ranking of priority levels from high to low;

Select each partition corresponding to the first business attribute before the designated serial number from the priority ranking as the data division standard of the first index table, wherein the first business attribute includes all business attributes of the source database Wherein, the first business attribute includes an index item attribute;

Dividing the first index table into sub-indexes corresponding to each partition according to the data division standard of the first index table;

Judging whether the data volume of the source database corresponding to each sub-index is within a preset single transmission volume;

If yes, divide the data of the source database into a first specified number of first fragmented data according to each of the sub-indexes, where the first specified number is the number of each partition corresponding to the first business attribute .
16. The computer device according to claim 16, wherein after the step of determining whether the data volume of the source database corresponding to each of the sub-indexes is within a preset single transmission volume, the step comprises:

If the data volume corresponding to each of the sub-indexes is not within the preset single transmission volume, then the partition corresponding to the second service attribute is added, where the second service attribute is the service attribute of all the source databases divided by Attributes other than index item attributes, the second business attribute includes at least one;

According to each of the sub-indexes and the partitions corresponding to the second business attributes, the data of the source database is divided into a second specified amount of second piece of data, so that the data volume of each of the second piece of data is Within the preset single transmission amount, wherein the second specified number is a product of the number of partitions of the first service attribute and the number of partitions of the second service attribute.
A computer-readable storage medium having a computer program stored thereon, wherein the method for migrating data is realized when the computer program is executed by a processor, and the method for migrating data includes:

Acquiring business attributes of a source database, where the source database is a database storing data to be migrated, and the source database includes a first index table;

According to the partition of the first index table and the business attribute of the source database, dividing the data of the source database into a specified number of fragmented data according to a preset dividing manner;

Acquiring the corresponding relationship between each of the fragmented data and the data structure in the target database, wherein the target database is a database storing the migration data after migration;

According to the corresponding relationship, each piece of data is migrated from the source database to the target database in a preset migration manner.
The computer-readable storage medium according to claim 18, wherein the data of the source database is divided according to a preset division method according to the partition of the first index table and the business attribute of the source database The steps to form a specified number of fragmented data include:

Acquiring the priority rankings corresponding to the business attributes of all the source databases, where the priority ranking is a ranking of priority levels from high to low;

Select each partition corresponding to the first business attribute before the designated serial number from the priority ranking as the data division standard of the first index table, wherein the first business attribute includes all business attributes of the source database Wherein, the first business attribute includes an index item attribute;

Dividing the first index table into sub-indexes corresponding to each partition according to the data division standard of the first index table;

Judging whether the data volume of the source database corresponding to each sub-index is within a preset single transmission volume;

If yes, divide the data of the source database into a first specified number of first fragmented data according to each of the sub-indexes, where the first specified number is the number of each partition corresponding to the first business attribute .
The computer-readable storage medium according to claim 19, wherein after the step of determining whether the data volume of the source database corresponding to each of the sub-indexes is within a preset single transmission volume, the step comprises :

If the data volume corresponding to each of the sub-indexes is not within the preset single transmission volume, then the partition corresponding to the second service attribute is added, where the second service attribute is the service attribute of all the source databases divided by Attributes other than index item attributes, the second business attribute includes at least one;

According to each of the sub-indexes and the partitions corresponding to the second business attributes, the data of the source database is divided into a second specified amount of second piece of data, so that the data volume of each of the second piece of data is Within the preset single transmission amount, wherein the second specified number is a product of the number of partitions of the first service attribute and the number of partitions of the second service attribute.