CN115237890A - Data migration method, device, medium and equipment for database splitting - Google Patents

Abstract

The application discloses a data migration method, a device, a medium and equipment for database splitting, wherein the method comprises the following steps: modifying the processing request of the point to be transformed into a butt transition user so as to obtain target data in a target data table of a source database; determining migration data partition time based on the current moment, acquiring first migration data in a target data table before the migration data partition time, and migrating the first migration data to a target database; and closing the write permission of the transition user to the target data table of the source database, acquiring second migration data after the migration data division time in the target data table, migrating the second migration data to the target database, and modifying the processing request of the point to be modified into a new user. When the data migration is carried out, the shutdown is not needed, the influence on the system is reduced to the minimum, the stability of the system during the data migration is improved, and the new database is different from the source database, so that the problem of wrong access of the new database and the old database does not exist.

Description

Data migration method, device, medium and equipment for database splitting

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a data migration method, apparatus, medium, and device for database splitting.

Background

With the increase of the service systems, more and more data are stored in the database, and the time for inquiring the data is longer and longer, so that the decoupling and splitting of the service systems are necessarily carried out, namely the splitting of the database according to the service systems is carried out. The database splitting is to split and migrate part of data of a source database into a new database, and the splitting of the database must be performed as follows: the service can not be interrupted due to the splitting, the usability index can not be reduced due to the splitting, and the resource loss caused by the splitting can not be caused.

However, the silent splitting without sensing at all of the system does not exist, and how to reduce the split sensing to the maximum and the split stability becomes the key of the database splitting.

When data migration is performed on new and old databases, two main approaches are currently used: stopping the machine to perform data migration and performing data migration online without stopping the machine. In the scheme of stopping the data migration, the database cannot normally provide service to the outside during the migration, the service is interrupted, and the stability is greatly influenced.

The scheme of on-line data migration without shutdown mainly comprises main-standby switching which is also often used as a disaster tolerance means for database exception, and by using the scheme, switching at the second level can be almost achieved, and perception is reduced to the greatest extent. However, the operation requires the old database name to be consistent with the new database name, otherwise confusion in database management is caused, the old database and the new database exist at the same time and provide services at the same time after the database is successfully split, and the problem of database access error is caused because the old database name is consistent with the new database name.

Therefore, a data migration method is needed to improve the stability of the system during data migration and avoid the access errors of the old and new databases.

Disclosure of Invention

In view of this, the present invention provides a data migration method, apparatus, medium, and device for database splitting, and mainly aims to solve the problems of poor stability and wrong access to new and old databases during data migration of the existing database splitting.

According to an aspect of the present application, there is provided a data migration method for database splitting, the method comprising:

modifying a processing request of a point to be modified into a butt joint transition user to obtain target data in a target data table of a source database, wherein the transition user has the right of reading and writing the target data table of the source database, and the point to be modified is a processing node matched with the target data table in a service system;

determining migration data partition time based on the current time, acquiring first migration data in the target data table before the migration data partition time, and migrating the first migration data to a data table of a target database;

closing the write permission of the transition user to the target data table of the source database, acquiring second migration data in the target data table after the migration data division time, and migrating the second migration data to the data table of the target database;

and modifying the processing request of the point to be transformed into a new user, wherein the new user has the authority of reading and writing the data table in the target database.

Optionally, the step of determining migration data partition time based on the current time, obtaining first migration data in the target data table before the migration data partition time, and migrating the first migration data to the data table of the target database includes:

acquiring data stored in the target data table before the current moment as stock data;

reading the stock data from the target data table, and writing the read stock data into a data table of the target database;

after the current time, acquiring data which is newly created and stored in the target data table and update data of the stored data in the target data table in a first preset time interval at intervals of a first preset time interval respectively to serve as first incremental data and first full update data in the first preset time interval;

reading first incremental data and first full update data generated in the first preset time period from the target data table, and writing the read first incremental data and the first full update data into the data table of the target database.

Optionally, the reading the stock data from the target data table and writing the read stock data into the data table of the target database includes:

acquiring time information of the stock data, and creating a time index according to the time information;

dividing the stock data into a plurality of groups of data according to the time index;

and for each group of data, reading the group of data from the target data according to the time sequence, and writing the read group of data into a data table of the target database.

Optionally, the migrating the second migration data to the data table of the target database, where the migrating the second migration data includes second incremental data and second full update data, and the obtaining of the second migration data in the target data table after the migration data division time includes:

respectively acquiring data newly created after the last first preset time period and before the time of closing the write permission of the transition user on the target data table of the source database and stored in the target data table and update data of stored data in the target data table as second incremental data and second full update data;

and respectively reading the second incremental data and the second full updating data from the target data table, and writing the read second incremental data and the second full updating data into the data table of the target database.

Optionally, before modifying the processing request of the point to be transformed to the docking transition user, the method further includes:

determining a target data table in a source database according to the migrated service system;

and acquiring processing nodes matched with the target data table in all service systems, and taking one processing node as a point to be modified.

Optionally, after the modifying the processing request of the point to be transformed to the docking transition user, the method further includes:

opening the read-write permission of the transition user to read the target data table of the source database:

monitoring whether the processing requests of all points to be transformed are all in butt joint with transition users by adopting a monitoring platform;

if the processing requests of the processing nodes of all the service systems are matched within the second preset time period, verifying that no missing point to be modified exists.

Optionally, after verifying that there are no missing points to be modified and the stock data is migrated, closing the write permission of the transition user on the source database target data table.

According to another aspect of the present application, a data migration apparatus for database splitting is provided, including:

the transition user docking module modifies a processing request of a point to be modified into a docking transition user to acquire target data in a source database target data table, wherein the transition user has the authority of reading and writing the source database target data table, and the point to be modified is a processing node matched with the target data table in a service system;

the first migration data migration module determines migration data partition time based on the current moment, acquires first migration data in the target data table before the migration data partition time, and migrates the first migration data into the data table of the target database;

the second migration data migration module is used for closing the write permission of the transition user to the target data table of the source database, acquiring second migration data in the target data table after the migration data division time, and migrating the second migration data to the data table of the target database;

and the new user docking module modifies the processing request of the point to be transformed into a docking new user, wherein the new user has the authority of reading and writing the data table in the target database.

Optionally, the step of determining migration data partition time based on the current time, obtaining first migration data in the target data table before the migration data partition time, and migrating the first migration data to the data table of the target database includes:

acquiring data stored in the target data table before the current moment as stock data;

reading the stock data from the target data table, and writing the read stock data into a data table of the target database;

after the current moment, acquiring data which is newly created and stored in the target data table and update data of the stored data in the target data table at intervals of a first preset time period as first incremental data and first full update data in the first preset time period;

and respectively reading first incremental data and first full update data generated in the first preset time period from the target data table, and writing the read first incremental data and the first full update data into the data table of the target database.

Optionally, the reading the stock data from the target data table and writing the read stock data into the data table of the target database includes:

acquiring time information of the stock data, and creating a time index according to the time information;

dividing the stock data into a plurality of groups of data according to the time index;

and for each group of data, reading the group of data from the target data according to the time sequence, and writing the read group of data into a data table of the target database.

Optionally, the migrating the second migration data to the data table of the target database, where the migrating the second migration data includes second incremental data and second full update data, and the obtaining of the second migration data in the target data table after the migration data division time includes:

respectively acquiring data newly created after the last first preset time period and before the time of closing the write permission of the transition user to the target data table of the source database and stored in the target data table and update data of stored data in the target data table as second incremental data and second full update data;

and respectively reading the second incremental data and the second full updating data from the target data table, and writing the read second incremental data and the second full updating data into the data table of the target database.

Optionally, before modifying the processing request of the point to be transformed to the docking transition user, the method further includes:

determining a target data table in a source database according to the migrated service system;

and acquiring processing nodes matched with the target data table in all service systems, and taking one processing node as a point to be modified.

Optionally, after the modifying the processing request of the point to be transformed to the docking transition user, the method further includes:

opening the read-write permission of the transition user to read the target data table of the source database:

monitoring whether the processing requests of all points to be transformed are all in butt joint with transition users by adopting a monitoring platform;

if so, closing the read-write permission of the original user to the source database target data table, and verifying that no missed points to be transformed exist if the processing requests of the processing nodes of all the service systems are subjected to matching processing within a second preset time period.

Optionally, after verifying that there are no missing points to be modified and the stock data is migrated, closing the write permission of the transition user on the source database target data table.

According to another aspect of the present application, a storage medium is provided, where at least one executable instruction is stored in the storage medium, and the executable instruction causes a processor to perform operations corresponding to the data migration method for database splitting.

According to another aspect of the present application, there is provided a computer device including: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;

the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the data migration method for database splitting.

By the technical scheme, the technical scheme provided by the embodiment of the invention at least has the following advantages:

compared with the prior art, the method, the device, the equipment and the medium for migrating the data for splitting the database are characterized in that a point to be modified is determined according to a target data table corresponding to a service system to be migrated, a processing request of the point to be modified is modified to be a transition user, the processing request of the point to be modified is made to obtain data in the target data table in a source database through the transition user, meanwhile, first migration data is determined in the target data table, the first migration data is migrated to the data table of the target database, the writing authority of the transition user on the target data table in the source database is closed, second migration data is migrated to the data table of the target database, then, the processing request of the point to be modified is modified to obtain the data in the target database through a new user, new second migration data is not generated during the writing authority of the target data table of the source database is closed, the second migration data is few, the time for migrating the second migration data to the target database is short, the influence on the system is reduced to the lowest under the condition of no outage, the stability of the migration of the source database is improved, and the problems of the old database and the old database access errors are different from the old database.

Compared with the prior art, the data migration method for splitting the database is characterized in that omission detection is carried out on the points to be reconstructed, the points to be reconstructed which are not omitted are determined to be absent, after data migration, the processing request of the points to be reconstructed is in butt joint with a new database, normal operation of a system after migration is guaranteed, and stability of the system is improved.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

Various additional advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

fig. 1 illustrates a flowchart of a data migration method for database splitting according to an embodiment of the present application;

FIG. 2 is a flowchart illustrating another data migration method for database splitting according to an embodiment of the present application;

FIG. 3 is a flow chart illustrating a further data migration method for database splitting according to an embodiment of the present application;

FIG. 4 is a flowchart illustrating a further data migration method for database splitting according to an embodiment of the present application;

FIG. 5 is a block diagram illustrating components of a data migration apparatus for database splitting according to an embodiment of the present application;

fig. 6 shows a schematic structural diagram of a computer device according to an embodiment of the present invention.

Wherein the content of the first and second substances,

in fig. 5: 502-transition user docking module; 504-a first migration data migration module; 506-a second migration data migration module; 508-new user docking module;

in fig. 6: 602-a processor; 604-a communication interface; 606-a memory; 608-a communication bus; 610-procedure.

Detailed Description

The invention will be described in detail hereinafter with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.

To further explain the technical means and effects of the present invention adopted to achieve the predetermined object, the following detailed description of the embodiments, structures, features and effects according to the present invention will be made with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "an embodiment" refers to not necessarily the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

The embodiment of the application provides a data migration method for database splitting, which aims at the problems that at present, database data migration is performed, data migration is performed when a database is stopped, the database cannot normally provide external services during the migration, services are interrupted, the influence on stability is great, main and standby switching is performed online without stopping, and database access errors can be caused due to the fact that an old database name is consistent with a new database name, and as shown in fig. 1, the method comprises the following steps:

102: modifying the processing request of the point to be modified into a butt joint transition user to obtain target data in a target data table of a source database, wherein the transition user has the right of reading and writing the target data table of the source database, and the point to be modified is a processing node matched with the target data table in a service system;

in the embodiment of the invention, as the data stored in the database is increased and the reading speed of the data is influenced, the database is split into the source database and the new database. Before the data is not split, the processing nodes of the service system acquire data in the source database through the original users matched with the source database, after the data is split, part of the data is migrated to the new database, and when some processing nodes of the service system need to acquire the migrated data, the processing nodes need to acquire the data in the new database through the new users matched with the new database. Therefore, the migration of the database includes not only the migration of the data, but also the docking of the processing node of the service system that acquires the migrated data with the user matching the database, and the docking of the original user matching the source database with the new user matching the new database, so the correct modification of the docking of the processing node of the service system is very important.

In the embodiment of the invention, the original user has the read-write permission of the source database, and the new user has the read-write permission of the new (called target in the application) database. In order to ensure correct modification of the processing node interface of the business system, a transition user is created, the transition user is set to have the right of accessing a target data table in a source database, the target data table is a data table to be migrated, is determined according to the migrated business system, and is not limited to commodity service, insurance claim settlement, medical service and the like. The transition user accesses the source database, namely the same database used by the original user, so that the problem that different databases are used to influence data access in the transformation process is solved, the processing requests of the points to be transformed are modified into the butt transition user, after the transition user operates in a normal environment, the processing requests of the points to be transformed are verified to be stably butted with a transition user system, and after data migration is completed, the processing requests of the points to be transformed are butted into new users matched with a target database, so that the processing requests of the processing nodes are ensured to be butted correctly.

The source database and the target database are warehouse databases, and are used for storing a large amount or a huge amount of data in a business operation process, wherein the business operation process includes but is not limited to content of operations such as insurance business, product update, claim settlement operation and the like, for example, code content and data content corresponding to the operations.

The data stored in the database may be public service data, customer data, etc., and since the types of the data are numerous, the database splitting method includes, but is not limited to, application scenarios such as public service, financial service, logistics management, etc.

104: determining migration data partition time based on the current time, acquiring first migration data in a target data table before the migration data partition time, and migrating the first migration data to a data table of a target database;

in the embodiment of the invention, the data in the target data table is divided into the first migration data and the second migration data, and the first migration data is generated earlier than the second migration data, so that the first migration data in the target data table is migrated while the processing request of the point to be modified is docked as the transition user. The first migration data includes, but is not limited to, data generated during operations such as insurance business, product sales, after-sales of products, product inventory management, claim settlement operations, and the like.

In another embodiment of the present invention, for further limitation and explanation, as shown in fig. 2, the migrating data includes stock data, first incremental data and first full update data, determining a migration data partition time based on a current time, acquiring first migrating data in a target data table before the migration data partition time, and migrating the first migrating data to a data table of a target database, and includes:

202: acquiring data stored in a target data table before the current moment as stock data;

204: reading stock data from the target data table, and writing the read stock data into a data table of a target database;

206: after the current time, acquiring data which is newly created and stored in a target data table and updating data of stored data in the target data table in a first preset time interval at intervals of a first preset time interval respectively to serve as first incremental data and first full updating data in the first preset time interval;

208: and respectively reading first incremental data and first full update data generated in a first preset time period from the target data table, and writing the read first incremental data and the first full update data into the data table of the target database.

In the embodiment of the invention, the first migration data comprises stock data, first incremental data and first total update data, the data stored in the target data table before the current time is taken as the stock data by taking the current time as a reference, the stock data is gradually migrated into the data table of the target database according to a preset rule, and the stock data is migrated at one time. Setting a first preset time period, after the current moment, acquiring newly created data in a target data table in the first preset time period at intervals of one first preset time period as first incremental data, and migrating the first incremental data in the first preset time period to a data table of a target database; every other first preset time interval, obtaining the update data of the stored data in the target data table in the first preset time interval as first full update data, and migrating the first full update data in the first preset time interval to the data table of the target database, wherein the first preset time interval is set according to needs, for example, 1 hour, 10 hours, and if the data is less, the time interval can also be 1 day. Because the stock data is more, the migration of the stock data is synchronously migrated with the first incremental data and the first full update data. The inventory data, the first incremental data, and the first full-scale update data each include, but are not limited to, data generated during operations such as insurance operations, product sales, after-sales products, product inventory management, claim settlement operations, and the like.

In one embodiment, the first preset time period is 1 day, the stock data is migrated point by point according to a preset rule, and the first incremental data and the first full update data in the first preset time period are migrated to the target data table every 1 day interval.

In one embodiment, a DATAX tool is employed to perform the migration of the inventory data, the first incremental data, and the first full update data.

In another embodiment of the present invention, for further limitation and explanation, as shown in fig. 3, reading stock data from the target database, and writing the read stock data into the data table of the target database includes:

302: acquiring time information of stock data, and creating a time index according to the time information;

304: dividing the stock data into a plurality of groups of data according to the time index;

306: and for each group of data, reading the group of data from the target data according to the time sequence, and writing the read group of data into a data table of the target database.

In the embodiment of the invention, because the stock data is many, the stock data is migrated according to the preset rule of time sequence, the stock data is divided into a plurality of groups of data according to the time index of the stock data, the size of each group of data is determined according to the migration speed, and each group of data is migrated into the target data table group by group according to the time sequence.

106: closing the write permission of the transition user to the target data table of the source database, acquiring second migration data in the target data table after the migration data division time, and migrating the second migration data to the data table of the target database;

in the embodiment of the invention, after the stock data in the first migration data are migrated, the write permission of the transition user on the target data table of the source database is closed, and the data generated in the target data table after the migration data are divided into the second migration data.

In another embodiment of the present invention, for further limitation and description, as shown in fig. 4, the migrating the second migration data to the data table of the target database includes:

402: respectively acquiring data which is newly created and stored in the target data table after the last first preset time period and before the time of closing the write permission of the transition user on the target data table of the source database and updating data of stored data in the target data table as second incremental data and second full updating data;

404: and respectively reading the second incremental data and the second full updating data from the target data table, and writing the read second incremental data and the second full updating data into the data table of the target database.

In the embodiment of the invention, data which is newly created and stored in a target data table in a time period after the last first preset time period of first incremental data and first full update data and before the time of closing the write permission of a transition user on the target data table of a source database is taken as second incremental data, and the second incremental data is migrated to the data table of the target database; and taking the updating data of the stored data in the target data table in the period after the last first preset period and before the time of closing the write permission of the transition user to the target data table of the source database as second full updating data, and migrating the second full updating data to the data table of the target database. Because the time interval after the last first preset time interval and before the time of closing the write permission of the transition user to the source database target data table is less than the first preset time interval, the second incremental data and the second full update data are less, the second incremental data and the second full update data are synchronously migrated, and the migration time is short. The second incremental data and the second full-scale update data include, but are not limited to, data generated during operations such as insurance business, product sales, after-sales of products, product inventory management, claim settlement, and the like.

108: and modifying the processing request of the point to be transformed into a butt joint new user, wherein the new user has the authority of reading and writing the data table in the target database.

In the embodiment of the invention, after the second migration data migration is completed, the data in the target data table are completely migrated, and the processing request of the point to be modified is modified to acquire the data in the target database through a new user, so that the processing request of the point to be modified can acquire related data conveniently, and the stable operation of the system after the migration is ensured.

The application provides a data migration method for splitting a database, compared with the prior art, the method comprises the steps of determining a point to be modified according to a target data table corresponding to a service system to be migrated, modifying a processing request of the point to be modified into a transition user, enabling the processing request of the point to be modified to acquire data in the target data table in a source database through the transition user, simultaneously determining first migration data in the target data table, migrating the first migration data into the data table of the target database, closing the writing authority of the transition user on the target data table in the source database, migrating second migration data into the data table of the target database, modifying the processing request of the point to be modified into data in the target database through a new user, during the period of closing the writing authority, generating no new second migration data, and because the second migration data are few, the migration data migration time for the second migration data to migrate to the target database is short, the influence on the system is reduced to the minimum under the condition of no shutdown, the stability of the system when the data is improved, the migration data is not different from that the source database and the new database accesses the old database, and the old database is different from the old database, and the problem that the old database accesses the old database is also is wrong.

In another embodiment of the present invention, for further definition and explanation, before modifying the processing request of the point to be transformed to the docking transition user, the method further includes:

determining a target data table in a source database according to the migrated service system;

and acquiring processing nodes matched with the target data table in all the service systems, and taking one processing node as a point to be modified.

Before data migration, processing requests of processing nodes of all service systems acquire data in a source database through an original user, the data are stored in a data table, and after the data migration is completed, the processing requests of the processing nodes of the service systems matched with the data table need to read the data in a new database through a new user. Therefore, in the data migration process, it is necessary to determine the processing node of the service system that matches the migration data table, and perform corresponding docking modification on the matching processing node. If a processing node is missed and a new database is not docked, the processing request of the processing node cannot be matched, stable and healthy operation of the whole system is affected, and therefore modification of the matched processing node is required during data migration.

Specifically, according to a service system to be migrated, an association table belonging to the service to be migrated is sorted, a target data table to be migrated is confirmed, processing nodes of all associated service systems using the target data table are sorted according to the association relationship between the target data table and the service system, and one processing node is used as a point to be modified.

In another embodiment of the present invention, for further definition and explanation, after modifying the processing request of the point to be transformed into the docking transition user, the method further includes:

opening the read-write permission of the transition user to read the target data table of the source database:

monitoring whether the processing requests of all points to be transformed are all in butt joint with transition users by adopting a monitoring platform;

if the processing requests of the processing nodes of all the service systems are matched within the second preset time period, verifying that no missing point to be modified exists.

Specifically, after the point to be modified is determined, in order to further verify whether the determined point to be modified is correct or omitted, the point to be modified is verified, verification is performed in two steps, firstly, the database execution condition of a monitoring platform monitoring system is adopted through a production formal environment, whether processing requests of the point to be modified are all in butt joint with transition users is judged, whether the point to be modified obtains data in a source database through the original database exists or not, after the point to be modified is determined to be in butt joint through the monitoring platform, the read-write permission of the original user on a target data table in the source database is closed, the running health condition of the whole system is judged, and when the whole system does not give an alarm abnormally, the point to be modified is determined not to be omitted. The monitoring platform is a real-time and near-full monitoring system, which focuses on monitoring the Java application. At present, the system is widely applied to a middleware (MVC, RPC, database, cache and the like) framework, provides system performance indexes, health conditions, monitoring alarms and the like for each service line, and adopts a cat monitoring platform in a preferred embodiment.

In one embodiment, if the processing requests of all the points to be modified are not all connected with the transition user, the processing requests of the points to be modified of the non-connected transition users are modified into the connected transition users, and it is ensured that the processing requests of the points to be modified are all connected with the transition users.

In one embodiment, in a second preset time period, if the processing request of the point to be modified does not obtain the matching processing, whether the processing node of which the processing request does not obtain the matching processing is the node to be modified is judged according to the incidence relation between the target data table and all the service systems, and if the processing node of which the processing request does not obtain the matching processing is the node to be modified, the processing request of the processing node of which the processing request does not obtain the matching processing is modified into a docking transition user until the processing requests of the point to be modified all obtain the matching processing in the second preset time period.

Compared with the prior art, the data migration method for splitting the database is characterized in that omission detection is carried out on the points to be reconstructed, the points to be reconstructed which are not omitted are determined to be absent, after data migration, the processing request of the points to be reconstructed is in butt joint with a new database, normal operation of a system after migration is guaranteed, and stability of the system is improved.

In another embodiment of the present invention, for further limitation and description, after verifying that there are no missing points to be transformed and the stock data is migrated, the write permission of the transition user to the source database target data table is closed.

Specifically, after a processing request of a point to be modified is modified into a butt joint transition user, omission verification of the point to be modified is carried out, migration of first migration data is carried out simultaneously, when verification of the point to be modified is not omitted and stock data in the first migration data is migrated completely, time consumption of first incremental data is stable, write permission of the transition user on a source database target data table is closed under the condition that the conditions are met, migration of second migration data is carried out, the second migration data are generated between the last preset time period of the first migration data and the time when the transition user closes the write permission on the source database target data table, the data volume is small, migration time of the second migration data is short, and influence on a system is low.

Further, as an implementation of the method shown in fig. 1, an embodiment of the present invention provides a data migration apparatus for splitting a database, and as shown in fig. 5, the apparatus includes:

the transition user docking module 502 modifies the processing request of the point to be modified into a docking transition user to acquire target data in the target data table of the source database, wherein the transition user has the right to read and write the target data table of the source database, and the point to be modified is a processing node matched with the target data table in the service system;

the first migration data migration module 504 determines migration data partition time based on the current time, obtains first migration data in the target data table before the migration data partition time, and migrates the first migration data into the data table of the target database;

the second migration data migration module 506 is configured to close the write permission of the transition user to the source database target data table, acquire second migration data in the target data table after the migration data division time, and migrate the second migration data to the data table of the target database;

the new user docking module 508 modifies the processing request of the point to be transformed to dock the new user, where the new user has the right to read and write the data table in the target database.

The application provides a data migration device for splitting a database, compared with the prior art, a point to be modified is determined according to a target data table corresponding to a service system to be migrated, a processing request of the point to be modified is modified to be connected with a transition user, the processing request of the point to be modified obtains data in the target data table in a source database through the transition user, meanwhile, first migration data is determined in the target data table, the first migration data is migrated into the data table of the target database, the writing authority of the transition user on the target data table in the source database is closed, second migration data is migrated into the data table of the target database, then the processing request of the point to be modified is modified to obtain data in the target database through a new user, no new second migration data is generated during closing of the writing authority, the migration time of the second migration data to the target database is short due to the fact that the second migration data is few, the migration data migration time of the second migration data to the target database is short, the influence on the system is reduced to the lowest under the condition of no shutdown, the stability of the system during data migration, the problem that the new database is different from the source database to the old database and the old database, the problem of accessing errors of the new database is also different from the old database.

In one embodiment, the first migration data includes stock data, first incremental data, and first full update data, determining migration data partition time based on the current time, acquiring first migration data in the target data table before the migration data partition time, and migrating the first migration data to the data table of the target database, including:

acquiring data stored in a target data table before the current moment as stock data;

reading stock data from the target data table, and writing the read stock data into a data table of a target database;

after the current time, acquiring data which is newly created and stored in a target data table and updating data of stored data in the target data table in a first preset time interval at intervals of a first preset time interval respectively to serve as first incremental data and first full updating data in the first preset time interval;

and respectively reading first incremental data and first full update data generated in a first preset time period from the target data table, and writing the read first incremental data and the first full update data into the data table of the target database.

In one embodiment, reading the stock data from the target data table and writing the read stock data into the data table of the target database includes:

acquiring time information of stock data, and creating a time index according to the time information;

dividing the stock data into a plurality of groups of data according to the time index;

and for each group of data, reading the group of data from the target data according to the time sequence, and writing the read group of data into a data table of the target database.

In one embodiment, the migrating the second migration data to the data table of the target database includes:

respectively acquiring data newly created and stored in the target data table after the last first preset time period and before the permission time for closing the transition user to write the target data table of the source database and updating data of stored data in the target data table as second incremental data and second full updating data;

and respectively reading the second incremental data and the second full updating data from the target data table, and writing the read second incremental data and the second full updating data into the data table of the target database.

In one embodiment, before modifying the processing request of the point to be transformed into the docking transition user, the method further comprises the following steps:

determining a target data table in a source database according to the migrated service system;

and acquiring processing nodes matched with the target data table in all the service systems, and taking one processing node as a point to be modified.

In one embodiment, after modifying the processing request of the point to be transformed into the docking transition user, the method further includes:

opening the read-write permission of the transition user to read the target data table of the source database:

monitoring whether the processing requests of all points to be modified are all in butt joint with transition users by using a monitoring platform;

if the processing requests of the processing nodes of all the service systems are matched within the second preset time period, verifying that no missing point to be modified exists.

In one embodiment, if the processing requests of all the points to be modified are not all in butt joint with the transition user, the processing requests of the points to be modified of the non-butt joint transition user are modified into the butt joint transition user, and it is ensured that the processing requests of the points to be modified are all in butt joint with the transition user.

In one embodiment, in a second preset time period, if the processing request of the point to be modified does not obtain the matching processing, whether the processing node of which the processing request does not obtain the matching processing is the node to be modified is judged according to the incidence relation between the target data table and all the service systems, and if the processing node of which the processing request does not obtain the matching processing is the node to be modified, the processing request of the processing node of which the processing request does not obtain the matching processing is modified into a docking transition user until the processing requests of the point to be modified all obtain the matching processing in the second preset time period.

In one embodiment, the whole system ensures that all operations have update time, and ensures that the data update can be screened and synchronized;

the whole system delete operation preferentially recommends offline, and then uses a double-write scheme (double-write is carried out only under the condition of using transition users, namely two users delete at the same time);

indexes exist in the updating time of all target data tables, and the performance of updating and synchronizing the full data is improved.

Compared with the prior art, the data migration device for splitting the database is characterized in that omission detection is carried out on points to be reconstructed, the points to be reconstructed are determined not to be omitted, after data migration, processing requests of the points to be reconstructed are butted with a new database to provide guarantee, normal operation of a system after migration is guaranteed, and stability of the system is improved.

In one embodiment, after verifying that no missing points to be reformed exist and the stock data is migrated, closing the write permission of the transition user on the target data table of the source database.

According to an embodiment of the present invention, a storage medium is provided, where the storage medium stores at least one executable instruction, and the computer executable instruction may execute the data migration method for database splitting in any of the method embodiments described above.

Fig. 6 is a schematic structural diagram of a computer device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the computer device.

As shown in fig. 6, the computer apparatus may include: a processor (processor) 602, a communication Interface 604, a memory 606, and a communication bus 608.

Wherein: the processor 602, communication interface 604, and memory 606 communicate with one another via a communication bus 608.

A communication interface 604 for communicating with network elements of other devices, such as clients or other servers.

The processor 602 is configured to execute the program 610, and may specifically perform relevant steps in the above-described data migration method for database splitting.

In particular, program 610 may include program code comprising computer operating instructions.

The processor 602 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement an embodiment of the invention. The computer device includes one or more processors, which may be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.

The memory 606 stores a program 610. Memory 606 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.

The program 610 may specifically be configured to cause the processor 602 to perform the following operations:

modifying the processing request of the point to be modified into a butt joint transition user to obtain target data in a target data table of a source database, wherein the transition user has the right of reading and writing the target data table of the source database, and the point to be modified is a processing node matched with the target data table in a service system;

determining migration data partition time based on the current time, acquiring first migration data in a target data table before the migration data partition time, and migrating the first migration data to a data table of a target database;

closing the write permission of the transition user to the target data table of the source database, acquiring second migration data after the migration data division time in the target data table, and migrating the second migration data to the data table of the target database;

and modifying the processing request of the point to be transformed into a butt joint new user, wherein the new user has the authority of reading and writing the data table in the target database.

It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

The above embodiments are only exemplary embodiments of the present application, and are not intended to limit the present application, and the protection scope of the present application is defined by the claims. Various modifications and equivalents may be made to the disclosure by those skilled in the art within the spirit and scope of the disclosure, and such modifications and equivalents should also be considered as falling within the scope of the disclosure.

Claims (10)

1. A data migration method for database splitting is characterized by comprising the following steps:

modifying a processing request of a point to be modified into a butt joint transition user to obtain target data in a target data table of a source database, wherein the transition user has the right of reading and writing the target data table of the source database, and the point to be modified is a processing node matched with the target data table in a service system;

determining migration data partition time based on the current time, acquiring first migration data in the target data table before the migration data partition time, and migrating the first migration data to a data table of a target database;

closing the write permission of the transition user to the target data table of the source database, acquiring second migration data in the target data table after the migration data division time, and migrating the second migration data to the data table of the target database;

and modifying the processing request of the point to be transformed into a new user, wherein the new user has the authority of reading and writing the data table in the target database.

2. The data migration method for database splitting according to claim 1, wherein the first migration data includes stock data, first incremental data, and first full update data, the determining a migration data partition time based on the current time, obtaining first migration data in the target data table before the migration data partition time, and migrating the first migration data to a data table of a target database includes:

acquiring data stored in the target data table before the current moment as stock data;

reading the stock data from the target data table, and writing the read stock data into a data table of the target database;

after the current time, acquiring data which is newly created and stored in the target data table and update data of the stored data in the target data table in a first preset time interval at intervals of a first preset time interval respectively to serve as first incremental data and first full update data in the first preset time interval;

and respectively reading first incremental data and first full update data generated in the first preset time period from the target data table, and writing the read first incremental data and the first full update data into the data table of the target database.

3. The data migration method for database splitting according to claim 2, wherein said reading said stock data from said target data table and writing the read stock data into a data table of said target database comprises:

acquiring time information of the stock data, and creating a time index according to the time information;

dividing the stock data into a plurality of groups of data according to the time index;

and for each group of data, reading the group of data from the target data according to the time sequence, and writing the read group of data into a data table of the target database.

4. The method for data migration for database splitting according to claim 3, wherein the second migration data includes second incremental data and second full update data, and the obtaining second migration data in the target data table after the migration data partition time and migrating the second migration data into the data table of the target database includes:

respectively acquiring data newly created after the last first preset time period and before the time of closing the write permission of the transition user on the target data table of the source database and stored in the target data table and update data of stored data in the target data table as second incremental data and second full update data;

and respectively reading the second incremental data and the second full updating data from the target data table, and writing the read second incremental data and the second full updating data into the data table of the target database.

5. The data migration method for database splitting according to claim 2, wherein before modifying the processing request of the point to be transformed into the docking transition user, further comprising:

determining a target data table in a source database according to the migrated service system;

and acquiring processing nodes matched with the target data table in all service systems, and taking one processing node as a point to be modified.

6. The data migration method for database splitting according to claim 5, wherein after modifying the processing request of the point to be transformed into the docking transition user, further comprising:

opening the read-write permission of the transition user to read the target data table of the source database:

monitoring whether the processing requests of all points to be transformed are all in butt joint with transition users by adopting a monitoring platform;

if the processing requests of the processing nodes of all the service systems are matched within the second preset time period, verifying that no missing point to be modified exists.

7. The data migration method for database splitting according to claim 6, wherein after verifying that there are no missing points to be transformed and the stock data migration is completed, closing the write permission of the transition user to the source database target data table.

8. A data migration apparatus for database splitting, comprising:

the transition user docking module modifies a processing request of a point to be modified into a docking transition user to acquire target data in a source database target data table, wherein the transition user has the right of reading and writing the source database target data table, and the point to be modified is a processing node matched with the target data table in a service system;

the first migration data migration module is used for determining migration data partition time based on the current moment, acquiring first migration data in the target data table before the migration data partition time, and migrating the first migration data to the data table of the target database;

the second migration data migration module is used for closing the write permission of the transition user on the target data table of the source database, acquiring second migration data in the target data table after the migration data division time, and migrating the second migration data to the data table of the target database;

and the new user docking module modifies the processing request of the point to be transformed into a docking new user, wherein the new user has the authority of reading and writing the data table in the target database.

9. A storage medium having stored therein at least one executable instruction, the executable instruction causing a processor to perform operations corresponding to the data migration method for database splitting according to any one of claims 1-7.

10. A computer device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;

the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation corresponding to the data migration method for database splitting as claimed in any one of claims 1-7.

Images