CN117076386A

CN117076386A - Data increment migration method and related device

Info

Publication number: CN117076386A
Application number: CN202311054407.8A
Authority: CN
Inventors: 张凡; 苏磊
Original assignee: Tencent Technology Beijing Co Ltd
Current assignee: Tencent Technology Beijing Co Ltd
Priority date: 2023-08-21
Filing date: 2023-08-21
Publication date: 2023-11-17

Abstract

The embodiment of the application discloses a data increment migration method and a related device, wherein the method comprises the following steps: determining an operation type corresponding to the target operation record and a data change condition corresponding to the target operation record, and determining a target migration mode corresponding to the target operation record according to at least one of the operation type and the data change condition, wherein the target migration mode is one of serial migration and parallel migration, and migrating the target operation record from the source database to the target database based on the target migration mode. In general, for the target operation record causing the change of the unique index key, serial migration can be adopted as a corresponding target migration mode, so that the consistency of the data configured with the unique index in the target database and the source database is ensured, namely the reliability of data migration is ensured; for the target operation record which does not cause the change of the unique index key, the parallel migration can be adopted as the corresponding target migration mode, so that the data migration speed is improved.

Description

Data increment migration method and related device

Technical Field

The present application relates to the field of database technologies, and in particular, to a method and an apparatus for incremental migration of data.

Background

MongoDB is a database based on distributed file storage that aims to provide an extensible high-performance data storage solution for WEB applications. MongoDB is a product that is interposed between a relational database and a non-relational database, which is the most functionally rich and most like relational database in the non-relational database.

In practical applications, migration of data stored in one mongo db database (source database) to another mongo db database (target database) needs to be implemented through full-scale migration and incremental migration. Specifically, the source database may migrate all of the currently stored data to the target database by total migration; for the data newly written in the original database in the whole migration period, the data needs to be migrated to the target database through incremental migration; in delta migration, oplog (a log for recording operations performed on data) after the full-volume migration start time point may be acquired from a source database and written to a target database to update data in the target database accordingly.

However, when the related technology is adopted to perform incremental migration on the data with the unique index in the source database, the data in the target database is inconsistent with the data in the source database due to improper processing on the data with the unique index; also, in some cases, migration speed is slower for uniquely indexed data.

Disclosure of Invention

The embodiment of the application provides a data increment migration method and a related device, which are used for ensuring the consistency of unique index data and improving the migration speed.

The first aspect of the application provides a data increment migration method, which comprises the following steps:

determining an operation type corresponding to the target operation record and a data change condition corresponding to the target operation record; the target operation record is used for representing the operation related to the target data configured with the unique index in the source database, and the data change condition is the change condition of the target data after the operation represented by the target operation record is executed;

determining a target migration mode corresponding to the target operation record according to at least one of the operation type and the data change condition; the target migration mode is one of serial migration and parallel migration, the serial migration is used for indicating that after the operation represented by the target operation record is completed in the target database, other operation records related to the data configured with the unique index are migrated, and the parallel migration is used for indicating that the target operation record and other operation records are migrated to the target database concurrently;

and migrating the target operation record from the source database to the target database based on the target migration mode.

A second aspect of the present application provides a data delta migration apparatus, the apparatus comprising:

the information determining module is used for determining the operation type corresponding to the target operation record and the data change condition corresponding to the target operation record; the target operation record is used for representing the operation related to the target data configured with the unique index in the source database, and the data change condition is the change condition of the target data after the operation represented by the target operation record is executed;

the migration mode determining module is used for determining a target migration mode corresponding to the target operation record according to at least one of the operation type and the data change condition; the target migration mode is one of serial migration and parallel migration, the serial migration is used for indicating that after the operation represented by the target operation record is completed in the target database, other operation records related to the data configured with the unique index are migrated, and the parallel migration is used for indicating that the target operation record and other operation records are migrated to the target database concurrently;

and the data migration module is used for migrating the target operation record from the source database to the target database based on the target migration mode.

A third aspect of the application provides a computer apparatus comprising a processor and a memory:

The memory is used for storing a computer program;

the processor is configured to execute the steps of the data delta migration method according to the first aspect described above according to the computer program.

A fourth aspect of the present application provides a computer readable storage medium storing a computer program for performing the steps of the data delta migration method of the first aspect described above.

A fifth aspect of the application provides a computer program product or computer program comprising computer instructions stored on a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the steps of the data delta migration method described in the first aspect above.

From the above technical solutions, the embodiment of the present application has the following advantages:

in the embodiment of the application, the target migration mode corresponding to the target operation record is determined by determining the operation type corresponding to the target operation record and the data change condition corresponding to the target operation record, and according to at least one of the operation type and the data change condition, the target migration mode is one of serial migration and parallel migration, and the target operation record is migrated from the source database to the target database based on the target migration mode. In general, for the target operation record causing the change of the unique index key, serial migration can be adopted as a corresponding target migration mode, so that the consistency of the data configured with the unique index in the target database and the source database is ensured, namely the reliability of data migration is ensured; for the target operation record which does not cause the change of the unique index key, the parallel migration can be adopted as the corresponding target migration mode, so that the data migration speed is improved.

Drawings

Fig. 1 is a schematic view of a scenario of a data increment migration method according to an embodiment of the present application;

FIG. 2 is a flowchart of a method for migrating data increments according to an embodiment of the present application;

FIG. 3 is a flowchart of another method for migrating data increments according to an embodiment of the present application;

fig. 4 is a schematic diagram of a data migration flow of a MongoDB according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a data delta migration apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a terminal device according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a server according to an embodiment of the present application.

Detailed Description

In order to make the present application better understood by those skilled in the art, the following description will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

As an example, assuming that the data a in the source database needs to be migrated to the target database, the migration may be performed by a full-scale migration manner, but in the process of migrating the data a from the source database to the target database by the full-scale migration manner, the source database may also write new data B, for the data B, the migration by an incremental migration manner is needed, specifically, an oplog of an operation performed on the data B after a full-scale migration start time point may be obtained from the source database, and the oplog is written to the target database, so that the target database updates the data therein according to the oplog.

In order to solve the above technical problems, an embodiment of the present application provides a data increment migration method and a related device, where the method includes: determining an operation type corresponding to a target operation record and a data change condition corresponding to the target operation record, wherein the target operation record is used for representing an operation related to target data configured with a unique index in a source database, and the data change condition is a change condition of the target data after the operation represented by the target operation record is executed; determining a target migration mode corresponding to the target operation record according to at least one of the operation type and the data change condition, wherein the target migration mode is one of serial migration and parallel migration, the serial migration is used for indicating that after the operation represented by the target operation record is completed in the target database, other operation records related to the data configured with the unique index are migrated, and the parallel migration is used for indicating that the target operation record and other operation records are migrated to the target database concurrently; and migrating the target operation record from the source database to the target database based on the target migration mode.

As an example, in the process of migrating data a from a source database to a target database by a full migration method, it is assumed that the source database will write new data B, determine an operation type M and a data change condition N of the target operation record corresponding to the data B, determine a target migration method corresponding to the target operation record according to the operation type M and/or the data change condition N, migrate the target operation record corresponding to the data B from the source database to the target database based on serial migration or parallel migration as the target migration method, so that the target database updates the data therein according to the target operation record.

Therefore, for the target operation record which causes the change of the unique index key, serial migration can be adopted as a corresponding target migration mode, so that the consistency of the data configured with the unique index in the target database and the source database is ensured, namely the reliability of data migration is ensured; for the target operation record which does not cause the change of the unique index key, the parallel migration can be adopted as the corresponding target migration mode, so that the data migration speed is improved.

Referring to fig. 1, the diagram is a schematic view of a scenario of a data increment migration method according to an embodiment of the present application, where the application scenario includes a server 100.

The server 100 determines an operation type corresponding to a target operation record of the source database and a data change condition corresponding to the target operation record; the target operation record is used for representing the operation related to the target data configured with the unique index in the source database, and the data change condition is the change condition of the target data after the operation represented by the target operation record is executed. As an example, assuming that a target operation record is field 1 in update data a, the operation type corresponding to the target operation record is an update operation, and the data change condition is that data a is changed to data A1 through the update operation.

The server 100 determines a target migration mode corresponding to the target operation record according to at least one of the operation type and the data change condition; the target migration mode is one of serial migration and parallel migration, the serial migration is used for indicating that after the operation represented by the target operation record is completed in the target database, other operation records related to the data configured with the unique index are migrated, and the parallel migration is used for indicating that the target operation record and other operation records are migrated to the target database concurrently. As an example, assuming that the operation type of the target operation record is an update operation and the data change condition is that a unique index key in the target data changes, it may be determined that the target migration mode corresponding to the target operation record is serial migration.

The server 100 migrates the target operational record of the source database from the source database to the target database based on the target migration manner.

As an example, when the target migration mode is serial migration, the target operation record needs to be migrated from the source database to the target database, and in the process of migrating the target operation record, other operation records do not execute migration operation, and after the migration of the target operation record is completed, other operation records are migrated.

As another example, when the target migration manner is parallel migration, the target operation record may be migrated from the source database to the target database simultaneously with other operation records.

The data increment migration method provided by the embodiment of the application can be applied to a server with data processing capability, wherein the server can be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, and a cloud server for providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, basic cloud computing services such as big data and artificial intelligent platforms.

The data increment migration method provided by the embodiment of the application designs a cloud technology and a database technology.

Cloud technology (Cloudtechnology) refers to a hosting technology that unifies serial resources such as hardware, software, network, etc. in a wide area network or a local area network to realize calculation, storage, processing and sharing of data.

Cloud technology (cloud technology) is based on the general terms of network technology, information technology, integration technology, management platform technology, application technology and the like applied by cloud computing business modes, and can form a resource pool, and the cloud computing business mode is flexible and convenient as required. Cloud computing technology will become an important support. Background services of technical networking systems require a large amount of computing, storage resources, such as video websites, picture-like websites, and more portals. Along with the high development and application of the internet industry, each article possibly has an own identification mark in the future, the identification mark needs to be transmitted to a background system for logic processing, data with different levels can be processed separately, and various industry data needs strong system rear shield support and can be realized only through cloud computing.

The Database (Database), which can be considered as an electronic filing cabinet, is a place for storing electronic files, and users can perform operations such as adding, inquiring, updating, deleting and the like on the data in the files. A "database" is a collection of data stored together in a manner that can be shared with multiple users, with as little redundancy as possible, independent of the application.

A database management system (DBMS for short) is a computer software system designed for managing databases, and generally has basic functions of storage, interception, security, backup, etc. The database management system may classify according to the database model it supports, such as relational, XML (extensible markup language); or by the type of computer supported, e.g., server cluster, mobile phone; or by the query language used, such as SQL (structured query language), XQuery, or by the performance impact emphasis, such as maximum-scale, highest speed of operation, or other classification means, regardless of which classification means is used, some DBMSs can support multiple query languages across categories, for example, simultaneously.

The related data collection and processing in the application should be strictly according to the requirements of relevant national laws and regulations when the example is applied, obtain the informed consent or independent consent of the main body of personal information, and develop the subsequent data use and processing behaviors within the authorized range of laws and regulations and the main body of personal information.

Referring to fig. 2, the flow chart of a data increment migration method according to an embodiment of the present application is shown.

Referring to fig. 2, the method for migrating data increments provided in the embodiment of the present application may include:

s201: and determining the operation type corresponding to the target operation record and the data change condition corresponding to the target operation record.

A source database means a source database that provides incremental migration of data, i.e., data needs to be migrated from the source database, which may specifically be MongoDB. The target database means a target database for data increment migration, that is, data needs to be migrated into the target database, and the target database can be MongoDB.

It can be understood that when the source database performs data migration to the target database, two migration modes can be divided, one migration mode is full migration, and the full migration is used for migrating data stored in the source database before the full migration starts; another migration approach is incremental migration, which is used to migrate new data written by the source database during the full migration.

Full migration refers to the process of migrating all data of one database to another database; incremental migration refers to the process of updating new data written by a source database in the full migration process to another database on the premise that the full migration is performed on a certain database.

The target operation record means a record of an operation on target data configured with a unique index in the source database, and as an example, the target operation record may be to "update" the target data Q to data X, assuming that an update operation is performed on the target data Q in the source database; the data change condition means a change condition of the target data after the operation corresponding to the target operation record is performed, and as an example, assuming that the target data is W and the target operation record is "delete" the target data W, the data change condition means that the target data W is deleted via the "delete" operation.

The unique index is an index built up of one or more combinations of fields on the table, the values of which (or of which) in combination are not repeatable in the table. For example, assuming that a computer can log in to a website only once, a unique index can be established for the ID of the computer and the ID of the website, and when the computer logs in to the website once, the computer cannot log in again. That is, the unique index ensures that the data written to the source database is a unique value.

S202: and determining a target migration mode corresponding to the target operation record according to at least one of the operation type and the data change condition.

The target migration mode is one of serial migration and parallel migration, the serial migration is used for indicating that after the operation represented by the target operation record is completed in the target database, other operation records related to the data configured with the unique index are migrated, and the parallel migration is used for indicating that the target operation record and other operation records are migrated to the target database concurrently.

As an example, assuming that the operation record includes D, E, F, and the target operation record is G, if serial migration is performed, migrating the target operation record G to the target database first, and then migrating the operation records D, E, F to the target database; if parallel migration is performed, the target operation record G and the operation records D, E, F are migrated together to the target database.

It will be appreciated that the target operation record may conflict with other operation records, so as to ensure correctness of the migrated data, it may be determined whether the target operation record may be migrated in parallel with other operation records by at least one of information of an operation type and a data change condition, that is, determining that the target migration mode is serial migration or parallel migration.

It should be noted that, as an example, serial migration may be used as the target migration mode when it is determined that the unique index key of the target data is changed according to at least one of the operation type and the data change condition.

S203: and migrating the target operation record from the source database to the target database based on the target migration mode.

It can be appreciated that, because there may be a conflict between different operation records, that is, migration cannot be performed simultaneously, serial migration is required for a target operation record that has a conflict with other operation records, so as to ensure consistency between data of the target database and data of the source database; for the situation that no conflict exists among different operation records, a parallel migration mode can be adopted to improve the efficiency of data transmission.

Based on the data delta migration method provided in the foregoing embodiment, when the operation type is an update operation, the target data will change based on the update operation, and as a possible implementation manner, step S202 may include:

a1: when the operation type is update operation, determining whether the unique index key of the target data changes according to the data change condition.

The unique index key means a key (key) corresponding to a unique index, and each unique index key may have corresponding target data.

It can be understood that, because the value corresponding to the unique index key is unique, when the unique index key changes, if a parallel migration mode is still adopted, data migration may be caused to occur in error, so as to solve the technical problem, when the operation type is update, whether the unique index key of the target data changes needs to be determined according to the data change condition, so that whether the target operation record is migrated from the source database to the target database in a serial migration or parallel migration mode is determined later.

As an example, step A1 may include:

b1: and judging whether a target field exists in the target operation record.

The target field means a field in the target operation record where the data changes, for example, when the target operation record is oplog, the oplog field may include a "set" field.

As an example, assuming that a set in the source database is uniquely indexed with fields a and b, a specific set representation may be as follows:

where "a" 1 and "b" 1 denote that field a and field b, respectively, are unique indices.

B2: if the target field exists and the unique index key data is included in the target field, determining that the unique index key of the target data changes.

Unique index key data means data in a database that is unique and non-repeatable, typically identified by a unique index key unique: { }, within which is the data content that is unique.

It will be appreciated that when the target field exists in the target operation record, the target data is considered to be changed, whether the target field contains unique index key data needs to be further determined, and if so, the update operation is considered to be an update operation for the unique index key, that is, the unique index key of the target data is changed.

If the target field exists, but the target field does not include the unique index key data, it is determined that the unique index key of the target data is unchanged, and the corresponding target migration mode is parallel migration.

As an example, assuming that a set in the source database is uniquely indexed with fields a and b, a representation specifically containing the target field may be as follows:

{"ts":Timestamp(1685540542,1),"t":NumberLong(1),"h":NumberLong(0),"v":2,"op":"u","ns":"somedb.somecol","ui":UUID("79f34e01-79ac-49c0-8ab1-cd8217af3d9a"),"o2":{"_id":ObjectId("64774eb439f68405caa6daa1")},"wall":ISODate("2023-05-31T13:42:22.133Z"),"o":{"$v":1,"$set":{"a":1,"b":1}}}，

{"ts":Timestamp(1685540542,1),"t":NumberLong(1),"h":NumberLong(0),"v":2,"op":"u","ns":"somedb.somecol","ui":UUID("79f34e01-79ac-49c0-8ab1-cd8217af3d9a"),"o2":{"_id":ObjectId("64774eb439f68405caa6daa1")},"wall":ISODate("2023-05-31T13:42:22.133Z"),"o":{"$v":1,"$set":{"a":1,"b":1}}}。

where "$set" indicates a target field, "$set": { "a":1, "b":1} indicates that the target field contains unique index key data.

B3: if the target field does not exist, the unique index key of the target data is determined to be changed.

It can be understood that when the operation type is an update operation, if the target field does not exist, the unique index key of the target data can be directly considered to be changed, that is, update is realized through replacement operation.

As an example, assuming that a set in the source database is uniquely indexed with fields a and b, a representation that specifically does not contain the target field may be as follows:

{"ts":Timestamp(1685540583,1),"t":NumberLong(1),"h":NumberLong(0),"v":2,"op":"u","ns":"somedb.somecol","ui":UUID("79f34e01-79ac-49c0-8ab1-cd8217af3d9a"),"o2":{"_id":ObjectId("64774eb439f68405caa6daa1")},"wall":ISODate("2023-05-31T13:43:03.673Z"),"o":{"_id":ObjectId("64774eb439f68405caa6daa1"),"a":0,"b":1}}。

the set does not contain a target field, but the operation type is update operation, so that the change of the unique index key of the target data can be directly determined.

It should be appreciated that in this embodiment, whether the unique index key has changed may be determined in two ways, one is by determining whether a target field exists in the target operation record and determining whether the unique index key data is included in the target field to determine whether the unique index key has changed; and when the target operation record does not contain the target field, the unique index key can be directly considered to be changed, so that the search is not needed manually, and the migration efficiency is improved.

A2: if the unique index key of the target data changes, determining that the target migration mode is serial migration; if the unique index key of the target data is unchanged, determining that the target migration mode is parallel migration.

It can be understood that when the unique index key changes, the target data corresponding to the unique index key can be considered to change, and the corresponding target migration mode should be serial migration, so as to ensure the consistency of the data between the target database and the source database; and when the unique index key is unchanged, the target migration mode can be parallel migration so as to improve the migration speed.

It should be noted that, the operation type corresponding to the target operation record is not limited to the update operation, and as a possible implementation manner, when the operation type is the delete operation, the target data may be changed based on the delete operation, and step S202 may include: when the operation type is delete operation delete, the target migration mode is determined to be serial migration. It will be appreciated that when the operation type is a delete operation, in order to ensure consistency between the target database and the source database, a serial migration manner is required to migrate the target operation record from the source database to the target database.

As an example, assuming that a set in the source database is uniquely indexed with fields a and b, the target operation record is characterized as a delete operation, the specific representation may be as follows:

{"ts":Timestamp(1686051769,1),"t":NumberLong(1),"h":NumberLong(0),"v":2,"op":"d","ns":"somedb.somecol","ui":UUID("f9b0cb8a-2a38-459c-947e-90fb06c9348f"),"wall":ISODate("2023-06-06T11:42:49.660Z"),"o":{"_id":ObjectId("647f1bb44436cc1ae649a60a")}}。

wherein, the operation type is a delete operation, and the target migration mode is serial migration.

It should be noted that, the operation type corresponding to the target operation record is not limited to the update operation and the delete operation, and as another possible implementation manner, when the operation type is the insert operation, the target data may be changed based on the insert operation, and step S202 may include: when the operation type is insert operation insert, the target migration mode is determined to be parallel migration. It will be appreciated that when the operation type is an insert operation, in order to increase the data migration speed, the target operation record may be migrated from the source database to the target database in a parallel migration manner.

Based on the data increment migration method provided in the foregoing embodiment, the target migration manner includes serial migration and parallel migration, serial migration means that data migration is performed one by one according to a certain order, as an example, it is assumed that the operation record includes V, C, and the target operation record is X, if the target migration manner is serial migration, it is necessary to first migrate the target operation record X, then migrate the operation record V, and finally migrate the operation record C; parallel migration means that multiple data sets are migrated at the same time, and the data sets may include target operation records, that is, the target operation records and operation records of other sets may be migrated to the target database together.

In some possible implementation manners, step S203 may include:

c1: and when the target migration mode is serial migration, locking the target operation record.

Wherein the locking process is used to block subsequent transmissions of other operational records associated with the data configured with the unique index. It will be understood that when the target migration mode is serial migration, it is necessary to migrate the target operation record from the source database to the target database before migrating other operation records, but because there may be multiple operation records related to the uniquely indexed data, if other operation records migrate in parallel with the target operation record or migrate to the target database before the target operation record, the migration of the target operation record may fail, further resulting in inconsistent data of the target database and data of the source database, so in order to ensure consistency of data between the target database and the source database, it is necessary to lock the target operation record when the target migration mode is serial migration, and block the operation record related to the uniquely indexed data.

C2: the target operation record is transferred from the source database to the target database, and an operation of target operation record characterization is performed in the target database for the target data.

It should be understood that in the embodiment of the present application, the migration of data is implemented by transmitting the target operation record from the source database to the target database, so that the operation represented by the target operation record is performed on the target data in the target database, and as an example, assuming that the target data is Y, the operation represented by the target operation record is to update the target data Y to the data U, when the target operation record is transmitted from the source database to the target database, the update operation indicated by the target operation record is performed on the target data Y in the target database, and the target data Y is updated to the data U.

And C3: and after the operation of the target operation record characterization is completed in the target database, unlocking the target operation record.

It will be appreciated that after the operation represented by the target operation record is completed in the target database, other operation records may be continuously executed, so that the locking process of the target operation record needs to be released to ensure that the operation represented by the subsequent operation record is normally executed.

As an example, recording may be performed for a target operation as follows in table 1:

table 1: performing operations of a target operation record characterization

Wherein, insert represents a write operation, delete represents a delete operation, update represents an update operation, replace represents a replace operation, oplog represents a target operation record, _id represents a primary key identification, and UniqueKey represents a unique index key.

Barrier means a barrier provided to a target database and operations after Barrier need to wait for operations before execution of the operations before Barrier to complete. Referring to table 1 above, for example, assuming that the operation characterized by the target operation record before the barrier-1 is being executed, the operation characterized by the operation record after the barrier-1 needs to be executed after the execution of the operation characterized by the target operation record is completed.

Based on table 1, when the target migration mode is serial migration, the operations of the operation record characterization need to be executed sequentially, that is, the operation of the next operation record characterization is executed after the execution of the operation of the target operation record characterization is completed, so as to ensure the consistency of the data.

In other possible implementations, step S203 may include:

d1: when the target migration mode is parallel migration, determining a target data set to which the target operation record belongs according to the primary key identifier corresponding to the target operation record.

The primary key identifier means an identifier corresponding to a data set, that is, a plurality of operation records may be included in one data set, and the data set corresponds to only one primary key identifier.

Each operation record in the same data set corresponds to the same main key identifier and is arranged according to the corresponding time stamp sequence. It should be understood that different operation records are operation records at different time points, and may be arranged according to the corresponding time stamp order in order to avoid confusion of operation records.

D2: the target data set is transmitted through one of a plurality of parallel data transmission channels.

The data transmission channel is used for realizing data transmission from the source database to the target database. It should be understood that when the target migration manner is parallel migration, a plurality of data transmission channels are included, in order to avoid data loss in the transmission process, a target data set to which the target operation record belongs needs to be determined through a primary key identifier corresponding to the target operation record, and the data is transmitted through the data transmission channel corresponding to the target data set, so that the data migration speed is improved, and meanwhile, the integrity of the data is ensured.

Based on the data increment migration method provided in the foregoing embodiment, the embodiment of the present application further provides a data increment migration method, and referring to fig. 3, the fig. is a flowchart of another data increment migration method provided in the embodiment of the present application.

When incremental migration is performed on the data with the unique index in the source database at present, the data in the target database is inconsistent with the data in the source database due to improper processing of the data with the unique index; also, in some cases, migration speed is slower for uniquely indexed data.

Given that uk fields are added to oplog in a source database to represent information of uniquely indexed data, so that collision detection of the unique index is further realized, modification of a kernel of the database is required, but the original database is not universal, that is, in the original database, the problem of data inconsistency still exists for the uniquely indexed data.

In order to solve the problem that in the data increment migration process, the data of the unique index is inconsistent and the migration speed of the data of the unique index is slow, firstly, in the embodiment of the application, the database may be a mongo db database, then the source database may be mongo db1, the target database may be mongo db2, the whole data is represented as D, and the data migration flow of the mongo db may refer to fig. 4, which is a schematic diagram of the data migration flow of the mongo db provided by the embodiment of the application.

As shown in fig. 4, it is first necessary to establish a connection between MongoDB1 and MongoDB2, and then migrate the full-size data D from MongoDB1 to MongoDB2 by way of full-size migration, and in the process of full-size migration, migrate the newly written incremental data H from MongoDB1 to MongoDB2 by way of incremental migration.

In the process of full-scale migration, referring to fig. 3, the method for migrating data increments provided in the embodiment of the present application may include:

s301: and obtaining a target operation record oplog.

S302: judging whether the last operation record corresponding to oplog has locking processing or not, if so, executing step S303; if not, step S304 is performed.

S303: wait for the last operation record to be successfully executed, and re-execute step S302.

S304: when the operation type is an updating operation, judging whether an $set field exists in the oplog, and judging whether the $set field comprises a unique index key; or, judging whether the operation type is a deletion operation.

If the operation type is an update operation, there is a $set field in oplog, and the $set field includes a unique index key, step S305 is performed; if the operation type is the update operation, there is no $set field in oplog, then step S305 is performed; if the operation type is an update operation, there is a $set field in oplog, and the $set field does not include a unique index key, then step S306 is performed; if the operation type is the delete operation, step S305 is executed; if the operation type is not the delete operation or the update operation, step S306 is performed.

S305: the oplog is locked to block other operation records related to the data configured with the unique index, and then step S306 is performed.

S306: and executing oplog, wherein the oplog corresponding to the same primary key identification_id is executed according to the time sequence.

When step S306 is performed by step S305, it can be understood that the serial migration; when step S306 is directly performed, it can be understood that the migration is parallel.

S307: after the oplog is successfully executed, unlocking the oplog.

Referring to fig. 5, the structure of a data increment migration apparatus according to an embodiment of the present application is shown.

Based on the data increment migration method provided in the foregoing embodiment, an embodiment of the present application further provides a data increment migration apparatus, and in conjunction with fig. 5, the apparatus 500 may include:

the information determining module 501 is configured to determine an operation type corresponding to the target operation record and a data change condition corresponding to the target operation record; the target operation record is used for representing the operation related to the target data configured with the unique index in the source database, and the data change condition is the change condition of the target data after the operation represented by the target operation record is executed;

the migration mode determining module 502 is configured to determine a target migration mode corresponding to the target operation record according to at least one of the operation type and the data change condition; the target migration mode is one of serial migration and parallel migration, the serial migration is used for indicating that after the operation represented by the target operation record is completed in the target database, other operation records related to the data configured with the unique index are migrated, and the parallel migration is used for indicating that the target operation record and other operation records are migrated to the target database concurrently;

The data migration module 503 is configured to migrate the target operation record from the source database to the target database based on the target migration mode.

As an example, migration style determination module 502 includes:

the judging unit is used for determining whether the unique index key of the target data changes according to the data change condition when the operation type is update operation;

the migration mode determining unit is used for determining that the target migration mode is serial migration if the unique index key of the target data changes; if the unique index key of the target data is unchanged, determining that the target migration mode is parallel migration.

As an example, the judging unit is specifically configured to:

judging whether a target field exists in the target operation record;

if the target field exists and the target field comprises the unique index key data, determining that the unique index key of the target data changes;

if the target field does not exist, the unique index key of the target data is determined to be changed.

As an example, the migration manner determining module 502 is specifically configured to:

and when the operation type is a deletion operation, determining that the target migration mode is serial migration.

And when the operation type is the insert operation, determining that the target migration mode is parallel migration.

As an example, the data migration module 503 is specifically configured to:

when the target migration mode is serial migration, locking the target operation record; the locking process is used to block the transmission of subsequent other operational records associated with the data configured with the unique index;

transmitting the target operation record from the source database to the target database, and executing the operation of target operation record characterization aiming at the target data in the target database;

and after the operation of the target operation record characterization is completed in the target database, unlocking the target operation record.

As an example, the data migration module 503 is specifically configured to:

when the target migration mode is parallel migration, determining a target data set to which the target operation record belongs according to a primary key identifier corresponding to the target operation record; each operation record in the same data set corresponds to the same main key identifier and is arranged according to the corresponding time stamp sequence;

transmitting the target data set through one data transmission channel of a plurality of parallel data transmission channels; the data transmission channel is used for realizing data transmission from the source database to the target database.

The data increment migration device provided by the embodiment of the application has the same beneficial effects as the data increment migration method provided by the embodiment, so that the description is omitted.

The embodiment of the application also provides a computer device, which can be a terminal device or a server, and the terminal device and the server provided by the embodiment of the application are introduced from the aspect of hardware materialization.

Referring to fig. 6, fig. 6 is a schematic structural diagram of a terminal device according to an embodiment of the present application. As shown in fig. 6, for convenience of explanation, only the portions related to the embodiments of the present application are shown, and specific technical details are not disclosed, please refer to the method portions of the embodiments of the present application. The terminal can be any terminal equipment including a mobile phone, a tablet personal computer, a personal digital assistant, a sales terminal (PointofSales, POS), a vehicle-mounted computer and the like, and takes the terminal as a computer as an example:

fig. 6 is a block diagram showing a part of the structure of a computer related to a terminal provided by an embodiment of the present application. Referring to fig. 6, a computer includes: radio frequency (RadioFrequency, RF) circuitry 1210, memory 1220, input unit 1230 (including touch panel 1231 and other input devices 1232), display unit 1240 (including display panel 1241), sensors 1250, audio circuitry 1260 (which may connect speaker 1261 and microphone 1262), wireless fidelity (WiFi) module 1270, processor 1280, and power supply 1290. Those skilled in the art will appreciate that the computer architecture shown in fig. 6 is not limiting and that more or fewer components than shown may be included, or that certain components may be combined, or that different arrangements of components may be utilized.

Memory 1220 may be used to store software programs and modules, and processor 1280 may execute the various functional applications and data processing of the computer by executing the software programs and modules stored in memory 1220. The memory 1220 may mainly include a storage program area that may store an operating system, application programs required for at least one function (such as a sound playing function, an image playing function, etc.), and a storage data area; the storage data area may store data created according to the use of the computer (such as audio data, phonebooks, etc.), and the like. In addition, memory 1220 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.

Processor 1280 is a control center of the computer and connects various parts of the entire computer using various interfaces and lines, performing various functions of the computer and processing data by running or executing software programs and/or modules stored in memory 1220, and invoking data stored in memory 1220. In the alternative, processor 1280 may include one or more processing units; preferably, the processor 1280 may integrate an application processor and a modem processor, wherein the application processor primarily handles operating systems, user interfaces, application programs, etc., and the modem processor primarily handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 1280.

In an embodiment of the present application, the processor 1280 included in the terminal further has the following functions:

Optionally, the processor 1280 is further configured to perform steps of any implementation of the data delta migration method provided by the embodiment of the present application.

Referring to fig. 7, fig. 7 is a schematic structural diagram of a server 1300 according to an embodiment of the present application. The server 1300 may vary considerably in configuration or performance and may include one or more Central Processing Units (CPUs) 1322 (e.g., one or more processors) and memory 1332, one or more storage media 1330 (e.g., one or more mass storage devices) that store applications 1342 or data 1344. Wherein the memory 1332 and storage medium 1330 may be transitory or persistent. The program stored on the storage medium 1330 may include one or more modules (not shown), each of which may include a series of instruction operations on a server. Further, the central processor 1322 may be configured to communicate with the storage medium 1330, and execute a series of instruction operations in the storage medium 1330 on the server 1300.

The server 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input/output interfaces 1358, and/or one or more operating systems, such as a Windowserver ^TM ，MacOSX ^TM ，Unix ^TM ,Linux ^TM ，FreeBSD ^TM Etc.

The steps performed by the server in the above embodiments may be based on the server structure shown in fig. 7.

Wherein CPU1322 is configured to perform the following steps:

Optionally, CPU1322 may also be configured to perform the steps of any one implementation of the data delta migration method provided by embodiments of the present application.

The embodiments of the present application further provide a computer readable storage medium storing a computer program for executing any one of the foregoing data incremental migration methods according to the foregoing embodiments.

Embodiments of the present application also provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium and executes the computer instructions to cause the computer device to perform any one of the data delta migration methods described in the foregoing respective embodiments.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-only memory (ROM), a random access memory (RandomAccessMemory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing a computer program.

It should be understood that in the present application, "at least one (item)" means one or more, and "a plurality" means two or more. "and/or" for describing the association relationship of the association object, the representation may have three relationships, for example, "a and/or B" may represent: only a, only B and both a and B are present, wherein a, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b or c may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.

The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of data delta migration, the method comprising:

determining an operation type corresponding to a target operation record and a data change condition corresponding to the target operation record; the target operation record is used for representing the operation related to the target data configured with the unique index in the source database, and the data change condition is the change condition of the target data after the operation represented by the target operation record is executed;

determining a target migration mode corresponding to the target operation record according to at least one of the operation type and the data change condition; the target migration mode is one of serial migration and parallel migration, the serial migration is used for indicating that other operation records related to the data configured with the unique index are migrated after the operation represented by the target operation record is completed in a target database, and the parallel migration is used for indicating that the target operation record and other operation records are concurrently migrated to the target database;

2. The method according to claim 1, wherein the determining, according to the at least one of the operation type and the data change condition, a target migration manner corresponding to the target operation record includes:

when the operation type is update operation, determining whether a unique index key of the target data changes according to the data change condition;

if the unique index key of the target data changes, determining that the target migration mode is the serial migration; and if the unique index key of the target data is unchanged, determining that the target migration mode is the parallel migration.

3. The method of claim 2, wherein determining whether the unique index key of the target data has changed based on the data change condition comprises:

judging whether a target field exists in the target operation record;

if the target field exists and the target field comprises unique index key data, determining that the unique index key of the target data changes;

and if the target field does not exist, determining that the unique index key of the target data changes.

4. The method according to claim 1, wherein the determining, according to the at least one of the operation type and the data change condition, a target migration manner corresponding to the target operation record includes:

and when the operation type is a deletion operation, determining the target migration mode as the serial migration.

5. The method according to claim 1, wherein the determining, according to the at least one of the operation type and the data change condition, a target migration manner corresponding to the target operation record includes:

and when the operation type is an insertion operation, determining the target migration mode as the parallel migration.

6. The method of any one of claims 1 to 5, wherein the migrating the target operation record from the source database to the target database based on the target migration manner comprises:

when the target migration mode is the serial migration, locking the target operation record; the locking process is used for blocking the transmission of subsequent operation records related to the data configured with the unique index;

Transmitting the target operation record from the source database to the target database, and executing the operation characterized by the target operation record aiming at the target data in the target database;

and after the operation represented by the target operation record is completed in the target database, unlocking the target operation record.

7. The method of any one of claims 1 to 5, wherein the migrating the target operation record from the source database to the target database based on the target migration manner comprises:

when the target migration mode is the parallel migration, determining a target data set to which the target operation record belongs according to a main key identifier corresponding to the target operation record; each operation record in the same data set corresponds to the same main key identifier and is arranged according to the corresponding time stamp sequence;

transmitting the target data set through one of a plurality of parallel data transmission channels; the data transmission channel is used for realizing data transmission from the source database to the target database.

8. A data delta migration apparatus, the apparatus comprising:

the migration mode determining module is used for determining a target migration mode corresponding to the target operation record according to at least one of the operation type and the data change condition; the target migration mode is one of serial migration and parallel migration, the serial migration is used for indicating that other operation records related to the data configured with the unique index are migrated after the operation represented by the target operation record is completed in a target database, and the parallel migration is used for indicating that the target operation record and other operation records are concurrently migrated to the target database;

9. A computer device, the computer device comprising a processor and a memory;

the memory is used for storing a computer program;

the processor is configured to perform the data delta migration method of any one of claims 1 to 7 according to the computer program.

10. A computer readable storage medium for storing a computer program for executing the data delta migration method of any one of claims 1 to 7.