CN115221142A

CN115221142A - Data migration processing method and device, electronic equipment and storage medium

Info

Publication number: CN115221142A
Application number: CN202110429507.9A
Authority: CN
Inventors: 郭占圳
Original assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Current assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Priority date: 2021-04-21
Filing date: 2021-04-21
Publication date: 2022-10-21

Abstract

According to the data migration processing method, the data migration processing device, the electronic equipment and the storage medium, after data of a source data source are migrated to a target data source, data read-write service is not directly switched from the source data source to the target data source, a second service mode is adopted as buffering/transition, the second service mode is a mode for providing data read-write service for double-end data sources based on a source end and a target end, test running detection on whether the target data source is abnormal in service is supported, double-write service is provided for the second service mode, and the same data can be synchronously written into the double-end data sources. When the service is abnormal (does not meet the second switching condition) in the second service mode and the source data source needs to be switched back to for service rollback, the source data source is written with full data and has no data loss, so that the service can be effectively and quickly switched back to the source data source for timely loss stopping, and the influence on the data read-write service during data migration is reduced.

Description

Data migration processing method and device, electronic equipment and storage medium

Technical Field

The present application belongs to the field of information technology, and in particular, to a data migration processing method and apparatus, an electronic device, and a storage medium.

Background

The database is a warehouse for organizing, storing and managing data according to a data structure, is a collection of a large amount of organized, sharable and uniformly managed data stored in a computer for a long time, and can provide service object data or metadata read-write service for upper-layer services.

When the reliability, throughput and the like of the used database cannot meet the service requirements along with the improvement of the traffic, the database needs to be switched to a new database with stronger function and more reliability, which relates to the problems of data migration and database switching of the service after the data migration. The traditional solution is to migrate data stored in an original database to a new database based on a certain technology, such as a replication technology, and to switch the online traffic of the data read-write service to the new database when migration is completed.

However, in this method, after the online traffic of the data read-write service is switched to the new database, if a problem is found, the service cannot be switched back to the original database quickly to stop loss in time.

Disclosure of Invention

In view of this, the present application provides a data migration processing method, an apparatus, an electronic device, and a storage medium, which are used to solve the above problems in the conventional manner, reduce the influence on the data read-write service during data migration, and implement a service-lossless data migration processing scheme as far as possible.

The specific technical scheme is as follows:

a data migration processing method comprises the following steps:

migrating the data of the source data source to the target data source based on the first service mode; the first service mode is a mode for providing data read-write service by utilizing the source data source;

determining whether a first switching condition is satisfied;

if the first switching condition is met, switching the first service mode into a second service mode; the second service mode is a mode for providing data read-write service by utilizing the source data source and the target data source, and provides service for synchronously writing the same data into the source data source and the target data source;

determining whether a second handover condition is satisfied; wherein, satisfying the second switching condition, at least including the data read-write service in the second service mode is not abnormal;

if the second switching condition is met, switching the second service mode into a third service mode, wherein the third service mode is a mode for providing data read-write service by using the target data source;

and if the second switching condition is not met, switching the second service mode to the first service mode.

Optionally, the migrating the data of the source data source to the destination data source includes:

migrating the stock data of the source data source to the target data source;

and if the source data source generates incremental data in the data migration process, migrating the incremental data to the target data source.

Optionally, if the source data source generates incremental data in the data migration process, the determining whether the first switching condition is met includes:

verifying the consistency of the stock data in the source data source and the target data source;

verifying the consistency of the incremental data in the source data source and the destination data source;

if the stock data are consistent between the source data source and the target data source, and the incremental data are consistent between the source data source and the target data source, determining that the first switching condition is met;

and if the stock data is inconsistent between the source data source and the target data source and/or the incremental data is inconsistent between the source data source and the target data source, determining that the first switching condition is not met.

Optionally, the second service mode includes a first sub-mode and/or a second sub-mode;

the first sub-mode is a mode of providing data reading and writing services by using the source data source and providing data writing services by using the destination data source, and the second sub-mode is a mode of providing data writing services by using the source data source and providing data reading and writing services by using the destination data source;

the switching the first service mode to a second service mode includes:

switching the first service mode to the first sub-mode;

or, the first service mode is switched to the second sub-mode;

or, the first service mode is switched to the first sub-mode; and determining whether a third switching condition is met, and if the third switching condition is met, switching the first sub-mode to the second sub-mode.

Optionally, if the second service mode includes the first sub-mode, the determining whether a second switching condition is satisfied includes:

determining whether the data read-write service under the first sub-mode is abnormal; and at least verifying whether the data written into the source data source and the destination data source based on the first sub-mode are consistent;

if the data read-write service under the first sub-mode is not abnormal and the data written into the source data source and the target data source based on the first sub-mode are consistent, determining that the second switching condition is met;

and if the data read-write service under the first sub-mode is abnormal and/or the data written into the source data source and the target data source based on the first sub-mode are inconsistent, determining that the second switching condition is not met.

Optionally, if the second service mode includes the second sub-mode, the determining whether the second switching condition is satisfied includes:

determining whether the data read-write service under the second sub-mode is abnormal; and at least verifying whether the data written into the source data source and the destination data source based on the second sub-mode are consistent;

if the data read-write service under the second sub-mode is not abnormal, and the data written into the source data source and the target data source based on the second sub-mode is consistent, determining that the second switching condition is met;

and if the data read-write service under the second sub-mode is abnormal and/or the data written into the source data source and the target data source based on the second sub-mode are inconsistent, determining that the second switching condition is not met.

Optionally, if the second service mode includes the first sub-mode and the second sub-mode;

the determining whether a second handover condition is satisfied includes:

if the third switching condition is not met, determining that the second switching condition is not met;

if the third switching condition is met and the first sub-mode is switched to the second sub-mode in response to the third switching condition being met, then:

if the data read-write service under the second sub-mode is not abnormal and the data written into the source data source and the target data source based on the second sub-mode are consistent, determining that the second switching condition is met;

if the data read-write service is abnormal in the second sub-mode and/or the data written into the source data source and the target data source based on the second sub-mode are inconsistent, determining that the second switching condition is not met;

wherein satisfying the third handover condition comprises: and the data read-write service under the first sub-mode is not abnormal, and the data written into the source data source and the target data source based on the first sub-mode is consistent.

Optionally, the writing the same data into the source data source and the destination data source synchronously includes:

and synchronously writing the same data into the source data source and the destination data source in a serial mode.

Optionally, the writing the same data into the source data source and the destination data source synchronously in a serial manner includes:

writing the same data to the source data source;

if the writing is successful, writing the same data into the target data source when the writing of the same data into the source data source is completed;

and if the writing fails, refusing to write the same data into the target data source.

Optionally, the method further includes:

determining whether the data read-write service in the third service mode is abnormal;

and if the abnormality exists, switching the third service mode to the first service mode.

A data migration processing apparatus comprising:

the migration unit is used for migrating the data of the source data source to the target data source based on the first service mode; the first service mode is a mode for providing data read-write service by utilizing the source data source;

a first determination unit configured to determine whether a first switching condition is satisfied;

a first switching unit, configured to switch the first service mode to a second service mode if the first switching condition is satisfied; the second service mode is a mode for providing data read-write service by utilizing the source data source and the target data source, and provides service for synchronously writing the same data into the source data source and the target data source;

a second determination unit configured to determine whether a second handover condition is satisfied; the data read-write service is not abnormal in the second service mode at least;

a second switching unit, configured to switch the second service mode to a third service mode if the second switching condition is met, where the third service mode is a mode in which the destination data source is used to provide data read-write service;

a third switching unit, configured to switch the second service mode to the first service mode if the second switching condition is not satisfied.

An electronic device, comprising:

a memory for storing a set of computer instructions;

a processor for implementing the method of any preceding claim by executing a set of instructions stored on the memory.

A computer readable storage medium having stored therein a set of computer instructions which, when executed by a processor, implement a method as in any above.

According to the data migration processing method, the data migration processing device, the electronic equipment and the storage medium, after data of a source data source are migrated to a target data source, data read-write service is not directly switched from the source data source to the target data source, a second service mode is adopted as buffering/transition, the second service mode is a mode for providing data read-write service for double-end data sources based on a source end and a target end, test running detection on whether the target data source is abnormal in service is supported, double-write service is provided for the second service mode, and the same data can be synchronously written into the double-end data sources. When the service is abnormal (does not meet the second switching condition) in the second service mode and the source data source needs to be switched back to the source data source for service rollback, the source data source does not have data loss due to the fact that full data is written in the source data source, synchronization of incremental data generated after migration is not needed to be performed from the target data source, and therefore the service can be effectively switched back to the source data source for timely loss stopping, and influences on data read-write service during data migration are reduced.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

FIG. 1 is a schematic diagram of processing logic for a data migration process in the prior art;

fig. 2 is a schematic flowchart of a data migration processing method according to an embodiment of the present application;

fig. 3 is another schematic flow chart of a data migration processing method provided in an embodiment of the present application;

FIG. 4 is a schematic diagram of processing logic of a data migration process according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of a data migration processing apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a data migration processing apparatus according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.

As shown in fig. 1, in the conventional technology, when data migration processing is performed, data stored in an original database is migrated to a new database, and then online traffic of a data read-write service is switched to the new database when migration is completed, and as a result of research by the inventors, the method has at least the following technical problems:

1) In the data migration process, incremental data is generated in an original database, so that when migration is completed and online traffic of a service is switched to a new database, consistency of the incremental data generated in the data migration process between the new database and the original database cannot be guaranteed, and external data service is easily damaged;

2) After the online flow of the data read-write service is switched to the new database, if a problem is found, the service cannot be switched back to the original database to stop loss in time, and incremental data generated during the service providing period of the new database needs to be synchronized to the original database and then switched back.

In order to solve at least some technical problems in the conventional technology, the application discloses a data migration processing method, a data migration processing device, an electronic device, and a storage medium, which are used for reducing the influence on data read-write services during data migration and realizing a service-lossless data migration processing scheme as far as possible.

Referring to fig. 2, a schematic flow chart of the data migration processing method provided in the embodiment of the present application is shown, where the data migration processing method may be applied to a data read-write service scenario of business object data or metadata, exemplarily, may be applied to a metadata read-write service scenario of a distributed cloud storage environment, and may be specifically applied to a corresponding physical machine deployed in the scenario, where the physical machine may be but is limited to a terminal device such as a mobile phone, a tablet computer, a personal PC (e.g., a notebook, an all-in-one machine, and a desktop computer) having data processing and communication functions, or a corresponding physical machine such as a private cloud/public cloud platform having data processing and communication functions, and a local area network/internet server.

Also, the method may be implemented in a single physical machine or across physical machines, and the processing logic of the method may be implemented in, but is not limited to, a tool, plug-in, or service/microservice form.

As shown in fig. 2, in this embodiment, the data migration processing method includes:

step 201, migrating the data of the source data source to the destination data source based on the first service mode.

The first service mode is a mode for providing data read-write service by using a source data source.

The source data source and the destination data source may be, but are not limited to, a database or a data table.

When the source data source cannot meet the service requirement due to the performance in the aspects of reliability, throughput and the like, the data of the source data source is migrated to a new data source which has stronger function and is more reliable, namely a target data source. During the migration process, an outward data read-write service (i.e., a first service mode) is still provided based on the source data source, so as to avoid service interruption.

In implementation, when data of the source data source needs to be migrated to the destination data source, a migration condition may be further set, and when the migration condition is satisfied, data migration between the source data source and the destination data source based on the first service mode is triggered.

The migration condition is satisfied, but not limited to, any one of the following:

1) Reaching a set time node;

2) And detecting that the service flow of the data read-write service is reduced to a certain threshold value, and entering a low peak period.

Optionally, the data migration from the source data source to the destination data source may also be triggered based on manual operations.

In this step, migrating the data of the source data source to the destination data source based on the first service mode specifically includes:

1) Migrating stock data of a source data source to a target data source;

the storage data here refers to data existing in the source data source when the data migration is started.

2) If the source data source generates incremental data in the data migration process, the incremental data is migrated to the target data source.

Wherein the incremental data comprises newly written data and/or modified data to the source data source.

In the data migration process, the source data source still provides service outwards, incremental data may be generated, and in order to avoid data inconsistency between the target data source and the source data source and further avoid service loss after the data read-write service is switched to the target data source, the incremental data is also migrated when data migration is performed. In implementation, the incremental data generated by the source data source in the data migration process can be migrated to the destination data source for multiple times in a circulating manner until the data of the destination data source and the data of the source data source are consistent at the current moment, so as to ensure the synchronization of the incremental data generated again in the incremental data migration process to the destination data source.

Step 202, determining whether a first switching condition is met.

Optionally, if the source data source does not generate incremental data in the data migration process, step 202 may be implemented as: verifying the consistency of the stock data in a source data source and a target data source; if the stock data are consistent between the source data source and the target data source, determining that a first switching condition is met; otherwise, if the two switching conditions are not consistent, the first switching condition is not met.

If the source data source generates incremental data in the data migration process, step 202 may be implemented as:

verifying the consistency of stock data in a source data source and a target data source;

verifying the consistency of the incremental data in the source data source and the target data source;

if the stock data are consistent in the source data source and the target data source, and the incremental data are consistent in the source data source and the target data source, determining that a first switching condition is met;

In an implementation, preferably, when migration of the stock data in the source data source is completed, consistency check of the migrated stock data in the source data source and the destination data source may be performed in real time, and the check mode may be a mode of comparing data records in the source data source and the destination data source one by one, or a mode of performing Digest calculation on data records in the source data source and the destination data source respectively according to a predetermined Digest Algorithm (Message-Digest Algorithm) and comparing Digest values, which is not limited herein. If the verification shows that the stock data is inconsistent between the source data source and the destination data source, determining inconsistent data records between the destination end and the source end based on the verification process, and correcting corresponding data at the destination end by taking the data of the source data source as a reference until the stock data is consistent between the destination end and the source end.

Similarly, if incremental data is generated in the source data source in the data migration process, when the migration of the incremental data is completed, consistency verification of the migrated incremental data between the source data source and the destination data source can be performed in real time by adopting any one of the verification methods. The checksum of the stock data and the migration of the incremental data may be performed in series or in parallel, which is not limited in this embodiment.

Step 203, if the first switching condition is satisfied, switching the first service mode to the second service mode.

The second service mode is a mode for providing data read-write service by using a source data source and a destination data source, and the mode provides service for synchronously writing the same data into the source data source and the destination data source.

After judgment, if the first switching condition is met, it indicates that the migration of stock data and incremental data in the source data source is completed, and the data in the target data source is consistent with the data in the source data source, at this time, in response to the first switching condition being met, the application does not directly switch the data read-write service from the source data source to the target data source (i.e., switch to providing the data read-write service by using the target data source), but switches the original first service mode for providing the data read-write service by using the source data source to a second service mode for providing the data read-write service by using both the source data source and the target data source.

The second service mode is essentially an intermediate mode between a first service mode for providing data read-write service by using the source data source and a third service mode for providing data read-write service by using the destination data source, and is used for providing a buffering/transition effect for the switching process from the first service mode to the third service mode.

The second service mode provides a double-write service, which can synchronously write the same data into a double-end data source, specifically, for each piece of business object data generated by an upper layer business or each piece of metadata generated by a metadata service for providing metadata support for the upper layer business, in the second service mode, based on a preset synchronization strategy, the piece of business object data or the metadata is written into both a source data source and a destination data source.

The second service mode may be implemented, but is not limited to, as any one of the following:

1) A first sub-mode is included;

in this implementation, the second service mode only includes the first sub-mode, and the first sub-mode is a mode for providing data read-write service by using the source data source and providing data write service by using the destination data source.

2) A second sub-mode is included;

in this implementation, the second service mode only includes a second sub-mode, and the second sub-mode is a mode for providing data writing service by using the source data source and providing data reading and writing service by using the destination data source.

3) The first sub-mode and the second sub-mode are included.

In this implementation, the second service mode includes both the first sub-mode and the second sub-mode.

And under the condition that the second service mode comprises the first sub-mode and the second sub-mode, the data read-write service is in the first sub-mode and the second sub-mode based on the strategy time sharing. That is, at a particular time, the data read and write service is in a corresponding particular mode of the two sub-modes based on the policy.

Corresponding to the above several implementation manners of the second service mode, in this step 203, when the first service mode is switched to the second service mode in response to the first switching condition being met, the following may be implemented:

switching the first service mode to the first sub-mode;

or, switching the first service mode to the second sub-mode;

or, switching the first service mode to the first sub-mode; and determining whether a third switching condition is met, and if the third switching condition is met, further switching the first sub-mode into the second sub-mode.

Under the condition that the second service mode is only realized to comprise the first sub-mode, when the data read-write service is in the mode, the data is synchronously written into the data sources of the source end and the destination end based on the double-write mode, and the data read service is provided by using the source data source, so that the mode supports the detection of whether the service is abnormal or not for the write service provided by the destination data source; similarly, in the case that the second service mode is implemented to include only the second sub-mode, when the data read-write service is in the mode, the detection of whether the read-write service provided by the target data source is abnormal or not is supported; under the condition that the second service mode is realized to comprise the first sub-mode and the second sub-mode, when the data read-write service is in the mode, whether the service abnormality of the data read-write service provided by the target data source is detected is supported.

For the case that the second service mode is implemented to include the first sub-mode and the second sub-mode, the third switching condition is specifically a condition that at least the data read-write service in the first sub-mode is abnormal.

For example, optionally, the third switching condition is set as: and detecting that the data read-write service is not abnormal under the first sub-mode.

Or, optionally, the third switching condition is set as: and detecting that the data read-write service is not abnormal in the first sub-mode, and writing the data of the source data source and the data of the destination data source based on the first sub-mode to be consistent. Based on the set third switching condition, in implementation, after the first Service mode is switched to the first sub-mode, service state information such as a sla (Service Level Agreement) state code of the data read-write Service in the first sub-mode may be monitored and obtained, and whether the Service is abnormal or not may be determined according to the Service state information, for example, whether the sla state code distribution of the Service is normal or not may be determined, so as to determine whether the Service is abnormal or not. If the service is not abnormal, at least verifying whether the data written into the source data source and the target data source in the first sub-mode are consistent or not based on a mode of comparing data records one by one or a mode of calculating abstract values of the data records and comparing the abstract values, and if so, switching the first sub-mode into a second sub-mode; otherwise, if the service is abnormal in the first sub-mode and/or the data written in the source data source is inconsistent with the data written in the destination data source, the first sub-mode is not switched to the second sub-mode.

And step 204, determining whether a second switching condition is met.

And the second switching condition is met, and the data read-write service is not abnormal in at least a second service mode. For example, the second switching condition may be set to: detecting that the data read-write service is not abnormal in the second service mode; alternatively, the second switching condition is set to: and detecting that the data read-write service is not abnormal in the second service mode, and writing the data of the source data source and the data of the destination data source are consistent based on the second service mode.

Aiming at the condition that the second switching condition is set to detect that the data read-write service is not abnormal in the second service mode, if the second service mode only comprises the first sub-mode, service state information such as the sla state code of the data read-write service in the first sub-mode can be monitored and obtained, whether the data read-write service is abnormal in the first sub-mode is judged based on the obtained service state information, and if the data read-write service is not abnormal, the second switching condition is met; otherwise, the condition is not satisfied; similarly, if the second service mode only includes the second sub-mode, it may be determined whether the second switching condition is satisfied based on the detection of whether the data read-write service in the second sub-mode is abnormal or not; and if the second service mode comprises a first sub-mode and a second sub-mode, correspondingly determining whether a second switching condition is met based on the detection of whether the data read-write service is abnormal under the first sub-mode and the second sub-mode.

Aiming at the situation that the second switching condition is set to detect that the data read-write service is not abnormal in the second service mode and the data written into the source data source and the destination data source are consistent based on the second service mode, wherein:

if the second service mode includes only the first sub-mode, it may be determined whether the second switching condition is satisfied by: determining whether the data read-write service is abnormal in the first sub-mode; and at least verifying whether the data written into the source data source and the destination data source based on the first sub-mode are consistent; if the data read-write service under the first sub-mode is not abnormal and the data written into the source data source and the target data source based on the first sub-mode are consistent, determining that a second switching condition is met; otherwise, if the data read-write service under the first sub-mode is abnormal and/or the data written into the source data source and the target data source based on the first sub-mode are inconsistent, the second switching condition is not satisfied.

Similarly, if the second service mode includes only the second sub-mode, it may be determined whether the second switching condition is satisfied by: determining whether the data read-write service is abnormal in the second sub-mode; at least verifying whether the data written into the source data source and the destination data source based on the second sub-mode are consistent; if the data read-write service under the second sub-mode is not abnormal and the data written into the source data source and the target data source based on the second sub-mode are consistent, determining that a second switching condition is met; otherwise, if the data read-write service under the second sub-mode is abnormal and/or the data written into the source data source and the target data source based on the second sub-mode are inconsistent, determining that the second switching condition is not met.

If the second service mode includes the first sub-mode and the second sub-mode, it may be determined whether the second switching condition is satisfied by:

if the third switching condition is not met (the third switching condition is not met, and the data reading and writing service is abnormal under the first sub-mode at least can be indicated), directly judging that the second switching condition is not met;

if the third switching condition is met and the first sub-mode is switched to the second sub-mode in response to the third switching condition being met, then: determining whether the data read-write service is abnormal in the second sub-mode; and at least verifying whether the data written into the source data source and the destination data source based on the second sub-mode are consistent; if the data read-write service under the second sub-mode is not abnormal and the data written into the source data source and the target data source based on the second sub-mode are consistent, determining that a second switching condition is met; otherwise, if the data read-write service is abnormal in the second sub-mode and/or the data written into the source data source and the target data source based on the second sub-mode are inconsistent, determining that the second switching condition is not met.

In implementation, an operation duration (for example, one day, two days, etc.) may be set for the second service mode in advance, the service state of the data read-write service is monitored in a time period corresponding to the operation duration, and if the service in the time period is not abnormal, consistency check is further performed on the data written into the dual-end data source in the dual-write service period. And for the second service mode comprising the first sub-mode and the second sub-mode, the second service mode can be subdivided into a plurality of operation time periods respectively set for the first sub-mode and the second sub-mode, and the first sub-mode and the second sub-mode are respectively monitored for service states and checked for data on the basis of no abnormal service in the corresponding operation time periods.

In this embodiment, at least verifying whether data written into the source data source and the destination data source based on a certain mode (e.g., the first sub-mode and the second sub-mode) are consistent may refer to performing consistency check on the full data of the destination data source based on the full data of the source data source (the method includes performing consistency check on data written into the source end and the destination end data source in the mode), or may also refer to performing consistency check only on data written into the source end and the destination end data source based on the dual write service in the mode in a targeted manner, which is not limited in this embodiment.

Step 205, if the second switching condition is met, switching the second service mode to a third service mode, where the third service mode is a mode for providing data read-write service by using the destination data source;

if the second switching condition is met, it is at least indicated that the data read-write service in the second service mode is not abnormal, and since the second service mode is a mode for providing data service based on the source data source and the destination data source at the same time, a trial operation process is essentially provided for the destination data source, so that it can be further indicated that the corresponding data service (such as write service or read-write service) provided by the destination data source in the trial operation stage is not abnormal.

In this case, in response to the second switching condition being satisfied, the application switches the second service mode to a third service mode that provides the data read-write service by using the destination data source, so as to complete the switching of the data read-write service from the source data source to the destination data source.

Specifically, if the second service mode only includes the first sub-mode, the first sub-mode is switched to the third service mode in response to the second switching condition being satisfied; if the second service mode only comprises the second sub-mode, responding to the second switching condition, and switching the second sub-mode into a third service mode; and if the second service mode comprises the first sub-mode and the second sub-mode, responding to the second switching condition being met, and switching the second sub-mode to a third service mode.

Step 206, if the second switching condition is not satisfied, the second service mode is switched to the first service mode.

Conversely, if the second switching condition is not satisfied, for example, the second switching condition is not satisfied due to an abnormal data read-write service in the second service mode or inconsistency of data written into the source data source and the destination data source based on the second service mode, the second service mode is switched back to the first service mode, that is, the mode for providing the data read-write service by using the source data source and the destination data source is switched back to the mode for providing the data read-write service by using the source data source, so as to implement the service rollback, where the rollback at this time is essentially equivalent to rolling back from the trial run stage of the destination data source to the mode for providing the service based on the source data source.

Specifically, if the second service mode only includes the first sub-mode, in response to that the second switching condition is not satisfied, switching the first sub-mode back to the first service mode for service rollback; if the second service mode only comprises the second sub-mode, responding to that the second switching condition is not met, and switching the second sub-mode back to the first service mode for service rollback; and if the second service mode comprises the first sub-mode and the second sub-mode, responding to that the second switching condition is not met, and switching the first sub-mode or the second sub-mode back to the first service mode for service rollback.

The second service mode is provided with double writing services and is responsible for synchronously writing each incremental data generated after the data migration into the source data source and the target service source, so that the full data of the services are maintained in the source data source, the incremental data generated further after the data migration is not lost, and the second-level switching of the services from the second service mode to the first service mode can be effectively supported.

In the data migration processing method disclosed in this embodiment, after migrating data of a source data source to a destination data source, a data read-write service is not directly switched from the source data source to the destination data source, but a second service mode is used as buffering/transition, the second service mode is a mode for providing data read-write service for a source-end and destination-end-based dual-port data source, and supports test operation detection on whether the destination data source is abnormal in service, and the second service mode provides a dual-write service, and can write the same data into the dual-port data source synchronously. When the service is abnormal (does not meet the second switching condition) in the second service mode and the source data source needs to be switched back to the source data source for service rollback, the source data source does not have data loss due to the fact that full data is written in the source data source, synchronization of incremental data generated after migration is not needed to be performed from the target data source, and therefore the service can be effectively switched back to the source data source for timely loss stopping, and influences on data read-write service during data migration are reduced.

In practical applications, when the second service mode synchronously writes data to the dual-end data source based on the dual-write mode, a write failure may occur in a data source at a certain end due to program code bug, full disk space, interface protocol state problems, and the like, which may easily cause inconsistency of data written to the dual ends in the second service mode, and further easily cause service damage after switching the data read-write service to a target data source.

To solve this problem, when the present application is implemented, it is preferable to set the second switching condition to: and detecting that the data read-write service is not abnormal in the second service mode, and the data written into the source data source and the destination data source based on the second service mode are consistent. Based on the setting, the mode of finally providing the data read-write service based on the target data source can be switched to enter on the premise that the data service (such as the write service or the read-write service) provided by the target data source in the second service mode is not abnormal, and the mode of finally providing the data read-write service based on the target data source can be switched to enter on the premise that the data of the target data source is consistent with the data of the source data source, so that the problem that the data read-write service is damaged when the data read-write service is switched to the target data source due to inconsistency with the data of the source data source can be avoided as much as possible.

In addition, when the data is synchronously written to the double-end data source based on the double-writing service provided by the second service mode, the double-end data writing operation can be carried out in a serial synchronization mode or a parallel synchronization mode.

Preferably, in implementation, a serial synchronization mode can be adopted for double-ended data writing operation. In the serial synchronization mode, firstly, data is written into a source data source, and if the data is successfully written, the data is continuously written into a target data source when the data is written into the source data source; otherwise, if the data fails to be written into the source data source, the data is refused to be written into the destination data source.

Based on the above serial synchronization method, in the dual write service provided based on the second service mode, the case where the data writing from the source data source is allowed to succeed and the data writing from the destination data source is allowed to fail, but the case where the data writing from the source data source is not allowed to fail and the data writing from the destination data source is not allowed to succeed, so that it can be ensured that the data from the source data source can be used as a reference at any time before entering the third service mode (providing data read-write service using the destination data source), thereby supporting the consistency verification of the data written in the destination data source, and supporting the fast (e.g. second level) switchback from the second service mode to the first service mode when the second switching condition is not satisfied.

In an optional embodiment of the present application, referring to a flowchart of the data migration processing method shown in fig. 3, after step 205, the method may further include:

step 207, determining whether the data read-write service in the third service mode is abnormal;

as described above, the third service mode is a mode for providing a data read-write service using a destination data source.

And step 208, if the abnormality exists, switching the third service mode to the first service mode.

After the third service mode is switched to enter, the service state of the data read-write service in the third service mode is further detected, and whether the service is abnormal or not is determined.

If the data read-write service in the third service mode is abnormal, when the third service mode is switched back to the first service mode for service rollback, whether incremental data are generated in a target data source during the service operation in the third service mode needs to be determined, if the incremental data are generated, the incremental data need to be synchronized to the source data source, and then the service rollback operation based on mode switching is executed.

It should be noted that, in the present application, a second service mode is adopted as a transition between switching from a first service mode to a third service mode and correspondingly switching a data read-write service from a source data source to a destination data source, wherein the second service mode simultaneously utilizes the source data source and the destination data source to provide the data read-write service, so that, during the second service mode, a test run is essentially performed on data services (such as write services or read-write services) provided by the destination data source, and the third service mode finally providing the data read-write service based on the destination data source is switched to when no abnormality occurs in the test run, and once the service is abnormal during the test run, the mode is switched back to the mode providing the data read-write service based on the source data source to perform service rollback, so that, if the data read-write service provided by the destination data source is abnormal, a high probability is found in the run stage, and then the data read-write event is repaired, and once the third service mode enters the third service mode, it indicates that the data service provided by the destination data source passes the test run stage, and thus, the data read-write event is only a small probability.

Therefore, compared with the prior art, the method and the device have the advantages that the probability of service rollback in the first service mode due to service exception in the third service mode is greatly reduced, so that the influence on data read-write service during data migration is reduced, and a service-lossless data migration processing scheme is realized as far as possible.

An example of an application of the method of the present application is provided below.

The system architecture diagram of this example is shown in FIG. 4, which includes:

a data service (service) for providing an outward data read-write service, such as a metadata data read-write service, by using a database;

the database A is a source database;

the database B is a target database;

migration tools (tools), which are tools implemented based on the method of the present application, are used to execute processing logic of the method of the present application to perform corresponding data migration processing (switching of services between different databases on the basis of data migration and migration) when migrating data of the database a to the database B.

Referring to fig. 4 in combination, in this example, the flow of performing the data migration process by using the migration tool includes:

1. a writing and A reading: initially, providing a data read-write service (a first service mode) by using a database A;

2. double write, a read (first sub-mode):

21 Migration tool migration of inventory data in database a to database B when migration conditions (e.g., at low peak of service traffic) are met; checking whether the data in the databases A and B are consistent; if the data in the databases A and B are consistent after verification, entering step 22), otherwise, correcting the data in the database B by taking the database A as a reference until the data in the databases A and B are consistent, and entering step 22);

22 Migrating incremental data generated in the database A in the migration process to the database B, and at least checking whether the migrated incremental data are consistent in the databases A and B; if the data in the database A are inconsistent, correcting the data in the database B by taking the database A as a reference until the data in the database A are consistent;

step 22) is executed in a circulating manner until the data in the databases A and B at the current moment are consistent, the mode is switched into a double-writing mode and a double-reading mode, the database A is used for providing reading and writing services, and the database B is used for providing writing services;

23 Observing whether the sla state code distribution of the service is normal or not, if so, rolling back the write service to the database A, namely switching to A write and A read modes; if the data is normal, entering the step 3;

3. double write, B read (second sub-mode):

31 The migration tool takes the database A as a reference, and at least checks whether the data written into the databases A and B in the double-writing and A-reading modes are consistent or not;

32 Step 31) is executed in a circulating way, so that consistency check is performed on incremental data continuously written into the databases A and B at least during the check period, until the data in the databases A and B are consistent, the read service is switched to the database B, the dual-write service to the databases A and B is maintained, and a dual-write mode and a dual-read mode are entered;

33 Observing whether the sla state code distribution of the service is normal or not, if so, rolling back the read-write service to the database A, namely switching to A write mode and A read mode; if so, entering the step 4;

4. b write, B read (third service mode):

41 The migration tool takes the database A as a reference and at least checks whether the data written into the databases A and B in the double-writing mode and the B-reading mode are consistent;

42 Step 41) is executed circularly, so that consistency check is carried out on incremental data which are continuously and doubly written into the databases A and B during the data check period of the step 41) at least, until the data in the databases A and B are consistent, the modes are switched to the writing mode and the reading mode of the database B, and data reading and writing services are provided by using the database B;

43 Observe whether the sla status code distribution of the service is normal, if so, roll back the read-write service to the database A (A write, A read mode), and if not, maintain the mode of providing the data read-write service by using the database B.

In the data migration process, service state detection and data consistency verification are carried out in the whole process, and under the condition that the service state and the consistency verification result meet the conditions, a first sub-mode (double writing and A reading) and a second sub-mode (double writing and B reading) are used as transition, the first service mode which provides read-write service by using the database A is gradually switched to a third service mode which provides read-write service by using the database B, the whole process of data in the double-end database can be guaranteed to be lossless, quick (such as second-level) service rollback to the database A under the condition that the whole process supports service abnormity is achieved, and the influence on the data read-write service during data migration is reduced;

in addition, in the embodiment, the second service mode is set to include the first sub-mode and the second sub-mode, and the first sub-mode and the second sub-mode are switched and controlled in stages based on service state detection and data verification, so that the read service abnormality or the write service abnormality can be effectively detected when the data service provided by the database B is abnormal, and further, defect modification can be conveniently and specifically performed on services with abnormality, such as program code bug of the read service, or interface protocol state problem of the read service and the database B can be specifically repaired.

Corresponding to the above data migration processing method, an embodiment of the present application further discloses a data migration processing apparatus, referring to a schematic structural diagram of the apparatus shown in fig. 5, where the apparatus may include:

a migration unit 501, configured to migrate data of a source data source to a destination data source based on a first service mode; the first service mode is a mode for providing data read-write service by utilizing a source data source;

a first determining unit 502 for determining whether a first switching condition is satisfied;

a first switching unit 503, configured to switch the first service mode to the second service mode if a first switching condition is met; the second service mode is a mode for providing data read-write service by utilizing a source data source and a target data source, and provides service for synchronously writing the same data into the source data source and the target data source;

a second determining unit 504 for determining whether a second handover condition is satisfied; the data read-write service is not abnormal in a second service mode at least;

a second switching unit 505, configured to switch the second service mode to a third service mode if a second switching condition is met, where the third service mode is a mode in which a destination data source is used to provide data read-write service;

a third switching unit 506, configured to switch the second service mode to the first service mode if the second switching condition is not satisfied.

In an optional implementation manner of the embodiment of the present application, the migration unit 501 is specifically configured to:

migrating stock data of a source data source to a target data source;

In an optional implementation manner of the embodiment of the present application, if a source data source generates incremental data in a data migration process, the first determining unit 502 is specifically configured to:

verifying the consistency of the incremental data in a source data source and a target data source;

if the stock data is consistent between the source data source and the destination data source, and the incremental data is consistent between the source data source and the destination data source, determining that a first switching condition is met;

In an optional implementation manner of the embodiment of the present application, the second service mode includes a first sub-mode and/or a second sub-mode;

the first sub-mode is a mode for providing data reading and writing services by using a source data source and providing data writing services by using a destination data source, and the second sub-mode is a mode for providing data writing services by using the source data source and providing data reading and writing services by using the destination data source;

the first switching unit 503 is specifically configured to:

switching the first service mode into a first sub-mode;

or, the first service mode is switched to a second sub-mode;

or, the first service mode is switched to a first sub-mode; and determining whether a third switching condition is met, and if the third switching condition is met, switching the first sub-mode to the second sub-mode.

In an optional implementation manner of the embodiment of the present application, if the second service mode includes the first sub-mode, the second determining unit 504 is specifically configured to:

determining whether the data read-write service is abnormal in the first sub-mode; and at least verifying whether the data written into the source data source and the destination data source based on the first sub-mode are consistent;

if the data read-write service under the first sub-mode is not abnormal and the data written into the source data source and the target data source based on the first sub-mode are consistent, determining that a second switching condition is met;

In an optional implementation manner of the embodiment of the present application, if the second service mode includes a second sub-mode, the second determining unit 504 is specifically configured to:

determining whether the data read-write service is abnormal in the second sub-mode; and at least verifying whether the data written into the source data source and the destination data source based on the second sub-mode are consistent;

if the data read-write service under the second sub-mode is not abnormal and the data written into the source data source and the target data source based on the second sub-mode are consistent, determining that a second switching condition is met;

In an optional implementation manner of the embodiment of the present application, if the second service mode includes the first sub-mode and the second sub-mode;

the second determining unit 504 is specifically configured to:

determining whether the data read-write service is abnormal in the second sub-mode; at least verifying whether the data written into the source data source and the target data source based on the second sub-mode are consistent;

if the data read-write service is abnormal in the second sub-mode and/or the data written in the source data source and the target data source based on the second sub-mode are inconsistent, determining that the second switching condition is not met;

wherein, satisfying the third switching condition includes: and the data read-write service under the first sub-mode is not abnormal, and the data written into the source data source and the destination data source based on the first sub-mode is consistent.

In an optional implementation manner of the embodiment of the present application, the service provided by the second service mode and used for synchronously writing the same data into the source data source and the destination data source is specifically configured to:

synchronously writing the same data into a source data source and a destination data source based on the following serial modes:

writing the same data into a source data source;

if the writing is successful, writing the same data into a target data source when the writing of the same data into a source data source is completed;

if the writing fails, the same data is refused to be written into the destination data source.

In an alternative implementation manner of the embodiment of the present application, referring to fig. 6, the apparatus may further include:

a third determining unit 507, configured to determine whether the data read-write service in the third service mode is abnormal;

the third switching unit 508 is configured to switch the third service mode to the first service mode if the data read-write service in the third service mode is abnormal.

The data migration processing apparatus disclosed in the embodiment of the present application is relatively simple in description because it corresponds to the data migration processing method disclosed in the above corresponding method embodiment, and for the relevant similarities, please refer to the description of the above corresponding method embodiment, and detailed description is omitted here.

The embodiment of the present application further discloses an electronic device, as shown in the schematic structural diagram of the electronic device shown in fig. 7, the electronic device at least includes:

a memory 701 for storing a set of computer instructions;

the set of computer instructions described above may be embodied in a computer program.

The memory 701 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The processor 702 is configured to implement the data migration processing method according to any one of the above method embodiments by executing the instruction set stored in the memory.

The processor 702 may be a Central Processing Unit (CPU), an application-specific integrated circuit (ASIC), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device.

In addition, the electronic device may further include a communication interface, a communication bus, and the like. The memory, the processor and the communication interface communicate with each other via a communication bus.

The communication interface is used for communication between the electronic device and other devices. The communication bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like, and may be divided into an address bus, a data bus, a control bus, and the like.

In this embodiment, after migrating data of a source data source to a destination data source by executing a computer instruction set stored in a memory, a processor in an electronic device does not directly switch a data read-write service from the source data source to the destination data source, but adopts a second service mode as buffering/transition, where the second service mode provides a data read-write service mode for a source-end and destination-end-based dual-port data source, and supports test run detection on whether a service is abnormal for the destination data source, and the second service mode provides a dual-write service, and can write the same data into the dual-port data source synchronously. When the service is abnormal (does not meet the second switching condition) in the second service mode and the source data source needs to be switched back to the source data source for service rollback, the source data source does not have data loss due to the fact that full data is written in the source data source, synchronization of incremental data generated after migration is not needed to be performed from the target data source, and therefore the service can be effectively switched back to the source data source for timely loss stopping, and influences on data read-write service during data migration are reduced.

In addition, an embodiment of the present application further discloses a computer-readable storage medium, in which a computer instruction set is stored, and when the computer instruction set is executed by a processor, the data migration processing method disclosed in any method embodiment above is implemented.

The computer readable storage medium is based on a computer instruction set stored in the computer readable storage medium, after data of a source data source is migrated to a target data source, a data read-write service is not directly switched from the source data source to the target data source, but a second service mode is adopted as buffering/transition, the second service mode provides a mode of data read-write service for a double-end data source based on a source end and a target end, test running detection on whether the target data source is abnormal in service is supported, the second service mode provides double-write service, and the same data can be synchronously written into the double-end data source. When the service is abnormal (does not meet the second switching condition) in the second service mode and the source data source needs to be switched back to the source data source for service rollback, the source data source does not have data loss due to the fact that full data is written in the source data source, synchronization of incremental data generated after migration is not needed to be performed from the target data source, and therefore the service can be effectively switched back to the source data source for timely loss stopping, and influences on data read-write service during data migration are reduced.

It should be noted that, in this specification, each embodiment is described in a progressive manner, and each embodiment focuses on differences from other embodiments, and portions that are the same as and similar to each other in each embodiment may be referred to.

For convenience of description, the above system or apparatus is described as being divided into various modules or units in terms of functions, respectively. Of course, the functionality of the various elements may be implemented in the same one or more pieces of software and/or hardware in the practice of the present application.

From the above description of the embodiments, it is clear to those skilled in the art that the present application can be implemented by software plus a necessary general hardware platform. Based on such understanding, the technical solutions of the present application may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments of the present application.

Finally, it is further noted that, herein, relational terms such as first, second, third, fourth, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising one of 8230; \8230;" 8230; "does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.

The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A data migration processing method is characterized by comprising the following steps:

determining whether a first switching condition is satisfied;

determining whether a second handover condition is satisfied; the data read-write service is not abnormal in the second service mode at least;

2. The method of claim 1, wherein migrating the data of the source data source to the destination data source comprises:

migrating the stock data of the source data source to the target data source;

3. The method of claim 2, wherein if the source data source generates incremental data during the data migration process, the determining whether a first switching condition is satisfied comprises:

4. The method of claim 1, wherein the second service mode comprises a first sub-mode and/or a second sub-mode;

the switching the first service mode to a second service mode includes:

switching the first service mode to the first sub-mode;

or, the first service mode is switched to the second sub-mode;

or, switching the first service mode to the first sub-mode; and determining whether a third switching condition is met, and if the third switching condition is met, switching the first sub-mode to the second sub-mode.

5. The method of claim 4, wherein if the second service mode comprises the first sub-mode, the determining whether a second handover condition is satisfied comprises:

determining whether the data read-write service is abnormal under the first sub-mode; and at least verifying whether the data written into the source data source and the destination data source based on the first sub-mode are consistent;

if the data read-write service under the first sub-mode is not abnormal, and the data written into the source data source and the target data source based on the first sub-mode is consistent, determining that the second switching condition is met;

6. The method of claim 4, wherein if the second service mode comprises the second sub-mode, the determining whether a second handover condition is satisfied comprises:

7. The method of claim 4, wherein if the second service mode comprises the first sub-mode and the second sub-mode;

the determining whether a second handover condition is satisfied includes:

determining whether the data read-write service is abnormal under the second sub-mode; and at least verifying whether the data written into the source data source and the destination data source based on the second sub-mode are consistent;

wherein, satisfying the third handover condition comprises: and the data read-write service under the first sub-mode is not abnormal, and the data written into the source data source and the target data source based on the first sub-mode is consistent.

8. The method of claim 1, wherein the synchronously writing the same data to the source data source and the destination data source comprises:

9. The method of claim 8, wherein said writing the same data to the source data source and the destination data source synchronously in a serial manner comprises:

writing the same data to the source data source;

10. The method of claim 1, further comprising:

11. A data migration processing apparatus, comprising:

a second determination unit configured to determine whether a second handover condition is satisfied; wherein, satisfying the second switching condition, at least including the data read-write service in the second service mode is not abnormal;

12. An electronic device, comprising:

a memory for storing a set of computer instructions;

a processor for implementing the method of any one of claims 1-10 by executing a set of instructions stored on the memory.

13. A computer-readable storage medium having stored therein a set of computer instructions which, when executed by a processor, implement the method of any one of claims 1-10.