CN116466894A

CN116466894A - Data migration method and device, storage medium and electronic equipment

Info

Publication number: CN116466894A
Application number: CN202310477431.6A
Authority: CN
Inventors: 郑迪; 李伟仁; 马思雨; 冯宇东
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2023-04-27
Filing date: 2023-04-27
Publication date: 2023-07-21

Abstract

The invention discloses a data migration method, a data migration device, a storage medium and electronic equipment. Relates to the technical field of big data. Wherein the method comprises the following steps: responding to a data migration request, and establishing a data link backup between an application server and first storage equipment; switching the data reading positions of M service nodes from a second storage device to a first storage device, and writing first data generated by an application server into the second storage device and the first storage device in a first double-writing mode, wherein the M service nodes are nodes in the application server; switching the first double-writing mode into a second double-writing mode, and writing second data generated by the application server into the first storage device and the second storage device; and cutting off the data link between the application server and the second storage device, and determining that the data migration is completed. The invention solves the technical problem that when the data of the old equipment is migrated to the new equipment, the service is required to be interrupted, so that the continuity of the equipment service is poor.

Description

Data migration method and device, storage medium and electronic equipment

Technical Field

The present invention relates to the field of big data technologies, and in particular, to a data migration method, a device, a storage medium, and an electronic apparatus.

Background

With the rapid development of electronic banking financial business, the number of users is rapidly increased, and when the performance and storage capacity of an application system cannot meet the business development, system upgrading and capacity expansion are required to meet new business requirements.

In the related art, during the process of device upgrade and capacity expansion, a shutdown window needs to be set, and service is interrupted, so that data of an old storage device is migrated to a new storage device, and the old storage device providing data storage service for an application server is switched to the new storage device, but setting the shutdown window and the service is influenced by user use. Meanwhile, if a system fault occurs, data can be lost, rapid emergency cannot be realized, and service is not available for a long time.

In view of the above problems, no effective solution has been proposed at present.

Disclosure of Invention

The embodiment of the invention provides a data migration method, a data migration device, a storage medium and electronic equipment, which at least solve the technical problem that when data of old equipment is migrated to new equipment, service is required to be interrupted, so that continuity of equipment service is poor.

According to an aspect of an embodiment of the present invention, there is provided a data migration method, including: responding to a data migration request, and establishing a data link between an application server and first storage equipment, wherein the data migration request is used for requesting to migrate data stored in second storage equipment to the first storage equipment, and the second storage equipment and the first storage equipment are used for providing data storage service for the application server; switching data reading positions of M service nodes from a second storage device to a first storage device, and writing first data generated by the application server into the second storage device and the first storage device by adopting a first double-writing mode, wherein the M service nodes are nodes in the application server, each service node is used for providing service for N users, and the first double-writing mode is that the data is written into the second storage device first and then is written into the first storage device, and M and N are positive integers; switching the first double-writing mode into a second double-writing mode, and writing second data generated by the application server into the first storage device and the second storage device, wherein the second double-writing mode is that the data is written into the first storage device and then written into the second storage device; and cutting off the data link between the application server and the second storage device, and determining that the data migration is completed.

Further, switching the data reading positions of the M service nodes from the second storage device to the first storage device includes: switching the data reading positions of S users in a target service node from a second storage device to a first storage device, wherein S is a positive integer smaller than N, and M service nodes comprise: the target service node; switching data reading positions of T users in a target service node from a second storage device to a first storage device, wherein the T users represent users except for S users in N users, and T is a positive integer smaller than N; f service node data reading positions are switched to a first storage device from a second storage device, wherein F represents service nodes except the target service node in M service nodes, and F is a positive integer smaller than M.

Further, the data migration method further comprises the following steps: in the process of switching the data reading positions of M service nodes from the second storage device to the first storage device, judging whether the data flow for accessing the first storage device is larger than a preset threshold value; judging whether the performance index data of the first storage device exceeds a preset index threshold value or not under the condition that the data flow of the first storage device for access is larger than the preset threshold value; and continuing data migration under the condition that the performance index data does not exceed the preset index threshold value.

Further, the data migration method further comprises the following steps: detecting whether data with writing failure exists, wherein the data with writing failure comprises the following steps: the first data is not written with the successful data of the first storage device; if the writing failure data exists, sending first alarm prompt information, and sending the writing failure data to a message queue; and writing the data in the message queue into the first storage device.

Further, after writing the first data generated by the application server to the second storage device and the first storage device, and/or after writing the second data generated by the application server to the first storage device and the second storage device, the method further includes: detecting consistency of stored data in the first storage device and the second storage device; and sending second alarm prompt information under the condition that the stored data in the first storage device and the second storage device are inconsistent.

Further, before establishing the data link between the application server and the first storage device, it includes: introducing target data in a first storage device to a target server in batches during a target time period, wherein the target time period comprises: the data flow of the service processed by the application server is smaller than a preset threshold, and the target data comprises: data stored in the second storage device prior to a target time; and importing the target data in the target server into a first storage device.

Further, importing the target data in the target server into a first storage device, including: verifying the target data in the target server to obtain a verification result, wherein verifying the target data in the target server comprises: checking whether the target data in the target server has data loss or not, and checking whether the data format of the target data in the target server meets a preset format or not; and under the condition that the verification result indicates that verification passes, importing the target data in the target server into the first storage device.

According to another aspect of the embodiment of the present invention, there is also provided a data migration apparatus, including: the method comprises the steps of establishing a link unit, wherein the link unit is used for responding to a data migration request, and establishing a data link between an application server and first storage equipment, wherein the data migration request is used for requesting to migrate data stored in second storage equipment to the first storage equipment, and the second storage equipment and the first storage equipment are used for providing data storage service for the application server; the first processing unit is used for switching data reading positions of M service nodes from a second storage device to a first storage device, and writing first data generated by the application server into the second storage device and the first storage device by adopting a first double-writing mode, wherein the M service nodes are nodes in the application server, each service node is used for providing service for N users, and the first double-writing mode is that the data is written into the second storage device first and then is written into the first storage device, and M and N are positive integers; the second processing unit is used for switching the first double-writing mode into a second double-writing mode and writing second data generated by the application server into the first storage device and the second storage device, wherein the second double-writing mode is that the data is written into the first storage device and then written into the second storage device; and the third processing unit is used for cutting off the data link between the application server and the second storage device and determining that the data migration is completed.

Further, the first processing unit includes: a first switching subunit, configured to switch, from a second storage device to a first storage device, data reading positions of S users in a target service node, where S is integer data smaller than N, and M service nodes include: the target service node; the second switching subunit is used for switching the data reading positions of T users in the target service node from the second storage device to the first storage device, wherein the T users represent users except the S users in the N users, and T is a positive integer smaller than N; and the third switching subunit is used for switching F service node data reading positions from the second storage device to the first storage device, wherein F represents service nodes except the target service node in M service nodes, and F is a positive integer smaller than M.

Further, the data migration apparatus further includes: the first judging unit is used for judging whether the data flow of accessing the first storage device is larger than a preset threshold value or not in the process of switching the data reading positions of the M service nodes from the second storage device to the first storage device; a second judging unit, configured to judge whether performance index data of the first storage device exceeds a preset index threshold value, if a data flow rate of accessing the first storage device is greater than the preset threshold value; and the migration unit is used for continuing data migration under the condition that the performance index data does not exceed the preset index threshold value.

Further, the data migration apparatus further includes: a first detection unit, configured to detect whether there is data with a write failure, where the data with the write failure includes: the first data is not written with the successful data of the first storage device; the first sending unit is used for sending first alarm prompt information and sending the writing failure data to a message queue under the condition that the writing failure data exists; and the writing unit is used for writing the data in the message queue into the first storage device.

Further, the data migration apparatus further includes: a second detecting unit configured to detect consistency of stored data in the first storage device and the second storage device after writing first data generated by the application server to the second storage device and the first storage device, and/or after writing second data generated by the application server to the first storage device and the second storage device; and the second sending unit is used for sending second alarm prompt information under the condition that the stored data in the first storage device and the second storage device are inconsistent.

Further, the data migration apparatus further includes: a first importing unit, configured to batch import, in a target period, target data in a first storage device to a target server before a data link between the application server and the first storage device is established, where the target period includes: the data flow of the service processed by the application server is smaller than a preset threshold, and the target data comprises: data stored in the second storage device prior to a target time; and the second importing unit is used for importing the target data in the target server into the first storage device.

Further, the second importing unit includes: the verification subunit is configured to verify the target data in the target server to obtain a verification result, where verifying the target data in the target server includes: checking whether the target data in the target server has data loss or not, and checking whether the data format of the target data in the target server meets a preset format or not; and the importing subunit is used for importing the target data in the target server into the first storage device under the condition that the verification result indicates that the verification is passed.

According to another aspect of the embodiment of the present invention, there is also provided an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the data migration method of any one of the above via execution of the executable instructions.

According to another aspect of the embodiments of the present invention, there is also provided a computer readable storage medium storing a computer program, where the computer program when executed controls a device in which the computer readable storage medium is located to perform the data migration method of any one of the above.

In the method, a data migration request is responded, a data link between an application server and first storage equipment is established, wherein the data migration request is used for requesting data stored in second storage equipment to be migrated to the first storage equipment, and the second storage equipment and the first storage equipment are used for providing data storage service for the application server; switching data reading positions of M service nodes from a second storage device to a first storage device, and writing first data generated by an application server into the second storage device and the first storage device by adopting a first double-writing mode, wherein the M service nodes are nodes in the application server, each service node is used for providing services for N users, the first double-writing mode is that the data is written into the second storage device first and then written into the first storage device, and M and N are positive integers; switching the first double-writing mode into a second double-writing mode, and writing second data generated by the application server into the first storage device and the second storage device, wherein the second double-writing mode is that the data is written into the first storage device and then written into the second storage device; and cutting off the data link between the application server and the second storage device, and determining that the data migration is completed. And further, the technical problem that when the data of the old equipment is migrated to the new equipment, the service is required to be interrupted, so that the continuity of the equipment service is poor is solved. In the invention, the data reading positions of a plurality of service nodes of the application server are gradually switched to the first storage device by adopting a double-writing mode in the process of switching the storage device of the application service from the second storage device to the first storage device, so that the technical effects that when data of old equipment is migrated to new equipment in the related technology, the service is required to be interrupted, the continuity of the equipment providing service is poor, and the condition of data loss easily occurs are avoided, thereby realizing the technical effects of reducing the loss rate of the data in the data migration process and improving the continuity of the storage device providing the data access service.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:

FIG. 1 is a flow chart of an alternative data migration method according to an embodiment of the present invention;

FIG. 2 is a flow chart of an alternative all user read new device dual write old and new device (write old then write new) in accordance with an embodiment of the present invention;

FIG. 3 is a flow chart of an alternative all user read new device dual write new and old device (write new then write old) in accordance with an embodiment of the present invention;

FIG. 4 is a flow chart of an alternative white list read new device dual write old device in accordance with an embodiment of the present invention;

FIG. 5 is a flow chart of an alternative new device to old device data ensuring strong consistency (write new then write old) in accordance with an embodiment of the present invention;

FIG. 6 is a flow chart of an alternative old device and new device data ensuring strong consistency (write old then write new) in accordance with an embodiment of the present invention;

FIG. 7 is a flow chart of an alternative data bulk export and re-import according to an embodiment of the present invention;

FIG. 8 is a schematic diagram of an alternative data migration apparatus according to an embodiment of the present invention;

fig. 9 is a schematic diagram of an electronic device according to an embodiment of the invention.

Detailed Description

In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

It should be noted that, the data migration method and the apparatus thereof in the present disclosure may be used in the case of switching an old storage device to a new storage device in the field of financial technology, and may also be used in any field other than the field of financial technology, where the performance adjustment method and the apparatus thereof in the present disclosure are not limited in the application field of the transaction system in the case of switching an old storage device to a new storage device.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, displayed data, service data, etc.) referred to in the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.

Example 1

In accordance with an embodiment of the present invention, an alternative method embodiment of a data migration method is provided, it being noted that the steps illustrated in the flowchart of the figures may be performed in a computer system, such as a set of computer executable instructions, and, although a logical order is illustrated in the flowchart, in some cases, the steps illustrated or described may be performed in an order other than that illustrated herein.

FIG. 1 is a flow chart of an alternative data migration method according to an embodiment of the present invention, as shown in FIG. 1, the method comprising the steps of:

step S101, a data link between an application server and a first storage device is established in response to a data migration request, wherein the data migration request is used for requesting migration of data stored in a second storage device to the first storage device, and the second storage device and the first storage device are used for providing data storage services for the application server.

The data migration request may be used to request migration of data stored in the second storage device to the first storage device, for example: when the performance of the application system and the storage capacity of the storage device cannot meet the service development, the system is required to be upgraded, and the capacity of the storage device is expanded to meet new service requirements. During the process of upgrading and expanding the storage device, the old storage device (corresponding to the second storage device) may need to be switched to the new storage device, and the data stored in the old storage device is migrated to the new storage device, so as to meet the requirements of normal data reading and data storage.

The application server may be a server for processing a service of a user, and the application server may read data from a storage device and store the data to the storage device, and when performing data migration, may first establish a data link between the application server and the first storage device.

The application server may read-only write to the old storage device (i.e., the second storage device) before or during the establishment of the data link between the application server and the first storage device.

In this embodiment, after adding a data source connection (corresponding to the data link described above) maintained to the database of the new storage device (i.e., the first storage device), a health check probe may be performed to verify the link connectivity of the application server to the data link of the new storage device, and the availability of the new storage device, in preparation for the application server to access the database of the new storage device for the next phase, wherein verifying the availability of the new storage device may include: verifying whether the database of the new storage device is normally accessible to the application server, and verifying network connectivity between the application server and the new storage device.

Step S102, switching the data reading positions of M service nodes from a second storage device to a first storage device, and writing first data generated by an application server into the second storage device and the first storage device by adopting a first double-writing mode, wherein the M service nodes are nodes in the application server, each service node is used for providing service for N users, and the first double-writing mode is that the data is written into the second storage device first and then written into the first storage device, wherein M and N are positive integers.

The above-mentioned M service nodes may be nodes in an application server, each service node may be configured to provide services for N users, in a process of switching a data reading position of the M service nodes from a second storage device to a first storage device, a part of users in a part of nodes may be selected first to perform a test (i.e. an internal test stage), the part of users are set to read data from the first storage device, then all users in the part of nodes are set to read data from the first storage device gradually, and finally all users in the M service nodes are set to read data from the first storage device gradually.

The first data may be storage data that is generated by the application server and needs to be stored in a process that the application server switches the data reading positions of the M service nodes from the second storage device to the first storage device.

In the process of switching the data reading positions of the M service nodes from the second storage device to the first storage device, the data (corresponding to the first data) which is generated by the application server and needs to be stored can be written into the first storage device and the second storage device, and the data which needs to be stored can be written into the second storage device first during double writing, and then written into the first storage device, namely written into the old storage device first and then written into the new storage device.

Fig. 2 is a flowchart of an optional dual write new and old device (write old before write new) for all users in M service nodes (e.g., gray service node and non-gray service node in fig. 2) of an application server, as shown in fig. 2, all users in M service nodes of the application server read the new device (i.e., new storage device), and data of the service nodes in the application server are dual written to the new and old devices, and the old device is written before the new device is written when the new device is written, wherein numerals 1 and 2 in fig. 2 indicate the sequence of writing data to the new and old devices, 1 indicates write before write, and 2 indicates write before, specifically, non-white list user and white list user in the non-gray service node and gray service node write to the old device before write to the new device.

Step S103, switching the first double-writing mode to a second double-writing mode, and writing second data generated by the application server into the first storage device and the second storage device, wherein the second double-writing mode is that the data is written into the first storage device and then written into the second storage device.

The second data may be data that needs to be stored and is generated by the application server in the process of writing data by adopting the second double-writing mode.

In this embodiment, after the data reading positions of the M service nodes are switched from the second storage device to the first storage device, and in the case that the application server can normally read data from the first storage device, the first dual-writing mode may also be switched to the second dual-writing mode, and in this stage, the second dual-writing mode may be adopted to write the first data to be stored generated by the application service into the first storage device and the second storage device, that is, the first data to be stored may be written into the first storage device first, then written into the second storage device, that is, written into the new storage device first, and then written into the old storage device.

Fig. 3 is a flowchart of an optional dual write new and old device (write new and write old) for all users in M service nodes (such as gray service node and non-gray service node in fig. 3) of an application server, as shown in fig. 3, all users in M service nodes read new devices (i.e. new storage devices), and data of service nodes in the application server are dual written to new and old storage devices, write new device and write old device first, (numerals 1 and 2 in fig. 3 indicate the sequence of writing data to new and old devices, 1 indicates write first, 2 indicates write then, and in particular, non-white list users and white list users in non-gray service node and gray service node write new device first and write old device later).

For example: all the flow reads data from the new storage device, writes the new and old storage devices twice, writes the new storage device first, writes the new storage device data successfully, and then writes the new storage device data into the old storage device (the old storage device may not guarantee the success of writing). At this point, the new storage device may act as the primary data source, but strong consistency of the old device with the new device still needs to be guaranteed in order to rollback.

And step S104, cutting off the data link between the application server and the second storage device, and determining that the data migration is completed.

Under the condition that second data generated by the application server can be normally written into the first storage device, data link between the application server and the second storage device can be cut off, the device which is determined to be the data storage service of the application server is formally switched into the first storage device, the data of the second storage device is formally migrated to the first storage device, namely the application server formally switches data access to the new storage device, connection between the application server and the old storage device can be not maintained any more, and the old storage device is formally disconnected.

Through the steps, in the embodiment, in the process of switching the storage device of the application service from the second storage device to the first storage device, the data reading positions of the plurality of service nodes of the application server are gradually switched to the first storage device step by step in a double-writing mode, so that the problem that the continuity of the service provided by the device is poor and the data loss is easy to occur due to the fact that the service is required to be interrupted when the data of the old device is migrated to the new device in the related technology is avoided, the technical effects of reducing the data loss rate in the data migration process and improving the continuity of the data access service provided by the storage device are realized. And further, the technical problem that when the data of the old equipment is migrated to the new equipment, the service is required to be interrupted, so that the continuity of the equipment service is poor is solved.

Optionally, switching the data reading positions of the M service nodes from the second storage device to the first storage device includes: switching the data reading positions of S users in the target service node from the second storage device to the first storage device, wherein S is a positive integer smaller than N, and M service nodes comprise: a target service node; switching data reading positions of T users in a target service node from a second storage device to a first storage device, wherein the T users represent users except S users in N users, and T is a positive integer smaller than N; and switching F service node data reading positions from the second storage device to the first storage device, wherein F represents service nodes except for the target service node in M service nodes, and F is a positive integer smaller than M.

For example: the gray white list user (corresponding to the above S users) is configured to read data from the new storage device, and can write the new and old storage devices (keep the data of the new and old devices consistent) when modifying (i.e. when storing data). In other cases (non-gray-scale white list users), data may be read from the old storage device (second storage device), double writing to the new and old storage devices. The double writing is to write old storage equipment first, and write new storage equipment (first storage equipment) after the old storage equipment data is written successfully, and this stage can be called an internal test point stage, and gray white list users account for 5% and the latest data of the old storage equipment is guaranteed preferentially. Assuming that the date at this stage is S, delta data between T-S is imported into the new storage device. And the consistency of the reference data of the new and old devices is ensured.

Fig. 4 is a flowchart of an alternative new device double writing new and old devices for whitelist reading, as shown in fig. 4, in an internal test stage, a whitelist user reads data from the new device, a non-whitelist user reads data from the old device, and the data writing sequence of all users in all service nodes is that the old device is written first and then the new device is written, wherein numerals 1 and 2 in fig. 4 indicate the data writing sequence, 1 indicates that the old device is written first and 2 indicates that the new device is written later, the non-whitelist user and the whitelist user in the non-gray service node and the gray service node all read data from the new device, and the non-whitelist user and the whitelist user in the non-gray service node write data into the old device first and then the new device.

After passing the internal test point stage, the gray-scale white list users can be gradually expanded to users of all gray-scale service nodes (corresponding to the target service nodes), and finally the users can be promoted to databases of all service node users which read new storage devices, and double-write new and old devices (such as the double-write new and old devices (write old and new after write old) of all users in fig. 2). The old storage device is written first at this stage, and the new storage device is written after the old storage device is written successfully (the online transaction of the new library can not be guaranteed to be written successfully), and the latest database data of the old storage device is guaranteed at this time so as to perform the switchback at any time.

By gradually switching the data reading positions of M service nodes of the application server to the first storage device, the condition that a long-time terminal service is needed when the storage device is switched in the related technology is avoided, and the technical effect of improving the continuity of service in the process of data migration from the old storage device to the new storage device is achieved.

Optionally, the data migration method further includes: in the process of switching the data reading positions of M service nodes from the second storage device to the first storage device, judging whether the data flow for accessing the first storage device is larger than a preset threshold value or not; judging whether the performance index data of the first storage device exceeds a preset index threshold value or not under the condition that the data flow of the first storage device for accessing is larger than the preset threshold value; and continuing data migration under the condition that the performance index data does not exceed the preset index threshold value.

In this embodiment, in the process of switching the data reading positions of the M service nodes from the second storage device to the first storage device, as the flow of the application server accessing the database of the first storage device increases, it may be verified whether the performance capacity of the new library can meet the requirement, that is, whether the data flow of accessing the first storage device is greater than a preset threshold, and if not, no processing may be performed; and if the performance index data of the first storage device exceeds the preset index threshold, continuing data migration, and if the performance index data of the first storage device does not exceed the preset index threshold, performing alarm prompt, wherein the performance index data can include, but is not limited to, response speed, memory occupancy rate and the like of the first storage device when the data of the first storage device is read.

In this embodiment, since the traffic is finally switched to the new storage device, this stage needs to ensure strong consistency of the data of the new and old devices, and the consistency of the data of the new and old storage devices can be verified. Preventing the new device from losing data.

Fig. 5 is a flowchart of an alternative method for ensuring strong consistency (writing new and then writing old) between new and old device data, according to an embodiment of the present invention, where, as shown in fig. 5, ensuring consistency between new and old device data may be performed by alarming and writing to a message queue, monitoring by a monitoring server, and performing near real-time synchronization based on the message server.

Numerals 1 to 5 in fig. 5 indicate the sequence of data processing of data, specifically, the data of the application server is written into the new device first, the old device is written after the writing is successful (such as the writing is successful in 1 in fig. 5), in the case of the writing failure (such as the writing failure in 2 in fig. 5) of the old device, an alarm (such as the alarm in 3 in fig. 5) can be sent to the monitoring server, the data of the writing failure can be written into the message queue (such as the writing message queue in 4 in fig. 5) and pushed to the message server, and after the writing failure data is written into the message server, the writing failure data can be synchronized to the old device in near real time (such as the near real time synchronization in fig. 5) through the message server.

In order to ensure strong consistency of data in the new device and the old device, as shown in fig. 5, day-to-day batch synchronization may also be performed on the new device and the old device, the day-to-day update data in the new device (e.g., the 1 day-to-day update data in fig. 5) may be exported to a batch server, and then checked by the batch server, and imported into the old device (e.g., the check import in fig. 5).

By monitoring the data flow and performance index of the access to the new storage device, the technical effect of improving the safety of the process data of the data migration and the reliability of the data access service provided by the new storage device is achieved.

Optionally, the data migration method further includes: detecting whether there is data with a write failure, wherein the data with the write failure comprises: the first data is not written with the successful data of the first storage device; under the condition that writing-in failure data exists, sending first alarm prompt information, and sending the writing-in failure data to a message queue; and writing the data in the message queue into the first storage device.

In this embodiment, in the process of writing the first data generated by the application server into the second storage device and the first storage device by adopting the first double-writing manner, if the first data is successfully written into the second storage device, whether the first data has data which is not successfully written into the first storage device or not may also be detected, and if the first data has data which is not successfully written into the first storage device, the first alarm prompt message may be sent, and the data which is not successfully written into the message queue may be sent, and then the data in the message queue may be written into the first storage device.

For example: when the writing of the data in the old library (namely, the database of the second storage device) is successful and the writing of the data in the new library (namely, the database of the first storage device) is failed, the application server can send an alarm (for example, send a first alarm prompt message) to the monitoring server, timely inform an operation and maintenance manager, and then send the data which is failed to be written into a message cluster (corresponding to the message queue) which can write the data into the new library in near real time. Meanwhile, the method can also carry out the daily end batch spam, derive the data updated by the old storage equipment (the second storage equipment) on the same day, check whether the data in the new storage equipment is up to date, and if the data in the new storage equipment is not up to date, the data updated by the old storage equipment on the same day can be imported into the new storage equipment in batch so as to update the data stored in the storage equipment, thereby realizing the technical effect of improving the reliability and the effectiveness of the data in the new storage equipment.

Optionally, after writing the first data generated by the application server to the second storage device and the first storage device, and/or after writing the second data generated by the application server to the first storage device and the second storage device, the method further includes: detecting consistency of stored data in the first storage device and the second storage device; and sending second alarm prompt information under the condition that the stored data in the first storage device and the second storage device are inconsistent.

In this embodiment, after the first data generated by the application server is written to the second storage device and the first storage device, and after the second data generated by the application server is written to the first storage device and the second storage device, consistency of the stored data in the first storage device and the second storage device may also be detected; and under the condition that the data stored in the first storage device and the data stored in the second storage device are inconsistent, a second alarm prompt message can be sent to prompt relevant operation and maintenance management personnel to process.

For example: and after the first data generated by the application server are written into the second storage device and the first storage device, sending an alarm prompt under the condition that the stored data in the first storage device and the second storage device are inconsistent, and updating the data stored in the first storage device through the stored data in the second storage device.

After writing the second data generated by the application server to the first storage device and the second storage device, an alert may be sent in the event of a non-coincidence of the stored data in the first storage device and the second storage device.

Fig. 6 is a flowchart of an alternative method for ensuring strong consistency (writing old and writing new) between old device and new device data according to an embodiment of the present invention, where, as shown in fig. 6, ensuring consistency between new device and old device data may be performed by alarming and writing to a message queue, monitoring by a monitoring server, and performing near real-time synchronization based on the message server, and numerals 1 to 5 in fig. 6 represent a sequence of data processing of data.

Numerals 1 to 5 in fig. 6 indicate the sequence of data processing of data, specifically, the data of the application server is written into the old device first, the new device is written after the writing is successful (for example, the writing is successful 1 in fig. 6), in the case that the writing of the new device fails (for example, the writing is failed 2 in fig. 6), an alarm (for example, the alarm 3 in fig. 6) can be sent to the monitoring server, the data of the writing failure can also be written into the message queue (for example, the writing message queue 4 in fig. 6) and pushed to the message server, and after the writing of the data of the writing failure into the message server, the writing failure data can also be synchronized to the new device in near real time (for example, the near real time synchronization in fig. 6) through the message server.

As shown in fig. 6, in order to ensure strong consistency of data in the new device and the old device, day-to-day batch synchronization may also be performed on data in the old device and the new device, the day-to-day update data in the old device (e.g., the 1 day-to-day update data in fig. 6) may be exported to a batch server, and then checked by the batch server, and the new device may be imported (e.g., the check import in fig. 6).

By detecting the nature of the data stored in the new storage device and the old storage device, the technical effects of reducing the error rate of the data in the storage device and improving the reliability of the data stored in the storage device are achieved.

Optionally, before establishing the data link between the application server and the first storage device, the method comprises: the method comprises the steps of importing target data in a first storage device into a target server in batches in a target time period, wherein the target time period comprises the following steps of: the data flow of the service processed by the application server is smaller than a preset threshold, and the target data comprises: data stored in the second storage device prior to the target time; and importing the target data in the target server into the first storage device.

The target time may be a time at which the establishment of the data link between the application server and the first storage device is started. After the new storage device is ready, the inventory data in the old storage device (second storage device) may be imported into the new storage device (first storage device) before the application server connects the new storage device. Since the old storage device is still providing online business access service for the application server, in order to avoid affecting online transactions, the data stored in the old storage device may be migrated to the new storage device in a batch export-to-import manner during non-business peak periods (corresponding to the target period described above).

FIG. 7 is a flow chart of an alternative data bulk export and re-import, according to an embodiment of the present invention, as shown in FIG. 7, where user data in an old device may be exported to a bulk server (corresponding to the target server described above) while being imported to a new device.

For example: the data before the T day can be exported to the batch server if the operation date is T, the batch server can import the data to the new storage device after the data verification is carried out, the batch server can verify the data, the integrity of the data, namely whether the data is lost or not, whether the data format is accurate or not and the like can be verified, and the technical effect of reducing the data loss rate in the data migration process is achieved.

Optionally, importing the target data in the target server into the first storage device includes: verifying the target data in the target server to obtain a verification result, wherein verifying the target data in the target server comprises: checking whether the target data in the target server have data loss or not, and checking whether the data format of the target data in the target server meets a preset format or not; and under the condition that the verification result indicates that the verification is passed, importing the target data in the target server into the first storage device.

For example: the data before the T day can be exported to a batch server (corresponding to the target server), after the batch server performs data verification, the data can be imported to a new storage device (corresponding to the first storage device), the batch server performs data verification, the integrity of the data can be verified, namely whether the data is lost or not, whether the data format is accurate or not can be verified, and the like, so that the technical effect of improving the reliability of the data in the new storage device is achieved.

In this embodiment, the data migration process may be controlled by a parameter dataSwitchFlag (preset parameter identifier), which may be dynamically modified in real time by the distributed configuration center. The data migration stage can be retracted, and the old storage equipment is switched back to provide data reading and data storage services for the application server, so that the data migration risk is ensured to be controllable.

The storage device for storing data in this embodiment may be a database or a cache, for example: redis, etc., through this embodiment, can ensure that the application system provides uninterrupted, high availability, the financial service level of no loss and quality of service, feel free to user's use, do not influence user experience to but also the test point, but real-time back cut reaches the risk controllability of data migration. And when the new storage device and the old storage device are switched to the new storage device, smooth transition of data is ensured, and data reading is that the storage service is continuously available.

Example two

The second embodiment provides an optional data migration apparatus, where each implementation unit in the data migration apparatus corresponds to each implementation step in the first embodiment.

Fig. 8 is a schematic diagram of an alternative data migration apparatus according to an embodiment of the present invention, the data migration apparatus further including: a link unit 81, a first processing unit 82, a second processing unit 83, and a third processing unit 84 are established.

Specifically, the link establishment unit 81 is configured to establish a data link between the application server and the first storage device in response to a data migration request, where the data migration request is used to request migration of data stored in the second storage device to the first storage device, and the second storage device and the first storage device are used to provide a data storage service for the application server;

the first processing unit 82 is configured to switch data reading positions of M service nodes from the second storage device to the first storage device, and write first data generated by the application server into the second storage device and the first storage device by adopting a first double-writing manner, where the M service nodes are nodes in the application server, each service node is configured to provide services for N users, and the first double-writing manner is that data is written into the second storage device first and then written into the first storage device, where M and N are positive integers;

a second processing unit 83, configured to switch the first dual-writing mode to a second dual-writing mode, and write second data generated by the application server into the first storage device and the second storage device, where the second dual-writing mode is that the data is written into the first storage device first and then written into the second storage device;

The third processing unit 84 is configured to cut off the data link between the application server and the second storage device, and determine that the data migration is completed.

In the data migration apparatus provided in the second embodiment of the present application, a data link between an application server and a first storage device may be established by establishing a link unit 81 in response to a data migration request, where the data migration request is used to request migration of data stored in a second storage device to the first storage device, the second storage device and the first storage device are used to provide data storage services for the application server, the data reading positions of M service nodes are switched from the second storage device to the first storage device by a first processing unit 82, and a first dual-writing mode is adopted to write first data generated by the application server into the second storage device and the first storage device, where M service nodes are nodes in the application server, each service node is used to provide services for N users, the first dual-writing mode is that data is written into the second storage device first, where M and N are positive integers, the first dual-writing mode is switched to the second dual-writing mode by a second processing unit 83, the second data generated by the application server is written into the first storage device and the second storage device, and the second dual-writing mode is cut off, and the data migration processing is performed between the second storage device and the first storage device by the second processing unit 84. And further, the technical problem that when the data of the old equipment is migrated to the new equipment, the service is required to be interrupted, so that the continuity of the equipment service is poor is solved. In this embodiment, in the process of switching the storage device of the application service from the second storage device to the first storage device, a double-writing manner is adopted to gradually switch the data reading positions of a plurality of service nodes of the application server to the first storage device, so that the technical effects that when data of old equipment is migrated to new equipment in the related technology, the service is required to be interrupted, the continuity of the equipment providing service is poor, and the situation of data loss is easy to occur are avoided, thereby realizing the technical effects of reducing the loss rate of data in the data migration process and improving the continuity of the storage device providing the data access service.

Optionally, in the data migration apparatus provided in the second embodiment of the present application, the first processing unit includes: a first switching subunit, configured to switch data reading positions of S users in the target service node from the second storage device to the first storage device, where S is integer data smaller than N, and the M service nodes include: a target service node; the second switching subunit is used for switching the data reading positions of T users in the target service node from the second storage device to the first storage device, wherein the T users represent users except the S users in the N users, and T is a positive integer smaller than N; and the third switching subunit is used for switching the F service node data reading positions from the second storage device to the first storage device, wherein F represents service nodes except the target service node in M service nodes, and F is a positive integer smaller than M.

Optionally, in the data migration apparatus provided in the second embodiment of the present application, the data migration apparatus further includes: the first judging unit is used for judging whether the data flow of accessing the first storage device is larger than a preset threshold value or not in the process of switching the data reading positions of the M service nodes from the second storage device to the first storage device; the second judging unit is used for judging whether the performance index data of the first storage device exceeds a preset index threshold value or not under the condition that the data flow for accessing the first storage device is larger than the preset threshold value; and the migration unit is used for continuing data migration under the condition that the performance index data does not exceed the preset index threshold value.

Optionally, in the data migration apparatus provided in the second embodiment of the present application, the data migration apparatus further includes: a first detection unit, configured to detect whether there is data with a write failure, where the data with the write failure includes: the first data is not written with the successful data of the first storage device; the first sending unit is used for sending first alarm prompt information and sending the data with the writing failure to the message queue under the condition that the data with the writing failure exists; and the writing unit is used for writing the data in the message queue into the first storage device.

Optionally, in the data migration apparatus provided in the second embodiment of the present application, the data migration apparatus further includes: a second detecting unit configured to detect consistency of stored data in the first storage device and the second storage device after writing first data generated by the application server to the second storage device and the first storage device, and/or after writing second data generated by the application server to the first storage device and the second storage device; and the second sending unit is used for sending the second alarm prompt information under the condition that the stored data in the first storage device and the second storage device are inconsistent.

Optionally, in the data migration apparatus provided in the second embodiment of the present application, the data migration apparatus further includes: a first importing unit, configured to batch import, in a target period, target data in a first storage device to a target server before a data link between the application server and the first storage device is established, where the target period includes: the data flow of the service processed by the application server is smaller than a preset threshold, and the target data comprises: data stored in the second storage device prior to the target time; and the second importing unit is used for importing the target data in the target server into the first storage device.

Optionally, in the data migration apparatus provided in the second embodiment of the present application, the second importing unit includes: the verification subunit is configured to verify target data in the target server to obtain a verification result, where verifying the target data in the target server includes: checking whether the target data in the target server have data loss or not, and checking whether the data format of the target data in the target server meets a preset format or not; and the importing subunit is used for importing the target data in the target server into the first storage device under the condition that the verification result indicates that the verification is passed.

The data migration apparatus may further include a processor and a memory, where the link establishment unit 81, the first processing unit 82, the second processing unit 83, the third processing unit 84, and the like are stored as program units, and the processor executes the program units stored in the memory to implement corresponding functions.

The processor includes a kernel, and the kernel fetches a corresponding program unit from the memory. The kernel can be provided with one or more than one, and the data reading positions of a plurality of service nodes of the application server are gradually switched to the first storage device by adopting a double-writing mode in the process of switching the storage device of the application service from the second storage device to the first storage device by adjusting kernel parameters, so that the problem that the continuity of the service provided by the device is poor and the data loss occurs easily due to the fact that the service is required to be interrupted when the data of the old device is migrated to the new device in the related technology is avoided, and the technical effects of reducing the data loss rate in the data migration process and improving the continuity of the data access service provided by the storage device are realized.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), which includes at least one memory chip.

Fig. 9 is a schematic diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 9, an embodiment of the present invention provides an electronic device 90, where the electronic device includes a processor, a memory, and a program stored on the memory and executable on the processor, and the processor implements a data migration method according to any one of the above when executing the program.

The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

In the foregoing embodiments of the present invention, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed technology content may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, for example, may be a logic function division, and may be implemented in another manner, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.

Claims

1. A method of data migration, comprising:

responding to a data migration request, and establishing a data link between an application server and first storage equipment, wherein the data migration request is used for requesting to migrate data stored in second storage equipment to the first storage equipment, and the second storage equipment and the first storage equipment are used for providing data storage service for the application server;

switching data reading positions of M service nodes from a second storage device to a first storage device, and writing first data generated by the application server into the second storage device and the first storage device by adopting a first double-writing mode, wherein the M service nodes are nodes in the application server, each service node is used for providing service for N users, and the first double-writing mode is that the data is written into the second storage device first and then is written into the first storage device, and M and N are positive integers;

switching the first double-writing mode into a second double-writing mode, and writing second data generated by the application server into the first storage device and the second storage device, wherein the second double-writing mode is that the data is written into the first storage device and then written into the second storage device;

And cutting off the data link between the application server and the second storage device, and determining that the data migration is completed.

2. The data migration method according to claim 1, wherein switching data reading positions of the M service nodes from the second storage device to the first storage device comprises:

switching the data reading positions of S users in a target service node from a second storage device to a first storage device, wherein S is a positive integer smaller than N, and M service nodes comprise: the target service node;

switching data reading positions of T users in a target service node from a second storage device to a first storage device, wherein the T users represent users except for S users in N users, and T is a positive integer smaller than N;

f service node data reading positions are switched to a first storage device from a second storage device, wherein F represents service nodes except the target service node in M service nodes, and F is a positive integer smaller than M.

3. The data migration method of claim 1, further comprising:

In the process of switching the data reading positions of M service nodes from the second storage device to the first storage device, judging whether the data flow for accessing the first storage device is larger than a preset threshold value;

judging whether the performance index data of the first storage device exceeds a preset index threshold value or not under the condition that the data flow of the first storage device for access is larger than the preset threshold value;

and continuing data migration under the condition that the performance index data does not exceed the preset index threshold value.

4. The data migration method of claim 1, further comprising:

detecting whether data with writing failure exists, wherein the data with writing failure comprises the following steps: the first data is not written with the successful data of the first storage device;

if the writing failure data exists, sending first alarm prompt information, and sending the writing failure data to a message queue;

and writing the data in the message queue into the first storage device.

5. The data migration method according to claim 1, characterized by further comprising, after writing the first data generated by the application server to the second storage device and the first storage device, and/or after writing the second data generated by the application server to the first storage device and the second storage device:

Detecting consistency of stored data in the first storage device and the second storage device;

and sending second alarm prompt information under the condition that the stored data in the first storage device and the second storage device are inconsistent.

6. The data migration method of claim 1, comprising, prior to establishing the data link between the application server and the first storage device:

introducing target data in a first storage device to a target server in batches during a target time period, wherein the target time period comprises: the data flow of the service processed by the application server is smaller than a preset threshold, and the target data comprises: data stored in the second storage device prior to a target time;

and importing the target data in the target server into a first storage device.

7. The data migration method of claim 6, wherein importing the target data in the target server into a first storage device comprises:

verifying the target data in the target server to obtain a verification result, wherein verifying the target data in the target server comprises: checking whether the target data in the target server has data loss or not, and checking whether the data format of the target data in the target server meets a preset format or not;

And under the condition that the verification result indicates that verification passes, importing the target data in the target server into the first storage device.

8. A data migration apparatus, comprising:

the method comprises the steps of establishing a link unit, wherein the link unit is used for responding to a data migration request, and establishing a data link between an application server and first storage equipment, wherein the data migration request is used for requesting to migrate data stored in second storage equipment to the first storage equipment, and the second storage equipment and the first storage equipment are used for providing data storage service for the application server;

the first processing unit is used for switching data reading positions of M service nodes from a second storage device to a first storage device, and writing first data generated by the application server into the second storage device and the first storage device by adopting a first double-writing mode, wherein the M service nodes are nodes in the application server, each service node is used for providing service for N users, and the first double-writing mode is that the data is written into the second storage device first and then is written into the first storage device, and M and N are positive integers;

The second processing unit is used for switching the first double-writing mode into a second double-writing mode and writing second data generated by the application server into the first storage device and the second storage device, wherein the second double-writing mode is that the data is written into the first storage device and then written into the second storage device;

and the third processing unit is used for cutting off the data link between the application server and the second storage device and determining that the data migration is completed.

9. A computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and wherein the computer program, when executed, controls a device in which the computer readable storage medium is located to perform the data migration method according to any one of claims 1 to 7.

10. An electronic device comprising one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data migration method of any of claims 1-7.