WO2024021554A1

WO2024021554A1 - Data migration method and device

Info

Publication number: WO2024021554A1
Application number: PCT/CN2023/074115
Authority: WO
Inventors: 吕洋; 景磊; 蔡庆明; 陈雪; 张旭
Original assignee: 京东科技信息技术有限公司
Priority date: 2022-07-29
Filing date: 2023-02-01
Publication date: 2024-02-01
Also published as: CN115639953A

Abstract

The present invention discloses a data migration method and device, which relate to the technical field of computers. The method comprises: switching a read-write address of new data from a data source end to a data destination end; launching a double-end read operation and reading new data from the data source end and the data destination end; in response to a current data migration state belonging to a first state, writing the new data to the data destination end in a serialized write mode; turning off the serialized write mode after a preset duration, and writing the new data to the data destination end; and migrating stored data from the data source end to the data destination end.

Description

Data migration method and device

Cross-references to related applications

This application claims priority to the Chinese patent application filed on July 29, 2022 with application number 202210911723.1 and the invention title "Data Migration Method and Device", the entire content of which is incorporated into this application by reference.

Technical field

The present disclosure relates to the field of computer technology, and specifically to data migration methods and devices.

Background technique

Cloud computing is a computing service. As a cloud storage service provider, its storage system is constantly being developed, upgraded or updated. When switching between old and new storage systems, the reliability of data reading and writing services is crucial. When switching between old and new storage systems, the existing data migration method is to migrate the existing data from the old storage system to the new storage system, and mirror the newly generated data read and write operations back to the source from the new storage system to the old storage system. Realize the continuity of data reading and writing services during the switching process between the old and new systems.

However, existing data migration methods suffer from poor reliability.

Contents of the invention

The present disclosure provides a data migration method, device, electronic device, and computer-readable storage medium.

In one or more embodiments of the present disclosure, a data migration method is provided, including: switching the read and write addresses of new data from the data source end to the data destination end, where the new data refers to the data generation end at Data generated during the data migration process; start a double-ended read operation to read new data from the data source and data destination; in response to the current data migration status being the first state, write the new data based on the serial writing method Enter the data destination; turn off the serialized writing mode after the preset time and write new data to the data destination; migrate the existing data from the data source to the data destination.

In one or more embodiments of the present disclosure, a data migration device is provided, including: a switching unit configured to switch the read and write addresses of new data from the data source end to the data destination end, wherein the new data Refers to the data generated by the data generation end during the data migration process; the double-ended reading unit is configured to enable a double-ended reading operation and reads new data from the data source and data destination; the first writing unit is is configured to write new data to the data destination based on the serialized writing method in response to the current data migration status belonging to the first status; the second writing unit is configured to turn off the serialized writing method after a preset period of time , writes new data to the data destination; the data migration unit is configured to migrate existing data from the data source to the data destination.

In one or more embodiments of the present disclosure, an electronic device is provided, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores information that can be The instructions executed by the at least one processor are executed by the at least one processor, so that the at least one processor can implement the method in any of the above embodiments.

In one or more embodiments of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to implement any of the above embodiments. Methods.

In one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, can implement the method in any of the above embodiments.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. in:

Figure 1 is an exemplary system architecture diagram in which embodiments of the present disclosure may be applied;

Figure 2 is a flow chart of an embodiment of a data migration method according to the present disclosure;

Figure 3 is a flow chart of another embodiment of a data migration method according to the present disclosure;

Figure 4 is a flow chart of an application scenario in the data migration method according to the present disclosure;

Figure 5 is another flow chart of an application scenario in the data migration method according to the present disclosure;

Figure 6 is a schematic flowchart of data writing in an application scenario in the data migration method according to the present disclosure;

Figure 7 is a schematic structural diagram of an embodiment of a data migration device according to the present disclosure;

FIG. 8 is a block diagram of an electronic device used to implement the data migration method according to an embodiment of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the present disclosure are included to facilitate understanding and should be considered to be exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

FIG. 1 shows an exemplary system architecture 100 to which embodiments of the data migration method or data migration apparatus of the present disclosure can be applied.

As shown in Figure 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is a medium used to provide communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

Users can use terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, and 103 may be user terminal devices, on which various client applications may be installed, such as data migration software, data maintenance applications, image applications, video applications, search applications, and financial applications. Applications etc.

The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting receiving server messages, including but not limited to smartphones, tablet computers, e-book readers, electronic players, laptop computers and desktop computers. etc.

The terminal devices 101, 102, and 103 may be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they can be various electronic devices. When the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, multiple software modules used to provide distributed services), or as a single software or software module. There are no specific limitations here.

The server 105 can switch the read and write address of new data from the data source to the data destination, Among them, new data refers to the data generated by the data generation end during the data migration process; the double-end read operation is enabled to read new data from the data source end and the data destination end; in response to the current data migration status belonging to the first state, the New data is written to the data destination based on the serial writing method; after a preset period of time, the serial writing method is turned off and new data is written to the data destination; the existing data at the data source is migrated to the data destination.

It should be noted that the data migration method provided by the embodiment of the present disclosure can be executed by the server 105, and accordingly, the data migration device can be provided in the server 105.

It should be understood that the number of terminal devices, networks and servers in Figure 1 is only illustrative. Depending on implementation needs, there can be any number of end devices, networks, and servers.

The data migration method and device provided by the disclosure include: switching the read and write address of new data from the data source to the data destination; starting a double-end read operation, reading new data from the data source and data destination, and responding When the current data migration state belongs to the first state, new data is written to the data destination based on the serialized writing method; after the preset time period, the serialized writing method is turned off and the new data is written to the data destination; the data is The existing data at the source end is migrated to the data destination end. This solution can be applied to data hot migration scenarios between large-scale heterogeneous storage systems. It can avoid long-tail data migration cycle problems and data consistency problems, and improve the availability and performance of data migration services. and data reliability.

Continuing to refer to FIG. 2 , a process 200 according to one embodiment of the data migration method of the present disclosure is shown, including the following steps:

Step 201: Switch the read-write address of the new data from the data source to the data destination, where the new data refers to the data generated by the data generation end during the data migration process.

In this embodiment, the execution subject of the data migration method (for example, the server shown in Figure 1) can switch the read and write address of the new data from the data source end to the data destination end. Among them, new data refers to data newly generated by the data generation end during the storage end switching process/stage and to be written to the data storage end. The data source refers to the data storage before data migration, and the data destination refers to the new data storage to which the data is to be migrated.

Step 202: Start a double-ended reading operation to read new data from the data source and data destination.

In this embodiment, the double-ended read operation can be enabled so that during the process of data live migration, New data can be read from the data source and data destination. This step does not change the process/operation of writing data, but only updates the process of reading data to double reading.

Step 203: In response to the current data migration state belonging to the first state, new data is written to the data destination based on the serialized writing method.

In this embodiment, if the current data migration state belongs to the first state stage of the storage side switching (that is, switching the read and write address of the new data from the data source to the data destination) stage, the new data can be transferred based on the serial write to the data destination in a standardized writing manner.

Step 204: Turn off the serialized writing mode after a preset period of time and write new data to the data destination.

In this embodiment, after switching the data writing operation to the data destination for a preset time period, the serial writing mode of the data destination can be turned off, and the new data can be written in the system's default way or the conventional way ( For example, parallel writing method, etc.) write data destination.

Step 205: Migrate existing data from the data source to the data destination.

In this embodiment, after switching the read and write operations of new data from the data source to the data destination, the existing data at the data source can be migrated to the data destination to complete all data (existing data and new data/ Incremental data) migration.

The data migration method provided in this embodiment switches the read and write address of new data from the data source to the data destination; starts a double-ended read operation, reads new data from the data source and data destination, and responds to the current The data migration state belongs to the first state. New data is written to the data destination based on the serial writing method; the serial writing method is turned off after the preset period of time, and the new data is written to the data destination; the data source end is Migrate existing data to the data destination. This solution can be applied to large-scale hot data migration scenarios between heterogeneous storage systems. It can avoid long-tail problems in the data migration cycle and data consistency problems, and improve the availability, performance and data of data migration services. reliability.

Optionally, reading new data from the data source and data destination includes: reading new data from the data destination first; in response to no new data being read from the data destination, reading new data from the data source. .

In this embodiment, when performing a double-ended read operation, new data can be read from the data destination first. If no new data is read from the data destination, new data can then be read from the data source. Since in heterogeneous storage systems, the last modification time (LastModified) of data is not trustworthy, for example, a user overwrites and uploads a data to the destination cluster, but the system time of the destination cluster It may be lagging behind the source cluster for some reason. If the last update time of the data is used to determine which side of the data to return to the source or destination, there will be a problem of not being able to read reliable data after the update. Therefore, determine the priority from The data destination reads new data to ensure the consistency of data reading.

Optionally, in response to the current data migration state belonging to the first state, writing the new data to the data destination based on the serialized writing method includes: in response to determining that the first lock exists, determining whether the first lock is valid, wherein the first One lock is used to lock the data writing permission; in response to determining that the first lock is valid, new data is written to the data destination in a serialized writing manner based on the first lock.

In this embodiment, if it is determined that the first lock exists, determine whether the first lock is valid, and if it is determined that the first lock is valid (such as the first lock lease is within the validity period, or the first lock is in an activated state), Write new data to the data destination based on serialized writing.

This embodiment performs data writing operations based on state locks, which can improve the security of the system and facilitate the maintenance of the operating sequence of the system.

Continuing to refer to FIG. 3 , a process 300 according to another embodiment of the data migration method of the present disclosure is shown, including the following steps:

Step 301: Switch the read and write address of the new data from the data source to the data destination.

Step 302: Start a double-end read operation to read new data from the data source and data destination, where the new data refers to the data generated by the data generation end during the data migration process.

Step 303: In response to the current data migration state belonging to the second state, new data is written to the data source based on the serialized writing method.

In this embodiment, if the current data migration state belongs to the second state of the storage-side switching (that is, switching the read-write address of the new data from the data source to the data destination) stage, the new data is written based on serialization. Write to the data source in input mode. The first state and the second state belong to different data storage switching states. The switching state may be in the first state or the second state based on instructions from a data migration technician.

Step 304: In response to the current data migration state belonging to the first state, new data is written to the data destination based on the serialized writing method.

Step 305: After a preset period of time, the serialized writing mode is turned off and new data is written to the data destination.

Step 306: Migrate the existing data at the data source to the data destination.

In this embodiment, steps 301, 302, 304, 305 and 306 are The description is consistent with the description of step 201, step 202, step 203, step 204, and step 205, and will not be described again here.

Compared with the embodiment described in Figure 2, the data migration method provided by this embodiment uses different writing ends at different stages of data migration to perform data migration operations in stages, which can improve the reliability of the data migration service.

Optionally, in response to the current data migration state belonging to the second state, writing the new data to the data source based on the serialized writing method includes: in response to determining that the second lock exists, determining whether the second lock is valid, wherein, The second lock is used to lock the data writing permission; in response to determining that the second lock is valid, based on the first lock, new data is written to the data source in a serialized writing manner.

In this embodiment, if it is determined that the second lock exists, it is determined whether the second lock is valid, and if it is determined that the second lock is valid (such as the second lock lease is within the validity period, or the second lock is in an activated state), Write new data to the data source based on serialized writing.

In some optional implementations of the embodiments described above in conjunction with Figures 2 and 3, before starting the double-ended read operation, the data migration method also includes: switching the read and write addresses of the new data from the data source. The switching operation to the data destination is verified; in response to determining that the verification is passed, a backend storage cluster is allocated for the switching operation, where the backend storage cluster includes the data source or the data destination.

In this embodiment, before starting the double-ended read operation, the data migration method also includes performing a verification operation on switching the read and write address of the new data from the data source to the data destination to verify the data migration operation/request. Is it legal or authorized? If it is determined that the verification is passed, a backend storage cluster is allocated for the switch operation.

In some optional implementations of the embodiments described above with reference to Figures 2 and 3, the data migration method also includes: closing the double-ended read operation and reading new data from the data destination.

In this embodiment, after migrating the writing operation of new data to the data destination, the double-ended reading operation can be turned off, and only new data is read from the data destination.

In some application scenarios, data migration can be implemented based on the proxy service. The proxy service can be located under the gateway of the source storage cluster (data source) and at the same level as the source storage cluster and destination storage cluster (data destination). . When performing data migration, by adjusting the gateway The forwarding rules cut user traffic to the proxy service. After the proxy service completes request preprocessing, it dynamically schedules read and write traffic according to the migration stage.

As shown in Figure 4, the migration agent service mainly consists of a preprocessing module, a protocol adaptation module, and a routing and forwarding module. Among them, the preprocessing module performs request context initialization, authenticates data migration requests (authentication/authorization), and audits log output; the protocol adaptation module is used to dispatch requests to the API (Application Programming Interface) corresponding to the proxy layer. interface); the routing and forwarding module dynamically routes different migration tasks to the corresponding back-end storage cluster (data source/data destination) based on the current migration progress.

As shown in Figure 5, the data migration process implemented by the proxy service consists of the following stages. During the migration process, the implementer dynamically adjusts the user's migration status in the proxy service according to the user's migration progress, thereby routing read and write traffic. The entire migration The process not only ensures the availability of the storage service, but also ensures the consistency of the data, and the user side is unaware of the entire migration process:

Reading data source and writing data source: When providing proxy access to users, the forwarding rules of the source cluster gateway need to be changed. For example, take Nginx (a high-performance hypertext transfer protocol and reverse proxy World Wide Web server) as an example. , changing the forwarding rules requires modifying the Nginx configuration file, adjusting the upstream (backend server) attributes, and executing Nginx reload instructions. Since Nginx may be deployed with multiple instances, part of the user traffic may be scheduled to the proxy service, and part of the traffic is still in the source storage cluster. Therefore, the traffic scheduled to the proxy service is first set to be routed to the source storage for both reading and writing. The cluster enables the proxy service to achieve a layer of transparent transmission and avoid risks in data consistency.

Dual-end reading of data and writing of data to the source: After the source-side cluster gateway completes the traffic redirection configuration for the user, the user's traffic status is changed to dual-end of reading data and writing of data to the source. This step does not change the writing The destination of the data only changes the reading data to double reading to avoid the linear consistency problem of not being able to read the data after writing. In the operation of reading data from both ends, the destination cluster is read first, and if the data exists, it is returned. Otherwise, the source cluster is read and the final result is returned directly (data exists or data is not found). In heterogeneous storage systems, the last modification time (LastModified) of data is not trustworthy. For example, a user overwrites and uploads data to the destination cluster, but the system time of the destination cluster may be different from the source cluster for some reason. Lagging behind, if the last update time of the data is used to determine which side of the data to return, there will be a reliability problem that the data cannot be read after the update. Therefore, the destination cluster is read first instead of reading from both ends at the same time.

Reading data from both ends and writing data source based on serialized writing: Turn on serialized writing, as shown in Figure 6. Before writing data, first obtain the resource lock and then write. The first lock is a priority lock. When the data migration phase is in the stage of "reading data from both ends and writing the data source end based on serialized writing", it can be called a left lock (second lock); when the data migration phase is When the stage is in the "double-end reading data and writing data destination based on serial writing method" stage, it is called the right lock (first lock), and the priority of the left lock is smaller than that of the right lock. The specific steps are:

If the data migration phase is in the "dual-end reading of data and writing of data source based on serialized writing", obtain the left lock of the resource to be written. If there is no current lock, or there is a left lock and the lease has expired , then acquire the left lock and write the data to the source. If the current left lock lease has not expired or there is a right lock, this request fails.

If the data migration phase is in the "dual-end reading of data and writing of data destination based on serialized writing", acquire the right lock of the resource to be written. If there is currently no unreleased right lock, acquire the lock and Write data to the destination.

Dual-end reading data and writing data destination: After completing the switching of writing data, wait for all proxy service instances to be ready after the preset time (there are some ongoing data requests that have not been completed during the switching process), you can Serialization is turned off. It is safe to turn off serialization writing at this time, because all nodes write to the destination cluster, and the linear consistency of data within a single system can be guaranteed.

After the above steps are completed, the user's incremental writes (new data generated during the migration process) have been safely migrated to the destination cluster. At this time, the existing data can be migrated. Since the data of the destination cluster can be determined to be the latest, Only data that does not exist in the destination cluster needs to be migrated.

After the user's existing data and incremental writes have been switched to the destination cluster, you can turn off dual reading to switch both reading and writing to the destination cluster. At this point, the entire data migration process is completed and the proxy service completes its task.

This method can be applied to large-scale data hot migration scenarios between heterogeneous storage systems, solving the problem of long-tail migration cycles and data consistency issues, and ensuring service availability, performance and data reliability.

With further reference to Figure 7, as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data migration device. The device embodiment is the same as the method shown in Figures 2 and 3. Corresponding to the embodiment, the device can be applied to various electronic devices.

As shown in FIG. 7 , the data migration device of this embodiment includes: a switching unit 701 , a double-ended reading unit 702 , a first writing unit 703 , a second writing unit 704 , and a data migration unit 705 . Among them, the switching unit is configured to switch the read and write address of new data from the data source end to the data destination end, where the new data refers to the data generated by the data generation end during the data migration process; the double-end reading unit is It is configured to enable a double-ended read operation and read new data from the data source and data destination; the first writing unit is configured to respond to the current data migration state belonging to the first state and write the new data based on serialization. The data destination is written in the input mode; the second writing unit is configured to turn off the serialized writing mode after a preset period of time and write new data to the data destination; the data migration unit is configured to transfer the data from the data source to the data destination. Existing data is migrated to the data destination.

In some embodiments, the double-ended reading unit includes: a first reading module configured to preferentially read new data from the data destination; a second reading module configured to respond to the unread data from the data destination. Get new data and read new data from the data source.

In some embodiments, the first writing unit includes: a first determination module configured to determine whether the first lock is valid in response to determining that the first lock exists, wherein the first lock is used to lock the data writing permission; The first writing module is configured to write new data to the data destination in a serialized writing manner based on the first lock in response to determining that the first lock is valid.

In some embodiments, the data migration device further includes: a third writing unit configured to write new data to the data source based on a serialized writing method in response to the current data migration state belonging to the second state.

In some embodiments, the third writing unit includes: a second determination module configured to determine whether the second lock is valid in response to determining that the second lock exists, wherein the second lock is used to lock the data writing permission; The second writing module is configured to write new data to the data source in a serialized writing manner based on the first lock in response to determining that the second lock is valid.

In some embodiments, before starting the double-ended read operation, the device further includes: verifying the switching operation of switching the read and write address of the new data from the data source to the data destination; in response to determining that the verification is passed, performing the switching operation Allocate a back-end storage cluster, where the back-end storage cluster includes the data source or data destination.

In some embodiments, the data migration device further includes: a shutdown unit configured to close the double-ended read operation and read new data from the data destination.

Each unit in the above-mentioned device 700 corresponds to the steps in the method described with reference to FIG. 2 and FIG. 3 . Therefore, the operations, features and achievable technical effects described above for the data migration method are also applicable to the device 700 and the units included therein, and will not be described again here.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

As shown in FIG. 8 , it is a block diagram of an electronic device 800 according to a data migration method according to an embodiment of the present disclosure. Electronic devices are intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are examples only and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in Figure 8, the electronic device includes: one or more processors 801, memory 802, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be mounted on a common motherboard or otherwise mounted as desired. The processor may process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used with multiple memories and multiple memories, if desired. Likewise, multiple electronic devices can be connected, each device providing part of the necessary operation (eg, as a server array, a set of blade servers, or a multi-processor system). Figure 8 takes a processor 801 as an example.

The memory 802 is the non-transitory computer-readable storage medium provided by the present disclosure. The memory stores instructions executable by at least one processor, so that the at least one processor executes the data migration method provided by the present disclosure. The non-transitory computer-readable storage medium of the present disclosure stores computer instructions, which are used to cause the computer to execute the data migration method provided by the present disclosure.

As a non-transitory computer-readable storage medium, the memory 802 can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions/modules corresponding to the data migration method in the embodiments of the present disclosure (for example, attached Switching unit 701 shown in Figure 7, double-ended Reading unit 702, first writing unit 703, second writing unit 704, data migration unit 705). The processor 801 executes various functional applications and data processing of the server by running non-transient software programs, instructions and modules stored in the memory 802, that is, implementing the data migration method in the above method embodiment.

The memory 802 may include a storage program area and a storage data area, wherein the storage program area may store an operating system and an application program required for at least one function; the storage data area may store data created according to the use of an electronic device for extracting video clips. Data etc. In addition, memory 802 may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 802 optionally includes memory located remotely from processor 801, and these remote memories may be connected via a network to electronic devices used to extract video clips. Examples of the above-mentioned networks include but are not limited to the Internet, intranets, local area networks, mobile communication networks and combinations thereof.

The electronic equipment of the data migration method may also include: an input device 803, an output device 804, and a bus 805. The processor 801, the memory 802, the input device 803 and the output device 804 can be connected through a bus 805 or other means. In Figure 8, the connection through the bus 805 is taken as an example.

The input device 803 may receive input numeric or character information and generate key signal input related to user settings and functional control of the electronic device used to extract the video clip, such as a touch screen, keypad, mouse, trackpad, touch pad, pointer An input device such as a stick, one or more mouse buttons, a trackball, or a joystick. Output devices 804 may include display devices, auxiliary lighting devices (eg, LEDs), tactile feedback devices (eg, vibration motors), and the like. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.

Various implementations of the systems and techniques described herein may be implemented in digital electronic circuitry, integrated circuit systems, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include implementation in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor The processor, which may be a special purpose or general purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device. An output device.

These computing programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine language Calculation program. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or means for providing machine instructions and/or data to a programmable processor ( For example, magnetic disks, optical disks, memories, programmable logic devices (PLD)), including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and may be provided in any form, including Acoustic input, voice input or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., A user's computer having a graphical user interface or web browser through which the user can interact with implementations of the systems and technologies described herein), or including such backend components, middleware components, or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communications network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

Computer systems may include clients and servers. Clients and servers are generally remote from each other and typically interact over a communications network. The relationship of client and server is created by computer programs running on corresponding computers and having a client-server relationship with each other.

It should be understood that various forms of the process shown above may be used, with steps reordered, added or deleted. For example, each step described in the present disclosure may be executed in parallel, sequentially, or in a different order, as long as the desired technical solutions disclosed in the present disclosure can be achieved. As a result, this article is not limited here.

The above-mentioned specific embodiments do not constitute a limitation on the scope of the present disclosure. It will be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions are possible depending on design requirements and other factors. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this disclosure shall be included in the protection scope of this disclosure.

Claims

A data migration method including:

Switch the read and write addresses of new data from the data source end to the data destination end, where the new data refers to the data generated by the data generation end during the data migration process;

Start a double-ended read operation to read the new data from the data source and the data destination;

In response to the current data migration state belonging to the first state, writing the new data to the data destination based on the serialized writing method;

Turn off the serialized writing mode after a preset period of time, and write the new data to the data destination;

Migrate existing data from the data source to the data destination.
The method according to claim 1, wherein reading the new data from the data source end and the data destination end includes:

Prioritize reading the new data from the data destination;

In response to the new data not being read from the data destination, the new data is read from the data source.
The method according to claim 1 or 2, wherein in response to the current data migration state belonging to the first state, writing the new data to the data destination based on a serialized writing method includes:

In response to determining that the first lock exists, determine whether the first lock is valid, wherein the first lock is used to lock the data write permission;

In response to determining that the first lock is valid, writing the new data to the data destination in a serialized writing manner based on the first lock.
The method according to any one of claims 1-3, wherein the method further includes:

In response to the current data migration state belonging to the second state, the new data is written to the data source end based on the serialized writing method.
The method according to claim 4, wherein in response to the current data migration state belonging to the second state, writing the new data to the data source end based on a serialized writing method includes:

In response to determining that the second lock exists, determine whether the second lock is valid, wherein the second lock is used to lock the data write permission;

In response to determining that the second lock is valid, writing the new data to the data source in a serialized writing manner based on the first lock.
The method according to any one of claims 1-5, wherein before starting the double-ended read operation, the method further includes:

Verify the switching operation of switching the read and write address of new data from the data source to the data destination;

In response to determining that the verification is passed, a backend storage cluster is allocated for the switching operation, where the backend storage cluster includes a data source end or a data destination end.
The method according to any one of claims 1 to 5, wherein, after migrating the existing data at the data source end to the data destination end, the method further includes:

Close the double-ended read operation and read the new data from the data destination.
A data migration device including:

A switching unit configured to switch the read-write address of new data from the data source end to the data destination end, where the new data refers to data generated by the data generation end during the data migration process;

A double-ended reading unit configured to enable a double-ended reading operation and read the new data from the data source end and the data destination end;

The first writing unit is configured to write the new data to the data destination based on the serialized writing method in response to the current data migration state belonging to the first state;

The second writing unit is configured to turn off the serialized writing mode after a preset period of time and write the new data to the data destination;

A data migration unit configured to migrate the existing data at the data source to the data destination.
The device according to claim 8, wherein the double-ended reading unit includes:

A first reading module configured to preferentially read the new data from the data destination;

The second reading module is configured to read the new data from the data source in response to the new data not being read from the data destination.
The device according to claim 8 or 9, wherein the first writing unit includes:

A first determination module configured to determine whether the first lock is valid in response to determining that a first lock exists, wherein the first lock is used to lock data writing permission;

The first writing module is configured to write the new data to the data destination in a serialized writing manner based on the first lock in response to determining that the first lock is valid.
The device according to any one of claims 8-10, wherein the device further includes:

The third writing unit is configured to write the new data to the data source end based on the serialized writing method in response to the current data migration state belonging to the second state.
The device according to claim 11, wherein the third writing unit includes:

A second determination module configured to determine whether the second lock is valid in response to determining that the second lock exists, wherein the second lock is used to lock the data write permission;

The second writing module is configured to write the new data to the data source end in a serialized writing manner based on the first lock in response to determining that the second lock is valid.
The device according to any one of claims 8-12, wherein before starting the double-ended read operation, the device further includes:

Verify the switching operation of switching the read and write address of new data from the data source to the data destination;

In response to determining that the verification is passed, a backend storage cluster is allocated for the switching operation, where the backend storage cluster includes a data source end or a data destination end.
The device according to any one of claims 8-12, wherein the device further includes:

The closing unit is configured to close the double-ended read operation and read the new data from the data destination end.
An electronic device including:

at least one processor; and

a memory communicatively connected to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of claims 1-7 Methods.
A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.
A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-7.