WO2024082857A1

WO2024082857A1 - Data migration method and system, and related apparatus

Info

Publication number: WO2024082857A1
Application number: PCT/CN2023/117343
Authority: WO
Inventors: 黄潘潘
Original assignee: 超聚变数字技术有限公司
Priority date: 2022-10-20
Filing date: 2023-09-06
Publication date: 2024-04-25
Also published as: CN115729463A

Abstract

A data migration method and system, and a related apparatus. The method comprises: when a server (100) receives a read request or a write request for first data, the server (100) triggering a migration operation for the first data, and then responding to the processing of the read request or the write request for the first data after the migration is completed. By means of the present solution, online migration of data can be realized, the efficiency of data migration can also be improved, and the user experience is good.

Description

Data migration method, system and related device

This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 20, 2022, with application number 202211292048.5 and application name “Methods, systems and related devices for data migration”, the entire contents of which are incorporated by reference in this application.

Technical Field

The embodiments of the present application relate to the technical field of data migration, and in particular, to a method, system and related apparatus for data migration.

Background technique

When the original storage medium capacity is insufficient, the old storage software is replaced with the new one, or the hardware life cycle expires, data migration is required. At present, with the rise of cloud computing, applications on the cloud or the replacement of cloud vendors, the storage methods selected by users are also changing, which also requires data migration.

How to improve the efficiency of data migration while achieving online migration is an urgent problem to be solved in this field.

Summary of the invention

The present application provides a method, system and related devices for data migration, which can not only realize online data migration, but also improve the efficiency of data migration and provide a good user experience.

In a first aspect, the present application provides a data migration method, which is applied to a computing device, comprising:

Receiving a read request or a write request for first data;

Processing the first data according to the read request, and migrating the first data from the source data storage area to the target storage area for storage;

Alternatively, the first data is processed according to the write request, and the first data is migrated from the source data storage area to the target storage area for storage;

The first data is stored in a source data storage area, and the source data storage area and the target storage area are located in the same computing device or in different computing devices.

In a possible implementation, in an embodiment of the present application, a migration operation of the first data may be triggered by a read request or a write request for the first data.

In a possible implementation, when the source data storage area and the target storage area are located in different computing devices, the first data can be sent and received between the different computing devices.

It can be seen from the above method that when the system receives a user's read request or write request for the first data, it can ensure that the user's read request or write request and migration task are implemented by the system only needing to read the first data once in the source data storage area.

In this way, compared with the method in which the system first receives the migration instruction and then receives the read request or the write request, the migration efficiency can be improved on the basis of realizing online migration.

In combination with the first aspect, in some implementations, after receiving a read request or a write request for the first data, the method further includes:

When it is determined that the first data needs to be migrated and the first data has not been migrated, the first data is processed according to the read request or the write request.

In one possible implementation, when the first data does not need to be migrated or has already been migrated, the first data can be directly processed for read requests or write requests without migrating the first data from the source data storage area to the target storage area.

In this way, the first data in various scenarios can be fully covered, various situations can be taken into consideration, and multiple migrations can be avoided.

In combination with the first aspect, in some implementations, the storage address of the first data in the source data storage area is a first address,

The processing of the first data according to the read request and migrating the first data from the source data storage area to the target storage area for storage specifically includes:

The first data may be read from the first address according to the read request and stored in a second address in the target storage area.

In a possible implementation, the first data may be stored in a block storage, object storage, or other storage methods, which are not limited thereto.

In a possible implementation, if the size of the first data corresponding to the read request is smaller than the size of the data block or the single object, the data block or the single object needs to be split according to the size of the data actually read so that the partial data after the splitting process is exactly the first data.

In a possible implementation manner, after the first data is migrated from the source data storage area to the target storage area for storage, the first data may be returned to the user.

In this way, a read request for the first data can be implemented and the efficiency of migrating the first data can be improved.

In combination with the first aspect, in some implementations, the storage address of the first data in the source data storage area is a first address, and the storage method of the first data is block storage.

The processing of the first data according to the write request and migrating the first data from the source data storage area to the target storage area specifically includes:

Determining whether the sizes of the first data and the data block are aligned;

In the case where it is determined that the first data is aligned with the size of the data block, the first data is read from the first address according to the write request and stored in the second address of the target storage area; and the first data stored in the first address and the second address is written;

The data block is data that is migrated once in the block storage; the alignment means that the size of the first data is an integer multiple of the size of the data block.

In a possible implementation, a prerequisite for responding to a write request is that write-write mutual exclusion must be satisfied, that is, the data targeted by two write requests cannot have an intersection.

In a possible implementation, after the write processing of the first data in the first address and the second address is completed, the user may be notified of the completion of the response to the write request.

This can avoid the situation where write-write mutual exclusion occurs.

In conjunction with the first aspect, in some implementations, when the computing device determines that the size of the data block of the first data is not aligned, the computing device continues to acquire a mutex lock of the data block;

Perform write processing on the first data stored at the first address according to the write request to obtain second data, and store the second data at the first address;

The second data is read from the first location and stored in a second address in the target storage area.

In this way, the problem that the size of the first data is not aligned with the size of the data of a single migration can be solved, thereby implementing write processing for the first data.

In combination with the first aspect, in some implementations, the storage method of the first data is object storage, and the storage address of the first data in the source data storage area is a first address.

According to the write request, the first data is read from the first address and stored in the second address in the target storage area, and a write process is performed on the first data stored in the first address and the second address.

In a possible implementation, the write processing operation of the object storage can only act on the entire object, so there is no need to perform alignment judgment between the first data and a single object.

In this way, a write request for the first data of the object storage type can be implemented, and the efficiency of migrating the first data can be improved.

In combination with the first aspect, in some embodiments, after the first data is migrated, the migration metadata record table is updated, and the migration metadata record table records the migration status of the first data. The migration status includes any one of the following: the first data needs to be migrated but has not been migrated, has been migrated, and does not need to be migrated.

In a possible implementation, after the first data is migrated from the source data storage area to the target storage area, the state of the migration metadata record table may be converted from "N" to "Y".

In a possible implementation, when the computing device does not receive a read request or a write request for the first data, a background migration task may be started to determine the data to be migrated, and the system begins to migrate the data blocks or single objects to be migrated in sequence in the background.

In a second aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method described in the first aspect.

In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the first aspect.

It is understandable that the electronic device provided in the second aspect and the computer-readable storage medium provided in the third aspect are both used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a data migration system according to an embodiment of the present application.

FIG. 2 is a schematic block diagram of the structure of a server 100 provided in an embodiment of the present application.

FIG3 is a flow chart of a method for data migration in response to a read request provided in an embodiment of the present application.

FIG4 is a flow chart of another data migration method in response to a read request provided in an embodiment of the present application.

FIG5 is a flow chart of a data migration method in response to a write request provided in an embodiment of the present application.

FIG6 is a flow chart of another method for data migration in response to a write request provided in an embodiment of the present application.

FIG. 7 is a flow chart of a background data migration method provided in an embodiment of the present application.

FIG8 is a flow chart of another background data migration method provided in an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, "/" means or, for example, A/B can mean A or B; "and/or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that can be recognized by the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

Data migration can refer to the process of selecting, preparing, extracting, and transforming data and permanently transferring it from one computer storage system to another. There are many reasons for data migration, which may include but are not limited to server or storage device replacement, maintenance or upgrade, application migration, website integration, disaster recovery, and data center migration. In some implementations, data migration can be understood as copying data stored in one computer storage system and copying it to another computer storage system.

At present, some data migration methods do not support online migration. For example, if a user needs to access the data being migrated, he needs to wait until the data migration is completed before accessing the data. In other words, the electronic device cannot respond to the read and write requests for the data during the data migration process.

Some data migration methods are less efficient. For example, when the system uses snapshots to migrate data, it is necessary to export data from the source data storage area and then import the exported data into the target storage area after the migration. If during the data migration process, a user operation triggers a write request resulting in new data, and the new data also needs to be migrated, then it is necessary to repeatedly export the new data from the source data storage area and then import the exported new data into the target storage area after the migration.

In order to improve the problems caused by the above data migration method, an embodiment of the present application provides a data migration method. In this method, data migration is divided into two forms: foreground migration and background migration.

The foreground migration method specifically refers to: firstly determining a data set to be migrated, wherein the data set to be migrated may be determined by a manager. When the system receives a read request or a write request for first data from a user, the system starts to migrate the first data. The first data belongs to the data set to be migrated.

When the system receives a read request for the operation of the first data from the user, the system reads the first data from the source data storage area, migrates it to the target storage area, and then returns the first data in the target storage area to the user.

When the system receives a write request from the user for an operation on the first data, the system reads the first data from the source data storage area, migrates it to the target storage area, then performs double-write processing on the first data in the source data storage area and the first data in the target storage area, and then returns the first data in the target storage area to the user; or the system first reads the first data, performs write processing, stores the first data after the write processing in the source data storage area, and then migrates the first data after the write processing to the source data storage area.

The background migration method specifically refers to: firstly determining the data to be migrated, and then the system starts to migrate the data blocks or single objects to be migrated in sequence in the background.

In some implementations, foreground migration has a higher priority than background migration. That is, when the system receives a read request or a write request from a user for the first data, the system may preferentially select the foreground migration method for migration. When the system does not receive any read request or write request, the system may only select the background migration method to migrate the data targeted by the migration instruction.

In some implementations, a storage address at which the first data is stored in the source data storage area may be referred to as a first address, and a storage address at which the first data is stored in the target data storage area may be referred to as a second address.

In some implementations, data after the first data is written may be referred to as second data.

The above method can ensure that when the system receives a user's read request or write request for the first data, the system only needs to read the first data once in the source data storage area to ensure that the user's read request or write request and migration task are implemented.

In this way, the migration efficiency can be improved compared with the method in which the system first receives the migration instruction and then receives the read request or write request. This is because the system cannot guarantee that the data received by the migration instruction and the data received by the read request or write request are the same data during the migration process. This will cause the system to read a certain data twice in the source data storage area when implementing the user's read request or write request and the migration task.

Therefore, the data migration method provided in the embodiment of the present application can not only realize the online migration of data, but also improve the efficiency of data migration and provide a good user experience.

In order to more clearly introduce the method provided in the embodiment of the present application, the data migration system framework provided in the embodiment of the present application is first introduced below.

Refer to Figure 1, which is a schematic diagram of the framework of a data migration system provided in an embodiment of the present application.

The data migration system may include but is not limited to a source data storage area, a target data storage area, a data migration tool, a storage medium, etc. The data migration system may be applied to application scenarios such as local data migration to the cloud, data migration between clouds, and cloud data migration to the local.

It is worth noting that the above-mentioned source data storage area and target data storage area may refer to the logical storage location of data blocks or single objects in the server, and the above-mentioned storage medium may refer to the physical storage location of data blocks or single objects in the server.

Among them, the source data storage area may refer to a database storing unmigrated data. Common databases may include but are not limited to relational database manager (My Structured Query Language, MYSQL), Oracle database, Microsoft SQL Server, etc.

The types of data stored in the source data storage area may include but are not limited to block storage, file storage, object storage, etc. For example, when the data stored in the source data storage area is stored in the form of block storage, the source data storage area may be used as a volume or a logical unit number (Logical Unit Number, lun), and the data stored in the source data storage area may be divided into data blocks of the same size (for example, a data block may be 1MB in size).

In some implementations, when data stored in a source data storage area is migrated, the data may be migrated in units of data blocks.

For another example, when the data stored in the source data storage area is stored in the form of object storage, the source data storage area can be used as a bucket, and the data stored in the source data storage area is a single object. In other words, data migration can be performed by unit amount of data.

The target data storage area may refer to a database storing migrated data. Similarly, the types of data stored in the target data storage area may include but are not limited to block storage, file storage, and object storage, etc. The detailed description of the storage type can refer to the above content and will not be repeated here.

In some implementations, the source data storage area and the target data storage area may be located on the same hardware device or on different hardware devices.

The rectangle a shown in the source data storage area and the rectangle b shown in the source data storage area shown in Figure 1 can represent a data block or a single object. In other words, to complete the migration task of the data set to be migrated in the source data storage area, multiple data blocks or single objects need to be migrated.

For example, in a complete migration task, the data block or single object of rectangle a can be migrated to rectangle b. The data block or single object represented by rectangle b is the data of the data block or single object of rectangle a after migration, and the data is consistent in content.

The data migration tool can capture requests, distribute data, store and update migration metadata record tables, and perform background migration tasks, etc.

Specifically, the capture of the request can be used to monitor a read request or a write request from a user for the first data, so as to start the foreground migration task.

The specific location where the capture of the request is implemented may be different. In some implementations, the capture of the request can be completed at the Device Mapper layer in Linux. When the server where the data is stored is connected to the corresponding client, the capture of the request can also be completed at the client. This is not limited in the embodiments of the present application.

In some implementations, after completing the capture of the request, the data migration tool obtains the logical address of the first data targeted by the request from the source data storage area, and reads the first data from the storage medium according to the logical address.

That is to say, the source data storage area and the target data storage area are logical storage locations of the first data, and the storage medium is the physical storage location of the first data.

The storage medium may refer to a carrier for storing the first data, such as a floppy disk, an optical disk, a high-density digital video disc (Digital Video Disc, DVD), a hard disk, a flash memory, a USB flash drive, a compact flash (Compact Flash, CF) card, a secure digital memory card (Secure Digital Memory Card, SD card), a multimedia card (Multi-Media Card, MMC card), a smart media (Smart Media, SM) card, a memory stick (Memory Stick), an extreme digital-picture card (Extreme Digital-Picture Card, xD card), etc.

The data distribution can be used to perform migration processing on the first data after the data migration tool reads the first data. In some implementations, the data migration tool can determine whether the first data needs to be migrated, that is, whether the first data belongs to the data set to be migrated. If the first data belongs to the data set to be migrated, the data migration tool can migrate the first data to the target storage area.

The migration metadata record table is used to record the migration status of the data (e.g., not migrated, migrated, and initial state). In some implementations, the data migration tool can store and update the migration metadata record table. Specifically, reference can be made to the migration metadata record tables shown in Table 1 and Table 2.

Table 1

Table 2

As shown in Table 1 and Table 2, Table 1 exemplarily shows a block storage migration metadata record table, and Table 2 exemplarily shows an object storage migration metadata record table.

The migration metadata record table may include an index and a migration status. The migration metadata record table may also include any one or more of the following items: the identifier of the server where the data is stored when it has not been migrated, the identifier of the server where the data is stored after it has been migrated, the logical location where the data is stored when it has not been migrated, and the logical location where the data is stored after it has been migrated, etc. In the embodiments of the present application, this is not limited.

The indexes shown in Table 1 and Table 2 may be used to represent a structure for sorting data in the source data storage area. The indexes may be used to quickly view specific data in the source data storage area.

For example, the index "Block 1" represents the index of data blocks of the same size (for example, 1MB) in data stored in the form of block storage, and the index "Obeject 1" represents the index of a single object in data stored in the form of object storage.

In some implementations, the index of data is related to the storage type of the data.

For example, if the storage type of certain data is block storage, the index of the data is volume ID + logical address; if the storage type of certain data is file storage, the index of the data is file system ID (File system ID) + directory (Directory) + file name (File name) + offset address (Offset); if the storage type of certain data is object storage, the index of the data is key (Key) + version number (Version).

The migration status shown in Table 1 and Table 2 indicates whether the data in the source data storage area is in a migration state.

The status "Y" indicates that the data indicated by the specific index in the row is currently in a state where migration is required and has been completed.

The status "N" indicates that the data indicated by the specific index in the row is currently in a state where the data needs to be migrated but has not been migrated, or needs to be migrated and is being migrated but has not been completely migrated.

The status "NA" indicates that the data indicated by the specific index in the row is currently in a state where migration is not required and is not in an unmigrated state.

In some implementations, the data migration tool can also change the state of the data in real time. For example, when data 1 needs to be migrated and after the migration is completed, the state of data 1 can be changed from "N" to "Y". Generally, data 1 will not be changed from "Y" to "N".

The same data indicated by the indexes in different migration metadata record tables may be in different states in different tables. For example, the state of data 1 in table A is "Y", and the state in table B is "N", that is, data 1 has been migrated from the logical location in table A where it was stored before migration to the logical location where it is stored after migration, but has not yet been migrated from the logical location in table B where it was stored before migration to the logical location where it is stored after migration. Logical location of storage.

It is worth noting that since the server 100 stores a migration metadata record table that records the status of the data of the migration unit, the migration metadata record table is updateable and persistent. Therefore, in the embodiment of the present application, breakpoint resumption and retry migration can be supported. For example, when a network problem occurs and the data cannot be copied, the data copying is interrupted, and the original migration task is not affected. When the network is restored, the data copying continues from the interruption point.

For another example, if a network problem causes replication to fail, and the system is responding to a write request and is writing a data block or object that has already been migrated, the corresponding status of the data block or object in the migration metadata record table is recorded as "N", indicating that the data block or object is indicated as a dirty block and needs to continue to be migrated. When the network is restored, the migration tool continues to restore the original migration process and migrates the unmigrated data and dirty blocks to the target. In other words, the data blocks or single objects processed at the moment the network problem occurs are reprocessed to achieve accurate migration tasks.

In some implementations, for the same source data storage area, multiple tasks can be migrated to different targets. Two different migration tasks for the same source data storage area are independent of each other and will not affect each other. For simultaneous migration of different source data storage areas, there will be multiple migration tasks in the background, and there will also be migration metadata tables for different migration tasks. Multiple migration tasks are also independent of each other and will not affect each other.

The background migration task is different from the foreground migration task. It refers to scanning the data to be migrated in real time and in sequence, determining the unmigrated unit data in the data to be migrated, and migrating the unmigrated unit data (such as data blocks or single objects) in the data to be migrated.

For example, the data to be migrated is 100 units of data, namely data 1 to data 100. If there is no read request or write request for any of the data from data 1 to data 100, the background migration task starts scanning from data 1 to data 100, and determines that data 1 is unmigrated data when scanning data 1, and migrates data 1. The remaining 99 data are migrated in the same way.

If there is a read request or write request for data 10, the foreground migration task will migrate data 10 first, and then the background migration task will scan from data 1 to data 100, and then migrate data 1-data 9, data 11-data 100 in order, and when scanning data 1, it will be determined that data 1 is not migrated data, and data 1 will be migrated. The remaining 98 data will be migrated in the same way, but when scanning data 10, it will be determined that data 10 is migrated data, and data 10 will not be migrated.

In this way, only the data blocks or single objects that have not been read or written in the foreground need to be migrated, and the data that has been read and written will not be migrated repeatedly, which reduces the reading and writing of data and improves the migration efficiency.

In order to more clearly introduce the method provided in the embodiment of the present application, the server 100 provided in the embodiment of the present application is as follows.

Refer to Figure 2, which is a schematic diagram of the structure of the server 100 provided in an embodiment of the present application.

As shown in FIG2 , the server 100 may include: one or more processors 101, a memory 102, a communication interface 103, a transmitter 105, a receiver 106, a coupler 107, and an antenna 108. The above components may be connected via a bus 104 or other means, and FIG2 takes the connection via a bus as an example. Among them:

In the embodiment of the present application, the processor 101 can be used to read and execute computer-readable instructions. Specifically, the processor 101 can be used to call a program stored in the memory 102, such as an implementation program of the data migration method provided in the embodiment of the present application on the server 100, and execute the instructions contained in the program. The processor 101 can also be used to determine whether migration is required based on the migration metadata record table.

The memory 102 is coupled to the processor 101 and is used to store various software programs and/or multiple sets of instructions. Specifically, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices.

The memory 102 may store information such as the data set to be migrated and the migration metadata record table.

The memory 102 may store an operating system (hereinafter referred to as system), such as embedded operating systems such as uCOS, VxWorks, RTLinux, etc. The memory 102 may also store a network communication program, which may be used to communicate with clients and other servers.

The communication interface 103 can be used for communication between the server 100 and other devices.

Specifically, the communication interface 103 may be a 3G communication interface, a long term evolution (LTE) (4G) communication interface, a 5G communication interface, a WLAN Communication interface, WAN communication interface, etc. Not limited to the wireless communication interface, the server 100 may also be configured with a wired communication interface 103 to support wired communication.

The transmitter 105 and the receiver 106 can be regarded as a wireless modem. The transmitter 105 can be used to transmit the signal output by the processor 101. The receiver 106 can be used to receive the signal.

In the server 100 , the number of the transmitter 105 and the number of the receiver 106 can be one or more.

The antenna 108 may be used to convert electromagnetic energy in the transmission line into electromagnetic waves in free space, or to convert electromagnetic waves in free space into electromagnetic energy in the transmission line.

The coupler 107 may be used to divide the mobile communication signal into multiple paths and distribute the signals to multiple receivers 106 .

It is understandable that the antenna 108 of the network device may be implemented as a large-scale antenna array.

It should be noted that the server 100 shown in FIG. 2 is only one implementation of the embodiment of the present application. In actual applications, the server 100 may also include more or fewer components, which is not limited here.

It is worth noting that, according to different needs of users and comprehensive considerations of other aspects, the server stored in the source data storage area and the target storage area can be the same or two different servers.

The data migration tool may be run on a server stored in a source data storage area, or on a server stored in a target storage area, or on other servers other than the above.

In some implementations, the data migration tool may be run simultaneously on the server stored in the source data storage area and the server stored in the target storage area. In the embodiments of the present application, this is not limited.

When the server stored in the source data storage area and the server stored in the target storage area in the data migration system in Figure 1 are the same server 100 shown in Figure 2, and the data migration tool runs on the server 100, the processor 101 can be used to respond to the user's read requests and write requests, and intercept and process the requests.

The processor 101 may also determine whether the data needs to be migrated according to the migration metadata record table. The processor 101 may also update the state information of the data in the migration metadata record table after the data migration is completed.

The memory 102 may be used to store information such as data to be migrated, data that has been migrated, and a migration metadata record table.

The communication interface 103 may be used to establish a communication connection with an electronic device used by a user.

The transmitter 105 can be used to return the data required by the user to the electronic device used by the user, the receiver 106 can be used to receive the user's read request or write request, and the receiver 106 can also be used to receive migration instructions for the first data. For example, the receiver 106 can also be used to receive migration instructions for the first data sent by the control center.

When the server stored in the source data storage area and the server stored in the target storage area in the data migration system in Figure 1 are not the same server 100 shown in Figure 2, the server stored in the source data storage area is server 110, and the server stored in the target storage area is server 120. The data migration tool can run on server 110 and server 120 at the same time, and server 100 includes server 110 and server 120.

The processor 101 of the server 110 can be used to respond to the user's read request and write request and intercept the request. The processor 101 of the server 110 can also determine whether the data needs to be migrated according to the migration metadata record table. The processor 101 of the server 110 can update the migration metadata record table after sending the data to be migrated.

The memory 102 of the server 110 may be used to store data to be migrated and information such as a migration metadata record table.

The communication interface 103 of the server 110 may be used to establish a communication connection with the server 120 .

The transmitter 105 of the server 110 may be used to send the data to be migrated to the server 120 .

The receiver 106 of the server 110 may be used to receive a read request or a write request from a user. The receiver 106 of the server 110 may also be used to receive a migration instruction for the first data sent by the control center.

The processor 101 of the server 120 may update the migration metadata record table after receiving the data to be migrated sent by the server 110 .

The memory 102 of the server 120 may be used to store information such as migrated data and migration metadata record tables.

The communication interface 103 of the server 120 may be used to establish a communication connection with the server 110 .

The transmitter 105 of the server 120 may be used to send a prompt message to the server 110 indicating that the data to be migrated has been migrated, and the receiver 106 of the server 120 may be used to receive the data to be migrated sent by the server 110 .

In some implementations, server 100 may be referred to as a computing device.

In some implementations, the electronic device used by the user may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device and/or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device.

A method for data migration provided in an embodiment of the present application is described below. This embodiment takes the case where the server 100 responds to a user's read request for data to be migrated as an example to introduce the method for data migration.

Fig. 3 shows a schematic flow chart of a data migration method in response to a read request provided in an embodiment of the present application. In this embodiment, the data storage mode is block storage, and the migration unit is a data block as an example.

S301. The server 100 responds to a user's read request for first data.

In some implementations, the user can input an operation to the server 100 according to his/her needs, and the server 100 can respond with a read request for the first data. The read request for the first data may include, but is not limited to: the logical address of the first data in the source data storage area, the index, and the data payload length, etc., wherein the data payload length refers to the effective payload data length excluding the header and the length in the read request information.

S302: Determine whether the first data needs to be migrated.

In some implementations, the server 100 may receive a migration instruction for the first data sent by the control center before responding to the read request.

The system may issue a migration instruction due to the remaining storage space of the server 100, the server 100 being upgraded, or the server 100 being damaged, or may issue a migration instruction based on an input operation by a manager. This is not limited in the embodiments of the present application.

Furthermore, if there is a migration instruction for the first data, the status of the first data in the migration metadata record table will not be "NA" but "Y" or "N".

When the server 100 determines that the first data needs to be migrated, that is, there is a migration instruction for the first data, and the status of the first data in the migration metadata record table is "Y" or "N", S303 is executed.

When the server 100 determines that the first data does not need to be migrated, that is, there is no migration instruction for the first data, and the status of the first data in the migration metadata record table is "NA", S304 is executed.

In some implementations, since the record unit in the migration metadata record table is in the size of a data block, the state of the first data may refer to:

If the size of the first data is greater than or equal to the data block, the state of the first data is the state of the first data divided into corresponding multiple data blocks; if the size of the first data is less than the data block, the state of the first data is the state of the data block corresponding to the first data.

S303: Determine whether the first data has been migrated.

In some implementations, the server 100 may determine whether the first data has been migrated based on the migration metadata record table.

For example, if in the migration metadata record table, the status corresponding to the index carried in the read request for the first data is "Y", it can be determined that the first data has completed migration; if in the migration metadata record table, the status corresponding to the index carried in the read request for the first data is "N", it can be determined that the first data has not been migrated or is in the process of migrating and has not yet completed migration.

When the server 100 determines that the first data has been migrated, the status of the first data in the migration metadata record table is "Y", and the first data does not need to be migrated from the source data storage area to the target storage area again, and S304 is executed.

When the server 100 determines that the first data has not been migrated, but it can be known from S303 that there is a migration instruction for the first data, the state of the first data in the migration metadata record table is "N", and S305 is executed.

Furthermore, in a possible implementation, if the first data is being migrated but has not been completely migrated, the migration metadata record table may record that the state of the first data may be "N+".

When the server 100 determines that the first data is being migrated but has not been completely migrated, the first data does not need to be migrated again, and S304 is executed.

S304: Read the first data from the source data storage area and return it to the user.

In some implementations, the server 100 may read the first data from the source data storage area based on the read request and return the first data to the user.

Specifically, if the first data is stored in the cloud and the user wants to read the first data, the cloud can obtain the logical address of the first data based on the read request of the first data sent by the local user, and then obtain the physical address of the first data based on the mapping relationship between the logical address and the physical address, and read the first data from the physical address, and then the cloud sends the read first data to the local user.

S305: Read the first data from the source data storage area, migrate the first data to the target storage area, update the migration metadata record table, and then return the first data to the user.

In some implementations, the server 100 may migrate the first data from the source data storage area to the target storage area. For example, when the source data storage area and the target storage area are stored on the same server 100, the logical address of the first data in the source data storage area is L1, and the physical address corresponding to the logical address L1 is W1, and the mapping relationship is that the logical address L1 maps the physical address W1.

The server 100 migrates the first data from the source data storage area to the target storage area, which may mean changing the logical address of the first data from the logical address L1 in the source data storage area to the logical address L2 in the target storage area, and changing the mapping relationship to map the logical address L2 to the physical address W1.

Optionally, the two physical addresses stored in the server 100 before and after the first data migration may be the same or different. In the embodiment of the present application, there is no limitation on this.

In some implementations, after completing the migration of the first data from the source data storage area to the target storage area, the server 100 may update the migration metadata record table.

For example, after the first data is migrated from the source data storage area to the target storage area, the state of the migration metadata record table may be converted from "N" to "Y".

In a possible implementation, if the size of the first data that the user wants to read is smaller than the data block, the server 100 needs to split the data block according to the size of the data actually read, so that the part of the data after the splitting process is exactly the first data.

For example, the size of the first data that the user wants to read is 0.5MB, and the size of the data block is 1MB. At this time, the size of the data block read out by the server 100 from the source data storage area is 1MB, wherein the 1MB data block includes 0.5MB of first data and 0.5MB of other data. The server 100 needs to divide the 1MB data block, select 0.5MB of first data therefrom, and return the 0.5MB of first data to the user.

Fig. 4 shows a flow chart of another data migration method in response to a read request provided in an embodiment of the present application. In this embodiment, the data storage mode is object storage, and the migration unit is a single object as an example.

S401. The server 100 responds to a user's read request for first data.

In some implementations, the user may perform input operations to the server 100 according to his/her own needs, and the server 100 may respond with a read request for the first data.

The read request for the first data may include but is not limited to: the logical address of the first data in the source data storage area, index and data payload length, etc., wherein the data payload length refers to the effective payload data length excluding the header and length in the read request information.

S402: Determine whether the first data needs to be migrated.

In some implementations, the server 100 may receive a migration request for the first data from the control center before responding to the read request. instruction.

When the server 100 determines that the first data needs to be migrated, that is, there is a migration instruction for the first data, and the status of the first data in the migration metadata record table is "Y" or "N", S403 is executed.

When the server 100 determines that the first data does not need to be migrated, that is, there is no migration instruction for the first data, and the status of the first data in the migration metadata record table is "NA", S404 is executed.

In some implementations, since the record unit in the migration metadata record table is based on the size of a single object, the state of the first data may refer to:

If the size of the first data is greater than or equal to a single object, the state of the first data is the state of the first data divided into corresponding multiple single objects; if the size of the first data is less than a single object, the state of the first data is the state of the single object corresponding to the first data.

S403: Determine whether the first data has been migrated.

For example, if in the migration metadata record table, the state corresponding to the index carried in the read request for the first data is "Y", it can be determined that the first data has been migrated;

If in the migration metadata record table, the state corresponding to the index carried in the read request for the first data is "N", it can be determined that the first data has not been migrated or is being migrated and the migration has not yet been completed.

When the server 100 determines that the first data has been migrated, the status of the first data in the migration metadata record table is "Y", and the first data does not need to be migrated from the source data storage area to the target storage area again, and S404 is executed.

When the server 100 determines that the first data has not been migrated, but it can be known from S403 that there is a migration instruction for the first data, the state of the first data in the migration metadata record table is "N", and S405 is executed.

When the server 100 determines that the first data is being migrated but has not been completely migrated, the first data does not need to be migrated again, and S404 is executed.

S404: Read the first data from the source data storage area and return it to the user.

S405: Read the first data from the source data storage area, migrate the first data to the target storage area, update the migration metadata record table, and then return the first data to the user.

The server 100 migrates the first data from the source data storage area to the target storage area, which may mean changing the logical address of the first data from the logical address L1 in the source data storage area to the logical address L2 in the target storage area, and changing the mapping relationship to the logical address L2 mapping relationship. Physical address W1.

In a possible implementation, if the size of the first data that the user wants to read is smaller than a single object, the server 100 needs to split the single object according to the size of the data actually read, so that the portion of data after the splitting process is exactly the first data. For example, the size of the first data that the user wants to read is 0.5MB, and the size of the single object is 1MB. At this time, the size of the single object read by the server 100 from the source data storage area is 1MB, where the 1MB single object includes 0.5MB of first data and 0.5MB of other data. The server 100 needs to split the 1MB single object, select 0.5MB of first data from it, and return the 0.5MB of first data to the user.

By implementing the above method, online migration can be realized, and it can be ensured that when the system receives a user's read request for the first data, the system only needs to read the first data once in the source data storage area to ensure that the user's read request and migration task are realized. In this way, the migration efficiency can be improved.

Another data migration method provided in an embodiment of the present application is described below. This embodiment takes the case where the server 100 responds to a user's write request for data to be migrated as an example to introduce the data migration method.

Fig. 5 shows a schematic flow chart of a data migration method in response to a write request provided in an embodiment of the present application. In this embodiment, the data storage mode is block storage, and the migration unit is a data block as an example.

S501. The server 100 responds to a user's write request for first data.

In some implementations, the user may perform input operations to the server 100 according to his/her own needs, and the server 100 may respond with a write request for the first data.

The write request for the first data may include but is not limited to: a logical address of the first data in the source data storage area, an index, a data payload length, and a modification content, etc.

In some implementations, there is a prerequisite for the server 100 to respond to the write request: the server 100 needs to satisfy write-write mutual exclusion in responding to the user's write request for the first data, that is, the data targeted by the two write requests cannot have an intersection.

If the data targeted by two write requests overlap, a mutually exclusive lock method is required to avoid write-write conflicts.

S502: Determine whether the first data needs to be migrated. If not, write the first data to the source data storage area, and store the written first data in the source data storage area.

In some implementations, the server 100 may receive a migration instruction for the first data issued by the system before responding to the write request.

When the server 100 determines that the first data does not need to be migrated, that is, there is no migration instruction for the first data, the status of the first data in the migration metadata record table is "NA", and the server 100 can directly process the first data according to the write request to obtain the first data after writing, and store the first data after writing in the source data storage area, that is, the logical address of the first data and the first data after writing is the same.

Optionally, the server 100 may notify the user of the completion of the write request. For example, if the user deletes document M, the server 100 may The server 100 notifies the user that the write request for the document M has been completed by displaying a prompt message (eg, a pop-up window) such as a text message "Document M has been deleted". In the embodiment of the present application, the server 100 is not limited to the way in which the write request has been completed.

When the server 100 determines that the first data needs to be migrated, that is, there is a migration instruction for the first data, and the status of the first data in the migration metadata record table is "Y" or "N", S503 is executed.

If the size of the first data is greater than or equal to the data block, the state of the first data is a state in which the first data is divided into a corresponding plurality of data blocks.

If the size of the first data is smaller than the data block, the state of the first data is the state of the data block corresponding to the first data.

S503: Determine whether the first data has been migrated. If so, double-write the first data in the source data storage area and the target storage area.

For example, if in the migration metadata record table, the status corresponding to the index carried in the write request for the first data is "Y", it can be determined that the first data has completed migration; if in the migration metadata record table, the status corresponding to the index carried in the write request for the first data is "N", it can be determined that the first data has not been migrated or is in the process of migrating and has not yet completed migration.

When the server 100 determines that the first data has completed migration, the status of the first data in the migration metadata record table is "Y", and the first data does not need to be migrated from the source data storage area to the target storage area again. The server 100 can process the first data in the source data storage area and the first data in the target storage area respectively according to the write request, and store the first data in the source data storage area after writing in the source data storage area, and store the first data in the target storage area after writing in the target storage area.

In this way, after the first data has been migrated, and the first data in the source data storage area has not been deleted, double-writing the first data in the source data storage area and the target storage area can maintain the content consistency of the first data. Optionally, the server 100 can notify the user of the completion of the write request, which is not described in detail here.

When the server 100 determines that the first data has not been migrated, but it can be known from S502 that there is a migration instruction for the first data, the state of the first data in the migration metadata record table is "N", and S504 is executed.

When the server 100 determines that the first data is being migrated but has not been completely migrated, the first data does not need to be migrated from the source data storage area to the target storage area again. The server 100 can process the first data in the source data storage area and the first data in the target storage area according to the write request, and store the first data in the source data storage area after writing in the source data storage area, and store the first data in the target storage area after writing in the target storage area. Optionally, the server 100 can notify the user of the completion of the write request, which will not be described in detail here.

S504: Determine whether the data block to be migrated is aligned with the first data.

In some implementations, when the data is stored in block storage, the migration unit is a data block, and the server 100 can determine whether the data block is aligned with the first data.

For example, if the size of the first data is 0.5MB and the size of the data block is 1MB, the data block is not aligned with the first data. In other words, when the size of the first data is smaller than the size of the data block, the data block is not aligned with the first data.

If the size of the first data is 1.5MB and the data block size is 1MB, the migration unit of the first data is 1.5, and the data block is not aligned with the first data. In other words, when the size of the first data is not an integer multiple of the data block, the data block is not aligned with the first data.

For another example, if the size of the first data is 1MB and the size of the data block is 1MB, the data block is aligned with the first data. In other words, when the size of the first data is equal to the size of the data block, the data block is aligned with the first data.

If the size of the first data is 2MB and the data block size is 1MB, the migration unit of the first data is 2, and the data block is aligned with the first data. In other words, when the size of the first data is an integer multiple of the data block, the data block is aligned with the first data.

When the server 100 determines that the data block is aligned with the first data, the size of the data block is consistent with the size of the first data or the size of the first data is the same as the size of the first data. The size is an integer multiple of the size of the data block, and S505 is executed.

When the server 100 determines that the data block is not aligned with the first data, there are two cases:

The first type is that the size of the first data is smaller than the data block size, in which case S506 is executed;

The second type is that the size of the first data is larger than the data block size and is not an integer multiple of the data block size. In this case, the first data can be divided into two parts, the first part can be data whose size is the maximum integer multiple of the data block size, and the second part can be data from the first data without the first part. The first part of the data is executed in S505, and the second part of the data is executed in S506.

S505: Migrate the first data to the target storage area, double-write the first data in the source data storage area and the target storage area, and update the migration metadata record table.

In some implementations, the server 100 may migrate the first data from a source data storage area to a target storage area.

For example, when the source data storage area and the target storage area are stored on the same server 100, the logical address of the first data in the source data storage area is L1, and the physical address corresponding to the logical address L1 is W1, and the mapping relationship is that the logical address L1 maps the physical address W1.

For another example, when the server stored in the source data storage area and the server stored in the target storage area are not the same server, that is, the server stored in the source data storage area is server 110, and the server stored in the target storage area is server 120, server 110 stored in the source data storage area sends the first data stored at logical address L1 and physical address W1 to server 120 stored in the target storage area, and server 120 stores the first data in the target storage area. The logical address of the first data in server 120 stored in the target storage area is L2, and the physical address is W2.

Optionally, the two physical addresses stored in the server before and after the migration of the first data may be the same or different. That is, the storage medium where the first data is physically stored in the source data storage area and the storage medium where the first data is physically stored in the target data storage area may be the same or different. In the embodiment of the present application, this is not limited.

In some implementations, the server 100 may double-write the first data in the source data storage area and the first data in the target storage area. Specifically, the server 100 may process the first data in the source data storage area and the first data in the target storage area respectively according to the write request, and store the first data in the source data storage area after writing in the source data storage area, and store the first data in the target storage area after writing in the target storage area.

In some implementations, after completing the double writing of the first data in the source data storage area and the first data in the target storage area, the server 100 may update the migration metadata record table. For example, the state of the first data in the migration metadata record table may be converted from "N" to "Y".

Optionally, the server 100 may notify the user of information on completion of the write request, which may be referred to the above content and will not be elaborated here.

S506: Write the first data in the source data storage area, migrate the written first data to the target storage area, and update the migration metadata record table.

In some implementations, the first data in S506 is not aligned with the size of the data block, so that the data targeted by other write requests may overlap with the first data targeted by this write request, that is, the same area may be written. Therefore, before writing the first data in the source data storage area, it is necessary to obtain a mutex lock in the data block. If the acquisition of the mutex lock fails, it is repeated until the acquisition succeeds.

The acquisition of the mutex lock successfully may mean stopping the write operation on the first data in other threads and only executing the write operation in S506. In this way, the write-write conflict phenomenon can be avoided.

In some implementations, the server 100 may write the first data in the source data storage area. Specifically, the server 100 may write the first data stored in the source data storage area according to the write request, and store the first data after writing in the source data storage area. In some implementations, the first data after writing may be referred to as the second data.

In some implementations, the server 100 may migrate the first data from the source data storage area to the target storage area. When the data storage area and the target storage area are stored on the same server 100, the logical address of the first data in the source data storage area is L1, and the physical address corresponding to the logical address L1 is W1, and the mapping relationship is that the logical address L1 maps the physical address W1.

For another example, when the server stored in the source data storage area and the server stored in the target storage area are not the same but on different servers 100, that is, the server stored in the source data storage area is server 110 and the server stored in the target storage area is server 120, server 110 stored in the source data storage area sends the first data stored in the logical address L1 to server 120 stored in the target storage area, and server 120 stores the first data in the target source target storage area. The logical address of the first data in server 120 stored in the target source target storage area is L2.

Optionally, the two physical addresses stored in the server 100 before and after the first data is migrated may be the same or different. That is, the storage medium where the first data is physically stored in the source data storage area and the storage medium where the first data is physically stored in the target data storage area may be the same or different. In the embodiment of the present application, this is not limited.

In some implementations, after completing the migration of the first data from the source data storage area to the target storage area, the server 100 may update the migration metadata record table. For example, the state of the first data in the migration metadata record table may be converted from "N" to "Y".

Optionally, the server 100 may notify the user of the completion of the write request, which will not be elaborated here.

By implementing the above method, online migration can be realized, and because the migration of data is triggered by a write request, it can be ensured that when the system receives a user's write request for first data, the system only needs to read the first data once in the source data storage area to ensure that the user's write request and migration task are realized. In this way, the migration efficiency can be improved.

Fig. 6 shows a flow chart of another data migration method in response to a write request provided in an embodiment of the present application. In this embodiment, the data storage mode is object storage, and the migration unit is a single object as an example.

S601: The server 100 responds to a user's write request for first data.

In some implementations, the user can input an operation to the server 100 according to his/her needs, and the server 100 can respond with a write request for the first data. The write request for the first data may include but is not limited to: the logical address of the first data in the source data storage area, index, data payload length, and modified content, etc.

S602: Determine whether the first data needs to be migrated. If not, write the first data in the source data storage area and store the written first data in the source data storage area.

When the server 100 determines that the first data does not need to be migrated, that is, there is no migration instruction for the first data, the status of the first data in the migration metadata record table is "NA", and the server 100 can directly write the first data according to the write request to obtain the first data after writing, and store the first data after writing in the source data storage area, that is, the logical address of the first data before the write process and the first data after the write process are the same.

When the server 100 determines that the first data needs to be migrated, that is, there is a migration instruction for the first data, and the status of the first data in the migration metadata record table is "Y" or "N", S603 is executed.

In some implementations, since the record unit in the migration metadata record table is based on the size of a single object, the state of the first data may mean: if the size of the first data is greater than or equal to a single object, then the state of the first data is the state of the first data divided into corresponding multiple single objects; if the size of the first data is smaller than a single object, then the state of the first data is the state of the single object corresponding to the first data.

S603: Determine whether the first data has been migrated, and if so, double-write the first data in the source data storage area and the target storage area.

When the server 100 determines that the first data has not been migrated, but it can be known from S602 that there is a migration instruction for the first data, the state of the first data in the migration metadata record table is "N", and S604 is executed.

When the server 100 determines that the first data is being migrated but has not been completely migrated, the first data does not need to be migrated from the source data storage area to the target storage area again. The server 100 can write the first data in the source data storage area and the first data in the target storage area according to the write request, and store the first data in the source data storage area after writing in the source data storage area, and store the first data in the target storage area after writing in the target storage area. Optionally, the server 100 can notify the user of the completion of the write request, which will not be described in detail here.

S604: Migrate the first data to the target storage area, double-write the first data in the source data storage area and the target storage area, and update the migration metadata record table.

In some implementations, since the first data is stored in an object storage manner and writing to an object generally involves writing to the entire object, there is no need to determine whether the first data is aligned with a single object as in S504. Instead, when the first data needs to be migrated and has not been migrated, the write operation is directly responded to and the migration task of the first data is completed.

For another example, when the server stored in the source data storage area is not the same as the server stored in the target storage area, that is, the source When the server stored in the data storage area is server 110 and the server stored in the target storage area is server 120, server 110 stored in the source data storage area sends the first data stored at logical address L1 and physical address W1 to server 120 stored in the target storage area, and server 120 stores the first data in the target storage area. The logical address of the first data in server 120 stored in the target storage area is L2, and the physical address is W2.

In some implementations, the server 100 may double-write the first data in the source data storage area and the first data in the target storage area.

Specifically, the server 100 can process the first data in the source data storage area and the first data in the target storage area respectively according to the write request, and store the first data in the source data storage area after writing in the source data storage area, and store the first data in the target storage area after writing in the target storage area.

Another data migration method provided in an embodiment of the present application is described below. This embodiment takes the case where there is no user read request or write request for the data to be migrated as an example to introduce the data migration method.

FIG7 shows a flow chart of a background data migration method provided in an embodiment of the present application. The flow chart takes block storage as an example. It is worth noting that if the source data storage area targeted by the background data migration method has a migration instruction, the following specific steps will no longer determine whether there is a migration instruction.

S701 , sequentially traverse the source data storage area to be migrated, and determine whether the currently scanned data block has been migrated.

In some implementations, the server 100 may sequentially traverse the source data storage area to be migrated. For example, if the source data storage area to be migrated is data 1 to data 100, which has 100 data blocks in total, the server 100 scans from data 1 to data 100, and determines whether each data block has been migrated.

In some implementations, the server 100 may determine whether the currently scanned data block has completed migration.

For example, if the status corresponding to the currently scanned data block is "Y" in the migration metadata record table, it can be determined that the currently scanned data block has completed migration;

If the state corresponding to the currently scanned data block is "N" in the migration metadata record table, it can be determined that the currently scanned data block has not been migrated or is being migrated and the migration has not yet been completed.

When the server 100 determines that the currently scanned data block has completed migration, the status of the currently scanned data block in the migration metadata record table is "Y", and the currently scanned data block does not need to be migrated from the source data storage area to the target storage area again, so the next data block of the current data block in the source data storage area is scanned.

For example, the source data storage area to be migrated is data 1-data 100, which has 100 data blocks in total. The server 100 determines that data 1 has been migrated, and then the server 100 scans data 2 to determine whether data 2 has been migrated.

When the server 100 determines that the currently scanned data block has not been migrated, the status of the currently scanned data block in the migration metadata record table is "N", and S502 is executed.

S702: Determine whether the currently scanned data block is being processed.

In some implementations, the server 100 may determine whether the currently scanned data block is being processed, that is, determine whether the currently scanned data block is being migrated.

In a possible implementation, if the currently scanned data block is in the process of being migrated but has not been completely migrated, the migration metadata record table may record that the status of the currently scanned data block may be "N+".

In the embodiment of the present application, in addition to the above method, there are other methods for determining whether a data block is being migrated, which are not limited to this.

When the server 100 determines that the currently scanned data block is being migrated and the currently scanned data block does not need to be migrated again from the source data storage area to the target storage area, S701 is executed to scan the next data block of the current data block in the source data storage area.

For example, the source data storage area to be migrated is data 1-data 100, which has 100 data blocks in total. The server 100 determines that data 1 is being migrated, and then the server 100 scans data 2 to determine whether data 2 has completed migration.

When the server 100 determines that the currently scanned data block is not being migrated, S703 is executed.

S703: Read the currently scanned data block from the source data storage area, migrate the data block to the target storage area, and update the migration metadata record table.

In some implementations, the server 100 may migrate the currently scanned data block from the source data storage area to the target storage area.

For example, when the source data storage area and the target storage area are stored on the same server 100, the logical address of the currently scanned data block in the source data storage area is L1, and the physical address corresponding to the logical address L1 is W1, and the mapping relationship is that the logical address L1 maps the physical address W1.

The server 100 migrates the currently scanned data block from the source data storage area to the target storage area, which may mean changing the logical address of the currently scanned data block from the logical address L1 in the source data storage area to the logical address L2 in the target storage area, and changing the mapping relationship to map the logical address L2 to the physical address W1.

Optionally, the two physical addresses of the currently scanned data block stored in the server 100 before and after the migration may be the same or different. In the embodiment of the present application, there is no limitation on this.

In some implementations, after completing the migration of the currently scanned data block from the source data storage area to the target storage area, the server 100 may update the migration metadata record table.

For example, after the currently scanned data block is migrated from the source data storage area to the target storage area, the state of the migration metadata record table may be converted from "N" to "Y".

FIG8 shows a flow chart of a background data migration method provided in an embodiment of the present application. The flow chart takes object storage as an example. It is worth noting that if the source data storage area targeted by the background data migration method has a migration instruction, the following specific steps will no longer determine whether there is a migration instruction.

S801 , sequentially traverse the source data storage area to be migrated, and determine whether the migration of the single object currently scanned has been completed.

In some implementations, the server 100 may sequentially traverse the source data storage area to be migrated.

For example, the source data storage area to be migrated is data 1 to data 100, which contains 100 objects in total. The server 100 scans from data 1 to data 100, and determines whether each object has been migrated.

In some implementations, the server 100 may determine whether the migration of a single object currently being scanned has been completed.

For example, if the status corresponding to the currently scanned single object is "Y" in the migration metadata record table, it can be determined that the currently scanned single object has completed migration;

If the status corresponding to the currently scanned single object is "N" in the migration metadata record table, it can be determined that the currently scanned single object has not been migrated or is being migrated and the migration has not yet been completed.

When the server 100 determines that the migration of the single object currently being scanned has been completed, the status of the single object currently being scanned in the migration metadata record table is "Y", and the single object currently being scanned does not need to be migrated from the source data storage area to the target storage area again, so the next data block of the current single object in the source data storage area is scanned.

For example, the source data storage area to be migrated is data 1-data 100, which contains 100 objects in total. The server 100 determines that data 1 has been migrated, and then the server 100 scans data 2 to determine whether data 2 has been migrated.

When the server 100 determines that the currently scanned single object has not been migrated, the status of the currently scanned single object in the migration metadata record table is "N", and S802 is executed.

S802: Determine whether the single object currently being scanned is being processed.

In some implementations, the server 100 may determine whether the currently scanned single object is being processed. Whether the object is being migrated.

In a possible implementation, if the currently scanned single object is in the process of being migrated but has not been completely migrated, the migration metadata record table may record that the status of the currently scanned single object may be "N+".

In the embodiment of the present application, in addition to the above method, there are other methods for determining whether a single object is being migrated, which are not limited to this.

When the server 100 determines that the currently scanned single object is being migrated and the currently scanned single object does not need to be migrated again from the source data storage area to the target storage area, S801 is executed to scan the next object of the current single object in the source data storage area.

For example, the source data storage area to be migrated is data 1-data 100, which contains 100 objects in total. The server 100 determines that data 1 is being migrated, and then the server 100 scans data 2 to determine whether data 2 has completed migration.

When the server 100 determines that the currently scanned single object is not being migrated, S803 is executed.

S803: Read the currently scanned single object from the source data storage area, migrate the single object to the target storage area, and update the migration metadata record table.

In some implementations, the server 100 may migrate a single object currently being scanned from a source data storage area to a target storage area.

For example, when the source data storage area and the target storage area are stored on the same server 100, the logical address of the currently scanned single object in the source data storage area is L1, and the physical address corresponding to the logical address L1 is W1, and the mapping relationship is that the logical address L1 maps to the physical address W1. The server 100 migrates the currently scanned single object from the source data storage area to the target storage area, which may refer to changing the logical address of the currently scanned single object from the logical address L1 in the source data storage area to the logical address L2 in the target storage area, and changing the mapping relationship to that the logical address L2 maps to the physical address W1.

Optionally, the two physical addresses of the single object currently being scanned stored in the server 100 before and after the migration may be the same or different. In the embodiment of the present application, there is no limitation on this.

In some implementations, after completing the migration of a single object currently being scanned from a source data storage area to a target storage area, the server 100 may update the migration metadata record table.

For example, after a single object currently being scanned is migrated from a source data storage area to a target storage area, the state of the migration metadata record table may be converted from "N" to "Y".

By implementing the above method, the corresponding data migration is completed during the data reading and writing process, and the remaining data is migrated through the background migration task.

This can improve the efficiency of online migration while ensuring that data migration is correct and complete, and reduce the read requests of the data source and extend the life of the storage medium.

The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: ROM or random access memory RAM, magnetic disk or optical disk, etc. The medium of the code.

In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

Claims

A data migration method, the method being applied to a computing device, characterized by comprising:

Receiving a read request or a write request for first data;

Processing the first data according to the read request, and migrating the first data from the source data storage area to the target storage area for storage;

Alternatively, the first data is processed according to the write request, and the first data is migrated from the source data storage area to the target storage area for storage;

The first data is stored in a source data storage area, and the source data storage area and the target storage area are located in the same computing device or in different computing devices.
The method according to claim 1, characterized in that: after receiving the read request or write request for the first data, the method further comprises:

When it is determined that the first data needs to be migrated and the first data has not been migrated, the first data is processed according to the read request or the write request.
The method according to any one of claims 1 to 2, characterized in that the storage address of the first data in the source data storage area is a first address,

The processing of the first data according to the read request and migrating the first data from the source data storage area to the target storage area for storage specifically includes:

The first data is read from the first address according to the read request and stored in a second address in the target storage area.
The method according to any one of claims 1 to 2, characterized in that the storage address of the first data in the source data storage area is a first address, and the storage mode of the first data is block storage,

The processing of the first data according to the write request and migrating the first data from the source data storage area to the target storage area specifically includes:

Determining whether the sizes of the first data and the data block are aligned;

In the case of determining that the first data is aligned with the size of the data block, reading the first data from the first address according to the write request and storing it in the second address of the target storage area; performing write processing on the first data stored in the first address and the second address;

The data block is data that is migrated once in the block storage; and the alignment means that the size of the first data is an integer multiple of the size of the data block.
The method according to claim 4, characterized in that: when the computing device determines that the size of the data block of the first data is not aligned, the computing device continues to acquire a mutex lock of the data block;

Perform write processing on the first data stored at the first address according to the write request to obtain second data, and store the second data in the first address;

The second data is read from the first address and stored in a second address in the target storage area.
The method according to claim 1 or 2 is characterized in that the storage mode of the first data is object storage, and the storage address of the first data in the source data storage area is a first address,

The processing of the first data according to the write request and migrating the first data from the source data storage area to the target storage area specifically includes:

According to the write request, the first data is read from the first address and stored in the second address in the target storage area, and a write process is performed on the first data stored in the first address and the second address.
The method according to claim 1 is characterized in that: after the first data is migrated, a migration metadata record table is updated, wherein the migration metadata record table records the migration status of the first data, the migration status includes the first data needs to be migrated but has not been Any of the following: being migrated, already migrated, or not needing to be migrated.
An electronic device, characterized in that it includes a memory and one or more processors; the memory is coupled to the one or more processors and is used to store computer program code, and the computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method as described in any one of claims 1-7.