CN115729463A

CN115729463A - Data migration method, system and related device

Info

Publication number: CN115729463A
Application number: CN202211292048.5A
Authority: CN
Inventors: 黄潘潘
Original assignee: XFusion Digital Technologies Co Ltd
Current assignee: XFusion Digital Technologies Co Ltd
Priority date: 2022-10-20
Filing date: 2022-10-20
Publication date: 2023-03-03
Also published as: WO2024082857A1

Abstract

The application provides a data migration method, a data migration system and a related device. According to the technical scheme, the online migration of the data can be realized, the efficiency of the data migration can be improved, and the user experience is good.

Description

Data migration method, system and related device

Technical Field

The embodiment of the application relates to the technical field of data migration, in particular to a method, a system and a related device for data migration.

Background

When the capacity of the original storage medium is short, the new and old storage software is replaced, and the life cycle of hardware is expired, data migration is required. At present, with the rise of cloud computing, the application is replaced by a cloud or a cloud manufacturer, the storage mode selected by a user is changed, and thus data migration is also required.

How to improve the efficiency of data migration in the case of realizing online migration is an urgent problem to be solved in the field.

Disclosure of Invention

The application provides a data migration method, a data migration system and a related device, which not only can realize online migration of data, but also can improve the efficiency of data migration, and have good user experience.

In a first aspect, the present application provides a data migration method, where the method is applied to a computing device, and the method is characterized by including:

receiving a read request or a write request for first data;

processing the first data according to the read request, and migrating the first data from the source data storage area to the target storage area for storage;

or processing the first data according to the write request, and migrating the first data from the source data storage area to the target storage area for storage;

wherein the first data is stored in a source data storage area that is located in the same computing device or in a different computing device than the target storage area.

In a possible implementation manner, the migration operation of the first data may be triggered by a read request or a write request for the first data.

In one possible implementation, when the source data storage area and the target data storage area are located in different computing devices, the sending and receiving operations of the first data can be performed between the different computing devices.

According to the method, when the system receives the read request or the write request of the user for the first data, the system only needs to read the first data once in the source data storage area, and the read request or the write request and the migration task of the user can be guaranteed to be realized.

Therefore, compared with a method that the system receives the migration instruction firstly and then receives the read request or the write request, the migration efficiency can be improved on the basis of realizing online migration.

With reference to the first aspect, in some embodiments, after receiving a read request or a write request for the first data, the method further comprises:

and under the condition that the first data needs to be migrated and the first data is not migrated, processing the first data according to the read request or the write request.

In a possible implementation manner, when the first data does not need to be migrated or the first data has already been migrated, a read request or a write request may be directly performed on the first data, and the first data is not migrated from the source data storage area to the target storage area.

Therefore, the first data under various scenes can be covered in an all-around manner, various situations of the first data are considered, and the situation of multiple times of migration is avoided.

With reference to the first aspect, in some embodiments, the storage address of the first data in the source data storage area is a first address,

the processing the first data according to the read request, and migrating the first data from the source data storage area to the target storage area for storage specifically includes:

the first data may be read from the first address and stored in a second address in the target storage area in accordance with the read request.

In a possible implementation manner, the storage manner of the first data may be a storage manner such as block storage, object storage, and the like, which is not limited in this respect.

In a possible implementation manner, if the size of the first data corresponding to the read request is smaller than the size of the data block or the single object, the data block or the single object needs to be cut according to the size of the actually read data, so that part of the data after the cutting process is exactly the first data.

In one possible implementation, the first data may be returned to the user after it has completed migration from the source data storage area to the target data storage area for storage.

Therefore, the read request aiming at the first data can be realized, and the efficiency of migrating the first data can be improved.

With reference to the first aspect, in some embodiments, the storage address of the first data in the source data storage area is a first address, the first data is stored in a block storage manner,

the processing the first data according to the write request, and migrating the first data from the source data storage area to the target storage area, specifically including:

judging whether the sizes of the first data and the data block are aligned or not;

in the case where it is determined that the first data is aligned with the size of the data block, reading the first data from the first address and storing in a second address in the target storage area according to the write request; performing write processing on the first data stored in the first address and the second address;

the data block is data which is migrated once in the block storage; the alignment means that the size of the first data is an integer multiple of the size of the data block.

In one possible implementation, the precondition for responding to the write request is that write mutex needs to be satisfied, that is, the data targeted by two write requests may not have intersection.

In a possible implementation manner, after the writing process of the first data in the first address and the second address is completed, the user may be notified of the completion of the response writing request.

This avoids the situation of write mutexes.

With reference to the first aspect, in some embodiments, in a case that the computing device determines that the sizes of the data blocks of the first data are not aligned, the computing device continuously acquires a mutual exclusion lock of the data blocks;

writing the first data stored in the first address according to the writing request to obtain second data, and storing the second data to the first address;

the second data is read from the first location and stored in a second address in the target storage area.

Therefore, the problem that the size of the first data is not aligned with the size of the data migrated at a single time can be solved, and the writing processing aiming at the first data is realized.

With reference to the first aspect, in some embodiments, the first data is stored as an object storage, a storage address of the first data in the source data storage area is a first address,

the processing the first data according to the write request, and migrating the first data from the source data storage area to the target storage area specifically includes:

according to the write request, the first data is read from the first address and stored in a second address in the target storage area, and the first data stored in the first address and the second address is subjected to write processing.

In one possible implementation, the write processing operation of the object storage can only act on the whole object, and therefore, the alignment judgment of the first data and the single object is not needed.

Therefore, the writing request of the first data aiming at the object storage type can be realized, and the efficiency of migrating the first data can be improved.

With reference to the first aspect, in some embodiments, after the first data completes migration, a migration metadata record table is updated, where the migration metadata record table records a migration status of the first data, and the migration status includes any one of that the first data needs to be migrated but is not migrated, has been migrated, and does not need to be migrated.

In one possible implementation, after the first data is migrated from the source data storage area to the target storage area, the state of the migration metadata record table may be converted from "N" to "Y".

In a possible implementation manner, when the computing device does not receive a read request and a write request of the first data, a background migration task may be started, the data to be migrated is determined, and the system starts to migrate the data blocks or the single objects to be migrated in sequence in the background.

In a second aspect, an embodiment of the present application provides an electronic device, including: one or more processors and memory; the memory is coupled to the one or more processors for storing computer program code comprising computer instructions which are invoked by the one or more processors to cause the electronic device to perform the method as described in the first aspect.

In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device is caused to perform the method according to the first aspect.

It is understood that the electronic device provided by the second aspect and the computer-readable storage medium provided by the third aspect are both used for executing the method provided by the embodiments of the present application. Therefore, the beneficial effects achieved by the method can refer to the beneficial effects in the corresponding method, and are not described herein again.

Drawings

Fig. 1 is a schematic diagram of a framework of a data migration system according to an embodiment of the present application.

Fig. 2 is a schematic block diagram of a structure of a server 100 according to an embodiment of the present disclosure.

Fig. 3 is a flowchart illustrating a data migration method for responding to a read request according to an embodiment of the present application.

Fig. 4 is a schematic flowchart of another data migration method for responding to a read request according to an embodiment of the present application.

Fig. 5 is a schematic flowchart of a data migration method responding to a write request according to an embodiment of the present application.

Fig. 6 is a flowchart illustrating another data migration method for responding to a write request according to an embodiment of the present application.

Fig. 7 is a schematic flowchart of a background data migration method according to an embodiment of the present application.

Fig. 8 is a flowchart illustrating another background data migration method according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and exhaustively described below with reference to the accompanying drawings. In the description of the embodiments herein, "/" means "or" unless otherwise specified, for example, a/B may mean a or B; the "and/or" in the text is only an association relation describing the association object, and indicates that three relations may exist, for example, a and/or B may indicate: three cases of a alone, a and B both, and B alone exist, and in addition, "a plurality" means two or more than two in the description of the embodiments of the present application.

In the following, the terms "first", "second" are used for descriptive purposes only and are not to be understood as implying or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of embodiments of this application, a "plurality" means two or more unless indicated otherwise.

The term "User Interface (UI)" in the following embodiments of the present application is a media interface for interaction and information exchange between an application program or an operating system and a user, and implements conversion between an internal form of information and a form acceptable to the user. The user interface is source code written by java, extensible markup language (XML) and other specific computer languages, and the interface source code is analyzed and rendered on the electronic equipment and finally presented as content which can be identified by a user. A common presentation form of the user interface is a Graphical User Interface (GUI), which refers to a user interface related to computer operations and displayed in a graphical manner. It may be a visual interface element such as text, an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a Widget, etc. displayed in the display of the electronic device.

Data migration (datamigration) may refer to the process of selecting, preparing, extracting, and transforming data, and permanently transferring the data from one computer storage system to another. There are a variety of reasons for data migration that may include, but are not limited to, server or storage device replacement, maintenance or upgrade, application migration, website integration, disaster recovery, and data center migration. In some implementations, data migration may be understood as the duplication of data stored in one computer storage system, copying into another computer storage system.

Currently, some data migration methods do not support online migration. For example, if a user needs to access data being migrated, the user needs to wait for the completion of the data migration before accessing the data. That is, the electronic device cannot respond to a read-write request for the data during the migration of the data.

There are also methods of data migration that are inefficient. For example, when the system migrates data in a snapshot manner, the data needs to be exported from the source data storage area, and then the exported data needs to be imported into the target storage area after migration. If new data occurs due to the fact that a write request is triggered by user operation in the process of migrating the data, and the new data needs to be migrated, the new data needs to be repeatedly exported from a source data storage area, and then the exported new data is imported into a target storage area after migration.

In order to solve the problems caused by the above data migration method, an embodiment of the present application provides a data migration method. In the method, data migration is divided into a foreground migration mode and a background migration mode.

The foreground migration method specifically includes: first, a data set to be migrated is determined, wherein the data set to be migrated may be determined by a manager. When the system receives a read request or a write request of a user for the first data, the system starts to migrate the first data. Wherein the first data belongs to a data set to be migrated.

When the system receives that the operation request of the user for the first data is a read request, the system reads the first data from the source data storage area, migrates the first data to the target storage area, and then returns the first data in the target storage area to the user.

When the system receives that an operation request of a user for first data is a write request, the system reads the first data from a source data storage area, migrates the first data to a target storage area, performs double write processing on the first data in the source data storage area and the first data in the target storage area, and returns the first data in the target storage area to the user; or the system reads the first data first, performs write processing, stores the first data after the write processing to the source data storage area, and then migrates the first data after the write processing to the source data storage area.

The background migration method specifically comprises the following steps: firstly, determining data to be migrated, and migrating the data blocks or the single objects to be migrated in sequence by the system in the background.

In some implementations, foreground migration is prioritized over background migration. That is, when the system receives a read request or a write request of the user for the first data, the system may preferentially select a foreground migration method for migration. When the system does not receive any read request or write request, the system can only select a background migration method to migrate the data for which the migration instruction is needed.

In some implementations, the storage address at which the first data is stored in the source data storage region can be referred to as a first address, and the storage address at which the first data is stored in the target data storage region can be referred to as a second address.

In some implementations, the data after the first data is subjected to the write process may be referred to as second data.

By the method, when the system receives a read request or a write request of a user for the first data, the system can ensure that the read request or the write request and the migration task of the user can be realized only by reading the first data once in the source data storage area.

Therefore, compared with a method that the system receives the migration instruction firstly and then receives the read request or the write request, the migration efficiency can be improved. This is because the system cannot guarantee that the data receiving the migration instruction and the data receiving the read request or the write request are the same data during the migration process, which may cause a phenomenon that the system may read a certain data twice in the source data storage area when the read request or the write request of the user and the migration task are implemented.

Therefore, the data migration method provided by the embodiment of the application can not only realize the online migration of the data, but also improve the efficiency of the data migration, and has good user experience.

In order to more clearly describe the method provided in the embodiment of the present application, a data migration system framework provided in the embodiment of the present application is described first below.

Referring to fig. 1, fig. 1 is a schematic diagram of a framework of a data migration system according to an embodiment of the present application.

The data migration system may include, but is not limited to, a source data storage area, a target data storage area, a data migration tool, and a storage medium, among others. The data migration system can be applied to application scenes that local data are migrated to a cloud end, data migration is carried out between the cloud ends, and cloud end data are migrated to the local area.

It should be noted that the source data storage area and the target data storage area may refer to a logical storage location of a data block or a single object in the server, and the storage medium may refer to a physical storage location of a data block or a single object in the server.

The source data storage area may refer to a database storing data that is not migrated, and common databases may include, but are not limited to, a relational database manager (My Structured Query Language, MYSQL), an Oracle database, a Microsoft SQL Server, and the like.

The type of data stored in the source data storage area may include, but is not limited to, block storage, file storage, object storage, and the like. For example, when the data stored in the source data storage area is stored in the form of block storage, the source data storage area may be a volume or a Logical Unit Number (lun), and the data stored in the source data storage area may be divided into data blocks of uniform size (e.g., the data blocks may be 1MB size data).

In some implementations, when data stored in the source data storage area is migrated, the data may be migrated in units of data blocks.

For another example, when the data stored in the source data storage area is stored in the form of object storage, the source data storage area may serve as a bucket, where the data stored in the source data storage area is a single object. That is, the data migration may be performed in units of data.

The target data storage area may refer to a database storing migrated data. Similarly, the type of the data stored in the target data storage area may include, but is not limited to, block storage, file storage, object storage, and the like, and for the detailed description of the storage type, reference may be made to the above contents, which are not described herein again.

In some implementations, the source data storage area and the target data storage area may be located on the same hardware device or may be located on different hardware devices.

The rectangle a shown in the source data storage area and the rectangle b shown in the source data storage area shown in FIG. 1 may represent a block of data or a single object. That is, if the migration task of the data set to be migrated in the source data storage area is to be completed, multiple data blocks or a single object need to be migrated.

For example, in a complete migration task, a data block or a single object in a rectangle a may be migrated into a rectangle b, where the data block or the single object represented by the rectangle b is data after the data block or the single object is migrated, and the data of the data block or the single object is consistent in content.

The data migration tool can realize the capture of the request, the distribution of the data, the storage and the update of the migration metadata record table, the background migration task and the like.

In particular, the capture of the request may be used to monitor a read request or a write request from a user for the first data, thereby starting the foreground migration task.

The specific location at which the acquisition of the request is accomplished may vary. In some embodiments, the capture of the request may be accomplished at the Device Mapper layer in Linux. When the server storing the data is connected with the corresponding client, the request can be captured at the client. This is not limited in the embodiments of the present application.

In some implementations, after completing the capture of the request, the data migration tool obtains a logical address of the first data targeted by the request from the source data storage area and reads the first data from the storage medium according to the logical address.

That is, the source data storage area and the target data storage area are logical storage locations of the first data, and the storage medium is a physical storage location of the first data.

The storage medium may refer to a carrier for storing the first data, such as a floppy disk, an optical disk, a Digital Video Disc (DVD), a hard disk, a Flash Memory, a usb disk, a Compact Flash (CF) Card, a Secure Digital Card (SD Card), a multimedia Card (MMC Card), a Smart Media (SM) Card, a Memory Stick (Memory Stick), an Extreme Digital Video Card (xD Card), and the like.

The distribution of the data may be used to perform a migration process on the first data after the data migration tool reads the first data. In some implementations, the data migration tool may determine whether the first data needs to be migrated, i.e., whether the first data belongs to the set of data to be migrated. If the first data belongs to the data set to be migrated, the data migration tool may migrate the first data to the target storage area.

The migration metadata record table is used to record the migration status (e.g., non-migrated, and initial status) of the data. In some implementations, the data migration tool may store and update the migration metadata record table. Specifically, the migration metadata record tables shown in table 1 and table 2 may be referred to.

TABLE 1

Index	State of state
		Block 1	Y
Block 2	N
		……	……
Block N	NA

TABLE 2

Index	State of state
		Object 1	Y
Object 2	N
		……	……
Object N	NA

As shown in table 1 and table 2, table 1 exemplarily shows a block storage migration metadata record table, and table 2 exemplarily shows an object storage migration metadata record table.

The migration metadata record table may include an index and a migration status, and the migration metadata record table may further include any one or more of the following items: the identifier of the server stored when the data is not migrated, the identifier of the server stored after the data has been migrated, the logical location stored when the data is not migrated, the logical location stored after the data has been migrated, and the like, which are not limited in the embodiment of the present application.

The indexes shown in tables 1 and 2 may be used to indicate a structure for sorting data in the source data storage area, and the index can be used to quickly view specific data in the source data storage area.

For example, the index "Block 1" indicates an index of a data Block divided into data blocks of the same size (for example, 1 MB) in the data stored in the form of a Block storage, and the index "object 1" indicates an index of a single object in the data stored in the form of an object storage.

In some implementations, the index of the data is related to the storage type of the data.

For example, if the storage type of a certain data is block storage, the index of the data is volume ID + logical address; if the storage type of a certain data is file storage, the index of the data is file system ID (Filesystem ID) + Directory (Directory) + file name (Filename) + Offset address (Offset); if the storage type of a certain data is object storage, the index of the data is Key (Key) + Version number (Version).

The migration status shown in tables 1 and 2 indicates whether or not the data in the source data storage area is in the migration status.

Where state "Y" represents the case where the data indicated by the particular index in the row is currently in need of being migrated and has completed being migrated.

The state "N" indicates that the data indicated by the particular index in the row is currently in a state that requires migration but does not perform migration, or that requires migration and is being migrated but is not completely migrated.

The state "NA" indicates that the data indicated by the particular index in the row is currently in a state that does not require migration, and is not in a non-migrated state.

In some implementations, the data migration tool can also change the state of the data in real-time. For example, after data 1 needs to be migrated and the migration is completed, the state of data 1 may be converted from "N" to "Y". In general, data 1 will not be converted from "Y" to "N".

The same data indicated by an index in a different migration metadata record table may be in a different state in a different table. For example, the state of data 1 in table a is "Y" and the state in table B is "N", that is, data 1 has been migrated from the logical location stored when it was not migrated in table a to the logical location stored after migration, but has not been migrated from the logical location stored when it was not migrated in table B to the logical location stored after migration.

It should be noted that, since the migration metadata record table is stored in the server 100, the state of the data of the migration unit is recorded, and the migration metadata record table has updating performance and durability, in this embodiment of the present application, breakpoint resuming and retry of migration can be supported. For example, when the data cannot be copied due to a network problem, the data copying is interrupted, and the original migration task is not affected. After the network is restored, the data continues to be copied from the interruption.

For example, when the system is responding to a write request and performing write processing on a data block or object that has already been migrated while a network is in a trouble and cannot be copied, the state of the data block or object corresponding to the migration metadata record table is recorded as "N", indicating that the data block or object indicates that the migration needs to be continued as a dirty block. And after the network is recovered, the migration tool continues to recover the original migration process and migrates the data and the dirty blocks which are not migrated to the target. That is, the data blocks or single objects processed at the moment when the network has a problem are reprocessed, so as to complete accurate migration tasks.

In some implementation manners, for the same source data storage area, migration of multiple tasks to different targets can be achieved, and two different migration tasks of the same source data storage area are independent from each other and cannot affect each other. For simultaneous migration of different source data storage areas, a background also has multiple migration tasks, and respective migration metadata tables for the different migration tasks are also provided, and the multiple migration tasks are independent from each other and cannot be influenced.

The background migration task is different from the foreground migration task, and refers to scanning data to be migrated in real time in sequence, determining unit data not to be migrated in the data to be migrated, and migrating the unit data not to be migrated (e.g., a data block or a single object) in the data to be migrated.

For example, the data to be migrated is 100 units of data from data 1 to data 100, if there is no read request or write request for any data from data 1 to data 100, the background migration task starts scanning from data 1 to data 100, and determines that data 1 is non-migrated data when scanning data 1, and migrates data 1, and the remaining 99 data are migrated in the same manner;

if a read request or a write request for the data 10 exists, the foreground migration task first migrates the data 10, then the background migration task starts scanning from the data 1 to the data 100, and then migrates the data 1 to the data 9 and the data 11 to the data 100 in sequence, and determines that the data 1 is non-migrated data when the data 1 is scanned, migrates the data 1, and migrates the rest 98 data in the same manner, but determines that the data 10 is migrated data when the data 10 is scanned, and does not migrate the data 10.

Therefore, only the data blocks or single objects which are not read and written by the foreground need to be migrated, the read and written data cannot be migrated repeatedly, the reading and writing of the data are reduced, and the migration efficiency is improved.

In order to more clearly describe the method provided by the embodiment of the present application, the following server 100 is provided by the embodiment of the present application.

Referring to fig. 2, fig. 2 is a schematic structural diagram of a server 100 according to an embodiment of the present disclosure.

As shown in fig. 2, the server 100 may include: one or more processors 101, memory 102, communication interface 103, transmitter 105, receiver 106, coupler 107, and antenna 108. The above components may be connected by a bus 104 or otherwise, and fig. 2 is an example of a connection by a bus. Wherein:

in embodiments of the present application, the processor 101 may be configured to read and execute computer readable instructions. Specifically, the processor 101 may be configured to call a program stored in the memory 102, for example, an implementation program of the data migration method provided in the embodiment of the present application, in the server 100, and execute instructions contained in the program. The processor 101 may also be configured to determine whether migration is required based on the migration metadata record table.

The memory 102 is coupled to the processor 101 for storing various software programs and/or sets of instructions. In particular, the memory 102 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state storage devices.

The memory 102 may store information such as data sets to be migrated and migration metadata records.

The memory 102 may store an operating system (hereinafter referred to as a system), such as an embedded operating system (os) like uCOS, vxWorks, and RTLinux. The memory 102 may also store network communication programs that may be used to communicate with clients and other servers.

The communication interface 103 may be used for the server 100 and other communication devices.

Specifically, the communication interface 103 may be a 3G communication interface, a Long Term Evolution (LTE) (4G) communication interface, a 5G communication interface, a WLAN communication interface, a WAN communication interface, or the like. Not limited to wireless communication interfaces, the server 100 may also be configured with a wired communication interface 103 to support wired communication.

The transmitter 105 and the receiver 106 may be considered as one wireless modem. The transmitter 105 may be used to transmit the signal output by the processor 101. The receiver 106 may be used to receive signals.

In the server 100, the number of the transmitters 105 and the receivers 106 may be one or more.

The antenna 108 may be used to convert electromagnetic energy in the transmission line to electromagnetic energy in free space or vice versa.

Coupler 107 may be used to multiplex the mobile communications signal for distribution to a plurality of receivers 106.

It can be appreciated that the antennas 108 of the network device can be implemented as a massive antenna array.

It should be noted that the server 100 shown in fig. 2 is only one implementation manner of the embodiment of the present application, and in practical applications, the server 100 may further include more or less components, which is not limited herein.

It should be noted that, according to different requirements of users and other comprehensive considerations, the servers stored in the source data storage area and the target data storage area may be the same server or may be two different servers.

The data migration tool may be executed on a server stored in the source data storage area, may be executed on a server stored in the target data storage area, and may be executed on other servers besides the above-mentioned cases.

In some embodiments, the data migration tool may run on both the server on which the source data storage area is stored and the server on which the target storage area is stored. In the embodiments of the present application, this is not limited.

When the server stored in the source data storage area and the server stored in the target data storage area in the data migration system in fig. 1 are the same server 100 shown in fig. 2, and the data migration tool runs on the server 100, the processor 101 may be configured to respond to a read request and a write request of a user, and intercept the requests.

The processor 101 may also determine whether the data needs to be migrated according to the migration metadata record table. The processor 101 may also update the state information of the data on the migration metadata record table after the data completes the migration.

The memory 102 may be used to store information such as data to be migrated, migrated data, and a migration metadata record table.

The communication interface 103 may be used to make a communication connection with an electronic device used by a user.

The transmitter 105 may be configured to return data required by the user to the electronic device used by the user, the receiver 106 may be configured to receive a read request or a write request from the user, and the receiver 106 may be further configured to receive a migration instruction for the first data, for example, the receiver 106 may be further configured to receive a migration instruction for the first data sent by the control center.

When the server stored in the source data storage area and the server stored in the target storage area in the data migration system in fig. 1 are not the same as the server 100 shown in fig. 2, the server stored in the source data storage area is the server 110, the server stored in the target storage area is the server 120, the data migration tool may run on the server 110 and the server 120 at the same time, and the server 100 includes the server 110 and the server 120.

The processor 101 of the server 110 may be configured to respond to a read request and a write request of a user and intercept the requests. The processor 101 of the server 110 may further determine whether the data needs to be migrated according to the migration metadata record table. The processor 101 of the server 110 may update the migration metadata record table after sending the data to be migrated is completed.

The memory 102 of the server 110 may be used to store information such as data to be migrated and a migration metadata record table.

The communication interface 103 of the server 110 may be used to communicatively connect with the server 120.

The transmitter 105 of the server 110 may be used to send data to be migrated to the server 120.

The receiver 106 of the server 110 may be configured to receive a read request or a write request of a user, and the receiver 106 of the server 110 may be further configured to receive a migration instruction for the first data sent by the control center.

The processor 101 of the server 120 may update the migration metadata record table after the receiving server 110 finishes sending the data to be migrated.

The memory 102 of the server 120 may be used to store information such as migrated data and migration metadata records.

The communication interface 103 of the server 120 may be used to communicatively connect with the server 110.

The transmitter 105 of the server 120 may be configured to send a notification to the server 110 that the data to be migrated has completed migration, and the receiver 106 of the server 120 may be configured to receive the data to be migrated sent by the server 110.

In some implementations, the server 100 may be referred to as a computing device.

In some implementations, the electronic device used by the user may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a Personal Digital Assistant (PDA), an Augmented Reality (AR) device, a Virtual Reality (VR) device, an Artificial Intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, and/or a smart city device, and the embodiment of the present application does not particularly limit the specific type of the electronic device.

A method of data migration provided in an embodiment of the present application is described below. This embodiment describes the data migration method by way of example in the case where the server 100 responds to a user's read request for data to be migrated.

Fig. 3 is a flowchart illustrating a data migration method responding to a read request provided in an embodiment of the present application. In this embodiment, the storage mode of data is block storage, and the migration unit is a data block as an example.

S301, the server 100 responds to the user' S read request for the first data.

In some embodiments, the user may perform an input operation to the server 100 according to the user's requirement, and the server 100 may respond to a read request for the first data. The read request of the first data may include, but is not limited to: a logical address of the first data in the source data storage area, an index, and a data payload length, etc., wherein the data payload length refers to a payload data length excluding a header and a length in the read request information.

S302, judging whether the first data needs to be migrated or not.

In some embodiments, the server 100 may receive a migration instruction for the first data issued by the control center before responding to the read request.

The system may be a migration instruction issued due to a remaining storage space of the server 100, or the server 100 performs upgrade update, or the server 100 is damaged, or may be a migration instruction issued based on an input operation of a manager. In the embodiments of the present application, this is not limited.

If there is a migration instruction for the first data, the state of the first data in the migration metadata record table is "Y" or "N" instead of "NA".

When the server 100 determines that the first data needs to be migrated, that is, there is a migration instruction for the first data, and the state of the first data in the migration metadata record table is "Y" or "N", S303 is executed.

When the server 100 determines that the first data does not need to be migrated, that is, there is no migration instruction for the first data, and the state of the first data in the migration metadata record table is "NA", S304 is executed.

In some implementations, since the record unit in the migration metadata record table is in units of the size of the data block, the state of the first data may refer to:

if the size of the first data is larger than or equal to the data block, the state of the first data is the state of the data block corresponding to the first data, and if the size of the first data is smaller than the data block, the state of the first data is the state of the data block corresponding to the first data.

S303, judging whether the first data is migrated or not.

In some implementations, the server 100 may determine whether the first data has completed the migration according to the migration metadata record table.

For example, if the state corresponding to the index carried in the read request for the first data in the migration metadata record table is "Y", it may be determined that the first data has completed migration; if the state corresponding to the index carried in the read request for the first data in the migration metadata record table is "N", it may be determined that the first data is not migrated or is being migrated and has not been migrated yet.

When the server 100 determines that the first data has completed the migration, the state of the first data in the migration metadata record table is "Y", and the first data does not need to be migrated from the source data storage area to the target storage area again, then S304 is executed.

When the server 100 determines that the first data is not migrated, but it is known from S303 that there is a migration instruction for the first data, the state of the first data in the migration metadata record table is "N", and S305 is executed.

Further, in a possible implementation manner, if the first data is being migrated but is not completely migrated, the migration metadata record table may record the state of the first data as "N +".

When the server 100 determines that the first data is being migrated but is not completely migrated, the first data does not need to be migrated again, and S304 is executed.

S304, reading the first data from the source data storage area and returning the first data to the user.

In some implementations, the server 100 can read the first data out of the source data storage area based on the read request and return the first data to the user.

Specifically, if the first data is stored in the cloud, and the user wants to read the first data, the cloud may obtain a logical address of the first data according to a read request of the first data sent by the local user, obtain a physical address of the first data according to a mapping relationship between the logical address and the physical address, read the first data from the physical address, and send the read first data to the local user.

S305, reading out the first data from the source data storage area, migrating the first data to the target storage area, updating the migration metadata record table, and returning the first data to the user.

In some implementations, the server 100 can migrate the first data from the source data storage area to the target storage area. For example, when the source data storage area and the target data storage area are stored in the same server 100, the logical address of the first data in the source data storage area is L1, the physical address corresponding to the logical address L1 is W1, and the mapping relationship is that the logical address L1 maps to the physical address W1.

Migrating the first data from the source data storage area to the target storage area by the server 100 may refer to changing a logical address of the first data from a logical address L1 in the source data storage area to a logical address L2 in the target storage area, and changing a mapping relationship to a logical address L2 to map the physical address W1.

Optionally, the two physical addresses stored in the server 100 before and after the first data migration may be the same or different. In the embodiments of the present application, this is not limited.

In some implementations, the server 100 may update the migration metadata record table after completing the migration of the first data from the source data storage area to the target storage area.

For example, after the first data is migrated from the source data storage area to the target storage area, the state of the migration metadata record table may be converted from "N" to "Y".

In a possible embodiment, if the size of the first data that the user wants to read is smaller than the data block, the server 100 needs to perform a cutting process on the data block according to the size of the actually read data, so that the partial data after the cutting process is exactly the first data.

For example, the first data size that the user wants to read is 0.5MB, and the data block size is 1MB, at this time, the data block size read by the server 100 from the source data storage area is 1MB, where the data block of 1MB includes the first data of 0.5MB and other data of 0.5MB, and the server 100 needs to divide the data block of 1MB size, select the first data of 0.5MB from the divided data blocks, and return the first data of 0.5MB to the user.

In some implementations, recording metadata may be understood as updating a migration metadata record table.

Fig. 4 is a flowchart illustrating another data migration method responding to a read request provided in an embodiment of the present application. In this embodiment, a storage manner of data is used as object storage, and a migration unit is a single object as an example.

S401, the server 100 responds to the user' S read request for the first data.

In some embodiments, the user may perform an input operation to the server 100 according to the user's own requirement, and the server 100 may respond to a read request for the first data.

The read request of the first data may include, but is not limited to: a logical address of the first data in the source data storage area, an index, and a data payload length, etc., wherein the data payload length refers to a payload data length excluding a header and a length in the read request information.

S402, judging whether the first data needs to be migrated or not.

The system may be a migration instruction issued due to the remaining accommodation space of the server 100, or the server 100 performs upgrade updating, or the server 100 is damaged, or the like, or may be a migration instruction issued based on an input operation of a manager. In the embodiments of the present application, this is not limited.

When the server 100 determines that the first data needs to be migrated, that is, there is a migration instruction for the first data, and the state of the first data in the migration metadata record table is "Y" or "N", S403 is executed.

When the server 100 determines that the first data does not need to be migrated, that is, there is no migration instruction for the first data, and the state of the first data in the migration metadata record table is "NA", S404 is executed.

In some implementations, since the record unit in the migration metadata record table is in units of the size of a single object, the state of the first data may refer to:

if the size of the first data is larger than or equal to a single object, the state of the first data is the state of the single object corresponding to the first data divided into the plurality of single objects, and if the size of the first data is smaller than the single object, the state of the first data is the state of the single object corresponding to the first data.

S403, judging whether the first data is migrated or not.

For example, if the state corresponding to the index carried in the read request for the first data in the migration metadata record table is "Y", it may be determined that the first data has completed migration;

if the state corresponding to the index carried in the read request for the first data in the migration metadata record table is "N", it may be determined that the first data is not migrated or is being migrated and has not been migrated yet.

When the server 100 determines that the migration of the first data has been completed, the state of the first data in the migration metadata record table is "Y", and the first data does not need to be migrated from the source data storage area to the target storage area again, then S404 is performed.

When the server 100 determines that the first data is not migrated, but it is known from S403 that there is a migration instruction for the first data, the state of the first data in the migration metadata record table is "N", and S405 is executed.

Further, in a possible implementation manner, if the first data is in the state of being migrated but not completely migrated, the migration metadata record table may record that the state of the first data may be "N +".

When the server 100 determines that the first data is being migrated but not completely migrated, the first data does not need to be migrated again, and S404 is executed.

And S404, reading the first data from the source data storage area and returning the first data to the user.

In some implementations, the server 100 can read the first data from the source data storage area based on the read request, and return the first data to the user.

S405, reading out the first data from the source data storage area, migrating the first data to the target storage area, updating a migration metadata record table, and returning the first data to a user.

In some implementations, the server 100 can migrate the first data from the source data storage area to the target storage area. For example, when the source data storage area and the target storage area are stored in the same server 100, the logical address of the first data in the source data storage area is L1, the physical address corresponding to the logical address L1 is W1, and the mapping relationship is that the logical address L1 maps the physical address W1.

Alternatively, the two physical addresses stored in the server 100 before and after the first data migration may be the same or different. In the embodiments of the present application, this is not limited.

In a possible implementation manner, if the size of the first data that the user wants to read is smaller than that of a single object, the server 100 needs to perform a cutting process on the single object according to the size of the actually read data, so that the partial data after the cutting process is exactly the first data. For example, the first data size that the user wants to read is 0.5MB, and the single object size is 1MB, at this time, the single object size that the server 100 reads from the source data storage area is 1MB, where the single object of 1MB includes the first data of 0.5MB and other data of 0.5MB, and the server 100 needs to divide the single object of 1MB size, select the first data of 0.5MB from the single object, and return the first data of 0.5MB to the user.

By implementing the method, online migration can be realized, and when the system receives a read request of a user for the first data, the system can ensure that the read request and the migration task of the user can be realized only by reading the first data in the source data storage area once. Thus, migration efficiency can be improved.

Another data migration method provided in the embodiment of the present application is described below. This embodiment describes the data migration method by taking as an example the case where the server 100 responds to a write request from a user for data to be migrated.

Fig. 5 is a flowchart illustrating a data migration method responding to a write request provided in an embodiment of the present application. In this embodiment, the storage mode of data is block storage, and the migration unit is a data block as an example.

S501, the server 100 responds to a write request for the first data from the user.

In some embodiments, the user may perform an input operation to the server 100 according to the user's own requirement, and the server 100 may respond to a write request for the first data.

The write request of the first data may include, but is not limited to: the logical address of the first data in the source data storage area, the index, the data payload length, and the modified content, among others.

In some implementations, the server 100 has a precondition in response to a write request: the server 100 responds to the write request of the user for the first data, and the write-write mutual exclusion needs to be satisfied, namely, the data targeted by the two write requests cannot have an intersection.

If the data aimed at by the two write requests have intersection, a locking and mutual exclusion mode needs to be adopted, so that the phenomenon of write-write conflict is avoided.

S502, judging whether the first data needs to be migrated or not, if not, writing the first data in the source data storage area, and storing the written first data in the source data storage area.

In some embodiments, the server 100 may receive a migration instruction for the first data issued by the system before responding to the write request.

If there is a migration instruction for the first data, the state of the first data in the migration metadata record table is not "NA", but "Y" or "N".

When the server 100 determines that the first data does not need to be migrated, that is, there is no migration instruction for the first data, and the state of the first data in the migration metadata record table is "NA", the server 100 may directly process the first data according to the write request to obtain the first data after writing, and store the first data after writing in the source data storage area, that is, the logical addresses of the first data and the first data after writing are the same.

Alternatively, the server 100 may notify the user of the completion of the write request. For example, the user deletes the document M, and the server 100 may notify the user that the server 100 has completed the user's write request for the document M by displaying a prompt message (e.g., displaying a pop-up window) such as the text message "deleted document M". In the embodiment of the present application, the manner in which the server 100 notifies the user that the write request has been completed is not limited.

When the server 100 determines that the first data needs to be migrated, that is, there is a migration instruction for the first data, and the state of the first data in the migration metadata record table is "Y" or "N", S503 is executed.

if the size of the first data is larger than or equal to the data block, the state of the first data is the state of the first data divided into a plurality of data blocks.

If the size of the first data is smaller than the data block, the state of the first data is the state of the data block corresponding to the first data.

S503, judging whether the first data is migrated or not, and if so, writing the first data in the source data storage area and the target data storage area in a double-writing mode.

For example, if the state corresponding to the index carried in the write request for the first data in the migration metadata record table is "Y", it may be determined that the first data has completed migration; if the state corresponding to the index carried in the write request for the first data in the migration metadata record table is "N", it may be determined that the first data is not migrated or is being migrated and has not been migrated yet.

When the server 100 determines that the migration of the first data has been completed, the state of the first data in the migration metadata record table is "Y", and the first data does not need to be migrated from the source data storage area to the target storage area again, the server 100 may process the first data in the source data storage area and the first data in the target storage area according to the write request, store the first data in the source data storage area after writing into the source data storage area, and store the first data in the target storage area after writing into the target storage area.

In this way, the dual-write source data storage area and the first data in the target storage area may maintain the content consistency of the first data after the first data has completed migration and the first data in the source data storage area has not been deleted. Optionally, the server 100 may notify the user of the information of completing the write request, which is not described herein again.

When the server 100 determines that the first data is not migrated, but it is known from S502 that a migration instruction for the first data exists, the state of the first data in the migration metadata record table is "N", and S504 is executed.

When the server 100 determines that the first data is in the migration state but is not completely migrated, the first data does not need to be migrated from the source data storage area to the target storage area again, and the server 100 may respectively process the first data in the source data storage area and the first data in the target storage area according to the write request, store the first data in the source data storage area after writing into the source data storage area, and store the first data in the target storage area after writing into the target storage area. Optionally, the server 100 may notify the user of the information of completing the write request, which is not described herein.

S504, whether the data block to be migrated is aligned with the first data is judged.

In some implementations, when the storage manner of the data is block storage, the migration unit is a data block, and the server 100 may determine that the data block is aligned with the first data.

For example, if the size of the first data is 0.5MB and the size of the data block is 1MB, the data block is not aligned with the first data. That is, when the size of the first data is smaller than the data block size, the data block is not aligned with the first data.

If the size of the first data is 1.5MB and the size of the data block is 1MB, the migration unit of the first data is 1.5, and the data block is not aligned with the first data. That is, when the size of the first data is not an integer multiple of the data block, the data block is not aligned with the first data.

For another example, if the size of the first data is 1MB and the size of the data block is 1MB, the data block is aligned with the first data. That is, when the size of the first data is equal to the data block size, the data block is aligned with the first data.

If the size of the first data is 2MB and the data block size is 1MB, the migration unit of the first data is 2, and the data block is aligned with the first data. That is, when the size of the first data is an integer multiple of the data block, the data block is aligned with the first data.

When the server 100 determines that the data block is aligned with the first data, the size of the data block coincides with the size of the first data or the size of the first data is an integer multiple of the size of the data block, S505 is performed.

When the server 100 determines that the data block is not aligned with the first data, two cases may be used:

first, the size of the first data is smaller than the size of the data block, at this time, S506 is executed;

second, the size of the first data is larger than the data block size and is not an integer multiple of the data block size, in this case, the first data may be divided into two parts, the first part may be data with the size of the largest integer multiple of the data block size, and the second part may be data with the first part removed from the first data. Wherein the first portion of data is executed S505 and the second portion of data is executed S506.

And S505, migrating the first data to the target storage area, double-writing the first data in the source data storage area and the target data storage area, and updating the migration metadata record table.

In some implementations, the server 100 can migrate the first data from the source data storage area to the target storage area.

For example, when the source data storage area and the target storage area are stored in the same server 100, the logical address of the first data in the source data storage area is L1, the physical address corresponding to the logical address L1 is W1, and the mapping relationship is that the logical address L1 maps the physical address W1.

For another example, when the server stored in the source data storage area and the server stored in the target storage area are not on the same server, that is, the server stored in the source data storage area is the server 110, and the server stored in the target storage area is the server 120, the server 110 stored in the source data storage area transmits the first data stored at the logical address L1 and the physical address W1 to the server 120 stored in the target storage area, and the server 120 stores the first data in the target storage area, where the logical address of the first data in the server 120 stored in the target storage area is L2 and the physical address is W2.

Optionally, the two physical addresses stored in the server before and after the first data migration may be the same or different. That is, the storage medium on which the first data stored in the source data storage area is physically stored may be the same as or different from the storage medium on which the first data stored in the target data storage area is physically stored. In the embodiments of the present application, this is not limited.

In some implementations, the server 100 can double write the first data in the source data storage area and the first data in the target storage area. Specifically, the server 100 may process the first data in the source data storage area and the first data in the target storage area according to the write request, store the first data in the source data storage area after writing into the source data storage area, and store the first data in the target storage area after writing into the target storage area.

In some implementations, the server 100 can update the migration metadata record table after completing the writing of the first data in the source data storage area and the first data in the target storage area. For example, the state of the first data in the migration metadata record table may be converted from "N" to "Y".

Optionally, the server 100 may notify the user of the information of completing the write request, which may refer to the above contents and is not described herein again.

S506, writing the first data in the source data storage area, migrating the first data after writing to the target storage area, and updating the migration metadata record table.

In some implementations, the size of the first data in S506 is not aligned with the size of the data block, so that there may be an overlapping area of the data for other write requests with respect to the first data for the current write request, that is, the data may be written to the same area. Therefore, a mutual exclusive lock within a data block needs to be acquired before writing the first data in the source data storage area. If the mutual exclusion lock is failed to be acquired, the acquisition is repeated until the acquisition is successful.

The obtaining of the mutex lock successfully may be to stop the write operation on the first data in the other threads, and only execute the write operation in S506. Thus, the phenomenon of write collision can be avoided.

In some implementations, the server 100 can write the first data in the source data storage area. Specifically, the server 100 may perform write processing on the first data stored in the source data storage area according to the write request, and store the first data after write into the source data storage area. In some implementations, the first data after the write may be referred to as second data.

The migration of the first data from the source data storage area to the target storage area by the server 100 may refer to changing a logical address of the first data from a logical address L1 in the source data storage area to a logical address L2 in the target storage area, and changing a mapping relationship to the logical address L2 to map the physical address W1.

For another example, when the server stored in the source data storage area and the server stored in the target storage area are not the same and are on different servers 100, that is, the server stored in the source data storage area is the server 110, and the server stored in the target storage area is the server 120, the server 110 stored in the source data storage area transmits the first data stored at the logical address L1 to the server 120 stored in the target storage area, the server 120 stores the first data in the target source target storage area, and the logical address of the first data in the server 120 stored in the target source target storage area is L2.

Alternatively, the two physical addresses stored in the server 100 before and after the first data migration may be the same or different. That is, the storage medium on which the first data stored in the source data storage area is physically stored may be the same as or different from the storage medium on which the first data stored in the target data storage area is physically stored. In the embodiments of the present application, this is not limited.

In some implementations, the server 100 may update the migration metadata record table after completing the migration of the first data from the source data storage area to the target storage area. For example, the state of the first data in the migration metadata record table may be converted from "N" to "Y".

Optionally, the server 100 may notify the user of the information of completing the write request, which is not described herein again.

By implementing the method, online migration can be realized, and because the migration of the data is triggered by the write request, when the system receives the write request of the user for the first data, the system can ensure that the write request and the migration task of the user can be realized only by reading the first data in the source data storage area once. Thus, migration efficiency can be improved.

Fig. 6 is a flowchart illustrating another data migration method responding to a write request provided in an embodiment of the present application. In this embodiment, a storage manner of data is used as object storage, and a migration unit is a single object as an example.

S601, the server 100 responds to a write request for the first data from the user.

In some embodiments, the user may perform an input operation to the server 100 according to the user's requirement, and the server 100 may respond to a write request for the first data. The write request of the first data may include, but is not limited to: the logical address of the first data in the source data storage area, the index, the data payload length, and the modified content, among others.

S602, judging whether the first data needs to be migrated, if not, writing the first data in the source data storage area, and storing the written first data in the source data storage area.

When the server 100 determines that the first data does not need to be migrated, that is, there is no migration instruction for the first data, and the state of the first data in the migration metadata record table is "NA", the server 100 may directly perform write processing on the first data according to the write request to obtain the first data after writing, and store the first data after writing into the source data storage area, that is, the logical addresses of the first data before write processing and the first data after write processing are the same.

When the server 100 determines that the first data needs to be migrated, that is, there is a migration instruction for the first data, and the state of the first data in the migration metadata record table is "Y" or "N", S603 is executed.

In some implementations, since the record unit in the migration metadata record table is in units of the size of a single object, the state of the first data may refer to: if the size of the first data is larger than or equal to a single object, the state of the first data is the state of the single object corresponding to the first data divided into the plurality of single objects, and if the size of the first data is smaller than the single object, the state of the first data is the state of the single object corresponding to the first data.

S603, judging whether the first data is migrated or not, and if so, writing the first data in the source data storage area and the target data storage area doubly.

In some implementations, the server 100 can determine whether the first data has completed migration according to the migration metadata record table.

In this way, the dual-write source data storage area and the first data in the target storage area can maintain the content consistency of the first data after the first data has completed the migration and the first data in the source data storage area has not been deleted. Optionally, the server 100 may notify the user of the information of completing the write request, which is not described herein again.

When the server 100 determines that the first data is not migrated, but it is known from S602 that there is a migration instruction for the first data, the state of the first data in the migration metadata record table is "N", and S604 is executed.

When the server 100 determines that the first data is in the migration state but is not completely migrated, the first data does not need to be migrated from the source data storage area to the target storage area again, and the server 100 may perform write processing on the first data in the source data storage area and the first data in the target storage area according to the write request, store the first data in the source data storage area after writing into the source data storage area, and store the first data in the target storage area after writing into the target storage area. Optionally, the server 100 may notify the user of the information of completing the write request, which is not described herein again.

S604, migrating the first data to the target storage area, writing the first data in the source data storage area and the target data storage area in a double-writing mode, and updating a migration metadata record table.

In some implementations, since the first data is stored as an object and the writing process on the object is generally performed on the entire object, there is no need to determine whether the first data is aligned with a single object like S504. And directly responding to the write operation and completing the migration task of the first data under the condition that the first data needs to be migrated and is not migrated yet.

In some implementations, the server 100 can double write the first data in the source data storage area and the first data in the target storage area.

Specifically, the server 100 may process the first data in the source data storage area and the first data in the target storage area according to the write request, store the first data in the source data storage area after writing into the source data storage area, and store the first data in the target storage area after writing into the target storage area.

Optionally, the server 100 may notify the user of the information of completing the write request, which is not described herein.

Another data migration method provided in the embodiment of the present application is described below. This embodiment describes a method of data migration by taking as an example a case where there is no read request and no write request by a user for data to be migrated.

Fig. 7 shows a flowchart of a background data migration method provided in an embodiment of the present application. The flow chart is illustrated by a block storage. It should be noted that, the source data storage area targeted by the background data migration method has a migration instruction, and whether the migration instruction exists is not determined in the following specific steps.

S701, traversing the source data storage areas to be migrated in sequence, and judging whether the currently scanned data blocks have been migrated.

In some implementations, the server 100 can traverse the source data storage areas to be migrated in a sequential order. For example, the source data storage area to be migrated is 100 data blocks from data 1 to data 100, the server 100 starts scanning from data 1 to data 100, and determines whether each data block has completed migration.

In some implementations, the server 100 may determine whether the currently scanned data block has completed migration.

For example, if the state corresponding to the currently scanned data block is "Y" in the migration metadata record table, it may be determined that the currently scanned data block has completed migration;

if the state corresponding to the currently scanned data block is "N" in the migration metadata record table, it may be determined that the currently scanned data block is not migrated or is in the process of being migrated and has not been migrated yet.

When the server 100 determines that the currently scanned data block has completed migration, the state of the currently scanned data block in the migration metadata record table is "Y", and the currently scanned data block does not need to be migrated from the source data storage area to the target storage area again, and then the next data block of the current data block in the source data storage area is scanned.

For example, the storage area of the source data to be migrated is 100 data blocks from data 1 to data 100, and if the server 100 determines that the data 1 has been migrated, the server 100 scans the data 2 and determines whether the data 2 has been migrated.

When the server 100 determines that the currently scanned data block is not migrated, and the status of the currently scanned data block in the migration metadata record table is "N", S702 is executed.

S702, judging whether the currently scanned data block is being processed.

In some implementations, the server 100 may determine whether a currently scanned data block is in process, i.e., whether a currently scanned data block is undergoing migration.

In a possible implementation manner, if the currently scanned data block is in the process of being migrated but is not completely migrated, the migration metadata record table may record that the state of the currently scanned data block may be "N +".

In the embodiment of the present application, there are other methods that can determine whether a data block is being migrated, which is not limited to the above method.

When the server 100 determines that the currently scanned data block is being migrated and the currently scanned data block does not need to be migrated from the source data storage area to the target storage area again, S701 is executed to scan a data block next to the current data block in the source data storage area.

For example, the source data storage area to be migrated is 100 data blocks from data 1 to data 100, and if the server 100 determines that data 1 is being migrated, the server 100 scans data 2 and determines whether data 2 has been migrated.

When the server 100 determines that the currently scanned data block is not being migrated, S703 is executed.

S703, reading out the currently scanned data block from the source data storage area, migrating the data block to the target storage area, and updating the migration metadata record table.

In some implementations, the server 100 can migrate a currently scanned data block from a source data storage area to a target storage area.

For example, when the source data storage area and the target storage area are stored in the same server 100, the logical address of the currently scanned data block in the source data storage area is L1, the physical address corresponding to the logical address L1 is W1, and the mapping relationship is that the logical address L1 maps the physical address W1.

The migration of the currently scanned data block from the source data storage area to the target storage area by the server 100 may refer to changing the logical address of the currently scanned data block from the logical address L1 in the source data storage area to the logical address L2 in the target storage area, and changing the mapping relationship to the logical address L2 to map the physical address W1.

Optionally, the two physical addresses stored in the server 100 before and after the migration of the currently scanned data block may be the same or different. In the embodiments of the present application, this is not limited.

In some implementations, the server 100 may update the migration metadata record table after completing the migration of the currently scanned data block from the source data storage area to the target storage area.

For example, after a currently scanned data block is migrated from a source data storage area to a target storage area, the state of the migration metadata record table may be changed from "N" to "Y".

Fig. 8 illustrates a flowchart of a background data migration method provided in an embodiment of the present application. The flow diagram takes object storage as an example. It should be noted that the source data storage area targeted by the background data migration method has a migration instruction, and whether the migration instruction exists is not determined in the following specific steps.

S801, traversing the source data storage areas to be migrated in sequence, and judging whether the currently scanned single object has completed migration.

In some implementations, the server 100 can traverse the source data storage areas to be migrated in a sequential order.

For example, the source data storage area to be migrated is 100 objects from data 1 to data 100, the server 100 starts scanning from data 1 to data 100, and determines whether each object has completed migration.

In some implementations, the server 100 may determine whether the currently scanned single object has completed the migration.

For example, if the state corresponding to the currently scanned single object in the migration metadata record table is "Y", it may be determined that the currently scanned single object has completed migration;

if the state corresponding to the currently scanned single object in the migration metadata record table is "N", it may be determined that the currently scanned single object is not migrated or is in the process of being migrated and has not yet been migrated.

When the server 100 determines that the currently scanned single object has completed migration, the state of the currently scanned single object in the migration metadata record table is "Y", and the currently scanned single object does not need to be migrated from the source data storage area to the target storage area again, and then the next data block of the currently scanned single object in the source data storage area is scanned.

For example, the source data storage area to be migrated is 100 objects including data 1 and data 100, and if the server 100 determines that the data 1 has been migrated, the server 100 scans the data 2 and determines whether the data 2 has been migrated.

When the server 100 determines that the currently scanned single object is not migrated, the state of the currently scanned single object in the migration metadata record table is "N", then S802 is performed.

S802, judging whether the single object scanned currently is in process.

In some implementations, the server 100 may determine whether the currently scanned single object is in process, i.e., whether the currently scanned single object is undergoing migration.

In a possible implementation manner, if the currently scanned single object is in the process of being migrated but is not completely migrated, the migration metadata record table may record that the state of the currently scanned single object may be "N +".

In the embodiment of the present application, there are other methods besides the above method that can determine whether a single object is being migrated, which is not limited to this.

When the server 100 determines that the currently scanned single object is undergoing migration and the currently scanned single object does not need to be migrated from the source data storage area to the target storage area again, S801 is executed to scan an object next to the currently scanned single object in the source data storage area.

For example, the source data storage area to be migrated is 100 objects of data 1 to data 100, and if the server 100 determines that data 1 is being migrated, the server 100 scans data 2 and determines whether data 2 has been migrated.

When the server 100 determines that the currently scanned single object is not undergoing migration, S803 is executed.

S803, read out the currently scanned single object from the source data storage area, migrate the single object to the target storage area, and update the migration metadata record table.

In some implementations, the server 100 can migrate a currently scanned single object from the source data storage area to the target storage area.

For example, when the source data storage area and the target storage area are stored on the same server 100, the logical address of the currently scanned single object in the source data storage area is L1, the physical address corresponding to the logical address L1 is W1, and the mapping relationship is that the logical address L1 maps the physical address W1. The migration of the currently scanned single object from the source data storage area to the target storage area by the server 100 may refer to changing a logical address of the currently scanned single object from a logical address L1 in the source data storage area to a logical address L2 in the target storage area, and changing a mapping relationship to a logical address L2 to map the physical address W1.

Alternatively, the two physical addresses stored in the server 100 before and after migration of the currently scanned single object may be the same or different. In the embodiments of the present application, this is not limited.

In some implementations, the server 100 may update the migration metadata record table after completing the migration of the currently scanned single object from the source data storage area to the target storage area.

For example, after a currently scanned single object is migrated from a source data storage area to a target storage area, the state of the migration metadata record table may be converted from "N" to "Y".

By implementing the method, the migration of the corresponding data is completed in the data reading and writing process, and the rest data is migrated through a background migration task.

Therefore, the online migration efficiency can be improved under the condition of ensuring the correct and complete data migration. And the read requests of the data source are reduced, and the service life of the storage medium is prolonged.

The embodiments of the present application can be combined arbitrarily to achieve different technical effects.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the procedures or functions described in accordance with the present application are generated, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), among others.

Those skilled in the art can understand that all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer readable storage medium and can include the processes of the method embodiments described above when executed. And the aforementioned storage medium includes: various media capable of storing program codes, such as ROM or RAM, magnetic or optical disks, etc.

In short, the above description is only an example of the technical solution of the present invention, and is not intended to limit the scope of the present invention. Any modifications, equivalents, improvements and the like made in accordance with the disclosure of the present invention are intended to be included within the scope of the present invention.

Claims

1. A data migration method applied to a computing device is characterized by comprising the following steps:

receiving a read request or a write request for first data;

processing the first data according to the reading request, and migrating the first data from a source data storage area to a target storage area for storage;

or processing the first data according to the write request, and migrating the first data from the source data storage area to a target storage area for storage;

2. The method of claim 1, wherein: after the receiving a read request or a write request for first data, the method further comprises:

and under the condition that the first data need to be migrated and the first data are not migrated, processing the first data according to the read request or the write request.

3. The method according to any one of claims 1-2, wherein the storage address of the first data in the source data storage area is a first address,

and reading the first data from the first address according to the read request and storing the first data in a second address in the target storage area.

4. The method according to any one of claims 1 to 2, wherein the storage address of the first data in the source data storage area is a first address, the storage manner of the first data is block storage,

in the case that the determination is that the first data is aligned with the size of the data block, reading the first data from the first address according to the write request and storing in a second address in the target storage area; performing write processing on the first data stored in the first address and the second address;

the data block is data which is migrated once in the block storage; the alignment means that the size of the first data is an integer multiple of the size of a data block.

5. The method of claim 4, wherein: in the case that the computing device determines that the sizes of the data blocks of the first data are not aligned, the computing device continuously acquires a mutual exclusion lock of the data blocks;

the second data is read from the first address and stored in a second address in the target storage area.

6. The method according to claim 1 or 2, wherein the first data is stored in an object storage manner, the storage address of the first data in the source data storage area is a first address,

and according to the write request, reading the first data from the first address and storing the first data in a second address in the target storage area, and performing write processing on the first data stored in the first address and the second address.

7. The method of claim 1, wherein: after the first data is migrated, updating a migration metadata record table, wherein the migration metadata record table records a migration state of the first data, and the migration state includes any one of migration required but not migrated, migration already, and migration not required of the first data.

8. An electronic device comprising memory and one or more processors; the memory, coupled with the one or more processors, to store computer program code comprising computer instructions that, when executed by the one or more processors, cause the electronic device to perform the method of any of claims 1-7.