WO2017032262A1

WO2017032262A1 - Data migration method and device

Info

Publication number: WO2017032262A1
Application number: PCT/CN2016/095854
Authority: WO
Inventors: 闫长海; 尹书威; 刘中巍; 丁辉; 周皓
Original assignee: 阿里巴巴集团控股有限公司; 闫长海; 尹书威; 刘中巍; 丁辉; 周皓
Priority date: 2015-08-27
Filing date: 2016-08-18
Publication date: 2017-03-02
Also published as: CN106487841A

Abstract

Disclosed is a data migration method. A sub-migration task of each fragmented file is acquired from a fragmented migration queue. That is, each fragmented file is migrated from a source network memory corresponding to a to-be-migrated file according to each sub-migration task, such that fragmented migration can be performed by fully utilizing network resources, thereby improving the migration efficiency of network storage resources, and saving resources of a device.

Description

Data migration method and device

Technical field

The present application relates to the field of communications technologies, and in particular, to a data migration method. The application also relates to a data migration device.

Background technique

With the continuous development of Internet technology, cloud storage has become one of the indispensable technologies. There are a lot of cloud storage services available on the market (such as Alibaba Cloud OSS...), which is very popular with customers. Therefore, with the increasing number of users, users will inevitably encounter the replacement of cloud storage services in the process of using cloud storage.

In the prior art, when a user wants to replace a cloud storage service, it is troublesome to migrate data in the cloud storage. Some users store large amounts of data (tens of T to PB). Migration by traditional methods requires a lot of machines for migration, migration time is very long, and migration costs are high.

Summary of the invention

The present application provides a data migration method for conveniently and quickly migrating data stored in a network, improving migration efficiency, and saving device resources. The method includes:

Obtaining a sub-migration task of each fragment file from the fragment migration queue, where the fragment file is generated by the file to be migrated, and the sub-migration task is generated according to the migration task corresponding to the file to be migrated;

Performing migration processing on each of the fragment files in the source network storage corresponding to the to-be-migrated file according to each of the sub-migration tasks.

Preferably, each of the fragment files is migrated in the source network storage corresponding to the to-be-migrated file according to each of the sub-migration tasks, specifically:

And requesting, by the source network, a data migration flow corresponding to each of the fragment files, where the data is The migration stream is used to transfer data from the source network storage to the memory;

The data migration stream is uploaded as a data input stream to a target network storage specified by the sub-migration task, the data input stream being used to transfer the data from the memory to the target network storage.

Preferably, before obtaining the sub-migration task of each fragment file from the fragment migration queue, the method further includes:

Obtaining the to-be-migrated task from the asynchronous migration queue;

Determining whether the file to be migrated corresponding to the task to be migrated has been migrated;

If the judgment result is yes, obtain other tasks to be migrated from the asynchronous migration queue;

If the result of the determination is no, the file to be migrated is sliced into a plurality of the fragment files according to the capacity of the file to be migrated and the preset fragmentation parameter, and corresponding to each of the fragment files is generated. Submigration task.

Preferably, before the obtaining the task to be migrated from the asynchronous migration queue, the method further includes:

Obtain migration configuration information;

Scan the file to be migrated in a file stored in the source network according to the migration configuration information;

The migration task is generated according to the scan result, and the migration task is added to the asynchronous migration task queue.

Preferably, after performing the migration processing on each of the fragment files in the source network storage corresponding to the to-be-migrated file according to each of the sub-migration tasks, the method further includes:

Determining whether all the fragment files corresponding to the file to be migrated are successfully migrated;

If yes, update and store the migration status of the file to be migrated.

Correspondingly, the present application also provides a data migration device, including:

Obtaining a module, obtaining a sub-migration task of each fragment file from the fragment migration queue, where the sub-migration task The slice file is generated by the file to be migrated, and the sub-migration task is generated according to the migration task corresponding to the file to be migrated;

The migration module performs migration processing on each of the fragment files in the source network storage corresponding to the to-be-migrated file according to each of the sub-migration tasks.

Preferably, the migration module is specifically configured to:

And requesting, by the source network, a data migration flow corresponding to each of the fragment files, where the data migration flow is used to transfer data from the source network storage to a memory;

Preferably, the method further comprises:

The determining module obtains the to-be-migrated task from the asynchronous migration queue, and determines whether the to-be-migrated file corresponding to the to-be-migrated task has been migrated;

If the determination result is yes, the determining module acquires other tasks to be migrated from the asynchronous migration queue;

If the determination result is no, the determining module processes the file to be migrated into a plurality of the fragment files according to the capacity of the file to be migrated and the preset fragmentation parameter, and generates and segments each of the segments. The sub-migration task corresponding to the slice file.

Preferably, the method further comprises:

Generating a module, acquiring the migration configuration information, scanning the to-be-migrated file in the file stored in the source network according to the migration configuration information, generating the migration task according to the scan result, and adding the migration task to the asynchronous migration In the task queue.

Preferably, the method further comprises:

The update module determines whether all the fragment files corresponding to the file to be migrated are successfully migrated, and updates and stores the migration status of the file to be migrated when the determination result is yes.

It can be seen that, by applying the technical solution of the present application, the sub-migration task of obtaining each fragment file from the fragment migration queue, that is, the fragmentation from the source network storage corresponding to the file to be migrated according to each sub-migration task The files are migrated, so that network resources can be fully utilized for fragment migration, which improves the migration efficiency of network storage resources and saves device resources.

DRAWINGS

1 is a schematic flowchart of a data migration method proposed by the present application;

FIG. 2 is a schematic structural diagram of a data migration device according to the present application.

detailed description

In view of the technical problems raised by the background art, the present application proposes a data migration method, as shown in FIG. 1, including the following steps:

S101: Obtain a sub-migration task of each fragment file from the fragment migration queue, where the fragment file is generated by the file to be migrated, and the sub-migration task is generated according to the migration task corresponding to the file to be migrated.

For the file migration request made by the user, the application first needs to obtain the migration configuration information based on the request, and after scanning the to-be-migrated file in the file stored in the source network according to the migration configuration information, and then generating the location according to the scan result. The migration task is described, and the migration task is added to the asynchronous migration task queue.

In the subsequent process, the present application needs to perform the fragmentation processing for each file to be migrated. Therefore, in the preferred embodiment of the present application, the task to be migrated needs to be obtained from the asynchronous migration queue before the step, and the determination is performed. Whether the file to be migrated corresponding to the task to be migrated has been migrated, and the following processing is performed based on the result:

(1) If the judgment result is yes, obtain other tasks to be migrated from the asynchronous migration queue;

(2) If the judgment result is no, according to the capacity of the file to be migrated and the preset fragmentation parameter The file to be migrated is sliced into a plurality of the fragment files, and a sub-migration task corresponding to each of the fragment files is generated.

S102: Perform migration processing on each of the fragment files in the source network storage corresponding to the to-be-migrated file according to each of the sub-migration tasks.

In the existing data dumping process, the data to be migrated must be downloaded from the source network storage to a temporary storage space, and then the data to be migrated is uploaded from the temporary space to the target network storage. Such a dumping method not only requires additional setting of the intermediate storage space for downloading and uploading, but also requires a certain amount of time for the downloading process and the uploading process. Therefore, in order to further improve the migration efficiency of the network storage, the technical solution of the present application removes two local IO operations in the existing migration process. In a preferred embodiment corresponding to the step, the data migration flow corresponding to each of the fragment files is first requested to the source network, and then the data migration stream is uploaded as a data input stream to the sub-migration task designation. Target network storage. Since the data migration stream is used to download data from the source network storage into the memory, and the data input stream is used to upload the data directly from the memory to the target storage, the technical solution of the present application obtains the points and points. After the data migration stream corresponding to the data to be migrated, the destination address of the data migration stream is directly directed to the target storage (ie, as an uploaded data input stream), so that the data transmission is all implemented in the memory without having to rely on The middle of the hard drive to the village. Therefore, the data of each fragment file can be directly downloaded and uploaded through the memory.

In order to ensure that the entire shard file is migrated at the end, the migration of the entire file is guaranteed. After the step, the preferred embodiment of the present application further determines whether all the fragment files corresponding to the file to be migrated are successfully migrated, and when the result is YES, the migration status of the file to be migrated is updated and stored. If so, continue to get the sub-migration task of each fragment file from the fragment migration queue.

To achieve the above technical purpose, the present application also proposes a data migration device, as shown in FIG. 2, including:

The obtaining module 210 is configured to obtain a sub-migration task of each fragment file from the fragment migration queue, where the fragment file is generated by the file to be migrated, and the sub-migration task is generated according to the migration task corresponding to the file to be migrated. ;

The migration module 220 performs migration processing on each of the fragment files in the source network storage corresponding to the to-be-migrated file according to each of the sub-migration tasks.

In a specific application scenario, the migration module is specifically configured to:

In specific application scenarios, it also includes:

Updating the module, determining whether all the fragment files corresponding to the file to be migrated are migrated into Work, and update and store the migration status of the file to be migrated when the judgment result is YES.

By applying the technical solution of the present application, the sub-migration task of each fragment file is obtained from the fragment migration queue, that is, each fragment file is migrated from the source network storage corresponding to the file to be migrated according to each sub-migration task. Therefore, the network resources can be fully utilized for fragment migration, the migration efficiency of the network storage resources is improved, and the equipment consumes resources.

Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by hardware, or by software plus a necessary general hardware platform. Based on such understanding, the technical solution of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several The instructions are for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform the methods described in various implementation scenarios of the present application.

A person skilled in the art can understand that the drawings are only a schematic diagram of a preferred implementation scenario, and the modules or processes in the drawings are not necessarily required to implement the application.

A person skilled in the art may understand that the modules in the apparatus in the implementation scenario may be distributed in the apparatus for implementing the scenario according to the implementation scenario description, or may be correspondingly changed in one or more devices different from the implementation scenario. The modules of the above implementation scenarios may be combined into one module, or may be further split into multiple sub-modules.

The above serial numbers are only for the description, and do not represent the advantages and disadvantages of the implementation scenario.

The above disclosure is only a few specific implementation scenarios of the present application, but the present application is not limited thereto, and any changes that can be made by those skilled in the art should fall within the protection scope of the present application.

Claims

A data migration method, comprising:

Obtaining a sub-migration task of each fragment file from the fragment migration queue, where the fragment file is generated by the file to be migrated, and the sub-migration task is generated according to the migration task corresponding to the file to be migrated;

Performing migration processing on each of the fragment files in the source network storage corresponding to the to-be-migrated file according to each of the sub-migration tasks.
The method according to claim 1, wherein each of the fragment files is migrated in a source network storage corresponding to the file to be migrated according to each of the sub-migration tasks, specifically:

And requesting, by the source network, a data migration flow corresponding to each of the fragment files, where the data migration flow is used to transfer data from the source network storage to a memory;

The data migration stream is uploaded as a data input stream to a target network storage specified by the sub-migration task, the data input stream being used to transfer the data from the memory to the target network storage.
The method according to claim 1, wherein before the sub-migration task of each fragment file is obtained from the fragment migration queue, the method further comprises:

Obtaining the to-be-migrated task from the asynchronous migration queue;

Determining whether the file to be migrated corresponding to the task to be migrated has been migrated;

If the judgment result is yes, obtain other tasks to be migrated from the asynchronous migration queue;

If the result of the determination is no, the file to be migrated is sliced into a plurality of the fragment files according to the capacity of the file to be migrated and the preset fragmentation parameter, and corresponding to each of the fragment files is generated. Submigration task.
The method of claim 1, further comprising: before acquiring the task to be migrated from the asynchronous migration queue, further comprising:

Obtain migration configuration information;

Scan the file to be migrated in a file stored in the source network according to the migration configuration information;

The migration task is generated according to the scan result, and the migration task is added to the asynchronous migration task queue.
The method according to claim 1, further comprising: after performing the migration processing on each of the fragment files in the source network storage corresponding to the file to be migrated according to each of the sub-migration tasks, further comprising:

Determining whether all the fragment files corresponding to the file to be migrated are successfully migrated;

If yes, update and store the migration status of the file to be migrated.
A data migration device, comprising:

Obtaining a module, the sub-migration task of each fragment file is obtained from the fragment migration queue, where the fragment file is generated by the file to be migrated, and the sub-migration task is generated according to the migration task corresponding to the file to be migrated;

The migration module performs migration processing on each of the fragment files in the source network storage corresponding to the to-be-migrated file according to each of the sub-migration tasks.
The device according to claim 6, wherein the migration module is specifically configured to:

And requesting, by the source network, a data migration flow corresponding to each of the fragment files, where the data migration flow is used to transfer data from the source network storage to a memory;

The data migration stream is uploaded as a data input stream to a target network storage specified by the sub-migration task, the data input stream being used to transfer the data from the memory to the target network storage.
The device of claim 6 further comprising:

The determining module obtains the to-be-migrated task from the asynchronous migration queue, and determines and migrates the to-be-migrated Whether the file to be migrated corresponding to the task has been migrated.

If the determination result is yes, the determining module acquires other tasks to be migrated from the asynchronous migration queue;

If the determination result is no, the determining module processes the file to be migrated into a plurality of the fragment files according to the capacity of the file to be migrated and the preset fragmentation parameter, and generates and segments each of the segments. The sub-migration task corresponding to the slice file.
The device of claim 6 further comprising:

Generating a module, acquiring the migration configuration information, scanning the to-be-migrated file in the file stored in the source network according to the migration configuration information, generating the migration task according to the scan result, and adding the migration task to the asynchronous migration In the task queue.
The device of claim 6 further comprising:

The update module determines whether all the fragment files corresponding to the file to be migrated are successfully migrated, and updates and stores the migration status of the file to be migrated when the determination result is yes.