WO2019033949A1

WO2019033949A1 - Data migration method, apparatus and device

Info

Publication number: WO2019033949A1
Application number: PCT/CN2018/098981
Authority: WO
Inventors: 彭文文
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2017-08-18
Filing date: 2018-08-06
Publication date: 2019-02-21
Also published as: CN109407964A

Abstract

Disclosed are a data migration method, apparatus and device. The method comprises: according to the capacity of a storage device to be migrated in a storage system, adding a target migration device to the storage system, so as to perform capacity expansion processing on the storage system; performing write operation-prohibiting processing on the storage device to be migrated, and migrating, to the target migration device, data stored in the storage device to be migrated; and removing, from the storage system, the storage device to be migrated, so as to perform capacity shrinking processing on the storage system. By means of the embodiments of the present application, in the case where a service is not interrupted and the quality of service of an online service is not affected, data migration and device replacement can be completed by means of capacity expansion processing and capacity shrinking processing without affecting a user's reading and writing of data in a storage system, and the speed of data migration can be increased and the time spent on data migration can be reduced.

Description

Data migration method, device and device

The present application claims the priority of the Japanese Patent Application Serial No. No. No. No. No. No. No. No. No. No. No.

Technical field

The present application relates to the field of computer technologies, and in particular, to a data migration method, apparatus, and device.

Background technique

Distributed data storage is a storage technology that distributes data across multiple independent storage devices and uses multiple storage devices to share the load. When the distributed data storage system stores data, the data is distributedly distributed to the corresponding storage device to implement distributed storage of the data. In some cases (for example, geographical migration of the entire server cluster corresponding to the distributed data storage system, over-service of the entire server cluster, or replacement of all or part of the servers in the entire server cluster, etc.), data migration is required. Does not affect the normal use of the business.

Usually, a new cluster is built using the replaced storage device, and the data of the original cluster is migrated to the new cluster. For example, as the hardware continues to evolve, the hardware performance of the device is higher and higher, if the server in the original cluster is SATA. (Serial Advanced Technology Attachment, serial high-tech configuration interface) server, and the current original cluster is out of warranty, you need to replace the SATA server in the original cluster with a HYBRID server, including the following: The original cluster provides normal external services, and at the same time, build In the new cluster of the HYBRID server, the data in the original cluster is copied from the SATA server in the original cluster to the HYBRID server of the new cluster in a snapshot manner. In the process of copying, the data is incrementally migrated for the newly written data in the original cluster. Through the double write function in the distributed data storage system, the data received during the migration of the above data is separately written into the new cluster and the original cluster. When the data stored in the new cluster catches up with the data stored in the original cluster, switch all read and write functions to the new cluster, and then modify the label attribute information to match the HYBRID server in the new cluster (ie, enable HYBRID server's hybrid storage write function), thus completing data migration and cluster replacement.

However, when the data stored in the new cluster is consistent with the data stored in the original cluster, all read and write functions need to be switched to the new cluster. In this process, when the data stored in the new cluster and the original cluster are consistent. Performing the data migration, the service of the original cluster is stopped, and the new cluster is started to provide services to the distributed data storage system. In the above switching process, the distributed data storage system cannot be used for a certain period of time, so that the user cannot Read and write data in a distributed data storage system.

Summary of the invention

The purpose of the embodiments of the present application is to provide a data migration method, device, and device, which can complete data migration and without disrupting service and quality of service without affecting online services through capacity expansion processing and volume reduction processing. The replacement of the device does not affect the user's reading and writing of data in the storage system, and can increase the speed of data migration and reduce the time spent on data migration.

To solve the above technical problem, the embodiment of the present application is implemented as follows:

A method for migrating data provided by an embodiment of the present application, where the method includes:

Adding a target migration device to the storage system to perform capacity expansion processing on the storage system according to the capacity of the storage device to be migrated in the storage system;

Performing a write-protection process on the to-be-migrated storage device, and migrating data stored in the to-be-migrated storage device to the target migration device;

Removing the storage device to be migrated in the storage system to perform a shrinking process on the storage system.

A data migration device provided by an embodiment of the present application, the device includes:

a capacity expansion module, configured to add a target migration device to the storage system to perform capacity expansion processing on the storage system according to a capacity of the storage device to be migrated in the storage system;

a data migration module, configured to perform a write-protection process on the to-be-migrated storage device, and migrate data stored in the to-be-migrated storage device to the target migration device;

And a shrinking module, configured to remove the to-be-migrated storage device in the storage system to perform a shrinking process on the storage system.

A data migration device is further provided by the embodiment of the present application, where the device includes:

Processor;

A memory arranged to store computer executable instructions that, when executed, cause the processor to:

It can be seen that the technical solution provided by the foregoing embodiment of the present application can be used to expand the storage system to add the target migration device to the storage system by using the capacity of the storage device to be migrated in the storage system, and then the migration storage device is to be migrated. Perform the write-protection process and migrate the data stored in the storage device to be migrated to the target migration device. Finally, the storage system is compressed to remove the storage device to be migrated from the storage system. Processing and shrinking processing can complete data migration and device replacement without interrupting service and quality of service that does not affect online services. It does not affect users' reading and writing of data in the storage system, and can improve data. The speed of migration reduces the time spent on data migration.

DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings to be used in the embodiments or the prior art description will be briefly described below. Obviously, the drawings in the following description are only It is a few embodiments described in the present application, and other drawings can be obtained from those skilled in the art without any inventive labor.

1 is an embodiment of a data migration method according to the present application;

2A is a schematic diagram of expansion of a storage system according to the present application;

2B is a schematic diagram of a storage system shrinkage according to the present application;

FIG. 3 is a schematic diagram of a data migration method according to the present application; FIG.

4A is a schematic diagram of another storage system expansion of the present application;

4B is a schematic diagram of another storage system of the present application;

FIG. 5A is a schematic diagram of another storage system expansion according to the present application; FIG.

FIG. 5B is a schematic diagram of another storage system shrinkage according to the present application; FIG.

6 is an embodiment of a data migration apparatus of the present application;

FIG. 7 is an embodiment of a data migration device according to the present application.

Detailed ways

The embodiment of the present application provides a data migration method, device, and device.

The technical solutions in the embodiments of the present application are clearly and completely described in the following, in which the technical solutions in the embodiments of the present application are clearly and completely described. The embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope shall fall within the scope of the application.

Embodiment 1

As shown in FIG. 1 , the embodiment of the present application provides a data migration method, where the execution entity of the method may be a server or a terminal device, where the server may be a metadata server in the storage system, or may be preset. For the server for data migration, the terminal device may be a personal computer or the like. The method may specifically include the following steps:

In step S101, the target migration device is added to the storage system to perform capacity expansion processing on the storage system according to the capacity of the storage device to be migrated in the storage system.

The storage system may be a distributed data storage system or a centralized data storage system. The storage device to be migrated may be any one of the storage systems or may be composed of multiple storage devices in the storage system, including all storage devices in the storage system. The storage device to be migrated may be a storage device that needs to be replaced. The target migration device may be a device for replacing the storage device to be migrated, and the storage space of the target migration device is greater than or equal to the storage space of the storage device to be migrated. The capacity expansion process may be the process of adding a storage device to the storage system and the service provided by the storage device.

In an implementation, a plurality of storage devices may be included in the storage system, and the data that needs to be stored may be stored by the storage device. In order to ensure the security of the data, the storage system stores multiple copies (for example, 3 copies or 5 copies, etc.) for the same data, and the plurality of data are respectively set in different storage devices, and the plurality of data are used as backup data. In this way, when the storage device where the data is located is abnormal, the storage device where the other backup data of the data is located can continue to provide services for the user.

With the continuous development of hardware, the upgrading of hardware devices is getting faster and faster, and the performance is getting better and better. For storage devices, the storage devices of different interfaces have different performances, for example, IDE (Integrated Drive Electronics, The storage device of the electronic integrated drive) interface, the SCSI (Small Computer System Interface) storage device, the storage device of the SATA interface, or the storage device of the HYBRID interface, etc., the performance of the corresponding storage device follows the sequence shown above. Enhance in turn. In order to provide users with related services, you can replace the storage devices with poor performance in the storage system. Considering that a large number of storage devices are included in the storage system, there are often more storage devices that need to be replaced, such as storage. All the storage devices in the system or some storage devices in the storage system, etc., so that the data stored in the storage device to be replaced (that is, the storage device to be migrated) needs to be migrated to the new storage device (that is, the target migration device). The service quality of the storage system can be processed by the expansion and processing of the storage system, such as the service that can be read and written in real time, in the process of data migration. As shown in Figure 2A, if you need to replace 7 storage devices in the storage system, you can calculate the storage capacity of the 7 storage devices that need to be replaced, such as 120TB or 100TB. You can select a target migration device with a storage capacity greater than or equal to the above storage capacity, and add the target migration device to the topology of the storage system. You can select the target migration device and start the storage system expansion process. In this case, you can add the target migration device to the storage system to obtain the expanded storage system, as shown in Figure 2A.

It should be noted that, in the storage system, a management device (such as a metadata server) and a data storage device at the management layer are often included, wherein the management device is used to manage data in the data storage device (such as data distribution and data). Data storage devices can be used to store data, such as queries, data deletions, and so on. If you need to replace the management device and the data storage device, you can replace the management device with the above processing method. After the management device is replaced, the replacement management device can replace the data storage device through the above processing. If you only need to replace the management device or data storage device, you can directly perform the above processing.

In step S102, the storage operation to be migrated is performed on the storage device to be migrated, and the data stored in the storage device to be migrated is migrated to the target migration device.

The write operation may be an operation of writing data to the storage device to be migrated.

In an implementation, the storage device may be provided with a permission setting option of a read operation and a permission setting option of a write operation. The above two options may be used to control a read operation and/or a write operation of the storage device, for example, when a certain storage is used. The data stored in the device does not allow the user to modify or delete, but when the user is allowed to download, copy or view, the permission setting option of the read operation can be set to allow, and the permission setting option of the write operation is set to be rejected, so that the user will The storage device could not be written. The storage device to be migrated needs to be replaced. When the user needs to write data to the storage system (that is, perform a write operation), the data can be written to the target migration device. To this end, the write to the migration storage device can be stopped. Operation, that is, the permission setting option of the read operation of the storage device to be migrated can be set to allow, and the permission setting option of the write operation is set to be rejected. After the foregoing processing, when the storage device needs to perform a write operation and allocates a storage device for the write operation, the write operation may be allocated to the target migration device, and the storage device to be migrated no longer performs the write operation, but at this time, Users can still view or download data stored in the storage device to be migrated.

Based on the above related content, in order to ensure the security of the data, multiple copies (such as 3 copies or 5 copies, etc.) are stored in the storage system for the same data, and the multiple copies of the data are respectively set in different storage devices. In order to not interrupt the online service and not affect the service quality of the online service, you can find the storage device where the data stored in the storage device to be migrated is located. During the data migration process, one or more copies of any data are always guaranteed. The data migration operation is not performed. For example, the storage device to be migrated includes 5 different data, and each data includes 3 copies, wherein any storage device A, storage device B, and storage device C store data. 1. Data 2, data 3, data 4, and data 5, when data migration occurs, storage device A, storage device B, and storage device C cannot simultaneously perform data migration, but only device A, storage device B, and storage device. C performs data migration in sequence, that is, any one of the storage devices performs data migration first, and then performs data migration on any of the remaining storage devices, and finally performs data migration on the remaining storage devices, or the storage device. A. Any two storage devices in storage device B and storage device C are first migrated. After the completion, the remaining Migration data storage device, or a storage device A, storage device B and storage device C prior to any storage device for a data migration is completed, then the remaining storage devices while data migration.

It should be noted that the triggering manner of performing the data migration operation may include multiple types. For example, a trigger condition may be preset, and when a storage device in the storage system satisfies the trigger condition, performing a data migration operation, where performing data migration Before the operation, you can select and set the storage device that needs to perform data migration, and the storage device that receives the migrated data. For example, you can set the trigger status for each storage device, and you can detect the status of each storage device in real time. When the state is converted to the above trigger state, data migration can be performed on the storage device. In addition to the foregoing triggering manners, a plurality of triggering modes may be included, which may be determined according to actual conditions, and are not described herein again.

In step S103, the storage device to be migrated is removed from the storage system to perform a shrinking process on the storage system.

The shrinking process may be the process of deleting the storage device from the storage system and the service provided by the storage device.

In the implementation, after the data migration in the foregoing step S102 is completed, it may be determined that the data stored in the storage device to be migrated has all migrated to the target migration device, and the data newly written in the storage system is written into the target migration device. The storage device to be migrated has lost its meaning in the storage system. Therefore, the storage system can be subjected to a shrinking process to remove the to-be-migrated storage device from the storage system, as shown in FIG. 2B. The storage system no longer includes the storage device to be migrated, but the target migration device, so that the data can be completed without capacity interruption and service quality without affecting the online service through capacity expansion processing and shrink processing. Migration improves the speed of data migration and reduces the time it takes to migrate data.

In addition, because the data storage format of the storage device (such as a disk) in the storage device or the storage device is different, the format of the data stored in the storage device is different. Therefore, if the storage device to be migrated and the target migration device have the above, If the data to be migrated is migrated to the target migration device, you can modify the configuration parameters of the migrated data to match the target migration device. If the storage device to be migrated and the target migration device do not exist, Different, you do not need to modify the configuration parameters of the migrated data.

It should be noted that the foregoing steps S101 to S103 may be applied to a storage device of a certain data storage type, such as a management device (such as a metadata server) or a data storage device of the management layer mentioned above. The storage device of the data storage type may be applied to the storage device of different data storage types. In this case, the storage device of the different data storage type may perform the processing of the foregoing steps S101 to S103. Priority, for example, as described above, the priority of the management device may be set to be higher than the priority of the data storage device, so that the management device may be replaced first, that is, the processing of steps S101 to S103 described above is performed, and then, The data storage device is replaced.

The embodiment of the present application provides a method for migrating data, and the capacity of the storage device to be migrated in the storage system is expanded to add the target migration device to the storage system, and then the write operation is prohibited. The data stored in the storage device to be migrated is migrated to the target migration device. Finally, the storage system is compressed to remove the storage device to be migrated from the storage system. Processing can complete data migration and device replacement without interrupting service and quality of service that does not affect online services. It does not affect the user's reading and writing of data in the storage system, and can improve the speed of data migration. Reduce the time it takes for data migration.

Embodiment 2

As shown in FIG. 3, the embodiment of the present application provides a data migration method. The execution entity of the method may be a server or a terminal device, where the server may be a metadata server in the storage system, or may be preset. A server for data migration, the terminal device may be a personal computer or the like.

For a more detailed description of the data migration method provided by the embodiment of the present application, the embodiment of the present application is described by taking all the storage devices in the storage system as an example, considering that the data is not lost when the HYBRID storage device is powered off. The data of the SATA storage device may be lost when the device is powered off. Therefore, the storage device to be migrated may be a SATA storage device, and the target migration device may be a HYBRID storage device. The storage device to be migrated is a metadata server and/or data storage in the storage system. A server, wherein the metadata server is used to manage metadata of data stored in the data storage server. For example, the data migration method provided by the embodiment of the present application may include the following steps:

In step S301, the target migration device is added to the storage system to perform capacity expansion processing on the storage system according to the capacity of the storage device to be migrated in the storage system.

The content of the step S301 is the same as that of the step S101 in the first embodiment. For the specific processing of the step S301, refer to the related content in the above step S101, and details are not described herein again.

In the step S302, the data to be migrated in the storage device to be migrated is determined according to a predetermined load balancing rule according to the storage space utilization ratio of the storage device to be migrated and the target migration device.

The storage space utilization can be used to represent the relationship between the used storage space and the unused storage space. The higher the storage space utilization, the more frequently the corresponding storage device is used, the data read operation and/or The more frequent the write operation. The load balancing rule may be a rule for transferring data in a storage device with a higher load to a storage device with a lower load. The purpose is to balance the load of the storage device to be migrated and the target migration device to achieve data. Balanced distribution, load balancing rules can be set according to actual conditions. The data to be migrated may be data that needs to be transferred to a storage device with a lower load, and the data to be migrated may be all data stored in the storage device to be migrated. Part of the data.

In the implementation, the processing manner of stopping the writing operation to the metadata server and the data storage server may include multiple types. The following provides a feasible processing manner, and the metadata server and the data storage server may be added to the blacklist respectively. The metadata server and the data storage server can perform a read operation and cannot perform a write operation. At this time, the user cannot write data to the metadata server and the data storage server, and thus, as shown in FIG. 4A, the write operation of the storage system The device can only be migrated by the target (in order to distinguish it from the aforementioned name, the target migration device is the target metadata server and the target data storage server). When the user needs to write data to the storage system, the write operation corresponding to the write data may be allocated to the target metadata server and the target data storage server, and the metadata server and the data storage server may provide the user with the storage thereof. Services such as reading or downloading data.

A storage system can be pre-configured with load balancing rules, such as the Rebalance mechanism. After the write-protected storage device is disabled, the storage system's Rebalance mechanism can be started. In this case, the target metadata server and target data are generated. The storage server's storage space utilization (or disk utilization) is lower, and the storage capacity of the metadata server and the data storage server is higher. This results in uneven load in the storage system. After the Rebalance mechanism is started, you can follow the The load balancing rule determines that the data to be migrated in the metadata server and the data storage server (that is, the storage device to be migrated) needs to be migrated to the target metadata server and the target data storage server (ie, the target migration device), for example, the data volume is 1 TB. Data A, etc.

In the step S303, the data to be migrated is migrated to the target migration device by using a predetermined load balancing rule, and the storage device to be migrated is set to the quasi-offline state, and the remaining data to be migrated is controlled to be migrated to the storage device to be migrated. Target migration device.

The quasi-downline state (ie, the ShutDown state) may be a state before the offline state, or may be understood as a state in which the offline line is prepared. The quasi-offline state requires the storage device to complete the predetermined operation. After the predetermined operation is completed, The storage device is offline.

In the implementation, after the Rebalance mechanism is started, and the data to be migrated in the storage device to be migrated is determined according to a predetermined load balancing rule, the data to be migrated stored in the metadata server and the data storage server are migrated to the target metadata server and Target data storage server.

For a storage device in a quasi-downline state, one or more operations that need to be performed, such as data migration operations, flow control of data migration, or network bandwidth control, may be preset. The storage device to be migrated can be set to the quasi-offline state. At this time, the storage device to be migrated is still in a state of being readable and unwritable (that is, a read operation can be performed but a write operation cannot be performed). The metadata server and/or the data storage server in the quasi-downline state can be found and determined, and the amount of data of the remaining data other than the data to be migrated in the stored data is counted, and then the corresponding target metadata server and/or can be determined. Or the target data storage server, the remaining data other than the data to be migrated in the metadata server and/or the data storage server may be migrated to the corresponding target metadata server and/or the target data storage server, and may be through a preset network. Bandwidth controls the amount of traffic migrated by data, such as 30MB/s or 35MB/s.

The processing of the foregoing step S303 can be various. The following provides a feasible processing manner. The processing of the foregoing step S303 may be specifically: according to the preset rack in which the storage device to be migrated is located in the storage system, respectively The storage device to be migrated in the preset rack is set to the quasi-offline state, and the storage device to be migrated is controlled to migrate the remaining data stored in the target migration device.

The preset rack (ie, the Rack) may include multiple storage devices, and the storage system may include multiple preset racks, and based on the storage system, multiple copies (for example, three copies) may be stored for the same data. Multiple pieces of data are respectively set in different storage devices, and multiple pieces of data are used as backup data for each other. Each preset rack can store one piece of data of any one of the data, for example, storage. The system includes three preset racks, namely rack 1, rack 2 and rack 3, and the storage system includes three types of data, namely data 1, data 2 and data 3, data 1, data 2 and Data 3 includes 3 pieces of data, respectively, and any of the racks 1, rack 2, and rack 3 stores data 1, data 2, and data 3, that is, data 1 is stored in rack 1. Data 2 and Data 3, Data 1, Data 2, and Data 3 are stored in the Rack 2, and Data 1, Data 2, and Data 3 are stored in the Rack 3.

In the implementation, in order to not interrupt the online service and not affect the quality of service of the online service, the data migration operation may be performed in units of preset racks, for example, based on the above example, in the rack 1, the rack 2 and the rack 3 If any of the preset racks stores data 1, data 2, and data 3, you can first set the metadata server or data storage server in rack 1 to the quasi-offline state, and the metadata server and / Or the remaining data other than the data to be migrated stored in the data storage server is migrated to the corresponding target metadata server and/or the target data storage server. After the migration is completed, the metadata server or the data storage server in the rack 2 may be all Set to the quasi-downline state, and migrate the remaining data except the data to be migrated stored in the metadata server and/or the data storage server to the corresponding target metadata server and/or target data storage server, and finally, the machine The metadata server or data storage server in shelf 3 is all set to the quasi offline state, and the metadata server and/or data storage is The remaining data other than the migration of migrating data traffic is to be stored in the respective target metadata server and / or the target data storage server, thereby completing the data migration operation. The metadata server or the data storage server is all set to the quasi offline state, and the remaining data other than the data to be migrated stored in the metadata server and/or the data storage server is migrated to the corresponding target metadata server and/or For the specific processing procedure in the target data storage server, refer to the related content above, and details are not described herein again.

It should be noted that, in the foregoing step S302, the purpose of the process of migrating data stored in the storage device to be migrated to the target migration device by using a predetermined load balancing rule according to the storage space utilization ratio of the storage device to be migrated and the target migration device In order to speed up the migration of data and reduce the time consumed for data migration, in actual applications, the above-described processing may be performed without performing the above processing.

The processing of the foregoing steps S302 to S303 is a specific processing manner of the step S102 in the first embodiment. In practical applications, data migration in the metadata server and the data storage server in the storage system is often encountered. In the corresponding target migration device, the storage device to be migrated may include a metadata server and a data storage server in the storage system. Accordingly, the target migration device may include a target metadata server and a target data storage server. The processing manner of the foregoing step S102 can also be implemented in the following manner, and specifically includes the following: performing a write inhibit operation on the metadata server, and migrating the data stored in the metadata server to the target metadata server; performing the data storage server on the data storage server The write operation processing is prohibited, and the data stored in the data storage server is migrated to the target data storage server. For the specific processing of the foregoing, refer to the related content of step S102 or step S302 and step S303, and details are not described herein again.

In addition, the above steps S301 to S303 and the following steps S304 to S305 may be performed on the metadata server and the data storage server, respectively.

In step S304, the storage device to be migrated is removed from the storage system to perform a shrinking process on the storage system.

The content of the step S304 is the same as the content of the step S103 in the first embodiment. For the specific processing of the step S304, refer to the related content in the above step S103, and details are not described herein again. The processing result of the above step S304 can be as shown in Fig. 4B.

It should be noted that, in the process of performing the foregoing data migration, data migration and device replacement may be performed on the metadata server, and then data migration and device replacement may be performed on the data storage server, and the metadata server and data may be simultaneously performed. The storage server performs the data migration and the device replacement, and the data storage server performs data migration and device replacement, and then performs data migration and device replacement on the metadata server. limited.

In addition, the data migration and device replacement for the metadata server can be handled by other means, for example, by writing the data written by the user to the target metadata server and the metadata server simultaneously by the double write function, when the target metadata server When the data stored in the data catches up with the data stored in the metadata server, all the read/write functions are switched to the target metadata server, and the processing of steps S301 to S304 described above can be performed for the data storage server.

In step S305, the tag attribute information that matches the storage device to be migrated is modified to the tag attribute information that matches the target migration device.

The tag attribute information is used to record information such as configuration parameters of the data, and the tag attribute information is adapted to the storage device.

In an implementation, since the storage device to be migrated (ie, the metadata server and/or the data storage server) is a SATA storage device, the target migration device (ie, the target metadata server and/or the target data storage server) is a HYBRID storage device, and two types. If the interface of the storage device is different, the format of the data to be migrated is different. Therefore, you need to modify the label attribute information (such as configuration parameters) of the migrated data after the data stored in the storage device to be migrated is migrated to the target migration device. Etc.) to match the target migration device.

In addition, the data migration method provided by the foregoing embodiment is an example of replacing all the storage devices in the storage system. In actual applications, the data migration method provided by the embodiment of the present application may also be applied to replacing the storage system. In the case of a part of the storage device, as shown in FIG. 5A, the storage device to be migrated is a part of the data storage server in the storage system, and the storage device to be migrated may also be a SATA storage device, and the target migration device may also be a HYBRID storage device.

Based on the foregoing, the data migration method provided by the embodiment of the present application may specifically include the following: adding a target data storage server to the storage system according to the capacity of the storage device to be migrated in the storage system (the data storage server in FIG. 5A) The storage system is expanded. The storage device to be migrated is forbidden. According to the storage load utilization of the storage device and the target migration device, the data to be migrated in the storage device to be migrated is determined according to a predetermined load balancing rule. The scheduled load balancing rule migrates the data to be migrated to the target migration device, and sets the to-be-migrated storage device in the preset chassis to the off-line according to the preset rack where the storage device to be migrated is located in the storage system. Status, which controls the storage device to be migrated to migrate the remaining data except the data to be migrated to the target migration device. The storage device to be migrated (the data storage server in FIG. 5A) is removed from the storage system to perform a shrinking process on the storage system, as shown in FIG. 5B. The tag attribute information that matches the storage device to be migrated is modified to the tag attribute information that matches the target data storage server (ie, the HYBRID storage device).

For the specific processing of the foregoing processing, refer to the related content in the foregoing Embodiment 1 and Embodiment 2, and details are not described herein again.

It should be noted that the foregoing steps S301 to S305 are based on the detailed description of the case where the storage device to be migrated is a serial high-tech interface SATA storage device and the target migration device is a hybrid HYBRID storage device. In practical applications, The storage device to be migrated and the target migration device may be any of the serial high-tech interface SATA storage device, the hybrid HYBRID storage device, and the SSD storage device, that is, if the storage device to be migrated is SATA. The storage device, the HYBRID storage device, and the SSD storage device may be any one of the types of storage devices, and the target migration device may be a SATA storage device, a HYBRID storage device, and an SSD storage device. Any type of storage device other than the type. For example, if the storage device to be migrated is a SATA storage device, the target migration device may be any one of a HYBRID storage device and an SSD storage device, that is, a HYBRID storage device or an SSD. Storage device.

Embodiment 3

The above is the data migration method provided by the embodiment of the present application. Based on the same idea, the embodiment of the present application further provides a data migration device, as shown in FIG. 6 .

The data migration device includes: a expansion module 601, a data migration module 602, and a reduction module 603, wherein:

The expansion module 601 is configured to add a target migration device to the storage system to perform capacity expansion processing on the storage system according to the capacity of the storage device to be migrated in the storage system;

The data migration module 602 is configured to perform a write-protection process on the to-be-migrated storage device, and migrate data stored in the to-be-migrated storage device to the target migration device.

The volume reduction module 603 is configured to remove the to-be-migrated storage device in the storage system to perform a shrinking process on the storage system.

In the embodiment of the present application, the data migration module 602 includes:

The data to be migrated determining unit is configured to determine data to be migrated in the to-be-migrated storage device according to a predetermined load balancing rule according to the storage space utilization ratio of the storage device to be migrated and the target migration device;

a data migration unit, configured to migrate the data to be migrated to the target migration device by using the predetermined load balancing rule, and set the to-be-migrated storage device to a quasi-offline state, and control the to-be-migrated storage device Remaining data other than the data to be migrated is migrated to the target migration device.

In the embodiment of the present application, the to-be-migrated storage device and the target migration device are any two of a serial high-tech configuration interface SATA storage device, a hybrid HYBRID storage device, and a solid-state hard disk SSD storage device.

In the embodiment of the present application, the device further includes:

The information modification module is configured to modify the tag attribute information that matches the to-be-migrated storage device to the tag attribute information that matches the target migration device.

In the embodiment of the present application, the data migration unit is configured to sequentially perform the to-be-migrated storage device in the preset rack according to the preset rack in which the storage device to be migrated is located in the storage system. The storage device to be migrated is configured to migrate the remaining data except the data to be migrated to the target migration device.

In the embodiment of the present application, the to-be-migrated storage device is a metadata server or a data storage server in the storage system, where the metadata server is configured to manage metadata of data stored in the data storage server.

In the embodiment of the present application, the to-be-migrated storage device includes a metadata server and a data storage server in the storage system, where the target migration device includes a target metadata server and a target data storage server.

Correspondingly, the data migration module 602 is configured to perform a write inhibit operation on the metadata server, and migrate data stored in the metadata server to the target metadata server; and the data storage server A write inhibit operation process is performed to migrate data stored in the data storage server to the target data storage server.

The embodiment of the present application provides a data migration device, which is configured to expand a storage system by using a capacity of the storage device to be migrated in the storage system to add the target migration device to the storage system, and then perform a write disable operation on the migration storage device. The data stored in the storage device to be migrated is migrated to the target migration device. Finally, the storage system is compressed to remove the storage device to be migrated from the storage system. Processing can complete data migration and device replacement without interrupting service and quality of service that does not affect online services. It does not affect the user's reading and writing of data in the storage system, and can improve the speed of data migration. Reduce the time it takes for data migration.

Embodiment 4

The above is the data migration device provided by the embodiment of the present application. Based on the same idea, the embodiment of the present application further provides a data migration device, as shown in FIG. 7 .

The data migration device may be provided by the above embodiment.

The migration device for data may vary considerably depending on configuration or performance, and may include one or more processors 701 and memory 702 in which one or more storage applications or data may be stored. Among them, the memory 702 can be short-term storage or persistent storage. An application stored in memory 702 may include one or more modules (not shown), each of which may include a series of computer executable instructions in a migration device for data. Still further, the processor 701 can be arranged to communicate with the memory 702 to execute a series of computer executable instructions in the memory 702 on the migration device of the data. The data migration device may also include one or more power sources 703, one or more wired or wireless network interfaces 704, one or more input and output interfaces 705, one or more keyboards 706.

Specifically in this embodiment, the data migration device includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each The modules may include a series of computer executable instructions in a migration device for data, and are configured to be executed by one or more processors. The one or more programs are included for performing the following computer executable instructions:

Optionally, the executable instructions, when executed, may also cause the processor to:

Determining data to be migrated in the to-be-migrated storage device according to a predetermined load balancing rule according to the storage space utilization ratio of the to-be-migrated storage device and the target migration device;

The data to be migrated is migrated to the target migration device by the predetermined load balancing rule, and the storage device to be migrated is set to a quasi offline state, and the data to be migrated is controlled by the to-be-migrated storage device. The remaining data outside is migrated to the target migration device.

Modifying the tag attribute information that matches the to-be-migrated storage device to the tag attribute information that matches the target migration device.

The storage device to be migrated in the preset rack is set to a quasi-offline state, and the storage device to be migrated is controlled according to the preset rack in which the storage device to be migrated is located in the storage system. Remaining data other than the data to be migrated is migrated to the target migration device.

The storage device to be migrated includes a metadata server and a data storage server in the storage system, and the target migration device includes a target metadata server and a target data storage server.

Correspondingly, the performing the forbidden write operation on the to-be-migrated storage device, and the data stored in the to-be-migrated storage device is migrated to the target migration device, including:

Performing a write inhibit operation on the metadata server, and migrating the data stored in the metadata server to the target metadata server;

Performing a write inhibit operation on the data storage server, and migrating the data stored in the data storage server to the target data storage server.

The embodiment of the present application provides a data migration device, which is configured to expand a storage system to add a target migration device to the storage system, and then perform a write-free operation on the migration storage device. The data stored in the storage device to be migrated is migrated to the target migration device. Finally, the storage system is compressed to remove the storage device to be migrated from the storage system. Processing can complete data migration and device replacement without interrupting service and quality of service that does not affect online services. It does not affect the user's reading and writing of data in the storage system, and can improve the speed of data migration. Reduce the time it takes for data migration.

The foregoing description of the specific embodiments of the specification has been described. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the embodiments and still achieve the desired results. In addition, the processes depicted in the figures are not necessarily in a particular order or in a sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

In the 1990s, improvements to a technology could clearly distinguish between hardware improvements (eg, improvements to circuit structures such as diodes, transistors, switches, etc.) or software improvements (for process flow improvements). However, as technology advances, many of today's method flow improvements can be seen as direct improvements in hardware circuit architecture. Designers almost always get the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented by hardware entity modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is an integrated circuit whose logic function is determined by the user programming the device. Designers can program themselves to "integrate" a digital system on a single PLD without having to ask the chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, today, instead of manually making integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used in programming development, but before compiling The original code has to be written in a specific programming language. This is called the Hardware Description Language (HDL). HDL is not the only one, but there are many kinds, such as ABEL (Advanced Boolean Expression Language). AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., are currently the most commonly used VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It should also be apparent to those skilled in the art that the hardware flow for implementing the logic method flow can be easily obtained by simply programming the method flow into the integrated circuit with a few hardware description languages.

The controller can be implemented in any suitable manner, for example, the controller can take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (eg, software or firmware) executable by the (micro)processor. In the form of logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, The Microchip PIC18F26K20 and the Silicone Labs C8051F320, the memory controller can also be implemented as part of the memory's control logic. Those skilled in the art will also appreciate that in addition to implementing the controller in purely computer readable program code, the controller can be logically programmed by means of logic gates, switches, ASICs, programmable logic controllers, and embedding. The form of a microcontroller or the like to achieve the same function. Such a controller can therefore be considered a hardware component, and the means for implementing various functions included therein can also be considered as a structure within the hardware component. Or even a device for implementing various functions can be considered as a software module that can be both a method of implementation and a structure within a hardware component.

The system, device, module or unit illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product having a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or A combination of any of these devices.

For the convenience of description, the above devices are described separately by function into various units. Of course, the functions of each unit may be implemented in the same software or software and/or hardware when implementing the present application.

Those skilled in the art will appreciate that embodiments of the present invention can be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or a combination of software and hardware. Moreover, the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (system), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory. Memory is an example of a computer readable medium.

Computer readable media includes both permanent and non-persistent, removable and non-removable media. Information storage can be implemented by any method or technology. The information can be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape storage or other magnetic storage devices or any other non-transportable media can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary storage of computer readable media, such as modulated data signals and carrier waves.

It is also to be understood that the terms "comprises" or "comprising" or "comprising" or any other variations are intended to encompass a non-exclusive inclusion, such that a process, method, article, Other elements not explicitly listed, or elements that are inherent to such a process, method, commodity, or equipment. An element defined by the phrase "comprising a ..." does not exclude the presence of additional equivalent elements in the process, method, item, or device including the element.

Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system, or computer program product. Thus, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment in combination of software and hardware. Moreover, the application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

The application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

The various embodiments in the specification are described in a progressive manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

The above description is only an embodiment of the present application and is not intended to limit the application. Various changes and modifications can be made to the present application by those skilled in the art. Any modifications, equivalents, improvements, etc. made within the spirit and scope of the present application are intended to be included within the scope of the appended claims.

Claims

A method for migrating data, characterized in that the method comprises:

Adding a target migration device to the storage system to perform capacity expansion processing on the storage system according to the capacity of the storage device to be migrated in the storage system;

Performing a write-protection process on the to-be-migrated storage device, and migrating data stored in the to-be-migrated storage device to the target migration device;

Removing the storage device to be migrated in the storage system to perform a shrinking process on the storage system.
The method according to claim 1, wherein the migrating the data stored in the storage device to be migrated to the target migration device comprises:

Determining data to be migrated in the to-be-migrated storage device according to a predetermined load balancing rule according to the storage space utilization ratio of the to-be-migrated storage device and the target migration device;

The data to be migrated is migrated to the target migration device by the predetermined load balancing rule, and the storage device to be migrated is set to a quasi offline state, and the data to be migrated is controlled by the to-be-migrated storage device. The remaining data outside is migrated to the target migration device.
The method according to claim 1, wherein the to-be-migrated storage device and the target migration device are any two of a serial high-tech configuration interface SATA storage device, a hybrid HYBRID storage device, and a solid-state hard disk SSD storage device. Kind.
The method of claim 3, wherein the method further comprises:

Modifying the tag attribute information that matches the to-be-migrated storage device to the tag attribute information that matches the target migration device.
The method according to claim 2, wherein the setting the storage device to be migrated to a quasi-offline state, and controlling the to-be-migrated storage device to migrate remaining data other than the data to be migrated to the Target migration equipment, including:

The storage device to be migrated in the preset rack is set to a quasi-offline state, and the storage device to be migrated is controlled according to the preset rack in which the storage device to be migrated is located in the storage system. Remaining data other than the data to be migrated is migrated to the target migration device.
The method according to claim 1, wherein the storage device to be migrated is a metadata server or a data storage server in the storage system, wherein the metadata server is used to manage the data storage server. Metadata for stored data.
The method according to claim 6, wherein the storage device to be migrated comprises a metadata server and a data storage server in the storage system, and the target migration device comprises a target metadata server and a target data storage server.

Correspondingly, the performing the forbidden write operation on the to-be-migrated storage device, and the data stored in the to-be-migrated storage device is migrated to the target migration device, including:

Performing a write inhibit operation on the metadata server, and migrating the data stored in the metadata server to the target metadata server;

Performing a write inhibit operation on the data storage server, and migrating the data stored in the data storage server to the target data storage server.
A data migration device, characterized in that the device comprises:

a capacity expansion module, configured to add a target migration device to the storage system to perform capacity expansion processing on the storage system according to a capacity of the storage device to be migrated in the storage system;

a data migration module, configured to perform a write-protection process on the to-be-migrated storage device, and migrate data stored in the to-be-migrated storage device to the target migration device;

And a shrinking module, configured to remove the to-be-migrated storage device in the storage system to perform a shrinking process on the storage system.
The device according to claim 8, wherein the data migration module comprises:

The data to be migrated determining unit is configured to determine data to be migrated in the to-be-migrated storage device according to a predetermined load balancing rule according to the storage space utilization ratio of the storage device to be migrated and the target migration device;

a data migration unit, configured to migrate the data to be migrated to the target migration device by using the predetermined load balancing rule, and set the to-be-migrated storage device to a quasi-offline state, and control the to-be-migrated storage device Remaining data other than the data to be migrated is migrated to the target migration device.
A data migration device, characterized in that the device comprises:

Processor;

A memory arranged to store computer executable instructions that, when executed, cause the processor to:

Adding a target migration device to the storage system to perform capacity expansion processing on the storage system according to the capacity of the storage device to be migrated in the storage system;

Performing a write-protection process on the to-be-migrated storage device, and migrating data stored in the to-be-migrated storage device to the target migration device;

Removing the storage device to be migrated in the storage system to perform a shrinking process on the storage system.