WO2021103596A1 - 一种数据迁移方法、装置和计算机可读存储介质 - Google Patents

一种数据迁移方法、装置和计算机可读存储介质 Download PDF

Info

Publication number
WO2021103596A1
WO2021103596A1 PCT/CN2020/103695 CN2020103695W WO2021103596A1 WO 2021103596 A1 WO2021103596 A1 WO 2021103596A1 CN 2020103695 W CN2020103695 W CN 2020103695W WO 2021103596 A1 WO2021103596 A1 WO 2021103596A1
Authority
WO
WIPO (PCT)
Prior art keywords
migration
file
migrations
task
target
Prior art date
Application number
PCT/CN2020/103695
Other languages
English (en)
French (fr)
Inventor
王帅阳
孟祥瑞
李文鹏
Original Assignee
浪潮电子信息产业股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 浪潮电子信息产业股份有限公司 filed Critical 浪潮电子信息产业股份有限公司
Publication of WO2021103596A1 publication Critical patent/WO2021103596A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Definitions

  • This application relates to the field of data storage technology, and in particular to a data migration method, data migration device, electronic equipment, and computer-readable storage medium.
  • the purpose of this application is to provide a data migration method, a data migration device, an electronic device and a computer-readable storage medium, which can improve the efficiency of data migration.
  • the specific plan is as follows:
  • This application discloses a data migration method, including:
  • the current system idle IO is the IO after excluding the IO actually occupied by the user and the reserved IO from the maximum IO of the system.
  • the determining the number of concurrent migration migrations based on the attributes of the current system idle IO and the preset migration IO required for each file migration task includes:
  • the smallest value among the first migration number and the second migration number is determined as the migration number.
  • the sending the migration task of the target migration file to the client according to the number of migrations includes:
  • Migration tasks include:
  • the new number of concurrent migrations is determined according to the attributes of the current system idle IO;
  • it also includes:
  • the method further includes:
  • stop scanning where the files to be migrated include target migration files.
  • the method further includes:
  • the processing information of the target migration file is recorded.
  • the reserved IO is 20% of the maximum IO of the system.
  • This application provides a data migration device, including:
  • the target migration file obtaining module is used to obtain the target migration file
  • the migration number adjustment module is used to determine the migration number of concurrent migrations according to the attributes of the current system idle IO and the preset migration IO required for each file migration task;
  • a migration task issuing module configured to send a migration task of the target migration file to the client according to the number of migrations, so that the client concurrently migrates the target migration file according to the migration task;
  • the current system idle IO is the IO after excluding the IO actually occupied by the user and the reserved IO from the maximum IO of the system.
  • This application provides an electronic device, including:
  • Memory used to store computer programs
  • the processor is used to implement the steps of the above data migration method when the computer program is executed.
  • the present application provides a computer-readable storage medium having a computer program stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above data migration method are realized.
  • This application provides a data migration method, including: obtaining a target migration file; determining the number of concurrent migration migrations according to the attributes of the current system idle IO and the preset migration IO required for each file migration task; The client sends the migration task of the target migration file, so that the client concurrently migrates the target migration file according to the migration task; among them, the current system idle IO is the IO after excluding the IO actually occupied by the user and the reserved IO from the maximum IO of the system.
  • this application allocates the maximum IO of the system to the remaining IO of the current system, the actual IO occupied by the user, and the reserved IO.
  • the reserved IO is used for buffering by the user. While the fast pool is quickly cleaned up to improve the overall performance of the storage system, the reserved IO is not set. Affect user usage; determine the number of concurrent migrations by using the attributes of the current system idle IO, adaptively adjust the number of concurrent migrations, improve the speed of file migration, and send the migration of the target migration file to the client according to the number of migrations Task, in this process, an asynchronous data storage method is adopted, which avoids the disadvantages of low migration efficiency and long migration period caused by related technologies through synchronous data storage, and improves the data migration efficiency.
  • This application also provides a data migration device, electronic equipment, and computer-readable storage medium, which all have the above-mentioned beneficial effects, and will not be repeated here.
  • FIG. 1 is a flowchart of a data migration method provided by an embodiment of the application
  • FIG. 2 is a schematic diagram of IO allocation in a storage system provided by an embodiment of the application.
  • FIG. 3 is a schematic diagram of a process for determining the number of migrations provided by an embodiment of the application
  • FIG. 4 is a flowchart of a migration task issuance provided by an embodiment of the application.
  • FIG. 5 is a specific data migration method provided by an embodiment of the application.
  • FIG. 6 is a schematic structural diagram of a data migration device provided by an embodiment of this application.
  • FIG. 7 is a structural diagram of an electronic device provided by an embodiment of this application.
  • FIG. 8 is a structural diagram of another electronic device provided by an embodiment of the application.
  • FIG. 1 is a flowchart of a data migration method provided by an embodiment of the application, which specifically includes:
  • the target migration file is the actual user data file.
  • S102 Determine the number of concurrent migration migrations according to the attributes of the current system idle IO and the preset migration IO required for each file migration task.
  • the current system idle IO is the IO after excluding the IO actually occupied by the user and the reserved IO from the maximum IO of the system.
  • Figure 2 is a schematic diagram of IO allocation of a storage system provided by an embodiment of this application.
  • the maximum IO of the system is recorded as max_io, and the remaining IO of the current system is recorded as left_io, and the IO actually occupied by the user is recorded as User_io and reserved IO are denoted as tmp_io.
  • the maximum IO of the system can be calculated from two aspects, namely iops and aggregate bandwidth.
  • the maximum IO of the system is the maximum IO capacity of the storage system. It can be further understood that the remaining IO of the current system is Identifies that the system can still increase the processing IO capability.
  • the concurrent migration capability is mainly based on the current system idle IO.
  • the actual IO occupied by the user includes the migration IO, because the user interface is called during the migration, which can be considered as a user operation.
  • the migration IO determines the number of concurrent migrations. After the migration IO is initiated, the actual IO occupied by the user increases, corresponding to the current system The remaining IO is reduced.
  • the reserved IO is not limited to 20% of the maximum IO of the system, 30%, 40% of the maximum IO of the system, or other values. Preferably, it is set to the system The maximum is 20%, so that under the condition of ensuring high data migration rate, it will not affect the user's operation.
  • the maximum IO of the system is calculated in iops, when the maximum iops of the system is 5000 iops, the reserved IO is 1000 iops, and the sum of the actual IO occupied by the user and the remaining IO of the current system is 4000 iops; when the maximum IO of the system is performed by the aggregate bandwidth
  • the maximum IO of the system is performed by the aggregate bandwidth
  • the aggregate bandwidth corresponding to the maximum IO of the system is 1000G
  • the reserved IO is 200G
  • the sum of the actual IO occupied by the user and the remaining IO of the current system is 800G
  • the current system IO can also be determined by both.
  • This embodiment will not be elaborated again, and the user can make settings according to the actual situation, as long as the purpose of this embodiment can be achieved.
  • this embodiment does not limit the method for determining the number of migrations. It can be calculated based on the iops of the remaining IO of the current system, or calculated based on the remaining aggregate bandwidth of the remaining IO of the current system, or calculated based on the remaining IO of the current system iops and remaining aggregate bandwidth are calculated.
  • calculating according to the IOPS of the current system remaining IO includes: determining the system idle iops, and dividing the system idle iops by the preset migration iops required to add a file migration task to obtain an integer value as the migration For example, if the system idle iops is 3000 iops, and the preset migration iops required to add a file migration task is 100 iops, then the number of migrations is determined to be 30.
  • the remaining aggregate bandwidth of the system is determined, and the remaining aggregate bandwidth of the system is divided by the preset migration aggregate bandwidth required for adding a file migration task as an integer value as the migration number, for example, system aggregation
  • the bandwidth is 500G
  • the preset migration aggregate bandwidth required to add a file migration task is 10G
  • the number of migrations is determined to be 50.
  • FIG. 3 is a schematic diagram of a process for determining the number of migrations provided in an embodiment of the application, including:
  • the system idle iops divide the system idle iops by the preset number of migration iops required to add a file migration task as the number of migrations. For example, if the system idle iops is 3000 iops, and the preset migration iops required to add a file migration task is 100 iops, then the first migration number is determined to be 30.
  • the migration number is 30 and the second migration number is 50, it is determined that the migration number is 30.
  • the file migration speed is adapted to the current storage performance according to the remaining amount of system performance, and the adaptive number of migrations can be obtained more accurately, thereby improving the efficiency of concurrent migration.
  • S103 Send a migration task of the target migration file to the client according to the number of migrations, so that the client concurrently migrates the target migration file according to the migration task.
  • the migration task of sending the target migration file to the client according to the number of migrations may specifically include: when the number of migrations is less than or equal to the number of files currently being processed, that is (the number of files to be migrated), then the processing is completed After a file is migrated, the new migration number is calculated. If the new migration number is less than or equal to the number of files currently being processed, the new migration number will be calculated after the next file migration is processed. , Until the calculated number of migrations is greater than the number of files currently being processed, the migration task of the target migration file is sent to the client, so that the client can concurrently process the target migration file according to the migration task.
  • step S103 it further includes: judging whether the file to be migrated has completed the concurrent migration; if it is completed, stop scanning, wherein the file to be migrated includes the target migration file. It can be seen that through regular scanning to determine the completion of the task, it is helpful to understand the completion of the file migration in real time.
  • the method further includes: after the concurrent migration of the target migration file is completed, recording the processing information of the target migration file.
  • the processing information includes but is not limited to the ID of the target migration file, migration start time, migration end time, and concurrent processing thread ID.
  • the data migration method provided in this embodiment can give full play to system storage performance, improve file classification data migration speed, improve file classification scanning thread processing capabilities, speed up the recovery of space capacity of the high-speed pool, and give full play to the role of the high-speed pool.
  • the maximum IO allocation of the storage performance system is defined by dividing the storage performance; the file migration speed is adapted to the current storage performance according to the current system idle IO; the user reserves the IO storage capacity to handle the user's burst IO.
  • the number of migrations is dynamically adjusted to control the migration speed. On the basis of making full use of system storage performance for data migration, it can ensure that user services are not affected.
  • this embodiment allocates the maximum IO of the system to the remaining IO of the current system, the actual IO occupied by the user, and the reserved IO.
  • the IO is reserved for user buffering, and the fast pool is quickly cleaned up to improve the overall performance of the storage system.
  • the reserved IO does not affect the user's use; by using the attributes of the current system idle IO to determine the number of concurrent migrations, adaptively adjust the number of concurrent migrations, improve the file migration speed, and send the target to the client according to the number of migrations
  • asynchronous data storage is adopted in this process, which avoids the disadvantages of low migration efficiency and long migration period caused by the synchronous data storage of related technologies, and improves the efficiency of data migration.
  • FIG. 4 is a flow chart of a migration task issuance provided by an embodiment of the application, including:
  • the concurrent number that is, the number of migrations
  • the concurrent number must be dynamically adjusted according to the current system idle IO of the cluster system storage performance to determine whether the number of data migration files currently in progress If the number is less than the number of migrations, if the number is greater than the number of migrations, wait for the file migration completion message, and if the number is less than the number of migrations, the migration task will be sent to the migration client, and then the next file will be processed.
  • step S1033 may specifically include: when a file to be migrated is completed, determining the new migration number for concurrent migration according to the attributes of the current system idle IO; if the number of all files to be migrated is less than the new migration number, then Send the migration task of the target migration file to the client.
  • the file migration speed is adapted to the current storage performance according to the remaining amount of system performance, that is, the current system idle IO, and the migration speed is controlled by dynamic adjustment of the number of migrations.
  • Figure 5 is a specific data migration method provided by an embodiment of the application, including:
  • the concurrent number that is, the number of migrations
  • the concurrent number must be dynamically adjusted according to the storage performance of the cluster system, to determine whether the number of data migration files currently in progress, that is, the number of files to be migrated in progress, is less than the threshold. If the number of migrations is greater than the threshold, wait for the file migration completion message. If the number is less than the threshold, the migration task will be sent to the client, and then continue to process the next file.
  • FIG. 6 is a data migration device provided by an embodiment of the present application. Schematic diagram of the structure of the data migration device, including:
  • the target migration file obtaining module 610 is used to obtain the target migration file
  • the migration number adjustment module 620 is configured to determine the migration number of concurrent migrations according to the attributes of the current system idle IO and the preset migration IO required for each file migration task;
  • the migration task issuing module 630 is configured to send the migration task of the target migration file to the client according to the number of migrations, so that the client concurrently migrates the target migration file according to the migration task;
  • the current system idle IO is the IO after excluding the IO actually occupied by the user and the reserved IO from the maximum IO of the system.
  • the migration number adjustment module 620 includes:
  • the first migration number determining unit is used to determine the first migration number according to the system idle iops and the migration iops required for each file migration task preset;
  • the second migration number determining unit is configured to determine the second migration number according to the remaining aggregate bandwidth of the system and the preset migration aggregate bandwidth required for each file migration task;
  • the migration number determining unit is used to determine the smallest value among the first migration number and the second migration number as the migration number.
  • the migration task issuing module 630 includes:
  • the judging unit is used to judge whether the number of files to be migrated currently being executed is less than the number of files to be migrated;
  • the first migration task issuing unit is configured to send the migration task of the target migration file to the client if the number of migration tasks is less than the number of migrations;
  • the second migration task issuing unit is configured to wait until the number of all files to be migrated is less than the number of migration if the number of migration tasks is greater than the number of migration, and send the migration task of the target migration file to the client.
  • the second migration task issuing unit includes:
  • the new migration number determining subunit is used to determine the new migration number of concurrent migrations according to the attributes of the current system idle IO when a file to be migrated is completed;
  • the second migration task issuing subunit is configured to send the migration task of the target migration file to the client if the number of all files to be migrated is less than the new migration number.
  • it further includes:
  • Scanning module used to scan the files to be migrated regularly
  • the judgment module is used to judge whether the files to be migrated have completed concurrent migration
  • the stop scanning module is used to stop scanning if it is completed, wherein the file to be migrated includes the target migration file.
  • it further includes:
  • the recording module is used to record the processing information of the target migration file after the concurrent migration of the target migration file is completed.
  • the reserved IO is 20% of the maximum IO of the system.
  • the embodiment of the data migration device part corresponds to the embodiment of the data migration method part, please refer to the description of the embodiment of the data migration method part for the embodiment of the data migration device part, which will not be repeated here.
  • FIG. 7 is a structural diagram of an electronic device provided by an embodiment of the application, including:
  • the memory 100 is used to store computer programs
  • the processor 200 is configured to implement the steps provided in the foregoing embodiments when executing a computer program.
  • the memory 100 includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system and computer-readable instructions
  • the internal memory provides an environment for running the operating system and the computer-readable instructions in the non-volatile storage medium.
  • the processor 200 provides computing and control capabilities for the electronic device.
  • the following steps can be implemented: obtaining the target migration file; according to the attributes of the current system idle IO and the preset requirements for each file migration task Migrate IO, determine the number of concurrent migrations; send the migration task of the target migration file to the client according to the number of migrations, so that the client can perform concurrent migration of the target migration file according to the migration task; among them, the current system idle IO is the system maximum IO IO after removing the IO actually occupied by the user and the reserved IO.
  • the processor 200 executes the computer subprogram stored in the memory 100, the following steps may be implemented: determine the first migration number according to the system idle iops and the preset migration iops required for each file migration task; Determine the second migration number according to the remaining aggregate bandwidth of the system and the preset migration aggregate bandwidth required for each file migration task; determine the smallest value of the first migration number and the second migration number as the migration number.
  • the processor 200 executes the computer subprogram stored in the memory 100, the following steps may be implemented: determine whether the number of files to be migrated currently being executed is less than the number of migrations; if it is less than the number of migrations, then Send the migration task of the target migration file to the client; if it is greater than the number of migration, wait until the number of all the files to be migrated is less than the number of migration, and send the migration task of the target migration file to the client.
  • the processor 200 executes the computer subprogram stored in the memory 100, the following steps may be implemented: when a file to be migrated is completed, determine the new number of concurrent migrations to be migrated according to the attributes of the current system idle IO; If the number of all the files to be migrated is less than the new number of migrations, the migration task of the target migration file is sent to the client.
  • the processor 200 executes the computer subprogram stored in the memory 100, the following steps may be implemented: scan the file to be migrated regularly; determine whether the file to be migrated has completed concurrent migration; if it is completed, stop scanning, where The migration file includes the target migration file.
  • FIG. 8 is a structural diagram of another electronic device provided by an embodiment of this application, and the electronic device further includes:
  • the input interface 300 is connected to the processor 200 and is used to obtain externally imported computer programs, parameters, and instructions, and store them in the memory 100 under the control of the processor 200.
  • the input interface 300 can be connected to an input device to receive parameters or instructions manually input by the user.
  • the input device can be a touch layer covered on the display screen, a button, a trackball or a touchpad provided on the terminal shell, or a keyboard, a touchpad, or a mouse.
  • the display unit 400 is connected to the processor 200 and is used for displaying data sent by the processor 200.
  • the display unit 400 may be a display screen on a PC, a liquid crystal display screen, an electronic ink display screen, or the like.
  • the network port 500 is connected to the processor 200 and is used to communicate with external terminal devices.
  • the communication technology used in the communication connection can be wired communication technology or wireless communication technology, such as mobile high-definition link technology (MHL), universal serial bus (USB), high-definition multimedia interface (HDMI), wireless fidelity technology (WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, communication technology based on IEEE802.11s, etc.
  • the following introduces a computer-readable storage medium provided by an embodiment of the present application.
  • the computer-readable storage medium described below and the method described above may refer to each other correspondingly.
  • This embodiment provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned data migration method are implemented.
  • the storage medium may include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.
  • a computer program is stored on the storage medium. When the computer program is executed by the processor, the following steps are implemented: obtain the target migration file; determine the concurrent migration based on the attributes of the current system idle IO and the preset migration IO required for each file migration task The number of migrations; according to the number of migrations, the migration task of the target migration file is sent to the client, so that the client can concurrently migrate the target migration file according to the migration task; among them, the current system idle IO is the maximum IO of the system, excluding the IO and actual occupied IO by the user. IO after IO is reserved.
  • the following steps can be implemented: determine the first migration according to the system idle iops and the migration iops required for each file migration task preset Number; Determine the second migration number according to the remaining aggregate bandwidth of the system and the preset migration aggregate bandwidth required for each file migration task; determine the smallest value of the first migration number and the second migration number as the migration number .
  • the following steps can be implemented: determine whether the number of all files to be migrated currently being executed is less than the number of files to be migrated; If the number of files to be migrated is less than the number of files to be migrated, then the migration task of the target migration file is sent to the client; if it is greater than the number of migrations, it waits until the number of all files to be migrated is less than the number of migration, and the migration task of the target migration file is sent to the client.
  • the following steps may be implemented: when a file to be migrated is completed, a new migration for concurrent migration is determined according to the attributes of the current system idle IO Number; if the number of all the files to be migrated is less than the number of new migrations, the migration task of the target migration file is sent to the client.
  • the following steps can be implemented: scan the file to be migrated regularly; determine whether the file to be migrated has completed concurrent migration; if it is completed, stop scanning, Among them, the files to be migrated include target migration files.
  • the steps of the method or algorithm described in the embodiments disclosed in this document can be directly implemented by hardware, a software module executed by a processor, or a combination of the two.
  • the software module can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or all areas in the technical field. Any other known storage media.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种数据迁移方法、装置和计算机可读存储介质，该方法包括：获取目标迁移文件；根据当前系统空闲IO的属性与预设的每个文件迁移任务需要的迁移IO，确定并发迁移的迁移个数；根据迁移个数向客户端发送目标迁移文件的迁移任务，以便客户端根据迁移任务对目标迁移文件进行并发迁移；其中，当前系统空闲IO为系统最大IO中去除用户实际占用IO和预留IO后的IO。本申请通过设置预留IO供用户缓冲使用，快速清理快速池提升存储系统整体性能的同时，不影响用户使用；通过利用当前系统空闲IO的属性确定并发迁移的迁移个数，自适应的调整并发迁移的迁移个数，提高数据迁移效率。

Description

一种数据迁移方法、装置和计算机可读存储介质
本申请要求于2019年11月29日提交中国专利局、申请号为201911204966.6、发明名称为“一种数据迁移方法、装置和计算机可读存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及数据存储技术领域，特别涉及一种数据迁移方法、数据迁移装置、电子设备和计算机可读存储介质。
背景技术
对于分布式文件系统，按照文件顺序对分级文件数据进行数据迁移，若当前文件的数据迁移完成后，才继续进行下一个文件的数据迁移，因此，导致文件迁移效率低下，数据迁移消耗周期长，快速池内数据不能及时迁移至低速池，影响整个存储系统的性能，针对此问题相关技术是增加并发迁移，配置固定的并发迁移数，但是此方案难以适配现场所有场景，例如在高并发迁移时会影响用户的IO操作；用户低IO操作时，迁移速度受自生并发迁移数配置限制，文件迁移效率较低。
因此，如何提供一种解决上述技术问题的方案是本领域技术人员目前需要解决的问题。
发明内容
本申请的目的是提供一种数据迁移方法、数据迁移装置、电子设备和计算机可读存储介质，能够提高数据迁移效率。其具体方案如下：
本申请公开了一种数据迁移方法，包括：
获取目标迁移文件；
根据当前系统空闲IO的属性与预设的每个文件迁移任务需要的迁移IO,确定并发迁移的迁移个数；
根据所述迁移个数向客户端发送所述目标迁移文件的迁移任务，以便所述客户端根据所述迁移任务对所述目标迁移文件进行并发迁移；
其中，所述当前系统空闲IO为系统最大IO中去除用户实际占用IO和预留IO后的IO。
可选的，所述根据当前系统空闲IO的属性与预设的每个文件迁移任务需要的迁移IO,确定并发迁移的迁移个数，包括：
根据系统空闲iops与预设的每个文件迁移任务需要的迁移iops，确定第一迁移个数；
根据系统剩余聚合带宽与预设的每个文件迁移任务需要的迁移聚合带宽，确定第二迁移个数；
确定所述第一迁移个数与所述第二迁移个数中最小的数值作为所述迁移个数。
可选的，所述根据所述迁移个数向客户端发送所述目标迁移文件的迁移任务，包括：
判断当前正在执行的所有的待迁移文件的数量是否小于所述迁移个数；
若小于所述迁移个数，则向所述客户端发送所述目标迁移文件的所述迁移任务；
若大于所述迁移个数，则进行等待，直至所有的所述待迁移文件的数量小于所述迁移个数，并向所述客户端发送所述目标迁移文件的所述迁移任务。
可选的，所述若大于所述迁移个数，则进行等待，直至所有的所述待迁移文件的数量小于所述迁移个数，并向所述客户端发送所述目标迁移文件的所述迁移任务，包括：
当完成一个待迁移文件时，根据当前系统空闲IO的属性确定并发迁移的新的迁移个数；
若所有的所述待迁移文件的数量小于所述新的迁移个数，则向所述客户端发送所述目标迁移文件的所述迁移任务。
可选的，还包括：
定时扫描待迁移文件；
对应的，所述根据所述迁移个数向客户端发送所述目标迁移文件的迁移任务，以便所述客户端根据所述迁移任务对所述目标迁移文件进行并发 迁移之后，还包括：
判断所述待迁移文件是否完成并发迁移；
若完成，则停止扫描，其中，所述待迁移文件包括目标迁移文件。
可选的，所述根据所述迁移个数向客户端发送所述目标迁移文件的迁移任务，以便所述客户端根据所述迁移任务对所述目标迁移文件进行并发迁移之后，还包括：
当所述目标迁移文件并发迁移完成后，记录所述目标迁移文件的处理信息。
可选的，所述预留IO是系统最大IO的20％。
本申请提供一种数据迁移装置，包括：
目标迁移文件获取模块，用于获取目标迁移文件；
迁移个数调整模块，用于根据当前系统空闲IO的属性与预设的每个文件迁移任务需要的迁移IO，确定并发迁移的迁移个数；
迁移任务下发模块，用于根据所述迁移个数向客户端发送所述目标迁移文件的迁移任务，以便所述客户端根据所述迁移任务对所述目标迁移文件进行并发迁移；
其中，所述当前系统空闲IO为系统最大IO中去除用户实际占用IO和预留IO后的IO。
本申请提供一种电子设备，包括：
存储器，用于存储计算机程序；
处理器，用于执行所述计算机程序时实现如上述数据迁移方法的步骤。
本申请提供一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如上述数据迁移方法的步骤。
本申请提供一种数据迁移方法，包括：获取目标迁移文件；根据当前系统空闲IO的属性与预设的每个文件迁移任务需要的迁移IO，确定并发迁移的迁移个数；根据迁移个数向客户端发送目标迁移文件的迁移任务，以便客户端根据迁移任务对目标迁移文件进行并发迁移；其中，当前系统空闲IO为系统最大IO中去除用户实际占用IO和预留IO后的IO。
可见，本申请将系统最大IO分配为当前系统剩余IO、用户实际占用IO和预留IO，预留IO供用户缓冲使用，快速清理快速池提升存储系统整体性能的同时，设置的预留IO不影响用户使用；通过利用当前系统空闲IO的属性确定并发迁移的迁移个数，自适应的调整并发迁移的迁移个数，提升了文件迁移速度，根据迁移个数向客户端发送目标迁移文件的迁移任务，在此过程中采用了异步数据存储的方式，避免了相关技术通过同步数据存储的方式造成的迁移效率低、迁移周期长的缺点，提高数据迁移效率。
本申请同时还提供的一种数据迁移装置、电子设备和计算机可读存储介质，均具有上述有益效果，在此不再进行赘述。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据提供的附图获得其他的附图。
图1为本申请实施例所提供的一种数据迁移方法的流程图；
图2为本申请实施例提供的一种存储系统IO分配示意图；
图3为本申请实施例提供的一种迁移个数确定的流程示意图；
图4为本申请实施例所提供的一种迁移任务下发的流程图；
图5为本申请实施例提供的一种具体的数据迁移方法；
图6为本申请实施例所提供的一种数据迁移装置的结构示意图；
图7为本申请实施例提供的一种电子设备的结构图；
图8为本申请实施例提供的另一种电子设备的结构图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
在常见的针对数据迁移进行的操作是增加并发迁移，配置固定的并发迁移数，但是此方案难以适配现场所有场景，例如在高并发迁移时会影响用户的IO操作；用户低IO操作时，迁移速度受自生并发迁移数配置限制，文件迁移效率较低。基于上述技术问题，本实施例提供一种数据迁移方法，具体请参考图1，图1为本申请实施例所提供的一种数据迁移方法的流程图，具体包括：
S101、获取目标迁移文件。
其中，目标迁移文件是实际用户数据文件。
S102、根据当前系统空闲IO的属性与预设的每个文件迁移任务需要的迁移IO,确定并发迁移的迁移个数。
其中，当前系统空闲IO为系统最大IO中去除用户实际占用IO和预留IO后的IO。
可以理解的是，请参考图2，图2为本申请实施例提供的一种存储系统IO分配示意图，系统最大IO记为max_io可以分配为当前系统剩余IO记为left_io、用户实际占用IO记为user_io和预留IO记为tmp_io，其中，系统最大IO可以从两个方面进行计算，分别为iops和聚合带宽，系统最大IO是存储系统最大IO能力，进一步可以理解的是，当前系统剩余IO是标识系统当前还能增加处理的IO能力，在进行数据迁移时，并发迁移能力主要依据的是当前系统空闲IO，当前系统空闲IO越大，对应的并发迁移能力越高，对应的迁移IO并不单独统计，用户实际占用IO包含迁移IO，因为迁移时调用用户接口，可认为是用户操作，迁移IO决定新增的迁移的并发数，迁移IO发起后，用户实际占用IO增加，对应的当前系统剩余IO减少。
针对预留IO本实施例进行进一步阐述，本实施例中不对预留IO进行限定它可以是系统最大IO的20％、系统最大IO的30％、40％或者其他数值，优选地，设置为系统最大的20％，以便在保证高数据迁移速率的条件下，能够不影响用户的操作。具体的，当系统最大IO以iops进行计算时，当系统最大的iops是5000iops，则预留IO为1000iops，用户实际占用IO与当前系统剩余IO的总和为4000iops；当系统最大IO以聚合带宽进行计 算时，系统最大IO对应的聚合带宽为1000G时，则预留IO为200G，用户实际占用IO与当前系统剩余IO的总和为800G；当然，当前系统IO也可以是以两者共同决定，具体的本实施例不再进行阐述，用户可根据实际情况进行设置，只要是能够实现本实施例的目的即可。
对应的，本实施例不对迁移个数的确定方式进行限定，可以是根据当前系统剩余IO的iops进行计算，或，根据当前系统剩余IO的剩余聚合带宽进行计算，或，根据当前系统剩余IO的iops和剩余聚合带宽进行计算。在一种可实现的实施方式中，根据当前系统剩余IO的IOPS进行计算包括：确定系统空闲iops，将系统空闲iops除以预设的增加一个文件迁移任务需要的迁移iops得到的整数值作为迁移个数，例如，系统空闲iops为3000iops，预设的增加一个文件迁移任务需要的迁移iops为100iops则，确定迁移个数为30。在另一种可实现的实施方式中，确定系统剩余聚合带宽，将系统剩余聚合带宽除以预设的增加一个文件迁移任务需要的迁移聚合带宽得到的整数值作为迁移个数，例如，系统聚合带宽为500G，预设的增加一个文件迁移任务需要的迁移聚合带宽为10G,则确定迁移个数为50。
在另一种可实现的实施方式中，针对迁移个数的确定，请参考图3，图3为本申请实施例提供的一种迁移个数确定的流程示意图，包括：
S1021、根据系统空闲iops与预设的每个文件迁移任务需要的迁移iops，确定第一迁移个数。
可以理解的是，确定系统空闲iops，将系统空闲iops除以预设的增加一个文件迁移任务需要的迁移iops得到的整数值作为迁移个数。例如，系统空闲iops为3000iops，预设的增加一个文件迁移任务需要的迁移iops为100iops则，确定第一迁移个数为30。
S1022、根据系统剩余聚合带宽与预设的每个文件迁移任务需要的迁移聚合带宽，确定第二迁移个数。
确定系统剩余聚合带宽，将系统剩余聚合带宽除以预设的增加一个文件迁移任务需要的迁移聚合带宽得到的整数值作为迁移个数，例如，系统聚合带宽为500G，预设的增加一个文件迁移任务需要的迁移聚合带宽为10G，则确定第二迁移个数为50。
S1023、确定第一迁移个数与第二迁移个数中最小的数值作为迁移个 数。
当第一迁移个数是30，第二迁移个数是50时，确定迁移个数为30。
可见，通过以系统空闲iops和系统剩余聚共同确定迁移个数，文件迁移速度根据系统性能剩余量自适应当前存储性能，更加准确的获取自适应的迁移个数，提高并发迁移的效率。
S103、根据迁移个数向客户端发送目标迁移文件的迁移任务，以便客户端根据迁移任务对目标迁移文件进行并发迁移。
其中根据迁移个数向客户端发送目标迁移文件的迁移任务，具体可以包括：当迁移个数小于或者等于当前正在处理的文件量的个数即(待迁移文件的数量)时，则在处理完成一个文件迁移后，再计算新的迁移个数，若新的迁移个数小于或者等于当前正在处理的文件量的个数时，则在处理完成下一个文件迁移后，再计算新的迁移个数，直至当计算得到的迁移个数大于当前正在处理的文件量的个数时，向客户端发送目标迁移文件的迁移任务，以便客户端根据迁移任务对目标迁移文件进行并发处理。
进一步的，还包括：定时扫描待迁移文件。其中，若待迁移文件处理完毕，本周期不需要进行迁移处理了。可以理解的是，待迁移文件包括在目标迁移文件之前的正在处理的文件，还包括目标迁移文件。对应的，步骤S103之后，还包括：判断待迁移文件是否完成并发迁移；若完成，则停止扫描，其中，待迁移文件包括目标迁移文件。可见，通过定时扫描，确定任务的完成情况，有利于实时了解文件迁移的完成情况。
进一步的，步骤S103之后，还包括：当目标迁移文件并发迁移完成后，记录目标迁移文件的处理信息。当目标迁移文件并发迁移完成后，记录下目标迁移文件的处理信息，其中，处理信息包括但是不限定于目标迁移文件的ID、迁移开始时间、迁移结束时间、并发处理的线程ID。
综上可知，本实施例提供的数据迁移方法，能够充分发挥系统存储性能，提升文件分级数据迁移速度，提升文件分级扫描线程处理能力，高速池空间容量回收得到提速，充分发挥高速池的作用，提升整系统存储能力，自管理，发挥系统迁移性能，同时不影响用户业务。本实施例通过界定划分存储性能系统最大IO分配；文件迁移速度根据当前系统空闲IO自适应当前存储性能；用户预留IO存储能力，处理用户突发IO。利用迁移个数 动态调整方式，控制迁移速度。在充分利用系统存储性能的进行数据迁移的基础上，能够保证用户业务不受影响。
基于上述技术方案，本实施例将系统最大IO分配为当前系统剩余IO、用户实际占用IO和预留IO，预留IO供用户缓冲使用，快速清理快速池提升存储系统整体性能的同时，设置的预留IO不影响用户使用；通过利用当前系统空闲IO的属性确定并发迁移的迁移个数，自适应的调整并发迁移的迁移个数，提升了文件迁移速度，根据迁移个数向客户端发送目标迁移文件的迁移任务，在此过程中采用了异步数据存储的方式，避免了相关技术通过同步数据存储的方式造成的迁移效率低、迁移周期长的缺点，提高数据迁移效率。
针对S103，本实施例进行进一步阐述，请参考图4，图4为本申请实施例所提供的一种迁移任务下发的流程图，包括：
S1031、判断当前正在执行的所有的待迁移文件的数量是否小于迁移个数。
S1032、若小于迁移个数，则向客户端发送目标迁移文件的迁移任务。
S1033、若大于迁移个数，则进行等待，直至所有的待迁移文件的数量小于迁移个数，并向客户端发送目标迁移文件的迁移任务。
可以理解的是，每次向客户端发送目标迁移文件的迁移任务前，都要根据集群系统存储性能的当前系统空闲IO动态调整并发数即迁移个数，判断当前正在进行的数据迁移文件数是否小于迁移个数，大于迁移个数进行等待文件迁移完成消息，小于迁移个数则向迁移客户端发送迁移任务，然后继续处理下一文件。
进一步的，步骤S1033具体可以包括：当完成一个待迁移文件时，根据当前系统空闲IO的属性确定并发迁移的新的迁移个数；若所有的待迁移文件的数量小于新的迁移个数，则向客户端发送目标迁移文件的迁移任务。
可见，文件迁移速度根据系统性能剩余量即当前系统空闲IO自适应当前存储性能，利用迁移个数动态调整方式，控制迁移速度。
请参考图5，图5为本申请实施例提供的一种具体的数据迁移方法，包 括：
扫描任务定时进行。
判断待处理文件是否处理完毕。即判断待迁移文件是否完成并发迁移。
如果处理完毕，判断本次处理的文件数据即待处理文件是否全部迁移完毕，迁移完毕结束本次扫描，否则进行等待，等本次扫描文件数据全部迁移完成后结束本次扫描。
文件未处理完毕时，每次发送迁移任务前，都要根据集群系统存储性能动态调整并发数即迁移个数，判断当前正在进行的数据迁移文件数即正在进行的待迁移文件数量是否小于阈值即迁移个数，大于阈值进行等待文件迁移完成消息，小于阈值则向客户端发送迁移任务，然后继续处理下一文件。
下面对本申请实施例提供的一种数据迁移装置进行介绍，下文描述的数据迁移装置与上文描述的数据迁移方法可相互对应参照，参考图6，图6为本申请实施例所提供的一种数据迁移装置的结构示意图，包括：
目标迁移文件获取模块610，用于获取目标迁移文件；
迁移个数调整模块620，用于根据当前系统空闲IO的属性与预设的每个文件迁移任务需要的迁移IO,确定并发迁移的迁移个数；
迁移任务下发模块630，用于根据迁移个数向客户端发送目标迁移文件的迁移任务，以便客户端根据迁移任务对目标迁移文件进行并发迁移；
其中，当前系统空闲IO为系统最大IO中去除用户实际占用IO和预留IO后的IO。
在一些具体的实施例中，迁移个数调整模块620，包括：
第一迁移个数确定单元，用于根据系统空闲iops与预设的每个文件迁移任务需要的迁移iops，确定第一迁移个数；
第二迁移个数确定单元，用于根据系统剩余聚合带宽与预设的每个文件迁移任务需要的迁移聚合带宽，确定第二迁移个数；
迁移个数确定单元，用于确定第一迁移个数与第二迁移个数中最小的数值作为迁移个数。
在一些具体的实施例中，迁移任务下发模块630，包括：
判断单元，用于判断当前正在执行的所有的待迁移文件的数量是否小于迁移个数；
第一迁移任务下发单元，用于若小于迁移个数，则向客户端发送目标迁移文件的迁移任务；
第二迁移任务下发单元，用于若大于迁移个数，则进行等待，直至所有的待迁移文件的数量小于迁移个数，并向客户端发送目标迁移文件的迁移任务。
在一些具体的实施例中，第二迁移任务下发单元包括：
新的迁移个数确定子单元，用于当完成一个待迁移文件时，根据当前系统空闲IO的属性确定并发迁移的新的迁移个数；
第二迁移任务下发子单元，用于若所有的待迁移文件的数量小于新的迁移个数，则向客户端发送目标迁移文件的迁移任务。
在一些具体的实施例中，还包括：
扫描模块，用于定时扫描待迁移文件；
判断模块，用于判断待迁移文件是否完成并发迁移；
停止扫描模块，用于若完成，则停止扫描，其中，待迁移文件包括目标迁移文件。
在一些具体的实施例中，还包括：
记录模块，用于当目标迁移文件并发迁移完成后，记录目标迁移文件的处理信息。
在一些具体的实施例中，预留IO是系统最大IO的20％。
由于数据迁移装置部分的实施例与数据迁移方法部分的实施例相互对应，因此数据迁移装置部分的实施例请参见数据迁移方法部分的实施例的描述，这里暂不赘述。
下面对本申请实施例提供的一种电子设备进行介绍，下文描述的电子设备与上文描述的数据迁移方法可相互对应参照。参见图7，图7为本申请实施例提供的一种电子设备的结构图，包括：
存储器100，用于存储计算机程序；
处理器200，用于执行计算机程序时可以实现上述实施例所提供的步 骤。
具体的，存储器100包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统和计算机可读指令，该内存储器为非易失性存储介质中的操作系统和计算机可读指令的运行提供环境。处理器200为电子设备提供计算和控制能力，执行存储器100中保存的计算机程序时，可以实现以下步骤：获取目标迁移文件；根据当前系统空闲IO的属性与预设的每个文件迁移任务需要的迁移IO，确定并发迁移的迁移个数；根据迁移个数向客户端发送目标迁移文件的迁移任务，以便客户端根据迁移任务对目标迁移文件进行并发迁移；其中，当前系统空闲IO为系统最大IO中去除用户实际占用IO和预留IO后的IO。
在一些实施例中，处理器200执行存储器100中保存的计算机子程序时，可以实现以下步骤：根据系统空闲iops与预设的每个文件迁移任务需要的迁移iops，确定第一迁移个数；根据系统剩余聚合带宽与预设的每个文件迁移任务需要的迁移聚合带宽，确定第二迁移个数；确定第一迁移个数与第二迁移个数中最小的数值作为迁移个数。
在一些实施例中，处理器200执行存储器100中保存的计算机子程序时，可以实现以下步骤：判断当前正在执行的所有的待迁移文件的数量是否小于迁移个数；若小于迁移个数，则向客户端发送目标迁移文件的迁移任务；若大于迁移个数，则进行等待，直至所有的待迁移文件的数量小于迁移个数，并向客户端发送目标迁移文件的迁移任务。
在一些实施例中，处理器200执行存储器100中保存的计算机子程序时，可以实现以下步骤：当完成一个待迁移文件时，根据当前系统空闲IO的属性确定并发迁移的新的迁移个数；若所有的待迁移文件的数量小于新的迁移个数，则向客户端发送目标迁移文件的迁移任务。
在一些实施例中，处理器200执行存储器100中保存的计算机子程序时，可以实现以下步骤：定时扫描待迁移文件；判断待迁移文件是否完成并发迁移；若完成，则停止扫描，其中，待迁移文件包括目标迁移文件。
由于电子设备部分的实施例与数据迁移方法部分的实施例相互对应，因此电子设备部分的实施例请参见数据迁移方法部分的实施例的描述，这里暂不赘述。
在上述实施例的基础上，作为优选实施方式，参见图8，图8为本申请实施例提供的另一种电子设备的结构图，该电子设备还包括：
输入接口300，与处理器200相连，用于获取外部导入的计算机程序、参数和指令，经处理器200控制保存至存储器100中。该输入接口300可以与输入装置相连，接收用户手动输入的参数或指令。该输入装置可以是显示屏上覆盖的触摸层，也可以是终端外壳上设置的按键、轨迹球或触控板，也可以是键盘、触控板或鼠标等。
显示单元400，与处理器200相连，用于显示处理器200发送的数据。该显示单元400可以为PC机上的显示屏、液晶显示屏或者电子墨水显示屏等。
网络端口500，与处理器200相连，用于与外部各终端设备进行通信连接。该通信连接所采用的通信技术可以为有线通信技术或无线通信技术，如移动高清链接技术(MHL)、通用串行总线(USB)、高清多媒体接口(HDMI)、无线保真技术(WiFi)、蓝牙通信技术、低功耗蓝牙通信技术、基于IEEE802.11s的通信技术等。
下面对本申请实施例提供的一种计算机可读存储介质进行介绍，下文描述的计算机可读存储介质与上文描述的方法可相互对应参照。
本实施例提供一种计算机可读存储介质，计算机可读存储介质上存储有计算机程序，计算机程序被处理器执行时实现如上述数据迁移方法的步骤。
该存储介质可以包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。该存储介质上存储有计算机程序，计算机程序被处理器执行时实现以下步骤：获取目标迁移文件；根据当前系统空闲IO的属性与预设的每个文件迁移任务需要的迁移IO,确定并发迁移的迁移个数；根据迁移个数向客户端发送目标迁移文件的迁移任务，以便客户端根据迁移任务对目标迁移文件进行并发迁移；其中，当前系统空闲IO为系统最大IO中去除用户实际占用IO和预留IO后的IO。
在一些实施例中，计算机可读存储介质中存储的计算机子程序被处理器执行时，可以实现以下步骤：根据系统空闲iops与预设的每个文件迁移任务需要的迁移iops，确定第一迁移个数；根据系统剩余聚合带宽与预设的每个文件迁移任务需要的迁移聚合带宽，确定第二迁移个数；确定第一迁移个数与第二迁移个数中最小的数值作为迁移个数。
在一些实施例中，计算机可读存储介质中存储的计算机子程序被处理器执行时，可以实现以下步骤：判断当前正在执行的所有的待迁移文件的数量是否小于迁移个数；若小于迁移个数，则向客户端发送目标迁移文件的迁移任务；若大于迁移个数，则进行等待，直至所有的待迁移文件的数量小于迁移个数，并向客户端发送目标迁移文件的迁移任务。
在一些实施例中，计算机可读存储介质中存储的计算机子程序被处理器执行时，可以实现以下步骤：当完成一个待迁移文件时，根据当前系统空闲IO的属性确定并发迁移的新的迁移个数；若所有的待迁移文件的数量小于新的迁移个数，则向客户端发送目标迁移文件的迁移任务。
在一些实施例中，计算机可读存储介质中存储的计算机子程序被处理器执行时，可以实现以下步骤：定时扫描待迁移文件；判断待迁移文件是否完成并发迁移；若完成，则停止扫描，其中，待迁移文件包括目标迁移文件。
由于计算机可读存储介质部分的实施例与方法部分的实施例相互对应，因此计算机可读存储介质部分的实施例请参见方法部分的实施例的描述，这里暂不赘述。
说明书中各个实施例采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似部分互相参见即可。对于实施例公开的装置而言，由于其与实施例公开的方法相对应，所以描述的比较简单，相关之处参见方法部分说明即可。
专业人员还可以进一步意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方 式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
结合本文中所公开的实施例描述的方法或算法的步骤可以直接用硬件、处理器执行的软件模块，或者二者的结合来实施。软件模块可以置于随机存储器(RAM)、内存、只读存储器(ROM)、电可编程ROM、电可擦除可编程ROM、寄存器、硬盘、可移动磁盘、CD-ROM、或技术领域内所公知的任意其它形式的存储介质中。
以上对本申请所提供的一种数据迁移方法、数据迁移装置、电子设备及计算机可读存储介质进行了详细介绍。本文中应用了具体个例对本申请的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本申请的方法及其核心思想。应当指出，对于本技术领域的普通技术人员来说，在不脱离本申请原理的前提下，还可以对本申请进行若干改进和修饰，这些改进和修饰也落入本申请权利要求的保护范围内。

Claims (10)

  1. 一种数据迁移方法，其特征在于，包括：
    获取目标迁移文件；
    根据当前系统空闲IO的属性与预设的每个文件迁移任务需要的迁移IO，确定并发迁移的迁移个数；
    根据所述迁移个数向客户端发送所述目标迁移文件的迁移任务，以便所述客户端根据所述迁移任务对所述目标迁移文件进行并发迁移；
    其中，所述当前系统空闲IO为系统最大IO中去除用户实际占用IO和预留IO后的IO。
  2. 根据权利要求1所述的数据迁移方法，其特征在于，所述根据当前系统空闲IO的属性与预设的每个文件迁移任务需要的迁移IO,确定并发迁移的迁移个数，包括：
    根据系统空闲iops与预设的每个文件迁移任务需要的迁移iops，确定第一迁移个数；
    根据系统剩余聚合带宽与预设的每个文件迁移任务需要的迁移聚合带宽，确定第二迁移个数；
    确定所述第一迁移个数与所述第二迁移个数中最小的数值作为所述迁移个数。
  3. 根据权利要求1所述的数据迁移方法，其特征在于，所述根据所述迁移个数向客户端发送所述目标迁移文件的迁移任务，包括：
    判断当前正在执行的所有的待迁移文件的数量是否小于所述迁移个数；
    若小于所述迁移个数，则向所述客户端发送所述目标迁移文件的所述迁移任务；
    若大于所述迁移个数，则进行等待，直至所有的所述待迁移文件的数量小于所述迁移个数，并向所述客户端发送所述目标迁移文件的所述迁移任务。
  4. 根据权利要求3所述的数据迁移方法，其特征在于，所述若大于所述迁移个数，则进行等待，直至所有的所述待迁移文件的数量小于所述迁移个数，并向所述客户端发送所述目标迁移文件的所述迁移任务，包括：
    当完成一个待迁移文件时，根据当前系统空闲IO的属性确定并发迁移的新的迁移个数；
    若所有的所述待迁移文件的数量小于所述新的迁移个数，则向所述客户端发送所述目标迁移文件的所述迁移任务。
  5. 根据权利要求1所述的数据迁移方法，其特征在于，还包括：
    定时扫描待迁移文件；
    对应的，所述根据所述迁移个数向客户端发送所述目标迁移文件的迁移任务，以便所述客户端根据所述迁移任务对所述目标迁移文件进行并发迁移之后，还包括：
    判断所述待迁移文件是否完成并发迁移；
    若完成，则停止扫描，其中，所述待迁移文件包括目标迁移文件。
  6. 根据权利要求1所述的数据迁移方法，其特征在于，所述根据所述迁移个数向客户端发送所述目标迁移文件的迁移任务，以便所述客户端根据所述迁移任务对所述目标迁移文件进行并发迁移之后，还包括：
    当所述目标迁移文件并发迁移完成后，记录所述目标迁移文件的处理信息。
  7. 根据权利要求1所述的数据迁移方法，其特征在于，所述预留IO是系统最大IO的20％。
  8. 一种数据迁移装置，其特征在于，包括：
    目标迁移文件获取模块，用于获取目标迁移文件；
    迁移个数调整模块，用于根据当前系统空闲IO的属性与预设的每个文件迁移任务需要的迁移IO,确定并发迁移的迁移个数；
    迁移任务下发模块，用于根据所述迁移个数向客户端发送所述目标迁移文件的迁移任务，以便所述客户端根据所述迁移任务对所述目标迁移文件进行并发迁移；
    其中，所述当前系统空闲IO为系统最大IO中去除用户实际占用IO和预留IO后的IO。
  9. 一种电子设备，其特征在于，包括：
    存储器，用于存储计算机程序；
    处理器，用于执行所述计算机程序时实现如权利要求1至7任一项所 述数据迁移方法的步骤。
  10. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1至7任一项所述数据迁移方法的步骤。
PCT/CN2020/103695 2019-11-29 2020-07-23 一种数据迁移方法、装置和计算机可读存储介质 WO2021103596A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911204966.6 2019-11-29
CN201911204966.6A CN111142788B (zh) 2019-11-29 2019-11-29 一种数据迁移方法、装置和计算机可读存储介质

Publications (1)

Publication Number Publication Date
WO2021103596A1 true WO2021103596A1 (zh) 2021-06-03

Family

ID=70517371

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/103695 WO2021103596A1 (zh) 2019-11-29 2020-07-23 一种数据迁移方法、装置和计算机可读存储介质

Country Status (2)

Country Link
CN (1) CN111142788B (zh)
WO (1) WO2021103596A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115904263A (zh) * 2023-03-10 2023-04-04 浪潮电子信息产业股份有限公司 一种数据迁移方法、系统、设备及计算机可读存储介质
CN116980641A (zh) * 2023-09-22 2023-10-31 江西云眼视界科技股份有限公司 视频迁移的异步处理方法、系统、计算机及存储介质

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111142788B (zh) * 2019-11-29 2021-10-15 浪潮电子信息产业股份有限公司 一种数据迁移方法、装置和计算机可读存储介质
CN111966301B (zh) * 2020-08-25 2023-01-10 苏州浪潮智能科技有限公司 分布式对象存储系统的迁移速度控制方法、装置及介质
CN113741810B (zh) * 2021-07-30 2023-08-11 苏州浪潮智能科技有限公司 一种数据迁移方法及装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101105738A (zh) * 2006-07-13 2008-01-16 国际商业机器公司 用于并行存储池迁移和备份的装置、系统和方法
CN103530388A (zh) * 2013-10-22 2014-01-22 浪潮电子信息产业股份有限公司 一种云存储系统中提升性能的数据处理方法
US20160196089A1 (en) * 2015-01-07 2016-07-07 Netapp, Inc. System and method for adaptive data transfers with limited resources
CN105843820A (zh) * 2015-01-16 2016-08-10 阿里巴巴集团控股有限公司 数据迁移方法及装置
CN105978949A (zh) * 2016-04-27 2016-09-28 乐视控股（北京）有限公司 离线数据上传方法及装置
CN111142788A (zh) * 2019-11-29 2020-05-12 浪潮电子信息产业股份有限公司 一种数据迁移方法、装置和计算机可读存储介质

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103297481A (zh) * 2012-03-05 2013-09-11 中兴通讯股份有限公司 互联交换方法和装置
CN103136043B (zh) * 2013-01-28 2016-05-25 华为技术有限公司 一种异步io的迁移方法、系统和网络节点
EP3026546A4 (en) * 2013-08-09 2016-08-17 Huawei Tech Co Ltd FILE PROCESSING METHOD, APPARATUS, AND STORAGE DEVICE
CN103500073B (zh) * 2013-10-08 2016-05-18 浪潮(北京)电子信息产业有限公司 一种数据块迁移方法及装置
CN103677959B (zh) * 2013-12-16 2017-05-31 国家计算机网络与信息安全管理中心 一种基于组播的虚拟机集群迁移方法及系统
CN104331253B (zh) * 2014-10-30 2017-12-15 浪潮电子信息产业股份有限公司 一种对象存储系统中对象迁移的计算方法
CN105045873B (zh) * 2015-07-17 2018-02-09 深圳市腾讯计算机系统有限公司 一种数据文件的推送方法、装置及系统
CN107506500A (zh) * 2017-10-09 2017-12-22 郑州云海信息技术有限公司 一种数据迁移的控制方法及相关装置
CN107844590B (zh) * 2017-11-17 2020-05-29 郑州云海信息技术有限公司 小文件迁移方法、装置、设备及计算机可读存储介质
CN108536539B (zh) * 2018-04-26 2021-12-17 东北大学 一种工业分布式数据采集系统中的任务调度方法
CN109343793B (zh) * 2018-09-11 2021-09-07 创新先进技术有限公司 数据迁移方法及装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101105738A (zh) * 2006-07-13 2008-01-16 国际商业机器公司 用于并行存储池迁移和备份的装置、系统和方法
CN103530388A (zh) * 2013-10-22 2014-01-22 浪潮电子信息产业股份有限公司 一种云存储系统中提升性能的数据处理方法
US20160196089A1 (en) * 2015-01-07 2016-07-07 Netapp, Inc. System and method for adaptive data transfers with limited resources
CN105843820A (zh) * 2015-01-16 2016-08-10 阿里巴巴集团控股有限公司 数据迁移方法及装置
CN105978949A (zh) * 2016-04-27 2016-09-28 乐视控股（北京）有限公司 离线数据上传方法及装置
CN111142788A (zh) * 2019-11-29 2020-05-12 浪潮电子信息产业股份有限公司 一种数据迁移方法、装置和计算机可读存储介质

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115904263A (zh) * 2023-03-10 2023-04-04 浪潮电子信息产业股份有限公司 一种数据迁移方法、系统、设备及计算机可读存储介质
CN116980641A (zh) * 2023-09-22 2023-10-31 江西云眼视界科技股份有限公司 视频迁移的异步处理方法、系统、计算机及存储介质
CN116980641B (zh) * 2023-09-22 2023-12-15 江西云眼视界科技股份有限公司 视频迁移的异步处理方法、系统、计算机及存储介质

Also Published As

Publication number Publication date
CN111142788B (zh) 2021-10-15
CN111142788A (zh) 2020-05-12

Similar Documents

Publication Publication Date Title
WO2021103596A1 (zh) 一种数据迁移方法、装置和计算机可读存储介质
WO2022062537A1 (zh) 数据压缩方法、装置及计算机可读存储介质
JP6961844B2 (ja) ストレージボリューム作成方法および装置、サーバ、並びに記憶媒体
WO2019169913A1 (zh) 一种数据处理的方法、装置、服务器和系统
US9110669B2 (en) Power management of a storage device including multiple processing cores
WO2023082560A1 (zh) 一种任务处理方法、装置、设备及介质
WO2018049899A1 (zh) 一种队列管理方法及装置
US20110141123A1 (en) Push Pull Adaptive Capture
JP5923627B2 (ja) 仮想プラットフォーム上でｉ／ｏチャネルを調整する方法及び装置
WO2021073546A1 (zh) 数据访问方法、装置和第一计算设备
US20070116049A1 (en) Dynamic power management for I/O resource pools
WO2021093365A1 (zh) 一种gpu显存管理控制方法及相关装置
WO2021169618A1 (zh) 一种虚拟机的迁移方法、装置、设备及介质
WO2015013981A1 (zh) 一种图像显示方法和装置
WO2016008338A1 (zh) 一种i/o请求处理方法及存储系统
WO2024055840A1 (zh) 图像渲染方法、装置、设备和介质
JP2012221340A (ja) 制御方法及びプログラム、並びにコンピュータ
US20230076468A1 (en) Per-lane power management of bus interconnects
CN110795323A (zh) 负载统计方法、装置、存储介质及电子设备
WO2024152609A1 (zh) 一种消息处理方法、装置、电子设备和存储介质
WO2024027140A1 (zh) 一种数据处理方法、装置、设备、系统及可读存储介质
WO2021120843A1 (zh) 云主机内存分配方法及云主机、设备及存储介质
CN117311628A (zh) 一种固态硬盘的命令处理方法、装置、设备及介质
CN112433848A (zh) 资源处理方法、装置、存储介质和缓存服务器
WO2023193527A1 (zh) 线程执行方法、装置、电子设备及计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20893708

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20893708

Country of ref document: EP

Kind code of ref document: A1