WO2019095576A1 - 数据迁移方法、装置、计算机设备和存储介质 - Google Patents

数据迁移方法、装置、计算机设备和存储介质 Download PDF

Info

Publication number
WO2019095576A1
WO2019095576A1 PCT/CN2018/077106 CN2018077106W WO2019095576A1 WO 2019095576 A1 WO2019095576 A1 WO 2019095576A1 CN 2018077106 W CN2018077106 W CN 2018077106W WO 2019095576 A1 WO2019095576 A1 WO 2019095576A1
Authority
WO
WIPO (PCT)
Prior art keywords
migration
task
current
stopped
migration object
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/077106
Other languages
English (en)
French (fr)
Inventor
纪璇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Publication of WO2019095576A1 publication Critical patent/WO2019095576A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/48Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806Task transfer initiation or dispatching
    • G06F9/4843Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/485Task life-cycle, e.g. stopping, restarting, resuming execution
    • G06F9/4856Task life-cycle, e.g. stopping, restarting, resuming execution resumption being on a different machine, e.g. task migration, virtual machine migration

Definitions

  • the application relates to a data migration method, apparatus, computer device and storage medium.
  • NAS Network Attached Storage
  • Migrate image files on NAS volumes to cloud storage for storage.
  • an abnormal situation such as a network disconnection or a device abnormality.
  • the inventor realizes that when an abnormality occurs, if the set migration process is not completed, the data needs to be re-migrated from the source storage device to the target storage device, and the migration workload is large and the migration efficiency is low.
  • a data migration method, apparatus, computer device, and storage medium are provided.
  • a data migration method including:
  • the current task state and the current migration state corresponding to the migration object when the migration is stopped are obtained;
  • the migration object is migrated according to the current task state corresponding to the migration object and the current migration state when the migration is stopped.
  • a data migration device comprising:
  • the migration object acquisition module is configured to obtain a migration object.
  • a task execution module configured to perform a corresponding migration task on the migration object according to preset task execution logic, to migrate the migration object, where the task execution logic includes execution logic between multiple tasks;
  • a status recording module configured to record a current task status and a current migration status corresponding to the migration object during the migration process
  • a state obtaining module configured to: when the migration is stopped during the execution of the data migration, acquire the current task state and the current migration state of the migration object when the migration is stopped, when the migration is restarted;
  • the migration module is configured to migrate the migration object according to the current task state and the current migration state of the migration object when the migration is stopped.
  • a computer device comprising a memory and one or more processors having computer readable instructions stored thereon, the computer readable instructions being executed by the processor and being executable by the processor.
  • One or more non-transitory computer readable storage mediums storing computer readable instructions, when executed by one or more processors, cause one or more processors to be implemented in any one embodiment of the present application The steps provided for the data migration method.
  • FIG. 1 is an implementation environment diagram of a data migration method provided in accordance with one or more embodiments.
  • FIG. 2 is a flow diagram of a method of data migration in accordance with one or more embodiments.
  • FIG. 3 is a flowchart of performing a corresponding task on a migration object according to preset task execution logic to migrate a migration object according to one or more embodiments.
  • FIG. 4 is a flowchart of performing a data collation task to match a migrated migration object with a pre-migration migration object according to one or more embodiments to obtain a matching result.
  • FIG. 5 is a flow chart of a data migration method in another embodiment.
  • FIG. 6 is a block diagram of a data migration device in accordance with one or more embodiments.
  • FIG. 7 is a block diagram of a task execution module in accordance with one or more embodiments.
  • FIG. 8 is a block diagram of a check execution unit in accordance with one or more embodiments.
  • FIG. 9 is a block diagram of a data migration unit in accordance with one or more embodiments.
  • Figure 10 is a block diagram of a computer device in accordance with one or more embodiments.
  • first, second and the like are used to describe various elements, unless otherwise specified, and are not limited by these terms unless specifically stated. These terms are only used to distinguish one element from another.
  • the first hash value may be referred to as a second hash value without departing from the scope of the present application, and similarly, the second hash value may be referred to as a first hash value.
  • the migration object before the migration refers to the migration object in the source storage device
  • the migrated migration object refers to the migration object stored on the target storage device after being migrated from the source storage device to the target storage device.
  • FIG. 1 is an implementation environment diagram of a data migration method provided in an embodiment. As shown in FIG. 1 , a source storage device 110 , a computer device 120 , and a target storage device 130 are included in the implementation environment.
  • the source storage device storage 110 stores a migration object that needs to be migrated.
  • the computer device 120 receives the migration instruction, acquires the migration object on the source storage device 110, and then migrates the migration object to the target storage device 130.
  • the computer device 120 may be a separate physical server or terminal, or may be a server cluster composed of multiple physical servers, and may be a cloud server that provides basic cloud computing services such as a cloud server, a cloud database, a cloud storage, and a CDN. No restrictions.
  • a data migration method is provided.
  • the data migration method may be applied to the computer device 120, and may specifically include the following steps:
  • Step S202 obtaining a migration object.
  • the migration object refers to data to be migrated, and the migration object may be data such as an image file, a video file, an audio file, and a text file.
  • the migration object may be data such as an image file, a video file, an audio file, and a text file.
  • the migration object can be obtained from the corresponding source storage device.
  • Step S204 Perform a corresponding migration task on the migration object according to the preset task execution logic to migrate the migration object, where the task execution logic includes execution logic between the multiple tasks.
  • the task execution logic refers to the relationship between the task and the task, for example, the execution order between the task and the task, the execution of the specific task, the execution result of the other task as the execution condition, the execution parameter of the specific task, the execution result of the other task, etc. .
  • the B task is performed, and if the C task fails to execute, the D task is executed.
  • the migration divide the migration process into multiple tasks and set the task execution logic between those tasks.
  • data migration can be divided into replication tasks, verification tasks, marking tasks, and repair tasks.
  • the replication task is used to copy the migration object from the source storage device to the target storage device.
  • the verification task is used to match the migrated migration object with the migration object before the migration to obtain a matching result. For example, the matching result is successful or the matching fails.
  • the marking task is used to mark the migration result of the migration object on the source storage device according to the matching result. When the matching is successful, the label migration result is successful for migration, and when the matching fails, the label migration result is migration failure.
  • the repair task is used to re-migrate the migrated object from the source storage device if the match fails.
  • step S206 during the migration process, the current task state corresponding to the migration object and the current migration state are recorded.
  • the current task status records the tasks currently in the migration process, such as downloading tasks or marking tasks.
  • the migration status refers to the execution status of the migration object, and may include, for example, executing, waiting for execution, pausing, and error.
  • the migration status may be adjusted as the task is executed. As the task progresses, the state of data migration may also change. Since migration generally has multiple migration objects, in order to improve the efficiency of migration, each task can open multiple processes to perform parallel migration of migrated objects. Therefore, after a task is completed, if the number of processes of the next task is insufficient, the migration state corresponding to the migration object is adjusted to wait for execution. For example, when the replication task is completed for a migration object, if the number of verification task processes is insufficient, the migration object of the migration object is waiting.
  • the number of processes corresponding to each task may be the same or different, and may be set according to the execution time of the task, and the length of the execution time may be positively correlated with the number of processes. For example, if the copy task takes a long time and the check task time is short, the process of copying the task is more than the process of checking the task.
  • the number of processes may be adjusted according to the traffic volume of the source storage device.
  • the migration status may be adjusted according to the migration adjustment instruction, and the migration stop instruction may be sent, and the migration instruction carrying the migration target of the migration object corresponding to the identifier of the migration object is not included in the migration instruction. Adjust to pause.
  • step S208 if the migration is stopped during the data migration, the current task state and the current migration state corresponding to the migration object when the migration is stopped are acquired when the migration is restarted.
  • the migration when the migration is stopped, the moment the migration is stopped, the migration may be stopped due to an abnormality, or may be caused by receiving an instruction to stop the migration.
  • the migration When re-migrating, obtain the current task status and current migration status of the migration object when the migration is stopped.
  • Step S210 The migration object is migrated according to the current task state corresponding to the migration object when the migration is stopped and the current migration state.
  • the migration object is migrated in combination with the current task state and the current migration state of the migration object when the migration is stopped.
  • the migration status may include executing, waiting for execution, pausing, and errors, and the like.
  • the migration task of the migration object is being executed.
  • the task that the migration object is in is not executing. If the task corresponding to the previous migration object is completed, the migration task will continue.
  • Pause means that the migration object needs to suspend the migration, and then resume the migration after the suspension condition is lifted.
  • the error indicates that the data matching with the migrated migration object before the migration fails or the migration of the migration object to the target storage device fails.
  • the repair task is required to re-migrate the data.
  • the task corresponding to the current task status corresponding to the migration object when the migration is stopped is executed; when the current migration status corresponding to the migration object when the migration is stopped is paused, If the suspension condition is not released when the migration is restarted, the suspended migration state is maintained, and the migration object is not migrated. If the suspension condition is released when the migration is restarted, the task corresponding to the current task state corresponding to the migration target is stopped when the migration is stopped. When the current migration status of the migration object is incorrect when the migration is stopped, perform the repair task to re-migrate the migration object from the source storage device.
  • the migrated task execution logic sequence is task A-task B-task C. If the task status when the migration stops abnormally is B and the migration status is waiting, the task B is executed for the migration object when the migration is restarted. If the exception occurs, the task status when stopping the migration is B, the migration status is paused, and the migration status is paused when the migration is stopped because the suspension condition is not migrated before July 18, if the migration is restarted at 7 On the 17th of the month, the migration object will not be migrated. If the migration is restarted on July 19th, task B is executed on the migrated object.
  • the migration object can be migrated to one or more target storage devices.
  • the migration target is migrated to the primary cluster and the disaster recovery cluster at the same time, which further improves the migration efficiency and enables the disaster recovery cluster to respond in time when the primary cluster sends a fault.
  • the migrated data can be stored in the form of object storage.
  • the migration log information is recorded during the migration process to make the entire migration process more transparent.
  • the migration object when the data migration is performed, the migration object is obtained, and the corresponding migration task is performed on the migration object according to the preset task execution logic to migrate the migration object, and during the migration process, the migration object is recorded. If the current task status and the current migration status are stopped during the data migration process, the migration object is migrated according to the current task status and current migration status of the migration object when the migration is stopped. Because the migration is divided into multiple tasks, and the task execution logic is set, and the current task status and the current migration status corresponding to the migration object are recorded during the migration process, after the migration is restarted, the migration object can be corresponding according to the migration stoppage. The current task status and the current migration status are migrated, which reduces the migration workload and improves the efficiency of the migration.
  • step S204 performs a corresponding task on the migration object according to the preset task execution logic, and the migration of the migration object may include the following steps:
  • the copying refers to sending the migration object stored on the source storage device to the target storage device, and also retaining the data on the source storage device.
  • the source storage device can be the same or different from the target storage device.
  • the migration object can be migrated from one partition of the device to another, or from one virtual machine to another.
  • the migration object and the migrated migration object may be checked according to the size and name of the data, and the content of the migration object. It is also possible to perform a hash operation based on the migration object before migration and the migration object attribute information after migration, and match the calculated hash values.
  • Hash calculation is a calculation method for transforming an input of arbitrary length into a certain length by a hash algorithm.
  • the hash algorithm can be MD5 (Message-digest Algorithm 5) algorithm and SHA (Secure Hash Standard, security Greek standard) algorithm, etc., specifically no restrictions.
  • the migration results include migration failure and migration success. If the result of the match is successful, the migration result is marked as successful on the source storage device. If the match result is a match failure, the migration result is marked as migration failure on the source storage device.
  • the migration object in the source storage device is re-migrated to the target storage device.
  • the download task may be performed, and the migration object is downloaded to the target storage device. If the download fails, the migration is unsuccessful and needs to be repaired. The task is re-migrated. If the download succeeds and the match is successful, the tag task is executed, and the migration result of the migration object on the source storage device is marked as successful.
  • the migration result on the source storage device may be obtained. If the migration result of the migration object on the source storage device is successful, the migration is performed on the target storage device. data.
  • FIG. 4 shows, in some embodiments, a step S304 of performing a data collation task to match a migrated migration object with a migration object before migration to obtain a matching result.
  • the method may include the following steps:
  • Step S402 performing hash calculation according to preset attribute information of the migration object before migration, to obtain a first hash value.
  • Step S404 performing hash calculation according to the preset attribute information of the migrated migration object, to obtain a second hash value.
  • step S406 the first hash value and the second hash value are matched. If they match, the matching result is successful. If not, the matching result is a failure.
  • the attribute information refers to the nature of the migration object, and may include, for example, size information of the migration object, a name, and the like. It can be understood that, when the hash calculation is performed, the type of the preset attribute information of the migration object before the migration and the migrated migration object are the same. For example, if the size information of the migration object before migration is used, the first hash value is calculated. , you must also use the size information of the migrated migration object when calculating the second hash value.
  • Hash calculation is a calculation method for transforming an input of arbitrary length into a certain length by a hash algorithm.
  • the hash algorithm can be MD5 (Message-digestAlgorithm5) and SHA (Secure Hash Standard, Secure Hash). Standard) algorithms, etc., are not limited.
  • the migration information of the migration object before migration and the migration object after migration may also be checked to further confirm whether the migration object has been correctly migrated to the target storage device.
  • the data migration method may further include the following steps:
  • Step S502 Receive a migration instruction, where the migration instruction carries a migration policy, where the migration policy includes a migration object identifier and a migration condition.
  • the migration object identifier may be a specific object identifier, or may be a character that can be used to identify the migration object, such as a data type or a data storage location.
  • Migration conditions refer to the circumstances under which the migration will occur.
  • the migration condition may include, for example, one or more of the time of migration, the priority of the migration object, or the data size requirement of the migration object.
  • the migration time is either to perform the migration immediately or to perform the migration at the preset time.
  • the priority of the migrated object is used to indicate which data is preferred for migration. For example, for the insurance industry, life insurance data is more important than property insurance, then life insurance data priority can be set higher than production insurance data priority.
  • the migration condition may also include a target storage address corresponding to the data type.
  • the target storage address corresponding to the life insurance data is the address of the A storage device
  • the target storage device corresponding to the production and insurance data is the address of the B storage device.
  • Step S504 Acquire attribute information of the migration object according to the migration target data identifier, and generate an index information sequence according to the attribute information and the migration condition.
  • the attribute information of the acquired migration object may be determined according to the migration condition. For example, if the migration condition includes which data types are preferentially migrated, the attribute information of the migration object includes the data type. If the migration condition includes a migration object with a small amount of data migration, the attribute information of the acquired migration object includes the size of the migration object. .
  • the index information sequence is generated according to the attribute information and the migration condition. The index information in the index information sequence is sorted according to the attribute information of the migration object and the migration condition. For example, when the migration condition is priority migration life insurance, the data of life insurance is ranked first.
  • the index information is arranged according to the amount of data of the migration object from large to small.
  • the index information sequence is obtained, the migration object is obtained according to the index information sequence. For example, the migration object is obtained according to the sorting of the index information.
  • the migration condition when the migration condition further includes a target storage address corresponding to the data type, the acquired attribute information includes a data type, and generates an index information sequence according to the target storage address corresponding to the data type, where the index information sequence includes each The target storage address corresponding to the migration object. Therefore, when the migration is performed, the migration object is migrated to the corresponding target storage device according to the target storage address corresponding to the migration target.
  • a data migration device is provided.
  • the data migration device can be integrated into the computer device 120, including:
  • the migration object acquisition module 602 is configured to acquire a migration object.
  • the task execution module 604 is configured to perform a corresponding migration task on the migration object according to the preset task execution logic to migrate the migration object, where the task execution logic includes execution logic between the multiple tasks.
  • the status record module 606 is configured to record the current task status and the current migration status corresponding to the migration object during the migration process.
  • the state obtaining module 608 is configured to: when the migration is stopped during the data migration, obtain the current task state and the current migration state of the migration object when the migration is stopped.
  • the migration module 610 is configured to migrate the migration object according to the current task state corresponding to the migration object when the migration is stopped and the current migration state.
  • the current migration status includes executing, waiting for execution, suspending, and error
  • the migration module is configured to: when the migration is stopped or the current migration status corresponding to the migration object is being executed or waiting to be executed, the migration object is executed when the migration is stopped.
  • the task corresponding to the current task status when the current migration status of the migration object is incorrect when the migration is stopped, the migration object is re-migrated from the source storage device; when the migration is stopped, the current migration status corresponding to the migration object is paused.
  • the suspension condition is released, and the task corresponding to the current task state corresponding to the migration object when the migration is stopped is executed.
  • the task execution module 604 includes:
  • the copy execution unit 702 is configured to perform a copy task to copy the migration object from the source storage device to the target storage device.
  • the verification execution unit 704 is configured to perform a data matching task to match the migrated migration object with the migration target before migration to obtain a matching result.
  • the tag execution unit 706 is configured to perform a tag task according to the matching result, and mark the migration result of the migration object on the source storage device.
  • the repair execution unit 708 is configured to perform a repair task to re-migrate the migration object from the source storage device if the matching fails.
  • the collation execution unit 704 includes:
  • the first calculating sub-unit 802 is configured to perform hash calculation according to preset attribute information of the migration object before migration to obtain a first hash value.
  • the second calculation sub-unit 804 performs hash calculation according to the preset attribute information of the migrated migration object to obtain a second hash value.
  • the matching sub-unit 806 is configured to match the first hash value and the second hash value. If they match, the matching result is successful. If not, the matching result is a failure.
  • the data migration apparatus may further include:
  • the migration instruction receiving module 902 is configured to receive a migration instruction, where the migration instruction carries a migration policy, and the migration policy includes a migration object identifier and a migration condition.
  • the index information generating module 904 is configured to obtain attribute information of the migration object according to the migration object data identifier, and generate an index information sequence according to the attribute information and the migration condition.
  • the migration object obtaining module 602 is specifically configured to acquire the migration object according to the sequence of index information.
  • Each of the above data migration devices may be implemented in whole or in part by software, hardware, and combinations thereof.
  • Each of the above modules may be embedded in or independent of the processor in the computer device, or may be stored in a memory in the computer device in a software form, so that the processor invokes the operations corresponding to the above modules.
  • FIG. 10 it is an internal structural diagram of a computer device in an embodiment, that is, the computer device may be a server or a terminal, and the computer device is connected to a processor, a non-volatile storage medium, and a system through a system connection bus. Memory and network interface.
  • the non-volatile storage medium can be a non-transitory computer-readable storage medium
  • the non-volatile storage medium of the computer device can store an operating system and computer readable instructions, which when executed, can cause the processor Perform a data migration method.
  • the processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device.
  • the internal memory can store computer readable instructions that, when executed by the processor, cause the processor to perform a data migration method.
  • the network interface of the computer device is used for network communication, such as receiving a migration instruction, sending a stop control instruction, and the like.
  • FIG. 10 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation of the computer device to which the solution of the present application is applied.
  • the specific computer device may It includes more or fewer components than those shown in the figures, or some components are combined, or have different component arrangements.
  • the data migration apparatus can be implemented in the form of a computer readable instruction that can be run on a computer device as shown in FIG. 10, non-volatile storage of the computer device
  • the medium may store various program modules constituting the data migration device, such as the migration object acquisition module 602, the task execution module 604, the state recording module 606, the state acquisition module 608, and the migration module 610 in FIG.
  • Computer program readable instructions are included in each program module for causing a computer device to perform the steps in the data migration method of various embodiments of the present application described in this specification.
  • the computer device may acquire the migration object by using the migration object acquisition module 602 in the data migration device as shown in FIG.
  • the status record module 606 records the current task status and the current migration status corresponding to the migration object during the migration process. If the migration is stopped during the data migration, the current task status and the current migration status of the migration object when the migration is stopped are obtained when the migration is restarted. The current task status and the current migration status of the migration object are stopped by the migration module 610 according to the migration. Migrate the migrated objects.
  • a computer device comprising a memory and one or more processors, the memory storing computer readable instructions, the computer readable instructions being executed by the processor, causing the one or more processors to perform the step of: acquiring a migration object;
  • the preset task execution logic performs a corresponding migration task on the migration object to migrate the migration object.
  • the task execution logic includes execution logic between multiple tasks.
  • the current task status corresponding to the migration object is recorded.
  • the current migration status if the migration is stopped during the data migration, when the migration is restarted, the current task status and the current migration status of the migration object when the migration is stopped are acquired; and, according to the current current of the migration object when the migration is stopped
  • the migration status is migrated by the task status and the current migration status.
  • One or more non-transitory computer readable storage mediums storing computer readable instructions, when executed by one or more processors, cause one or more processors to perform the steps of: acquiring a migration object; Performing a corresponding migration task on the migration object according to the preset task execution logic to migrate the migration object, the task execution logic includes execution logic between multiple tasks; during the migration process, recording the current task corresponding to the migration object Status and current migration status; if the migration is stopped during the data migration process, the current task status and current migration status of the migration object when the migration is stopped are obtained when the migration is restarted; and, according to the migration object corresponding to the migration stop The current task status and the current migration status migrate the migrated objects.
  • Non-volatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory can include random access memory (RAM) or external cache memory.
  • RAM is available in a variety of formats, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronization chain.
  • SRAM static RAM
  • DRAM dynamic RAM
  • SDRAM synchronous DRAM
  • DDRSDRAM double data rate SDRAM
  • ESDRAM enhanced SDRAM
  • Synchlink DRAM SLDRAM
  • Memory Bus Radbus
  • RDRAM Direct RAM
  • DRAM Direct Memory Bus Dynamic RAM
  • RDRAM Memory Bus Dynamic RAM

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Retry When Errors Occur (AREA)

Abstract

一种数据迁移方法,包括:获取迁移对象(S202);根据预设的任务执行逻辑对迁移对象执行对应的迁移任务,以对迁移对象进行迁移,任务执行逻辑包括多个任务之间的执行逻辑(S204);在进行迁移的过程中,记录迁移对象对应的当前任务状态以及当前迁移状态(S206);若在执行数据迁移的过程中停止迁移,则在重新开始迁移时,获取停止迁移前迁移对象对应的当前任务状态以及当前迁移状态(S208);根据停止迁移前迁移对象对应的当前任务状态以及当前迁移状态对迁移对象进行迁移(S210)。

Description

数据迁移方法、装置、计算机设备和存储介质
相关申请的交叉引用
本申请要求于2017年11月16日提交中国专利局,申请号为2017111405255,申请名称为“数据迁移方法、装置、计算机设备和存储介质”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
技术领域
本申请涉及数据迁移方法、装置、计算机设备和存储介质。
背景技术
随着科技的发展以及信息量的增大,在许多场景下都会存在对数据进行迁移的需求,例如当存储影像文件的NAS(NetworkAttached Storage,网络附属存储)卷过了标准保修服务期后,可以将NAS卷上的影像文件迁移到云存储中进行存储。将数据从源存储设备迁移到目标存储设备时,经常会出现停止迁移的情况,例如遭遇网络断开、设备异常等异常状况等。传统技术中,发明人意识到,当出现异常时,若设置的迁移过程未完成,需要将数据从源存储设备重新迁移到目标存储设备,迁移工作量大、迁移效率低。
发明内容
根据本申请的各种实施例,提供一种数据迁移方法、装置、计算机设备和存储介质。
一种数据迁移方法,包括:
获取迁移对象;
根据预设的任务执行逻辑对所述迁移对象执行对应的迁移任务,以对所述迁移对象进行迁移,所述任务执行逻辑包括多个任务之间的执行逻辑;
在进行迁移的过程中,记录所述迁移对象对应的当前任务状态以及当前迁移状态;
若在执行数据迁移的过程中停止迁移,则在重新开始迁移时,获取停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态;及
根据停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态对所述迁移对象进行迁移。
一种数据迁移装置,包括:
迁移对象获取模块,用于获取迁移对象;
任务执行模块,用于根据预设的任务执行逻辑对所述迁移对象执行对应的迁移任务,以对所述迁移对象进行迁移,所述任务执行逻辑包括多个任务之间的执行逻辑;
状态记录模块,用于在进行迁移的过程中,记录所述迁移对象对应的当前任务状态以及当前迁移状态;
状态获取模块,用于若在执行数据迁移的过程中停止迁移,则在重新开始迁移时,获取停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态;及
迁移模块,用于根据停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态对所述迁移对象进行迁移。
一种计算机设备,包括存储器和一个或多个处理器,所述存储器中储存有计算机可读指令,所述计算机可读指令被所述处理器执行时,计算机可读指令被处理器执行时实现本申请任意一个实施例中提供的数据迁移方法的步骤。
一个或多个存储有计算机可读指令的非易失性计算机可读存储介质,计算机可读指令被一个或多个处理器执行时,使得一个或多个处理器实现本申请任意一个实施例中提供的数据迁移方法的步骤。
本申请的一个或多个实施例的细节在下面的附图和描述中提出。本申请的其它特征、目的和优点将从说明书、附图以及权利要求书变得明显。
附图说明
为了更清楚地说明本申请实施例中的技术方案,下面将对实施例描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。
图1为根据一个或多个实施例中提供的数据迁移方法的实施环境图。
图2为根据一个或多个实施例中数据迁移方法的流程图。
图3为根据一个或多个实施例中根据预设的任务执行逻辑对迁移对象执行对应的任务, 以对迁移对象进行迁移的流程图。
图4为根据一个或多个实施例中执行数据核对任务对迁移后的迁移对象与迁移前的迁移对象进行匹配,得到匹配结果的流程图。
图5为另一个实施例中数据迁移方法的流程图。
图6为根据一个或多个实施例中数据迁移装置的框图。
图7为根据一个或多个实施例中任务执行模块的框图。
图8为根据一个或多个实施例中核对执行单元的框图。
图9为根据一个或多个实施例中数据迁移单元的框图。
图10为根据一个或多个实施例中计算机设备的框图。
具体实施方式
为了使本申请的技术方案及优点更加清楚明白,以下结合附图及实施例,对本申请进行进一步详细说明。应当理解,此处所描述的具体实施例仅仅用以解释本申请,并不用于限定本申请。
可以理解,如无例外说明,本申请所使用的术语“第一”、“第二”等可在本文中用于描述各种元件,但除非特别说明,这些元件不受这些术语限制。这些术语仅用于将第一个元件与另一个元件区分。举例来说,在不脱离本申请的范围的情况下,可以将第一哈希值称为第二哈希值,且类似地,可将第二哈希值称为第一哈希值。在本申请中,迁移前的迁移对象指源存储设备中的迁移对象,迁移后的迁移对象指从源存储设备中迁移到目标存储设备后存储在目标存储设备上的迁移对象。
图1为一个实施例中提供的数据迁移方法的实施环境图,如图1所示,在该实施环境中,包括源存储设备110、计算机设备120以及目标存储设备130。
源存储设备存储110上存储有需要进行迁移的迁移对象,当需要进行迁移时,计算机设备120接收迁移指令,获取源存储设备110上的迁移对象,然后将迁移对象迁移到目标存储设备130中。计算机设备120可以是独立的物理服务器或终端,也可以是多个物理服务器构成的服务器集群,可以是提供云服务器、云数据库、云存储和CDN等基础云计算服务的云服务器,本申请在此不做限制。
如图2所示,在一些实施例中,提出了一种数据迁移方法,该数据迁移方法可以应用于 上述的计算机设备120中,具体可以包括以下步骤:
步骤S202,获取迁移对象。
具体地,迁移对象指要进行迁移的数据,迁移对象可以是图像文件、视频文件、音频文件以及文本文件等数据。迁移对象可以有一个或多个,在一些实施例中可以包括一个NAS盘或多个NAS盘上存储的全部或者部分数据。当需要进行数据迁移时,可以从对应的源存储设备上获取迁移对象。
步骤S204,根据预设的任务执行逻辑对迁移对象执行对应的迁移任务,以对迁移对象进行迁移,任务执行逻辑包括多个任务之间的执行逻辑。
任务执行逻辑指任务与任务之间的关系,例如为任务与任务之间的执行顺序以及特定任务的执行以其他任务的执行结果作为执行条件、特定任务的执行参数根据其他任务的执行结果生成等。例如A任务完成后进行B任务,若C任务执行失败则执行D任务。在进行迁移前,将迁移的过程分为多个任务,并设置这些任务之间的任务执行逻辑。
在一实施例中,可以将数据迁移分为复制任务、核对任务、标记任务以及修复任务。复制任务用于将迁移对象从源存储设备复制到目标存储设备;核对任务用于对迁移后的迁移对象与迁移前的迁移对象进行匹配,得到匹配结果,匹配结果例如为匹配成功或者匹配失败;标记任务用于根据匹配结果在源存储设备上标记迁移对象的迁移结果,当匹配成功时,标记迁移结果为迁移成功,当匹配失败时,标记迁移结果为迁移失败。修复任务用于若匹配失败,从源存储设备中重新迁移迁移对象。
步骤S206,在进行迁移的过程中,记录迁移对象对应的当前任务状态以及当前迁移状态。
具体地,当前任务状态记录的是迁移过程中当前所在的任务,例如为下载任务或者标记任务等。迁移状态指迁移对象的执行状况,例如可以包括正在执行、等待执行、暂停以及错误等。
在一实施例中,迁移状态可以随着任务的执行调整。随着任务的进行,数据的迁移状态也可能在进行变化。由于迁移一般具有多个迁移对象,为了提高迁移效率,每个任务可以开启多个进程,以对迁移对象进行并行迁移。因此,一个任务完成后,若下一个任务的进程数不够,则迁移对象对应的迁移状态由正在执行调整为等待执行。例如,当对一个迁移对象完成复制任务后,若核对任务进程数不够,则该迁移对象的迁移对象为等待中。
在一实施例中,每个任务对应的进程数可以相同也可以不同,具体可以根据任务的执行 时间进行设定,执行时间的长度与进程个数可以呈正相关关系。例如,复制任务时间长,核对任务时间短,则进行复制任务的进程比核对任务的进程多。
在一实施例中,在迁移过程中,可以根据源存储设备的业务量调整进程数目。当业务大的时候,降低进程的数量,以降低源存储设备的压力。业务量小的时候,增加进程的数量,以提高迁移的速度。
在一实施例中,迁移状态可以是根据迁移调整指令调整的,可以发送停止迁移指令,停止迁移指令中携带需要停止迁移的迁移对象的标识,则该迁移对象的标识对应的迁移对象的迁移状态调整为暂停。
步骤S208,若在执行数据迁移的过程中停止迁移,则在重新开始进行迁移时,获取停止迁移时迁移对象对应的当前任务状态以及当前迁移状态。
具体地,停止迁移时指停止迁移的那一刻,停止迁移可以是由于发生了异常导致的,也可以是接收到停止迁移的指令是导致的。当重新进行迁移时,获取停止迁移时迁移对象对应的当前任务状态以及当前迁移状态。
步骤S210,根据停止迁移时迁移对象对应的当前任务状态以及当前迁移状态对迁移对象进行迁移。
具体地,获取到停止迁移时迁移对象对应的当前任务状态以及当前迁移状态后,结合停止迁移时迁移对象对应的当前任务状态以及当前迁移状态对迁移对象进行迁移。
在一实施例中,迁移状态可以包括正在执行、等待执行、暂停以及错误等。正在执行指迁移对象的迁移任务正在执行中,等待执行指迁移对象所在的任务均没有在执行,但若排序在前的迁移对象对应的任务完成后,则开始继续执行迁移任务。暂停指迁移对象需要暂停迁移,等暂停条件解除后再继续进行迁移。错误指当迁移前与迁移后的迁移对象的数据匹配失败或者到目标存储设备中下载迁移对象失败,需要进行修复任务对数据重新进行迁移。当停止迁移时迁移对象对应的当前迁移状态为正在执行或者等待执行时,执行停止迁移时迁移对象对应的当前任务状态对应的任务;当停止迁移时的迁移对象对应的当前迁移状态为暂停时,若重新开始迁移时,暂停条件还未解除,则保持暂停的迁移状态,不对该迁移对象进行迁移,若重新开始迁移时暂停条件解除,执行停止迁移时迁移对象对应的当前任务状态对应的任务。当停止迁移时迁移对象对应的当前迁移状态为错误时,执行修复任务从源存储设备中重新迁移迁移对象。
例如,若迁移的任务执行逻辑顺序为任务A-任务B-任务C。若发生异常停止迁移时的任务状态为B,迁移状态为等待中,则重新开始迁移时对迁移对象执行任务B。若发生异常导致停止迁移时的任务状态为B,迁移状态为暂停,而停止迁移时迁移状态为暂停是因为暂停条件为7月18日前对该数据不进行迁移,则若重新开始迁移是在7月17日,则不对该迁移对象进行迁移。若重新开始迁移时为7月19日,则对迁移对象执行任务B。
在一些实施例中,可以将迁移对象迁移到一个或多个目标存储设备中。例如,将迁移对象同时迁移到主集群以及灾备集群,进一步提升迁移效率,且在主集群发送故障的时候使灾备集群能够及时进行响应。迁移后的数据可以采用对象存储的方式进行存储。
在一些实施例中,迁移过程中记录迁移日志信息,使整个迁移过程更加透明化。
上述数据迁移方法,要进行数据迁移时,获取迁移对象,根据预设的任务执行逻辑对迁移对象执行对应的迁移任务,以对迁移对象进行迁移,并在进行迁移的过程中,记录迁移对象对应的当前任务状态以及当前迁移状态,若在执行数据迁移的过程中停止迁移,则在重新开始进行迁移时,根据停止迁移时迁移对象对应的当前任务状态以及当前迁移状态对迁移对象进行迁移。由于将迁移分为多个任务,且设置了任务执行逻辑,并在迁移的过程中记录迁移对象对应的当前任务状态以及当前迁移状态,因此在重新开始迁移后,可以根据停止迁移时迁移对象对应的当前任务状态以及当前迁移状态进行迁移,降低了迁移工作量,提高了迁移的效率。
在一些实施例中,如图3所示,步骤S204即根据预设的任务执行逻辑对迁移对象执行对应的任务,以对迁移对象进行迁移可以包括以下步骤:
S302,执行复制任务将迁移对象从源存储设备复制到目标存储设备。
具体地,复制是指将存储在源存储设备上的迁移对象发送到目标存储设备中,同时也可以保留源存储设备上的数据。源存储设备与目标存储设备可以相同也可以不同,例如可以将迁移对象从设备的一个分区迁移到另一个分区,或者从一台虚拟机迁移到另一台虚拟机。
S304,执行数据核对任务将迁移后的迁移对象与迁移前的迁移对象进行匹配,得到匹配结果。
具体地,可以根据数据的大小、名称等迁移对象属性信息以及迁移对象的内容对迁移前与迁移后的迁移对象进行核对。也可以分别根据迁移前的迁移对象以及迁移后的迁移对象属性信息进行哈希运算,将计算得到的哈希值进行匹配。哈希计算为把任意长度的输入通过散 列算法变换成一定长度的输出的计算方法,散列算法可以为MD5(Message-digest Algorithm5,信息摘要算法5)算法和SHA(Secure Hash Standard、安全哈希标准)算法等,具体不做限制。
S306,根据匹配结果执行标记任务,在源存储设备上标记迁移对象的迁移结果。
具体地,迁移结果包括迁移失败以及迁移成功。若匹配结果为匹配成功,则在源存储设备上标记迁移结果为迁移成功。若匹配结果为匹配失败,则在源存储设备上标记迁移结果为迁移失败。
S308,若匹配失败,执行修复任务从源存储设备中重新迁移迁移对象。
具体地,修复任务用于匹配失败时,重新将源存储设备中的迁移对象迁移到目标存储设备中。
在一些实施例中,若匹配成功,为了进一步确认迁移对象已经迁移到目标存储设备中,还可以执行下载任务,到目标存储设备中下载迁移对象,若下载失败,说明迁移未成功,需要执行修复任务重新迁移。若下载成功以及匹配成功,执行标记任务,标记源存储设备上迁移对象的迁移结果为迁移成功。
在一些实施例中,当业务服务器接收到获取迁移对象的数据请求时,可以获取源存储设备上的迁移结果,若源存储设备上的迁移对象的迁移结果为迁移成功,到目标存储设备上获取数据。
图4示出了在一些实施例中,步骤S304即执行数据核对任务对迁移后的迁移对象与迁移前的迁移对象进行匹配,得到匹配结果的实现流程图,具体可以包括以下步骤:
步骤S402,根据迁移前的迁移对象的预设属性信息进行哈希计算,得到第一哈希值。
步骤S404,根据迁移后的迁移对象的预设属性信息进行哈希计算,得到第二哈希值。
步骤S406,将第一哈希值以及第二哈希值进行匹配,若一致,则匹配结果为成功,若不一致,则匹配结果为失败。
具体地,属性信息指迁移对象的性质,例如可以包括迁移对象的大小信息以及名称等。可以理解,在进行哈希计算时,迁移前的迁移对象与迁移后的迁移对象的预设属性信息的类型是相同的,例如若利用了迁移前的迁移对象的大小信息计算第一哈希值,则在计算第二哈希值时也必须利用迁移后的迁移对象的大小信息。哈希计算为把任意长度的输入通过散列算法变换成一定长度的输出的计算方法,散列算法可以为MD5(Message-digestAlgorithm5, 信息摘要算法5)算法和SHA(Secure Hash Standard、安全哈希标准)算法等,具体不做限制。当得到第一哈希值以及第二哈希值时,比较第一哈希值与第二哈希值是否一致。
在一些实施例中,还可以对迁移前的迁移对象以及迁移后上的迁移对象的属性信息进行核对,以进一步确认迁移对象是否已经正确迁移到目标存储设备中。
在一些实施例中,如图5所示,数据迁移方法还可以包括以下步骤:
步骤S502,接收迁移指令,迁移指令携带迁移策略,迁移策略包括迁移对象标识以及迁移条件。
具体地,迁移对象标识可以为具体的对象标识,也可以为数据类型或者数据存储位置等可以用于标识迁移对象的字符。迁移条件指要在什么情况下进行迁移。迁移条件例如可以包括迁移的时间、迁移对象的优先级或者迁移对象的数据大小要求中的一个或多个。迁移的时间为立即执行迁移或者在预设时间执行迁移。迁移对象的优先级用于指示指哪一些数据优先进行迁移。例如,对于保险行业,寿险的数据比产险重要,则可以设置寿险数据优先级比产险数据优先级高。
在一些实施例中,迁移条件还可以包括数据类型对应的目标存储地址。例如寿险数据对应的目标存储地址为A存储设备的地址,产险数据对应的目标存储设备为B存储设备的地址。
步骤S504,根据迁移对象数据标识获取迁移对象的属性信息,并根据属性信息以及迁移条件生成索引信息序列。
具体地,获取的迁移对象的属性信息可以根据迁移条件确定。例如,若迁移条件包括哪些数据类型优先进行迁移,则迁移对象的属性信息包括数据类型,若迁移条件包括优先迁移数据量小的迁移对象时,则获取的迁移对象的属性信息包括迁移对象的大小。当得到属性信息后,根据属性信息以及迁移条件生成索引信息序列。索引信息序列中索引信息根据迁移对象的属性信息以及迁移条件进行排序。例如,当迁移条件为优先迁移寿险时,将寿险的数据的排序在前。当迁移条件为优先迁移数据量大的迁移对象时,按照迁移对象的数据量从大到小对索引信息进行排列。当得到索引信息序列时,根据索引信息序列获取迁移对象。如按照索引信息的排序获取迁移对象。
在一些实施例中,当迁移条件中还包括数据类型对应的目标存储地址时,获取的属性信息中包括数据类型,并根据数据类型对应的目标存储地址生成索引信息序列,索引信息序列中包括各个迁移对象对应的目标存储地址,因此,在进行迁移时,根据迁移对象对应的目标 存储地址将迁移对象迁移到对应的目标存储设备中。
应该理解的是,虽然上述流程图中的各个步骤按照箭头的指示依次显示,但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明,这些步骤的执行并没有严格的顺序限制,这些步骤可以以其它的顺序执行。而且,上述流程图中至少一部分步骤可以包括多个子步骤或者多个阶段,这些子步骤或者阶段并不必然是在同一时刻执行完成,而是可以在不同的时刻执行,这些子步骤或者阶段的执行顺序也不必然是依次进行,而是可以与其它步骤或者其它步骤的子步骤或者阶段的至少一部分轮流或者交替地执行。
如图6所示,在一些实施例中,提供了一种数据迁移装置,该数据迁移装置可以集成于上述的计算机设备120中,包括:
迁移对象获取模块602,用于获取迁移对象。
任务执行模块604,用于根据预设的任务执行逻辑对迁移对象执行对应的迁移任务,以对迁移对象进行迁移,任务执行逻辑包括多个任务之间的执行逻辑。
状态记录模块606,用于在进行迁移的过程中,记录迁移对象对应的当前任务状态以及当前迁移状态。
状态获取模块608,用于若在执行数据迁移的过程中停止迁移,则在重新开始迁移时,获取停止迁移时迁移对象对应的当前任务状态以及当前迁移状态。
迁移模块610,用于根据停止迁移时迁移对象对应的当前任务状态以及当前迁移状态对迁移对象进行迁移。
在一些实施例中,当前迁移状态包括正在执行、等待执行、暂停以及错误,迁移模块用于:当停止迁移时迁移对象对应的当前迁移状态为正在执行或者等待执行时,执行停止迁移时迁移对象对应的当前任务状态对应的任务;当停止迁移时迁移对象对应的当前迁移状态为错误时,从源存储设备中重新迁移迁移对象;当停止迁移时迁移对象对应的当前迁移状态为暂停时,若暂停条件解除,执行停止迁移时迁移对象对应的当前任务状态对应的任务。
在一些实施例中,如图7所示,任务执行模块604包括:
复制执行单元702,用于执行复制任务将迁移对象从源存储设备复制到目标存储设备。
核对执行单元704,用于执行数据核对任务将迁移后的迁移对象与迁移前的迁移对象进行匹配,得到匹配结果。
标记执行单元706,用于根据匹配结果执行标记任务,在源存储设备上标记迁移对象的 迁移结果。
修复执行单元708,用于若匹配失败,执行修复任务从源存储设备中重新迁移迁移对象。
在一些实施例中,如图8所示,核对执行单元704包括:
第一计算子单元802,用于根据迁移前的迁移对象的预设属性信息进行哈希计算,得到第一哈希值。
第二计算子单元804,根据迁移后的迁移对象的预设属性信息进行哈希计算,得到第二哈希值。
匹配子单元806,用于将第一哈希值以及第二哈希值进行匹配,若一致,则匹配结果为成功,若不一致,则匹配结果为失败。
在一些实施例中,如图9所示,数据迁移装置还可以包括:
迁移指令接收模块902,用于接收迁移指令,迁移指令携带迁移策略,迁移策略包括迁移对象标识以及迁移条件。
索引信息生成模块904,用于根据迁移对象数据标识获取迁移对象的属性信息,并根据属性信息以及迁移条件生成索引信息序列。
迁移对象获取模块602具体用于根据索引信息序列获取迁移对象。
关于数据迁移装置的具体限定可以参见上文中对于数据迁移方法的限定,在此不再赘述。上述数据迁移装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中,也可以以软件形式存储于计算机设备中的存储器中,以便于处理器调用执行以上各个模块对应的操作。
如图10所示,为一个实施例中计算机设备的内部结构图,即该计算机设备既可以是服务器也可以是终端,该计算机设备通过系统连接总线连接处理器、非易失性存储介质、内存储器和网络接口。非易失性存储介质可以是非易失性计算机可读存储介质,该计算机设备的非易失性存储介质可存储操作系统和计算机可读指令,该计算机可读指令被执行时,可使得处理器执行一种数据迁移方法。该计算机设备的处理器用于提供计算和控制能力,支撑整个计算机设备的运行。该内存储器中可储存有计算机可读指令,该计算机可读指令被处理器执行时,可使得处理器执行一种数据迁移方法。计算机设备的网络接口用于进行网络通信,如接收迁移指令,发送停止控制指令等。本领域技术人员可以理解,图10中示出的结构,仅仅是与本申请方案相关的部分结构的框图,并不构成对本申请方案所应用于其上的计算机设备的 限定,具体的计算机设备可以包括比图中所示更多或更少的部件,或者组合某些部件,或者具有不同的部件布置。
在一些实施例中,本申请提供的数据迁移装置可以实现为一种计算机可读指令的形式,计算机可读指令可在如图10所示的计算机设备上运行,计算机设备的非易失性存储介质可存储组成该数据迁移装置的各个程序模块,比如图6中的迁移对象获取模块602、任务执行模块604、状态记录模块606、状态获取模块608和迁移模块610。各个程序模块中包括计算机可读指令,计算机可读指令用于使计算机设备执行本说明书中描述的本申请各个实施例的数据迁移方法中的步骤。例如,计算机设备可以通过如图6所示的数据迁移装置中的迁移对象获取模块602获取迁移对象,通过任务执行模块604根据预设的任务执行逻辑对迁移对象执行对应的迁移任务,以对迁移对象进行迁移,任务执行逻辑包括多个任务之间的执行逻辑,通过状态记录模块606在进行迁移的过程中,记录迁移对象对应的当前任务状态以及当前迁移状态,通过状态获取模块608若在执行数据迁移的过程中停止迁移,则在重新开始迁移时,获取停止迁移时迁移对象对应的当前任务状态以及当前迁移状态,通过迁移模块610根据停止迁移时迁移对象对应的当前任务状态以及当前迁移状态对迁移对象进行迁移。
一种计算机设备,包括存储器和一个或多个处理器,存储器中储存有计算机可读指令,计算机可读指令被处理器执行时,使得一个或多个处理器执行以下步骤:获取迁移对象;根据预设的任务执行逻辑对迁移对象执行对应的迁移任务,以对迁移对象进行迁移,任务执行逻辑包括多个任务之间的执行逻辑;在进行迁移的过程中,记录迁移对象对应的当前任务状态以及当前迁移状态;若在执行数据迁移的过程中停止迁移,则在重新开始迁移时,获取停止迁移时迁移对象对应的当前任务状态以及当前迁移状态;及,根据停止迁移时迁移对象对应的当前任务状态以及当前迁移状态对迁移对象进行迁移。
一个或多个存储有计算机可读指令的非易失性计算机可读存储介质,计算机可读指令被一个或多个处理器执行时,使得一个或多个处理器执行以下步骤:获取迁移对象;根据预设的任务执行逻辑对迁移对象执行对应的迁移任务,以对迁移对象进行迁移,任务执行逻辑包括多个任务之间的执行逻辑;在进行迁移的过程中,记录迁移对象对应的当前任务状态以及当前迁移状态;若在执行数据迁移的过程中停止迁移,则在重新开始迁移时,获取停止迁移时迁移对象对应的当前任务状态以及当前迁移状态;及,根据停止迁移时迁移对象对应的当前任务状态以及当前迁移状态对迁移对象进行迁移。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程,是可以通过计算机可读指令来指令相关的硬件来完成,所述的计算机可读指令可存储于一非易失性计算机可读取存储介质中,该计算机可读指令在执行时,可包括如上述各方法的实施例的流程。其中,本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用,均可包括非易失性和/或易失性存储器。非易失性存储器可包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。易失性存储器可包括随机存取存储器(RAM)或者外部高速缓冲存储器。作为说明而非局限,RAM以多种形式可得,诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双数据率SDRAM(DDRSDRAM)、增强型SDRAM(ESDRAM)、同步链路(Synchlink)DRAM(SLDRAM)、存储器总线(Rambus)直接RAM(RDRAM)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)等。
以上实施例的各技术特征可以进行任意的组合,为使描述简洁,未对上述实施例中的各个技术特征所有可能的组合都进行描述,然而,只要这些技术特征的组合不存在矛盾,都应当认为是本说明书记载的范围。
以上实施例仅表达了本申请的几种实施方式,其描述较为具体和详细,但并不能因此而理解为对本申请专利范围的限制。应当指出的是,对于本领域的普通技术人员来说,在不脱离本申请构思的前提下,还可以做出若干变形和改进,这些都属于本申请的保护范围。因此,本申请专利的保护范围应以所附权利要求为准。

Claims (20)

  1. 一种数据迁移方法,包括:
    获取迁移对象;
    根据预设的任务执行逻辑对所述迁移对象执行对应的迁移任务,以对所述迁移对象进行迁移,所述任务执行逻辑包括多个任务之间的执行逻辑;
    在进行迁移的过程中,记录所述迁移对象对应的当前任务状态以及当前迁移状态;
    若在执行数据迁移的过程中停止迁移,则在重新开始迁移时,获取停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态;及
    根据停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态对所述迁移对象进行迁移。
  2. 根据权利要求1所述的方法,其特征在于,所述根据预设的任务执行逻辑对所述迁移对象执行对应的任务,以对所述迁移对象进行迁移包括:
    执行复制任务将所述迁移对象从源存储设备复制到目标存储设备;
    执行数据核对任务将迁移后的迁移对象与迁移前的迁移对象进行匹配,得到匹配结果,所述匹配结果包括匹配成功或匹配失败;及
    根据所述匹配结果执行标记任务,在所述源存储设备上标记所述迁移对象的迁移结果;
    若匹配失败,执行修复任务从所述源存储设备中重新迁移所述迁移对象。
  3. 根据权利要求1所述的方法,其特征在于,所述当前迁移状态包括正在执行、等待执行、暂停以及错误,所述根据停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态对所述迁移对象进行迁移包括:
    当停止迁移时所述迁移对象对应的当前迁移状态为正在执行或者等待执行时,执行停止迁移时所述迁移对象对应的当前任务状态对应的任务;
    当停止迁移时所述迁移对象对应的当前迁移状态为错误时,从源存储设备中重新迁移所述迁移对象;及
    当停止迁移时所述迁移对象对应的当前迁移状态为暂停时,若暂停条件解除,执行停止迁移时所述迁移对象对应的当前任务状态对应的任务。
  4. 根据权利要求2所述的方法,其特征在于,所述执行数据核对任务对迁移后的迁移对 象与迁移前的迁移对象进行匹配,得到匹配结果包括:
    根据迁移前的所述迁移对象的预设属性信息进行哈希计算,得到第一哈希值;
    根据迁移后的所述迁移对象的预设属性信息进行哈希计算,得到第二哈希值;及
    将所述第一哈希值以及所述第二哈希值进行匹配,若一致,则匹配结果为成功,若不一致,则匹配结果为失败。
  5. 根据权利要求1~4任一项所述的方法,其特征在于,还包括:
    接收迁移指令,所述迁移指令携带迁移策略,所述迁移策略包括迁移对象标识以及迁移条件;
    根据所述迁移对象数据标识获取所述各个迁移对象的属性信息,并根据所所述各个迁移对象对应的属性信息以及所述迁移条件生成索引信息序列;及
    所述获取迁移对象的步骤包括:
    根据所述索引信息序列获取迁移对象。
  6. 一种数据迁移装置,包括:
    迁移对象获取模块,用于获取迁移对象;
    任务执行模块,用于根据预设的任务执行逻辑对所述迁移对象执行对应的迁移任务,以对所述迁移对象进行迁移,所述任务执行逻辑包括多个任务之间的执行逻辑;
    状态记录模块,用于在进行迁移的过程中,记录所述迁移对象对应的当前任务状态以及当前迁移状态;
    状态获取模块,用于若在执行数据迁移的过程中停止迁移,则在重新开始迁移时,获取停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态;及
    迁移模块,用于根据停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态对所述迁移对象进行迁移。
  7. 根据权利要求6所述的装置,其特征在于,所述任务执行模块包括:
    复制执行单元,用于执行复制任务将所述迁移对象从源存储设备复制到目标存储设备;
    核对执行单元,用于执行数据核对任务将迁移后的迁移对象与迁移前的迁移对象进行匹配,得到匹配结果,所述匹配结果包括匹配成功或匹配失败;
    标记执行单元,用于根据所述匹配结果执行标记任务,在所述源存储设备上标记所述迁移对象的迁移结果;及
    修复执行单元,用于若匹配失败,执行修复任务从所述源存储设备中重新迁移所述迁移对象。
  8. 根据权利要求6所述的装置,其特征在于,所述当前迁移状态包括正在执行、等待执行、暂停以及错误,所述迁移模块用于:
    当停止迁移时所述迁移对象对应的当前迁移状态为正在执行或者等待执行时,执行停止迁移时所述迁移对象对应的当前任务状态对应的任务;
    当停止迁移时所述迁移对象对应的当前迁移状态为错误时,从源存储设备中重新迁移所述迁移对象;及
    当停止迁移时所述迁移对象对应的当前迁移状态为暂停时,若暂停条件解除,执行停止迁移时所述迁移对象对应的当前任务状态对应的任务。
  9. 根据权利要求7所述的装置,其特征在于,所述核对执行单元包括:
    第一计算子单元,用于根据迁移前的所述迁移对象的预设属性信息进行哈希计算,得到第一哈希值;
    第二计算子单元,用于根据迁移后的所述迁移对象的预设属性信息进行哈希计算,得到第二哈希值;及
    匹配子单元,用于将所述第一哈希值以及所述第二哈希值进行匹配,若一致,则匹配结果为成功,若不一致,则匹配结果为失败。
  10. 根据权利要求6~9任一项所述的装置,其特征在于,还包括:
    接收迁移指令,所述迁移指令携带迁移策略,所述迁移策略包括迁移对象标识以及迁移条件;
    根据所述迁移对象数据标识获取所述各个迁移对象的属性信息,并根据所所述各个迁移对象对应的属性信息以及所述迁移条件生成索引信息序列;及
    所述获取迁移对象的步骤包括:
    根据所述索引信息序列获取迁移对象。
  11. 一种计算机设备,包括存储器及一个或多个处理器,所述存储器中储存有计算机可读指令,所述计算机可读指令被所述一个或多个处理器执行时,使得所述一个或多个处理器执行以下步骤:
    获取迁移对象;
    根据预设的任务执行逻辑对所述迁移对象执行对应的迁移任务,以对所述迁移对象进行迁移,所述任务执行逻辑包括多个任务之间的执行逻辑;
    在进行迁移的过程中,记录所述迁移对象对应的当前任务状态以及当前迁移状态;
    若在执行数据迁移的过程中停止迁移,则在重新开始迁移时,获取停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态;及
    根据停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态对所述迁移对象进行迁移。
  12. 根据权利要求11所述的计算机设备,其特征在于,所述处理器所执行的所述根据预设的任务执行逻辑对所述迁移对象执行对应的任务,以对所述迁移对象进行迁移的包括:
    执行复制任务将所述迁移对象从源存储设备复制到目标存储设备;
    执行数据核对任务将迁移后的迁移对象与迁移前的迁移对象进行匹配,得到匹配结果,所述匹配结果包括匹配成功或匹配失败;
    根据所述匹配结果执行标记任务,在所述源存储设备上标记所述迁移对象的迁移结果;及
    若匹配失败,执行修复任务从所述源存储设备中重新迁移所述迁移对象。
  13. 根据权利要求11所述的计算机设备,其特征在于,所述当前迁移状态包括正在执行、等待执行、暂停以及错误,所述处理器所执行的所述根据停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态对所述迁移对象进行迁移包括:
    当停止迁移时所述迁移对象对应的当前迁移状态为正在执行或者等待执行时,执行停止迁移时所述迁移对象对应的当前任务状态对应的任务;
    当停止迁移时所述迁移对象对应的当前迁移状态为错误时,从源存储设备中重新迁移所述迁移对象;及
    当停止迁移时所述迁移对象对应的当前迁移状态为暂停时,若暂停条件解除,执行停止迁移时所述迁移对象对应的当前任务状态对应的任务。
  14. 根据权利要求12所述的计算机设备,其特征在于,所述处理器所执行的所述执行数据核对任务对迁移后的迁移对象与迁移前的迁移对象进行匹配,得到匹配结果包括:
    根据迁移前的所述迁移对象的预设属性信息进行哈希计算,得到第一哈希值;
    根据迁移后的所述迁移对象的预设属性信息进行哈希计算,得到第二哈希值;及
    将所述第一哈希值以及所述第二哈希值进行匹配,若一致,则匹配结果为成功,若不一致,则匹配结果为失败。
  15. 根据权利要求11~14任一项所述的计算机设备,其特征在于,所述处理器执行所述计算机可读指令时还用于实现以下步骤:
    接收迁移指令,所述迁移指令携带迁移策略,所述迁移策略包括迁移对象标识以及迁移条件;
    根据所述迁移对象数据标识获取所述各个迁移对象的属性信息,并根据所所述各个迁移对象对应的属性信息以及所述迁移条件生成索引信息序列;及
    所述获取迁移对象的步骤包括:
    根据所述索引信息序列获取迁移对象。
  16. 一个或多个存储有计算机可读指令的非易失性计算机可读存储介质,所述计算机可读指令被一个或多个处理器执行时,使得所述一个或多个处理器执行以下步骤:
    获取迁移对象;
    根据预设的任务执行逻辑对所述迁移对象执行对应的迁移任务,以对所述迁移对象进行迁移,所述任务执行逻辑包括多个任务之间的执行逻辑;
    在进行迁移的过程中,记录所述迁移对象对应的当前任务状态以及当前迁移状态;
    若在执行数据迁移的过程中停止迁移,则在重新开始迁移时,获取停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态;及
    根据停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态对所述迁移对象进行迁移。
  17. 根据权利要求16所述的存储介质,其特征在于,所述处理器所执行的所述根据预设的任务执行逻辑对所述迁移对象执行对应的任务,以对所述迁移对象进行迁移的包括:
    执行复制任务将所述迁移对象从源存储设备复制到目标存储设备;
    执行数据核对任务将迁移后的迁移对象与迁移前的迁移对象进行匹配,得到匹配结果,所述匹配结果包括匹配成功或匹配失败;
    根据所述匹配结果执行标记任务,在所述源存储设备上标记所述迁移对象的迁移结果;及
    若匹配失败,执行修复任务从所述源存储设备中重新迁移所述迁移对象。
  18. 根据权利要求16所述的存储介质,其特征在于,所述当前迁移状态包括正在执行、等待执行、暂停以及错误,所述处理器所执行的所述根据停止迁移时所述迁移对象对应的当前任务状态以及当前迁移状态对所述迁移对象进行迁移包括:
    当停止迁移时所述迁移对象对应的当前迁移状态为正在执行或者等待执行时,执行停止迁移时所述迁移对象对应的当前任务状态对应的任务;
    当停止迁移时所述迁移对象对应的当前迁移状态为错误时,从源存储设备中重新迁移所述迁移对象;及
    当停止迁移时所述迁移对象对应的当前迁移状态为暂停时,若暂停条件解除,执行停止迁移时所述迁移对象对应的当前任务状态对应的任务。
  19. 根据权利要求17所述的存储介质,其特征在于,所述处理器所执行的所述执行数据核对任务对迁移后的迁移对象与迁移前的迁移对象进行匹配,得到匹配结果包括:
    根据迁移前的所述迁移对象的预设属性信息进行哈希计算,得到第一哈希值;
    根据迁移后的所述迁移对象的预设属性信息进行哈希计算,得到第二哈希值;及
    将所述第一哈希值以及所述第二哈希值进行匹配,若一致,则匹配结果为成功,若不一致,则匹配结果为失败。
  20. 根据权利要求16~19任一项所述的存储介质,其特征在于,所述处理器执行所述计算机可读指令时还用于实现以下步骤:
    接收迁移指令,所述迁移指令携带迁移策略,所述迁移策略包括迁移对象标识以及迁移条件;
    根据所述迁移对象数据标识获取所述各个迁移对象的属性信息,并根据所所述各个迁移对象对应的属性信息以及所述迁移条件生成索引信息序列;及
    所述获取迁移对象的步骤包括:
    根据所述索引信息序列获取迁移对象。
PCT/CN2018/077106 2017-11-16 2018-02-24 数据迁移方法、装置、计算机设备和存储介质 Ceased WO2019095576A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201711140525.5A CN107943572B (zh) 2017-11-16 2017-11-16 数据迁移方法、装置、计算机设备和存储介质
CN201711140525.5 2017-11-16

Publications (1)

Publication Number Publication Date
WO2019095576A1 true WO2019095576A1 (zh) 2019-05-23

Family

ID=61931496

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/077106 Ceased WO2019095576A1 (zh) 2017-11-16 2018-02-24 数据迁移方法、装置、计算机设备和存储介质

Country Status (2)

Country Link
CN (1) CN107943572B (zh)
WO (1) WO2019095576A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112131179A (zh) * 2020-09-23 2020-12-25 平安科技(深圳)有限公司 任务状态检测方法、装置、计算机设备及存储介质
CN117370468A (zh) * 2023-11-09 2024-01-09 浙江智臾科技有限公司 数据同步方法以及集群系统

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110209344B (zh) * 2018-07-26 2022-02-18 腾讯科技(深圳)有限公司 迁移对象存储的方法、装置和计算机可读存储介质
CN109450997B (zh) * 2018-10-26 2022-09-30 平安科技(深圳)有限公司 数据跨终端迁移方法、装置、计算机设备及存储介质
CN109766392A (zh) * 2019-01-21 2019-05-17 北京字节跳动网络技术有限公司 数据处理方法和装置
CN109884897B (zh) * 2019-03-21 2022-03-25 中山大学 一种基于深度强化学习的无人机任务匹配与计算迁移方法
CN110335012A (zh) * 2019-05-30 2019-10-15 广东民航机场建设有限公司 项目任务状态切换方法、装置、计算机设备和存储介质
CN113032369A (zh) * 2021-03-26 2021-06-25 山东英信计算机技术有限公司 一种数据迁移方法、装置及介质
CN115168318A (zh) * 2021-04-06 2022-10-11 腾讯科技(深圳)有限公司 数据迁移的控制方法、装置和存储介质及电子设备
CN116244257B (zh) * 2021-12-08 2026-02-27 腾讯科技(深圳)有限公司 目标对象的迁移方法和装置、存储介质及电子设备
CN115129704A (zh) * 2022-07-22 2022-09-30 中国平安人寿保险股份有限公司 迁移数据校验方法及其系统、电子设备、存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103902593A (zh) * 2012-12-27 2014-07-02 中国移动通信集团河南有限公司 一种数据迁移的方法和装置
CN105528368A (zh) * 2014-09-30 2016-04-27 北京金山云网络技术有限公司 一种数据库迁移方法及装置
CN105824846A (zh) * 2015-01-09 2016-08-03 阿里巴巴集团控股有限公司 数据迁移方法及装置

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9344484B2 (en) * 2011-05-27 2016-05-17 Red Hat, Inc. Determining consistencies in staged replication data to improve data migration efficiency in cloud based networks

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103902593A (zh) * 2012-12-27 2014-07-02 中国移动通信集团河南有限公司 一种数据迁移的方法和装置
CN105528368A (zh) * 2014-09-30 2016-04-27 北京金山云网络技术有限公司 一种数据库迁移方法及装置
CN105824846A (zh) * 2015-01-09 2016-08-03 阿里巴巴集团控股有限公司 数据迁移方法及装置

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112131179A (zh) * 2020-09-23 2020-12-25 平安科技(深圳)有限公司 任务状态检测方法、装置、计算机设备及存储介质
CN112131179B (zh) * 2020-09-23 2023-11-21 平安科技(深圳)有限公司 任务状态检测方法、装置、计算机设备及存储介质
CN117370468A (zh) * 2023-11-09 2024-01-09 浙江智臾科技有限公司 数据同步方法以及集群系统

Also Published As

Publication number Publication date
CN107943572A (zh) 2018-04-20
CN107943572B (zh) 2019-09-03

Similar Documents

Publication Publication Date Title
WO2019095576A1 (zh) 数据迁移方法、装置、计算机设备和存储介质
KR101960993B1 (ko) 효율적인 애플리케이션-인식 재난 복구 기법
US20160077863A1 (en) Reliable and scalable image transfer for data centers with low connectivity using redundancy detection
WO2017049764A1 (zh) 数据读写方法及分布式存储系统
CN106775602B (zh) 一种代码发布方法及装置
CN110781028B (zh) 数据备份方法、数据恢复方法、装置及计算设备
CN106339177A (zh) 一种创建虚拟机的方法和装置
WO2019047482A1 (zh) 应用程序兼容性测试方法、装置、计算机设备和存储介质
WO2018233630A1 (zh) 故障发现
CN106897345B (zh) 一种数据存储的方法及装置
CN110471906A (zh) 数据库切换方法、装置及设备
WO2017032170A1 (zh) 镜像文件的导入方法及装置
US12086101B2 (en) Systems and methods for ingesting data files using multi-threaded processing
WO2017113694A1 (zh) 文件同步方法、装置及系统
CN108255576A (zh) 虚拟机热迁移异常处理方法、装置和存储介质
CN114416689A (zh) 数据迁移方法、装置、计算机设备、存储介质
CN116107994A (zh) 数据迁移系统、方法、电子设备及计算机可读存储介质
CN106789142B (zh) 资源配置的方法和装置
CN106339473A (zh) 文件复制方法及装置
CN102193841A (zh) 一种Subversion配置库的备份方法及装置
CN114490196A (zh) 数据库切换方法、系统、设备及介质
CN107402862A (zh) 用于在客户端实现资源编排的方法与设备
CN106484312A (zh) 一种虚拟机磁盘数据迁移方法及装置
CN109542860B (zh) 基于hdfs的业务数据管理方法、终端设备
CN114281818A (zh) 数据处理方法、装置、服务器和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18878523

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 13.08.2020)

122 Ep: pct application non-entry in european phase

Ref document number: 18878523

Country of ref document: EP

Kind code of ref document: A1