CN114489517A - Offline accelerated migration method, system, equipment and storage medium - Google Patents

Offline accelerated migration method, system, equipment and storage medium Download PDF

Info

Publication number
CN114489517A
CN114489517A CN202210135354.1A CN202210135354A CN114489517A CN 114489517 A CN114489517 A CN 114489517A CN 202210135354 A CN202210135354 A CN 202210135354A CN 114489517 A CN114489517 A CN 114489517A
Authority
CN
China
Prior art keywords
data
migrated
hard disk
migration
storage
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210135354.1A
Other languages
Chinese (zh)
Other versions
CN114489517B (en
Inventor
张晨光
徐源浩
马豹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Inspur Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN202210135354.1A priority Critical patent/CN114489517B/en
Publication of CN114489517A publication Critical patent/CN114489517A/en
Application granted granted Critical
Publication of CN114489517B publication Critical patent/CN114489517B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0662—Virtualisation aspects
    • G06F3/0664—Virtualisation aspects at device level, e.g. emulation of a storage device or system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G06F3/0677—Optical disk device, e.g. CD-ROM, DVD
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides an offline accelerated migration method, system, device and medium, wherein the method comprises the following steps: acquiring a storage format of at least one hard disk to be migrated, selecting a corresponding migration instruction according to the storage format of the hard disk to be migrated, and simultaneously reading data of the hard disk to be migrated into a transfer space; and acquiring the data of the hard disk to be migrated from the transfer space and sending the data to the target storage equipment through a network. According to the offline accelerated migration method provided by the invention, data in the hard disks are migrated to new storage equipment in a parallel mode, and the migration speeds of the migrated hard disks are regulated and controlled by means of a transfer space. The problem that the copy of the disk is slow by the migration tool is solved. The migration tool can intelligently identify the type of the disk file system and migrate data by using a corresponding command, and the copying speed can almost reach the limit. Meanwhile, a scheme for regulating and controlling the migration speed or the migration priority of different hard disks according to the requirements is also provided.

Description

Offline accelerated migration method, system, equipment and storage medium
Technical Field
The invention belongs to the field of computers, and particularly relates to an offline accelerated migration method, system, device and storage medium.
Background
In recent years, with the development of virtualization technology, openstack (an open source cloud computing management platform project) functions are increasingly abundant, the environment is more and more stable, and more government and enterprise units choose to deploy their business systems into openstack. For systems which are deployed in physical machines or vmware virtual machines before, migration to the openstack cloud platform is also more prone. Migration is generally two types, one is online migration and one is offline migration. Migrating a source node to a cloud platform under the condition that the online migration requires that services on the source node to be migrated are not interrupted; the offline migration refers to the migration to the cloud platform when the service of the source node can be stopped. This patent is presented for offline migration.
At present, a few tools and schemes for off-line migration and cloud-up are available, an open-source off-line migration tool commonly used in the industry is a recycling dragon, and the recycling dragon is selected to be used when a plurality of unit services migrate and cloud-up. However, the regenerated dragon has two disadvantages: 1. the operation is complex, the source node and the target node are migrated only by performing a series of operations, and the nodes are transferred; 2. only one disk can be migrated at a time, and if a system disk and a plurality of data disks in the source node need to be migrated, the operating system needs to be restarted and the migration operation is executed for multiple times.
Disclosure of Invention
In order to solve the above problems, the present invention provides an offline accelerated migration method, including:
acquiring a storage format of at least one hard disk to be migrated, selecting a corresponding migration instruction according to the storage format of the hard disk to be migrated, and simultaneously reading data of the hard disk to be migrated into a transfer space;
and acquiring the data of the hard disk to be migrated from the transfer space and sending the data to the target storage equipment through a network.
In some embodiments of the present invention, selecting a corresponding migration instruction, and reading data of the hard disk to be migrated to the transfer space at the same time includes:
establishing different storage addresses for different migration instructions in a memory, and setting a target address of the migration instruction as the storage address corresponding to the migration instruction;
and reading data from the hard disk to be migrated through a migration instruction matched with the hard disk to be migrated and storing the data into the storage address corresponding to the migration instruction.
In some embodiments of the present invention, acquiring data of the hard disk to be migrated from the transit space and sending the data to the target storage device through the network includes:
and acquiring the data of the hard disk to be migrated from the storage address by a first preset size and sending the data to the corresponding target storage equipment through a network.
In some embodiments of the present invention, acquiring data of the hard disk to be migrated from the transit space and sending the data to the target storage device through the network includes:
responding to the fact that the number of the storage addresses corresponding to the hard disks to be migrated is multiple, reading data from the storage addresses corresponding to the hard disks to be migrated according to the migration priority of the set multiple hard disks to be migrated and sending the data to target storage equipment through a network according to the priority; and
and responding to the network bandwidth vacancy, sequentially reading data from the storage addresses corresponding to the hard disks to be migrated according to the priority and sending the data to target storage equipment through a network.
In some embodiments of the present invention, selecting a corresponding migration instruction, and reading data of the hard disk to be migrated to the transfer space at the same time includes:
and establishing a shared memory in the memory, and reading and storing the data of the hard disk to be migrated corresponding to the shared memory by the migration instruction.
In some embodiments of the present invention, selecting a corresponding migration instruction, and reading the data of the hard disk to be migrated to the transfer space at the same time further includes:
responding to the plurality of hard disks to be migrated, and according to the migration priorities of the plurality of hard disks to be migrated, limiting migration instructions corresponding to the hard disks to be migrated to write data into the shared memory according to the priorities;
responding to the fact that the shared memory is idle or the data storage quantity in the preset time is maintained below a second preset size, and writing data into the shared memory according to the migration instruction corresponding to the hard disk to be migrated in the priority level; and
and in response to the existence of the transmission bandwidth set for the plurality of hard disks to be migrated, controlling the speed of writing data into the shared memory by the instruction corresponding to the hard disk to be migrated according to the ratio of the transmission bandwidth of the hard disk to be migrated.
In some embodiments of the present invention, acquiring data of the hard disk to be migrated from the transit space and sending the data to the target storage device through the network includes:
and reading corresponding data from the shared memory and sending the corresponding data to the target storage equipment corresponding to the data.
In another aspect, the present invention further provides an offline accelerated migration system, including:
the hard disk data reading module is configured to acquire a storage format of at least one hard disk to be migrated, select a corresponding migration instruction according to the storage format of the hard disk to be migrated, and read data of the hard disk to be migrated to a transfer space at the same time;
and the hard disk data sending module is configured to acquire the data of the hard disk to be migrated from the transfer space and send the data to a target storage device through a network.
Yet another aspect of the present invention also provides a computer apparatus, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method of any one of the above embodiments.
Yet another aspect of the present invention further provides a computer-readable storage medium, which stores a computer program, and the computer program realizes the steps of the method of any one of the above embodiments when executed by a processor.
According to the offline accelerated migration method provided by the invention, data in the hard disks are migrated to new storage equipment in a parallel mode, and the migration speeds of the migrated hard disks are regulated and controlled by means of a transfer space. The problem that the copying of the disk by the migration tool is slow is solved. The migration tool can intelligently identify the type of the disk file system and migrate data by using a corresponding command, and the copying speed can almost reach the limit. Meanwhile, a scheme for regulating and controlling the migration speed or the migration priority of different hard disks according to the requirements is also provided.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a method for offline accelerated migration according to an embodiment of the present invention;
fig. 2 is a schematic system structure diagram of an offline accelerated migration system according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a computer device according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
The invention relates to hard disk data migration application in the technical field of OpenStack virtualization, wherein a traditional migration method is generally used for migrating a disk by using dd commands, dd is a command on a Unix and Unix-like system, the main function is to convert and copy files, and dd can copy data among files, equipment, partitions and volumes. Data can be input or output from anywhere therein; but there are important differences in the output to the partitions. However, data migration by dd commands is full copy, if the disk is a 100G disk, even if the disk is empty data, 100G can be copied, and almost no disk is full in an actual scene on site, and the disks of the physical machine are all large, so that the migration speed is greatly reduced. In addition, because dd commands are supported by Unix systems, dd commands are usually called in a Shell script (Shell is a user interface of the system, and provides an interface for a user to interact with a kernel, and Shell scripts are script programs written for the Shell). Therefore, only one hard disk can be migrated at the same time. Resulting in inefficient migration.
As shown in fig. 1, to solve the above problem, the present invention provides an offline accelerated migration method, including:
step S1, acquiring a storage format of at least one hard disk to be migrated, selecting a corresponding migration instruction according to the storage format of the hard disk to be migrated, and reading data of the hard disk to be migrated to a transfer space simultaneously;
and step S2, acquiring the data of the hard disk to be migrated from the transfer space and sending the data to the target storage device through the network.
In the embodiment of the invention, when data reading migration is carried out on a disk, a migration instruction of an open source tool, namely, partclone, is selected to migrate data in the disk, wherein the partclone is an open source data copying tool and supports almost all common file systems. For the supported file system, when copying data, the blank blocks are skipped, and only the blocks with the data are copied, so that the copying speed can be greatly increased, and the copying speed is increased. For unsupported file systems, the data is copied using the partclone.dd command, which, although it is faster to copy without support, is also much faster than the normal dd command. Although partclone supports multiple operating systems, it has different commands corresponding to them according to different disk formats, and the maximum migration speed can be reached only by executing the appropriate migration command, and the data migration with the wrong command will report an error or copy the entire amount.
With the help of the partclone tool, in step S1, the migration tool developed according to the present invention first obtains the storage format of the hard disk to be migrated, and then selects different partclone migration instructions according to different storage formats to migrate the data of the corresponding hard disk.
In this embodiment, when migrating data through the migration instruction corresponding to the partclone, the present invention does not directly migrate the disk data to the target storage device, i.e. the target disk, through the corresponding partclone instruction, but transfers the destination address of the partcclone instruction migration to the transfer space. Specifically, the conventional use of partclone is as follows: ext 4-d-b-s/dev/sda 1-o/dev/sdb 1,/dev/sda 1 is the source address and/dev/sdb 1 is the destination address. In the present application, the use mode of the instruction needs to be changed according to the source code of the partclone, that is, the use mode of the source address is the same as the original use mode, but the destination address is fixed to a transfer space, which may be a corresponding disk or a cache or a memory. All partclone commands will send data to the same type of transit space. And data are distinguished in the transfer space according to different disks and different instructions.
In step S2, the corresponding disk data is read from the transit space by the corresponding data sending program in the migration tool developed according to the present invention, and then sent to other storage devices via the network or other data transmission channel. Typically a network.
In some embodiments of the present invention, selecting a corresponding migration instruction, and reading data of the hard disk to be migrated to the transfer space at the same time includes:
establishing different storage addresses for different migration instructions in a memory, and setting a target address of the migration instruction as the storage address corresponding to the migration instruction;
and reading data from the hard disk to be migrated through a migration instruction matched with the hard disk to be migrated and storing the data into the storage address corresponding to the migration instruction.
In this embodiment, the migration tool implemented by the offline accelerated migration method provided by the present invention runs in a LiveCD, which is an operating system (usually including some other software) that is stored in a bootable self-booting optical disc and can be executed without installation. After exiting the LiveCD and rebooting, the computer can be restored to the original operating system. The LiveCD system is completely loaded and run into the memory when running, so the invention creates a plurality of storage addresses in the LiveCD system (in the memory) by means of the LiveCD system, such as/dev/ext 2/sd1,/dev/ext 2/sd2; v/ext3/sd1,/dev/ext 3/sd 2; /dev/ext4/sd1,/dev/ext 4/sd 2; the device comprises a data storage space, a data transfer space, a transfer space and a transfer space, wherein the data storage space comprises/dev/fat 32/sd1,/dev/fat 32/sd2 and the like, a plurality of storage addresses are created based on the storage formats of the disks, and the condition that different to-be-transferred hard disks in the transfer space have corresponding storage addresses under the condition that the plurality of storage formats in the to-be-transferred hard disks are the same is met.
And further, calling different migration instructions according to the storage format of the hard disk to be migrated, and writing data into the storage address corresponding to the hard disk to be migrated. Specifically, if the current hard disk to be migrated has 5 hard disks in total, there are two ext2 format hard disks, two ntfs format hard disks, and one fat32 format hard disk. During migration, data of two hard disks with the format of ext2 are read and written into/dev/ext 2/sd1,/dev/ext 2/sd2 respectively through the partclon. ext2, and data of two hard disks with the format of ntfs are read from corresponding hard disks and written into/dev/ntfs/sd 1,/dev/ntfs/sd 2 through a partclon. ntfs command; similarly, a hard disk with the format of fat32 is written into/dev/fat 32/sd1 through a partclone 32 command.
In some embodiments of the present invention, a space size is set for each storage address, that is, addresses such as/dev/ntfs/sd 2, and the space size of the corresponding storage address can be flexibly allocated according to the size of the memory occupied by the LiveCD system, for example, the size of each storage address is set to 100MB, and after the corresponding storage address is fully written, the corresponding migration instruction does not write data to the corresponding storage address any more.
In some embodiments of the present invention, acquiring data of the hard disk to be migrated from the transit space and sending the data to the target storage device through the network includes:
and acquiring the data of the hard disk to be migrated from the storage address by a first preset size and sending the data to the corresponding target storage equipment through a network.
In this embodiment, the corresponding data sending program in the migration tool developed according to the method of the present invention reads the corresponding data from the storage address according to a certain size, and then sends the corresponding disk data to the target storage device through the network with the size data as the granularity. Specifically, data can be read from the corresponding storage space with a size of 1MB, and then sent out through the network, for example, files in the dev/fat32/sd1 are read with a granularity of 1MB, and then sent to a remote storage device through the network.
It should be noted that, when the migration instruction writes data into the migration instruction, the data needs to be written according to the above granularity, and the written data is marked according to the writing time or the data number as needed, so that the data sending program can conveniently read the data at the corresponding granularity and send the data to the destination storage device.
In some embodiments of the present invention, acquiring data of the hard disk to be migrated from the transit space and sending the data to the target storage device through the network includes:
responding to the fact that the number of the storage addresses corresponding to the hard disks to be migrated is multiple, reading data from the storage addresses corresponding to the hard disks to be migrated according to the migration priority of the set multiple hard disks to be migrated and sending the data to target storage equipment through a network according to the priority; and
and responding to the network bandwidth vacancy, sequentially reading data from the storage addresses corresponding to the hard disks to be migrated according to the priority and sending the data to target storage equipment through a network.
In this embodiment, when the data transmission program processes the data migration tasks of the multiple hard disks to be migrated, the data transmission program may read data from the storage addresses corresponding to the generation migration hard disks according to the set priorities of the multiple hard disks to be migrated, and transmit the data through the network. And when the data is read, the data of the corresponding storage address with high priority is read preferentially according to a certain granularity and is sent to the corresponding destination storage equipment.
Further, if the write amount of the data of the storage address corresponding to the hard disk to be migrated with the high priority is small, the data sending program cannot read the data to be sent within a certain time, or the data is read to hardly occupy the transmission bandwidth of the whole network, the data sending program may read the data in the storage address corresponding to the corresponding hard disk to be migrated from high to low according to the priority of the hard disk to be migrated and send the data to the destination storage device.
Specifically, there are many factors that affect reading of the hard disk to be migrated by the migration instruction, for example, the read access speed of the hard disk to be migrated, the optimization problem of the implementation logic of data migration of the migration instruction itself, and the like all affect the speed of writing data into the storage address corresponding to the hard disk to be migrated by the migration instruction. Even if the space of the storage address is set to be too small, the capability of the data sending program sending data through the network is high (i.e. the network bandwidth is sufficient and the read-write capability of a single hard disk is weak), for example, if the reading speed of the hard disk 1 with the highest priority is 300MB/s, and the network bandwidth is 1250MB/s, the hard disk with the highest priority cannot fully utilize the network bandwidth, and at this time, the data sending program can read the data of the corresponding hard disk to be migrated from the storage address of other hard disk to be migrated and send the data. And when the disk data reading capability is weak, but the priority is high, the effect of utilizing the network bandwidth in a self-adaptive manner can be realized through the mechanism, namely the network bandwidth which is the same as the reading speed of the hard disk to be migrated with the high priority is kept, and other bandwidths are reasonably distributed to the data migration of the hard disk to be migrated with other priorities.
In addition, when the user does not specify the priority of the hard disk to be migrated, the data sending program polls and acquires the corresponding hard disk data to be migrated from the corresponding storage address and sends the data to the corresponding destination storage device.
In some embodiments of the present invention, selecting a corresponding migration instruction, and reading data of the hard disk to be migrated to the transfer space at the same time includes:
and establishing a shared memory in the memory, and reading and storing the data of the hard disk to be migrated corresponding to the shared memory by the migration instruction.
In this embodiment, when the LiveCD is not used or does not support the corresponding system platform, the corresponding migration instruction is stored in the memory by establishing the shared memory, and the data read from the corresponding hard disk to be migrated is stored. The shared memory may be a customized storage queue that can distinguish hard disk data to be migrated, or a customized storage structure with a large data space. And has a certain size. For example, 1GB of memory space, and the size of data written into the shared memory by each migration instruction, that is, the granularity of the written data, for example, 10MB, needs to be set, and the corresponding data sending program reads and sends the data to the destination storage device in units of 10 MB.
Further, when the migration instruction stores data in the shared memory, a user-defined mark needs to be added to the corresponding data. For example, the model unique ID of the disk or the number in the data migration task at this time is added to the corresponding data.
In some embodiments of the present invention, selecting a corresponding migration instruction, and reading the data of the hard disk to be migrated to the transfer space at the same time further includes:
responding to the plurality of hard disks to be migrated, and according to the migration priorities of the plurality of hard disks to be migrated, limiting migration instructions corresponding to the hard disks to be migrated to write data into the shared memory according to the priorities;
responding to the condition that the shared memory is idle or the data storage quantity in preset time is maintained below a second preset size, and writing data into the shared memory according to the migration instruction corresponding to the hard disk to be migrated, which is operated according to the priority level; and
and in response to the existence of the transmission bandwidth set for the plurality of hard disks to be migrated, controlling the speed of writing data into the shared memory by the instruction corresponding to the hard disk to be migrated according to the ratio of the transmission bandwidth of the hard disk to be migrated.
In this embodiment, when there are multiple hard disks to be migrated, the number or speed of writing data into the shared memory by the migration instruction corresponding to the hard disk to be migrated may be limited according to the priority of the hard disk to be migrated.
In some embodiments of the present invention, the migration instruction (referring to the program developed again based on the open source code) also writes data into the shared memory at a certain granularity, and when the migration instruction with the highest priority writes data into the shared memory, other migration commands cannot write data into the shared memory, because the data read-write speed of the memory is much higher than the data read speed of the hard disk, therefore, the migration instruction with the highest priority does not always occupy the shared memory, and after the data is written by the migration instruction with the highest priority, and at this time, once the highest-level migration instruction or the high-level migration instruction can write data into the shared memory again, stopping the second-level migration instruction currently written into the shared memory from writing subsequent (the data of the current granularity can be written completely) data into the shared memory. Specifically, for example, there are two migration instructions, the priority of the migration instruction 1 is higher than that of the migration instruction 2, and the migration instruction 2 can write data after the migration instruction 1 has written the data. In some cases, the migration instruction 2 accumulates a large amount of written data, and when the write right of the shared memory is obtained (the migration instruction 1 has no data to write), the migration instruction 2 may continuously write data into the shared memory, but after the migration instruction 1 has the write condition (certain data is read from the hard disk), and after the migration instruction 2 finishes writing the corresponding granular data, the migration instruction 1 cancels the write right of the migration instruction 2, and continues to write data into the shared memory, and after the write is finished, the migration instruction 2 is switched to write data. And so on, so too are the cases above 2 migration instructions.
In some embodiments of the present invention, the granularity number of writing data into the shared memory by the corresponding migration instruction is further allocated in a quota allocation manner based on the reading speed of the hard disk to be migrated, the network bandwidth, the priority of the hard disk to be migrated, and the like.
Specifically, assuming that 10MB of data is written into the shared memory each time by a migration instruction as a granularity, there are 3 hard disks to be migrated, the read speed of the first hard disk to be migrated is 500MB/S, the read speed of the second hard disk to be migrated is 400MB/S, the read speed of the third hard disk to be migrated is 500MB/S, and the network bandwidth is still 1250MB/S bandwidth provided by a trillion network card, and the priority of the second hard disk to be migrated is higher than that of the third hard disk to be migrated, and the priority of the third hard disk to be migrated is higher than that of the first hard disk to be migrated, the amount of data written into the shared memory per second by the migration instruction corresponding to the second hard disk is not limited, the amount of data written into the shared memory per second by the migration instruction corresponding to the three hard disks to be migrated is 500MB, and the amount of data written into the shared memory per second by the migration instruction corresponding to the first hard disk to be migrated is 1250 + 400MB, and limiting the hard disk to be migrated to write data into the shared memory in a quota mode, counting and emptying the data after every second, and preferentially allowing a second hard disk to be migrated (with the highest priority) to write data into the shared memory.
In some embodiments of the present invention, acquiring data of the hard disk to be migrated from the transit space and sending the data to the target storage device through the network includes:
and reading corresponding data from the shared memory and sending the corresponding data to the target storage equipment corresponding to the data.
In this embodiment, the data sending program directly reads data from the shared memory (such as the above-mentioned customized sequential storage structure) in sequence, and sends the data to the corresponding storage device through the network.
In some embodiments of the present invention, the speed of the migration tool implemented according to the method in the above embodiments is compared with the speed of the conventional migration method as follows:
disk size parclone.xfs partclone.dd dd
20G 240MB/s 117MB/s 12MB/s
In practical tests, the network bandwidth is ten-trillion network, the disk data is copied by using the traditional dd command at the speed of about 12MB/s, and the disk data is copied by using the partclone.dd command at the speed of about 117MB/s, which is 8-10 times that of the dd command. Xfs copying with the partclone command can be up to 240MB/s because it skips empty data blocks, which also varies with the real data size in the disk. Therefore, the scheme greatly improves the migration speed.
As shown in fig. 2, another aspect of the present invention further provides an offline accelerated migration system, including:
the hard disk data reading module 1 is configured to obtain a storage format of at least one hard disk to be migrated, select a corresponding migration instruction according to the storage format of the hard disk to be migrated, and read data of the hard disk to be migrated to a transfer space at the same time;
and the hard disk data sending module 2 is configured to obtain the data of the hard disk to be migrated from the transfer space and send the data to a target storage device through a network.
As shown in fig. 3, yet another aspect of the present invention also provides a computer apparatus, including:
at least one processor 21; and
a memory 22, said memory 22 storing computer instructions 23 executable on said processor 21, said instructions 23 when executed by said processor 21 implementing the steps of the method of any of the above embodiments.
As shown in fig. 4, a further aspect of the present invention also provides a computer-readable storage medium 401, where the computer-readable storage medium 401 stores a computer program 402, and the computer program 402 implements the steps of the method according to any one of the above embodiments when being executed by a processor.
According to the offline accelerated migration method provided by the invention, data in the hard disks are migrated to new storage equipment in a parallel mode, and the migration speeds of the migrated hard disks are regulated and controlled by means of a transfer space. The problem that the copy of the disk is slow by the migration tool is solved. The migration tool can intelligently identify the type of the disk file system and migrate data by using a corresponding command, and the copying speed can almost reach the limit. Meanwhile, a scheme for regulating and controlling the migration speed or the migration priority of different hard disks according to the requirements is also provided.
As described above, the implementation manner provided by the embodiment of the present invention can implement high-speed controllability of data migration under multiple platforms. The migration can be performed by means of LiveCD, the hard disk data can also be migrated in a general mode, the data migration can be performed in an equal-flow parallel migration mode (priority is not specified), and the corresponding hard disk data can also be selected to be preferentially migrated in a priority mode.
The foregoing are exemplary embodiments of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of an embodiment of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements and the like that may be made without departing from the spirit or scope of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.

Claims (10)

1. An offline accelerated migration method, comprising:
acquiring a storage format of at least one hard disk to be migrated, selecting a corresponding migration instruction according to the storage format of the hard disk to be migrated, and simultaneously reading data of the hard disk to be migrated into a transfer space;
and acquiring the data of the hard disk to be migrated from the transfer space and sending the data to the target storage equipment through a network.
2. The method according to claim 1, wherein the selecting the corresponding migration instruction and reading the data of the hard disk to be migrated to the transfer space simultaneously comprises:
establishing different storage addresses for different migration instructions in a memory, and setting a target address of the migration instruction as the storage address corresponding to the migration instruction;
and reading data from the hard disk to be migrated through a migration instruction matched with the hard disk to be migrated and storing the data into the storage address corresponding to the migration instruction.
3. The method of claim 2, wherein the obtaining the data of the hard disk to be migrated from the staging space and sending the data to a target storage device through a network comprises:
and acquiring the data of the hard disk to be migrated from the storage address by a first preset size and sending the data to the corresponding target storage equipment through a network.
4. The method of claim 3, wherein the obtaining the data of the hard disk to be migrated from the staging space and sending the data to a target storage device through a network comprises:
responding to the fact that the number of the storage addresses corresponding to the hard disks to be migrated is multiple, reading data from the storage addresses corresponding to the hard disks to be migrated according to the migration priority of the set multiple hard disks to be migrated and sending the data to target storage equipment through a network according to the priority; and
and responding to the network bandwidth vacancy, sequentially reading data from the storage addresses corresponding to the hard disks to be migrated according to the priority and sending the data to target storage equipment through a network.
5. The method according to claim 1, wherein the selecting the corresponding migration instruction and reading the data of the hard disk to be migrated to the transfer space simultaneously comprises:
and establishing a shared memory in the memory, and reading and storing the data of the hard disk to be migrated corresponding to the shared memory by the migration instruction.
6. The method according to claim 5, wherein the selecting the corresponding migration instruction and reading the data of the hard disk to be migrated to the transfer space simultaneously further comprises:
responding to the plurality of hard disks to be migrated, and according to the migration priorities of the plurality of hard disks to be migrated, limiting migration instructions corresponding to the hard disks to be migrated to write data into the shared memory according to the priorities;
responding to the condition that the shared memory is idle or the data storage quantity in preset time is maintained below a second preset size, and writing data into the shared memory according to the migration instruction corresponding to the hard disk to be migrated, which is operated according to the priority level; and
and in response to the existence of the transmission bandwidth set for the plurality of hard disks to be migrated, controlling the speed of writing data into the shared memory by the instruction corresponding to the hard disk to be migrated according to the ratio of the transmission bandwidth of the hard disk to be migrated.
7. The method of claim 5, wherein the obtaining the data of the hard disk to be migrated from the staging space and sending the data to a target storage device through a network comprises:
and reading corresponding data from the shared memory and sending the corresponding data to the target storage equipment corresponding to the data.
8. An offline accelerated migration system, comprising:
the hard disk data reading module is configured to acquire a storage format of at least one hard disk to be migrated, select a corresponding migration instruction according to the storage format of the hard disk to be migrated, and read data of the hard disk to be migrated to a transfer space at the same time;
and the hard disk data sending module is configured to acquire the data of the hard disk to be migrated from the transfer space and send the data to a target storage device through a network.
9. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method of any one of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
CN202210135354.1A 2022-02-14 2022-02-14 Offline acceleration migration method, system, equipment and storage medium Active CN114489517B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210135354.1A CN114489517B (en) 2022-02-14 2022-02-14 Offline acceleration migration method, system, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210135354.1A CN114489517B (en) 2022-02-14 2022-02-14 Offline acceleration migration method, system, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN114489517A true CN114489517A (en) 2022-05-13
CN114489517B CN114489517B (en) 2023-09-08

Family

ID=81479910

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210135354.1A Active CN114489517B (en) 2022-02-14 2022-02-14 Offline acceleration migration method, system, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114489517B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116340295A (en) * 2023-05-25 2023-06-27 北京长河数智科技有限责任公司 Data migration method and device based on multi-type data sources

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200233601A1 (en) * 2017-09-05 2020-07-23 Huawei Technologies Co., Ltd. Solid-State Disk (SSD) Data Migration
CN113448688A (en) * 2021-06-29 2021-09-28 华云数据控股集团有限公司 Method, system and computer medium for automatically correcting network configuration in cloud migration scene
WO2021259094A1 (en) * 2020-06-23 2021-12-30 中兴通讯股份有限公司 Method and apparatus for migration of virtual machine across cloud platform, and storage medium and electronic apparatus
CN113986119A (en) * 2021-09-30 2022-01-28 贵州电网有限责任公司 Simplified data reading method and system for accelerating cloud migration speed

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200233601A1 (en) * 2017-09-05 2020-07-23 Huawei Technologies Co., Ltd. Solid-State Disk (SSD) Data Migration
WO2021259094A1 (en) * 2020-06-23 2021-12-30 中兴通讯股份有限公司 Method and apparatus for migration of virtual machine across cloud platform, and storage medium and electronic apparatus
CN113448688A (en) * 2021-06-29 2021-09-28 华云数据控股集团有限公司 Method, system and computer medium for automatically correcting network configuration in cloud migration scene
CN113986119A (en) * 2021-09-30 2022-01-28 贵州电网有限责任公司 Simplified data reading method and system for accelerating cloud migration speed

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116340295A (en) * 2023-05-25 2023-06-27 北京长河数智科技有限责任公司 Data migration method and device based on multi-type data sources
CN116340295B (en) * 2023-05-25 2023-09-12 北京长河数智科技有限责任公司 Data migration method and device based on multi-type data sources

Also Published As

Publication number Publication date
CN114489517B (en) 2023-09-08

Similar Documents

Publication Publication Date Title
US8726275B2 (en) Selective partial cloning of virtual machines in a virtual computing environment
CN101727331B (en) Method and equipment for upgrading client operating system of active virtual machine
JP2019101703A (en) Storage system and control software arrangement method
US10333859B2 (en) Multi-tenant resource coordination method
US20110078681A1 (en) Method and system for running virtual machine image
CN103793258A (en) Hypervisor-based server duplication system and method
JPH10283210A (en) Virtual machine movement control system between virtual machine systems
CN102200921A (en) Intelligent boot device selection and recovery
JP2010097533A (en) Application migration and power consumption optimization in partitioned computer system
US9875057B2 (en) Method of live migration
CN102541619A (en) Management device and method for virtual machine
US20210042045A1 (en) Storage system and resource allocation control method
US11693684B2 (en) Information processing system and information processing method
US8732427B2 (en) Systems and methods for collapsing a derivative version of a primary storage volume
US20190332275A1 (en) Information processing system and volume allocation method
CN112181736A (en) Distributed storage system and configuration method thereof
CN110119308B (en) System for managing large-scale container applications
WO2018158808A1 (en) Information system, management program, and program exchanging method for information system
CN103885811A (en) Device, system and method for system-wide online migration of virtual machine system
CN108475201A (en) A kind of data capture method in virtual machine start-up course and cloud computing system
CN114489517B (en) Offline acceleration migration method, system, equipment and storage medium
JP2014010540A (en) Data migration control device, method and system in virtual server environment
CN107832097B (en) Data loading method and device
US11755438B2 (en) Automatic failover of a software-defined storage controller to handle input-output operations to and from an assigned namespace on a non-volatile memory device
CN109343928B (en) Virtual memory file redirection method and system for virtual machine in virtualization cluster

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant