CN114489517B - Offline acceleration migration method, system, equipment and storage medium - Google Patents

Offline acceleration migration method, system, equipment and storage medium Download PDF

Info

Publication number
CN114489517B
CN114489517B CN202210135354.1A CN202210135354A CN114489517B CN 114489517 B CN114489517 B CN 114489517B CN 202210135354 A CN202210135354 A CN 202210135354A CN 114489517 B CN114489517 B CN 114489517B
Authority
CN
China
Prior art keywords
data
migrated
hard disk
migration
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210135354.1A
Other languages
Chinese (zh)
Other versions
CN114489517A (en
Inventor
张晨光
徐源浩
马豹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Inspur Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN202210135354.1A priority Critical patent/CN114489517B/en
Publication of CN114489517A publication Critical patent/CN114489517A/en
Application granted granted Critical
Publication of CN114489517B publication Critical patent/CN114489517B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0662—Virtualisation aspects
    • G06F3/0664—Virtualisation aspects at device level, e.g. emulation of a storage device or system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G06F3/0677—Optical disk device, e.g. CD-ROM, DVD
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application provides an offline acceleration migration method, an offline acceleration migration system, offline acceleration migration equipment and an offline acceleration migration medium, wherein the offline acceleration migration method comprises the following steps: acquiring a storage format of at least one hard disk to be migrated, selecting a corresponding migration instruction according to the storage format of the hard disk to be migrated, and simultaneously reading data of the hard disk to be migrated to a transfer space; and acquiring the data of the hard disk to be migrated from the transit space and sending the data to a target storage device through a network. According to the offline acceleration migration method provided by the application, data in the hard disk are migrated to the new storage device in a parallel mode, and the migration speed of the migrated plurality of hard disks is regulated and controlled by means of the transfer space. The problem that the migration tool copies the disk slowly is solved. The migration tool can intelligently identify the type of the disk file system and use the corresponding command to migrate the data, and the copying speed can almost reach the limit. Meanwhile, a scheme for regulating and controlling the migration speed or migration priority of different hard disks according to the requirement is also provided.

Description

Offline acceleration migration method, system, equipment and storage medium
Technical Field
The application belongs to the field of computers, and particularly relates to an offline acceleration migration method, an offline acceleration migration system, offline acceleration migration equipment and an offline acceleration migration storage medium.
Background
In recent years, with the development of virtualization technology, the functions of an openstack (an open-source cloud computing management platform project) are more and more abundant, the environment is more and more stable, and more government enterprises choose to deploy own business systems into the openstack. For systems previously deployed in physical machines or vmware virtual machines, migration onto the openstack cloud platform is also more favored. There are typically two types of migration, one is online migration and one is offline migration. On-line migration requires that the source node to be migrated is migrated to the cloud platform under the condition that the service on the source node is not interrupted; offline migration refers to migration onto a cloud platform in the event that source node traffic may be down. This patent is directed to offline migration.
At present, the offline migration cloud-up tool and the scheme are not much, the open source offline migration tool commonly used in the industry is a regeneration dragon, and the regeneration dragon can be selected for use in the cloud-up of many unit service migration. However, regenerative dragon suffers from two drawbacks: 1. the operation is complex, the source node and the target node can be migrated only by a series of operations, and the transfer node is needed; 2. only one disk can be migrated at a time, if the system disk and a plurality of data disks in the source node need to be migrated, the operating system needs to be restarted, and the migration operation is executed for a plurality of times.
Disclosure of Invention
In order to solve the above problems, the present application provides an offline acceleration migration method, including:
acquiring a storage format of at least one hard disk to be migrated, selecting a corresponding migration instruction according to the storage format of the hard disk to be migrated, and simultaneously reading data of the hard disk to be migrated to a transfer space;
and acquiring the data of the hard disk to be migrated from the transit space and sending the data to a target storage device through a network.
In some embodiments of the present application, selecting a corresponding migration instruction, and simultaneously reading the data of the hard disk to be migrated to the transfer space includes:
establishing different storage addresses for different migration instructions in a memory, and setting a target address of the migration instructions as the storage address corresponding to the migration instructions;
and reading data from the hard disk to be migrated through a migration instruction matched with the hard disk to be migrated, and storing the data in the storage address corresponding to the migration instruction.
In some embodiments of the present application, the obtaining, from the staging space, the data of the hard disk to be migrated and sending the data to a target storage device through a network includes:
and acquiring the data of the hard disk to be migrated from the storage address with a first preset size and sending the data to the corresponding target storage device through a network.
In some embodiments of the present application, the obtaining, from the staging space, the data of the hard disk to be migrated and sending the data to a target storage device through a network includes:
responding to a plurality of storage addresses corresponding to the hard disks to be migrated, according to the migration priorities of the plurality of hard disks to be migrated, reading data from the storage addresses corresponding to the hard disks to be migrated according to the height of the priorities, and sending the data to a target storage device through a network; and
and in response to the network bandwidth being idle, reading data from the storage addresses corresponding to the hard disk to be migrated in sequence according to the priority and sending the data to target storage equipment through a network.
In some embodiments of the present application, selecting a corresponding migration instruction, and simultaneously reading the data of the hard disk to be migrated to the transfer space includes:
and establishing a shared memory in the memory, and reading and storing the data of the hard disk to be migrated, which corresponds to the shared memory, into the shared memory through the migration instruction.
In some embodiments of the present application, selecting a corresponding migration instruction, and simultaneously reading the data of the hard disk to be migrated to the transfer space further includes:
responding to a plurality of hard disks to be migrated, and according to the migration priorities of the plurality of hard disks to be migrated, limiting migration instructions corresponding to the hard disks to be migrated to write data into the shared memory according to the height of the priorities;
responding to the idle shared memory or maintaining the data stock below a second preset size in preset time, and operating a migration instruction corresponding to the hard disk to be migrated according to the priority level to write data into the shared memory; and
and in response to the transmission bandwidths set for the plurality of hard disks to be migrated, controlling the speed of writing data into the shared memory by the instruction corresponding to the hard disk to be migrated according to the proportion of the transmission bandwidths of the hard disks to be migrated.
In some embodiments of the present application, the obtaining, from the staging space, the data of the hard disk to be migrated and sending the data to a target storage device through a network includes:
and reading corresponding data from the shared memory and sending the data to target storage equipment corresponding to the data.
The application also provides an offline acceleration migration system, which comprises:
the hard disk data reading module is configured to acquire a storage format of at least one hard disk to be migrated, select a corresponding migration instruction according to the storage format of the hard disk to be migrated, and read the data of the hard disk to be migrated to a transfer space at the same time;
and the hard disk data transmission module is configured to acquire the data of the hard disk to be migrated from the transit space and transmit the data to the target storage device through a network.
Yet another aspect of the present application provides a computer apparatus, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, which when executed by the processor, perform the steps of the method of any of the above embodiments.
Yet another aspect of the application also proposes a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method of any of the above embodiments.
According to the offline acceleration migration method provided by the application, data in the hard disk are migrated to the new storage device in a parallel mode, and the migration speed of the migrated plurality of hard disks is regulated and controlled by means of the transfer space. The problem that the migration tool copies the disk slowly is solved. The migration tool can intelligently identify the type of the disk file system and use the corresponding command to migrate the data, and the copying speed can almost reach the limit. Meanwhile, a scheme for regulating and controlling the migration speed or migration priority of different hard disks according to the requirement is also provided.
Drawings
In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for offline acceleration migration method according to an embodiment of the present application;
fig. 2 is a schematic system structure diagram of an offline acceleration migration system according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a computer device according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a computer readable storage medium according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the following embodiments of the present application will be described in further detail with reference to the accompanying drawings.
It should be noted that, in the embodiments of the present application, all the expressions "first" and "second" are used to distinguish two entities with the same name but different entities or different parameters, and it is noted that the "first" and "second" are only used for convenience of expression, and should not be construed as limiting the embodiments of the present application, and the following embodiments are not described one by one.
The application relates to a hard disk data migration application in the technical field of OpenStack virtualization, wherein dd is a command on a Unix and Unix-like system and is used for migrating a disk in a traditional migration method, the main function is to convert and copy files, and the dd can copy data among the files, devices, partitions and volumes. Data may be input or output from anywhere therein; but there are important differences in output to the partition. However, when dd commands migrate data, the data is copied in full quantity, if the data is 100G, the data is copied 100G even if the data is empty, and in actual scene of the scene, almost no disk is fully written, and the data of the physical machine is bigger, so that the migration speed is reduced. In addition, the dd command is usually called in a Shell script (Shell is a user interface of a system, provides an interface for a user to interact with a kernel, and is a script program written for the Shell) because the dd command is supported by the Unix system. So that only one hard disk can be migrated at a time. Resulting in inefficient migration.
As shown in fig. 1, to solve the above-mentioned problems, the present application provides an offline acceleration migration method, which includes:
step S1, acquiring a storage format of at least one hard disk to be migrated, selecting a corresponding migration instruction according to the storage format of the hard disk to be migrated, and simultaneously reading data of the hard disk to be migrated to a transfer space;
and S2, acquiring the data of the hard disk to be migrated from the transit space and sending the data to a target storage device through a network.
In the embodiment of the application, when data reading and migration are carried out on the disk, a migration instruction of an open source tool partclone is selected to migrate the data in the disk, and the partclone is an open source data copying tool and supports almost all commonly used file systems. For a supported file system, when copying data, the skip of blank blocks is supported, and only blocks with data are copied, so that the copying speed can be greatly improved, and the copying speed is improved. For unsupported file systems, the data will be copied using the partclone. Dd command, which, although not supported, copies faster than the normal dd command. The partclone supports multiple operating systems, but according to different disk formats, different commands correspond to the partclone, and the maximum migration speed can be achieved only by executing proper migration commands, and the wrong commands are used for migrating data, so that errors or full copies can be reported.
In step S1, a migration tool developed according to the present application firstly obtains a storage format of a hard disk to be migrated, and then selects different partclone migration instructions to migrate data of a corresponding disk according to different storage formats.
In this embodiment, when data is migrated by the migration instruction corresponding to the partclone, the present application does not adopt the original method of directly migrating disk data to the target storage device, i.e. the target disk by the corresponding partclone instruction, but transfers the destination address of the migration of the partclone instruction to the transfer space. Specifically, the conventional usage modes of partclone are as follows: ext4-d-b-s/dev/sda 1-o/dev/sdb 1,/dev/sda1 is the source address and/dev/sdb 1 is the destination address. In the application, the use mode of the instruction needs to be changed according to the source code of the part clone, namely the use mode of the source address is the same as the original use mode, but the destination address is fixed to a transfer space, and can be a corresponding magnetic disk or a cache or a memory. All partclone instructions will send data to the same type of staging space. And the data are distinguished in the transit space according to different magnetic discs and different instructions.
In step S2, the corresponding disk data is read out from the transfer space by the corresponding data transmission program in the migration tool developed according to the present application, and then transmitted to other storage devices through a network or other data transmission channels. Typically a network.
In some embodiments of the present application, selecting a corresponding migration instruction, and simultaneously reading the data of the hard disk to be migrated to the transfer space includes:
establishing different storage addresses for different migration instructions in a memory, and setting a target address of the migration instructions as the storage address corresponding to the migration instructions;
and reading data from the hard disk to be migrated through a migration instruction matched with the hard disk to be migrated, and storing the data in the storage address corresponding to the migration instruction.
In this embodiment, the migration tool implemented by the offline acceleration migration method provided by the present application is run in a LiveCD, which is an operating system (usually including some other software) stored in a bootable self-starting optical disc and executed without installation. After exiting the LiveCD and restarting, the computer can be restored to the original operating system. The LiveCD system will be fully loaded and run into memory when running, so the present application creates multiple memory addresses, e.g.,/dev/ext 2/sd1,/dev/ext2/sd2., with the LiveCD system and in the LiveCD system (in memory); the/dev/ext 3/sd1,/dev/ext3/sd 2; v/ext4/sd1, V/ext4/sd2; and creating a plurality of storage addresses based on the storage formats of the magnetic disks, wherein the storage addresses are different to-be-migrated hard disks in the transfer space under the condition that the storage formats of the plurality of to-be-migrated hard disks are the same.
Further, different migration instructions are called according to the storage format of the hard disk to be migrated, and data are written into the storage address corresponding to the hard disk to be migrated. Specifically, if the current hard disk to be migrated has 5 hard disks, there are two hard disks in ext2 format, two hard disks in ntfs format, and one hard disk in fat32 format. During migration, respectively reading and writing data of two hard disks with the format of ext2 into/dev/ext 2/sd1 and/dev/ext 2/sd2 through partclone. Ext2, and reading data of two hard disks with the format of ntfs from corresponding hard disks through partclone. Ntfs commands and writing the data into/dev/ntfs/sd 1 and/dev/ntfs/sd 2; a hard disk in the format of fat32 is similarly written with its data into/dev/fat 32/sd1 by a partclone. Fat32 command.
In some embodiments of the present application, the space size of each storage address, i.e., the address size of/dev/ntfs/sd 2, is set, and the space size of the corresponding storage address can be flexibly allocated according to the size of the memory occupied by the LiveCD system, for example, the size of each storage address is set to 100MB, and when the corresponding storage address is full, the corresponding migration instruction does not write data into the corresponding storage address any more.
In some embodiments of the present application, the obtaining, from the staging space, the data of the hard disk to be migrated and sending the data to a target storage device through a network includes:
and acquiring the data of the hard disk to be migrated from the storage address with a first preset size and sending the data to the corresponding target storage device through a network.
In this embodiment, the corresponding data transmitting program in the migration tool developed according to the method of the present application reads the corresponding data from the storage address according to a certain size, and then transmits the corresponding disk data to the target storage device through the network with the size data as granularity. Specifically, data may be read from the corresponding storage space in a size of 1MB, then sent out through the network, for example, files in 1MB granularity read/dev/fat 32/sd1, and then sent to a remote storage device through the network.
When the migration instruction writes data into the storage device, the data is also required to be written according to the granularity, the written data is marked according to the writing time or the data number as required, and the data is conveniently read by a data sending program according to the corresponding granularity and sent to the target storage device.
In some embodiments of the present application, the obtaining, from the staging space, the data of the hard disk to be migrated and sending the data to a target storage device through a network includes:
responding to a plurality of storage addresses corresponding to the hard disks to be migrated, according to the migration priorities of the plurality of hard disks to be migrated, reading data from the storage addresses corresponding to the hard disks to be migrated according to the height of the priorities, and sending the data to a target storage device through a network; and
and in response to the network bandwidth being idle, reading data from the storage addresses corresponding to the hard disk to be migrated in sequence according to the priority and sending the data to target storage equipment through a network.
In this embodiment, when the data transmission program processes the data migration tasks of the plurality of hard disks to be migrated, the data may be read from the storage addresses corresponding to the hard disks to be migrated according to the set priorities of the plurality of hard disks to be migrated and transmitted through the network. When the data is read, the data of the corresponding storage address with high priority is read preferentially according to a certain granularity and is sent to the corresponding destination storage equipment.
Further, if the writing amount of the storage address data corresponding to the hard disk to be migrated with high priority is small, the data to be transmitted cannot be read by the data transmitting program within a certain time, or the data is difficult to occupy the transmission bandwidth of the whole network, the data transmitting program can read the data in the storage address corresponding to the corresponding hard disk to be migrated from high to low according to the priority of the hard disk to be migrated and transmit the data to the target storage device.
Specifically, many factors affect the reading of the corresponding hard disk to be migrated by the migration instruction, for example, the reading access speed of the hard disk to be migrated, the optimization problem of the realization logic of the data migration of the migration instruction itself, and the like affect the speed of writing data into the storage address corresponding to the hard disk to be migrated by the migration instruction. Even because the space setting of the storage address is too small, the capability of the data transmission program for transmitting data through the network is higher (i.e. the network bandwidth is sufficient and the read-write capability of a single hard disk is weaker), for example, if the read speed of the hard disk 1 with the highest priority is 300MB/s and the network bandwidth is 1250MB/s, the maximum transmission speed of the multi-megainterface is 1250MB/s, so that the hard disk with the highest priority is difficult to fully utilize the network bandwidth, and at this time, the data transmission program can read the corresponding data of the hard disk to be migrated from the storage addresses of other hard disks to be migrated for transmission. And when the disk data reading capability is weaker, but the priority is higher, the effect of self-adapting utilization of network bandwidth can be realized through the mechanism, namely, the network bandwidth which is the same as the reading speed of the hard disk to be migrated with high priority is maintained, and other bandwidths are reasonably distributed to the data migration of the hard disk to be migrated with other priorities.
In addition, when the user does not specify the priority of the hard disk to be migrated, the data sending program polls from the corresponding storage address to acquire the corresponding hard disk data to be migrated and sends the corresponding hard disk data to the corresponding destination storage device.
In some embodiments of the present application, selecting a corresponding migration instruction, and simultaneously reading the data of the hard disk to be migrated to the transfer space includes:
and establishing a shared memory in the memory, and reading and storing the data of the hard disk to be migrated, which corresponds to the shared memory, into the shared memory through the migration instruction.
In this embodiment, when the LiveCD is not used or the LiveCD does not support the corresponding system platform, a mode of establishing a shared memory is adopted to store a corresponding migration instruction in the memory, and data read from the corresponding hard disk to be migrated is stored. The shared memory may be a custom memory queue that can distinguish between hard disk data to be migrated, or a custom memory structure with a larger data space. And has a certain size. For example, 1GB of memory space, and the size of data written into the shared memory by each migration instruction needs to be set, that is, the granularity of the written data, for example, 10MB, and the corresponding data sending program reads and sends the data to the target storage device in units of 10 MB.
Further, when the migration instruction stores data into the shared memory, a custom mark needs to be added into the corresponding data. For example, the model unique ID of the disk or the number in the data migration task at this time is added to the corresponding data.
In some embodiments of the present application, selecting a corresponding migration instruction, and simultaneously reading the data of the hard disk to be migrated to the transfer space further includes:
responding to a plurality of hard disks to be migrated, and according to the migration priorities of the plurality of hard disks to be migrated, limiting migration instructions corresponding to the hard disks to be migrated to write data into the shared memory according to the height of the priorities;
responding to the idle shared memory or maintaining the data stock below a second preset size in preset time, and operating a migration instruction corresponding to the hard disk to be migrated according to the priority level to write data into the shared memory; and
and in response to the transmission bandwidths set for the plurality of hard disks to be migrated, controlling the speed of writing data into the shared memory by the instruction corresponding to the hard disk to be migrated according to the proportion of the transmission bandwidths of the hard disks to be migrated.
In this embodiment, when the number of hard disks to be migrated is multiple, the number or speed of writing data into the shared memory according to the migration instruction corresponding to the hard disk to be migrated may be limited according to the priority of the hard disk to be migrated.
In some embodiments of the present application, the migration instruction (refer to a program developed again based on open source code) writes data into the shared memory with a certain granularity, when the migration instruction with the highest priority writes data into the shared memory, other migration instructions cannot write data into the shared memory, because the data reading and writing speed of the memory is far higher than the data reading speed of the hard disk, the migration instruction with the highest priority does not occupy the shared memory all the time, after the data is written by the migration instruction with the highest priority, other migration instructions can write data into the shared memory, and at this time, once the migration instruction with the highest level or the migration instruction with the highest level can write data into the shared memory again, the secondary migration instruction currently written into the shared memory stops writing subsequent data (when the data with the current granularity can be written) into the shared memory. Specifically, for example, there are two migration instructions, where the priority of migration instruction 1 is higher than that of migration instruction 2, and after migration instruction 1 is written, migration instruction 2 can write data. In some cases, the migration instruction 2 accumulates a large amount of write data, and when the write authority of the shared memory is obtained (the migration instruction 1 has no data to write), the migration instruction 2 can continuously write data into the shared memory, but after the migration instruction 1 has a write condition (a certain data is read from the hard disk), the write authority of the migration instruction 2 is cancelled after the migration instruction 2 writes the corresponding granularity data, the migration instruction 1 continues to write data into the shared memory, and after the write is completed, the write authority of the migration instruction 2 is switched. Similarly, the same is true for more than 2 migration instructions.
In some embodiments of the present application, the granularity number of writing data into the shared memory by the corresponding migration instruction is further allocated in a quota allocation manner based on the reading speed of the hard disk to be migrated, the network bandwidth, the priority of the hard disk to be migrated, and the like.
Specifically, assuming that the data to be migrated is written into the shared memory 10MB each time by using the migration command as granularity, there are 3 hard disks to be migrated, the read speed of the first hard disk to be migrated is 500MB/S, the read speed of the second hard disk to be migrated is 400MB/S, the read speed of the third hard disk to be migrated is 500MB/S, meanwhile, the network bandwidth is still 1250MB/S provided by the trillion network card, the priority of the second hard disk to be migrated is higher than that of the third hard disk to be migrated, the priority of the third hard disk to be migrated is higher than that of the first hard disk to be migrated, the amount of data written into the shared memory by the migration command corresponding to the second hard disk to be migrated is not limited, the amount of data written into the shared memory by the migration command corresponding to the three hard disks to be migrated is 500MB each second, the amount of data written into the shared memory by the migration command corresponding to the first hard disk to be migrated is 1250-400-500 MB, and the data is limited to be migrated by the shared memory in a fixed amount, and the data to be migrated is allowed to be written into the shared memory in a certain mode, and the data to be migrated is allowed to be written into the shared memory every second (the highest priority is allowed).
In some embodiments of the present application, the obtaining, from the staging space, the data of the hard disk to be migrated and sending the data to a target storage device through a network includes:
and reading corresponding data from the shared memory and sending the data to target storage equipment corresponding to the data.
In this embodiment, the data transmitting program directly reads data from the shared memory (the storage structure with sequence is defined as described above) sequentially and transmits the data to the corresponding storage device through the network.
In some embodiments of the present application, the speed of the migration tool implemented according to the method in the above embodiments is compared with the conventional migration method as follows:
in practical test, the network bandwidth is ten meganets, the speed of copying disk data is about 12MB/s by using a traditional dd command, the speed of copying disk data is about 117MB/s by using a partclone. Dd command, and the speed is 8-10 times of the dd command. The speed of copying by using the partclone. Xfs command can reach 240MB/s because empty data blocks can be skipped, and the real data size in the disk can also change. It can be seen that this solution greatly improves the migration rate.
As shown in fig. 2, another aspect of the present application further provides an offline acceleration migration system, including:
the hard disk data reading module 1 is configured to acquire a storage format of at least one hard disk to be migrated, select a corresponding migration instruction according to the storage format of the hard disk to be migrated, and read the data of the hard disk to be migrated to a transfer space at the same time;
and the hard disk data sending module 2 is configured to obtain the data of the hard disk to be migrated from the transit space and send the data to the target storage device through a network.
As shown in fig. 3, a further aspect of the present application also proposes a computer device, including:
at least one processor 21; and
a memory 22, said memory 22 storing computer instructions 23 executable on said processor 21, said instructions 23 when executed by said processor 21 implementing the steps of any of the methods of the above embodiments.
As shown in fig. 4, a further aspect of the present application also proposes a computer readable storage medium 401, said computer readable storage medium 401 storing a computer program 402, which when executed by a processor implements the steps of the method according to any of the above embodiments.
According to the offline acceleration migration method provided by the application, data in the hard disk are migrated to the new storage device in a parallel mode, and the migration speed of the migrated plurality of hard disks is regulated and controlled by means of the transfer space. The problem that the migration tool copies the disk slowly is solved. The migration tool can intelligently identify the type of the disk file system and use the corresponding command to migrate the data, and the copying speed can almost reach the limit. Meanwhile, a scheme for regulating and controlling the migration speed or migration priority of different hard disks according to the requirement is also provided.
As described above, the implementation manner provided by the embodiment of the application can realize high-speed controllability of data migration under various platforms. The migration can be carried out by means of LiveCD, the migration can be carried out on hard disk data in a general mode, the flow equalization type parallel migration (without priority assignment) can be carried out during data migration, and the corresponding hard disk data priority migration can be selected in a priority assignment mode.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The foregoing embodiment of the present application has been disclosed with reference to the number of embodiments for the purpose of description only, and does not represent the advantages or disadvantages of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will appreciate that: the above discussion of any embodiment is merely exemplary and is not intended to imply that the scope of the disclosure of embodiments of the application, including the claims, is limited to such examples; combinations of features of the above embodiments or in different embodiments are also possible within the idea of an embodiment of the application, and there are many other variations of the different aspects of the embodiments of the application as described above, which are not provided in detail for the sake of brevity. Therefore, any omission, modification, equivalent replacement, improvement, etc. of the embodiments should be included in the protection scope of the embodiments of the present application.

Claims (7)

1. An offline acceleration migration method, comprising:
acquiring a storage format of at least one hard disk to be migrated, selecting a corresponding migration instruction according to the storage format of the hard disk to be migrated, and simultaneously reading data of the hard disk to be migrated to a transfer space;
acquiring data of the hard disk to be migrated from the transit space and sending the data to a target storage device through a network;
the selecting a corresponding migration instruction, and simultaneously reading the data of the hard disk to be migrated to the transfer space includes:
establishing different storage addresses for different migration instructions in a memory, and setting a target address of the migration instructions as the storage address corresponding to the migration instructions;
reading data from a hard disk to be migrated through a migration instruction matched with the hard disk to be migrated, and storing the data in the storage address corresponding to the migration instruction;
the step of obtaining the data of the hard disk to be migrated from the transfer space and sending the data to the target storage device through a network comprises the following steps:
acquiring the data of the hard disk to be migrated from the storage address with a first preset size and sending the data to the corresponding target storage device through a network;
responding to a plurality of storage addresses corresponding to the hard disks to be migrated, according to the migration priorities of the plurality of hard disks to be migrated, reading data from the storage addresses corresponding to the hard disks to be migrated according to the height of the priorities, and sending the data to a target storage device through a network; and
and in response to the network bandwidth being idle, reading data from the storage addresses corresponding to the hard disk to be migrated in sequence according to the priority and sending the data to target storage equipment through a network.
2. The method of claim 1, wherein selecting the corresponding migration instruction, and simultaneously reading the data of the hard disk to be migrated to the transfer space comprises:
and establishing a shared memory in the memory, and reading and storing the data of the hard disk to be migrated, which corresponds to the shared memory, into the shared memory through the migration instruction.
3. The method of claim 2, wherein selecting the corresponding migration instruction, and simultaneously reading the data of the hard disk to be migrated to the transit space further comprises:
responding to a plurality of hard disks to be migrated, and according to the migration priorities of the plurality of hard disks to be migrated, limiting migration instructions corresponding to the hard disks to be migrated to write data into the shared memory according to the height of the priorities;
responding to the idle shared memory or maintaining the data stock below a second preset size in preset time, and operating a migration instruction corresponding to the hard disk to be migrated according to the priority level to write data into the shared memory; and
and in response to the transmission bandwidths set for the plurality of hard disks to be migrated, controlling the speed of writing data into the shared memory by the instruction corresponding to the hard disk to be migrated according to the proportion of the transmission bandwidths of the hard disks to be migrated.
4. The method of claim 2, wherein the obtaining the data of the hard disk to be migrated from the staging space and sending the data to a target storage device over a network comprises:
and reading corresponding data from the shared memory and sending the data to target storage equipment corresponding to the data.
5. An offline acceleration migration system, comprising:
the hard disk data reading module is configured to acquire a storage format of at least one hard disk to be migrated, select a corresponding migration instruction according to the storage format of the hard disk to be migrated, and read the data of the hard disk to be migrated to a transfer space at the same time;
the hard disk data transmission module is configured to acquire the data of the hard disk to be migrated from the transfer space and transmit the data to the target storage device through a network;
the hard disk data reading module is further configured to:
establishing different storage addresses for different migration instructions in a memory, and setting a target address of the migration instructions as the storage address corresponding to the migration instructions;
reading data from a hard disk to be migrated through a migration instruction matched with the hard disk to be migrated, and storing the data in the storage address corresponding to the migration instruction;
the hard disk data transmitting module is further configured to:
acquiring the data of the hard disk to be migrated from the storage address with a first preset size and sending the data to the corresponding target storage device through a network;
responding to a plurality of storage addresses corresponding to the hard disks to be migrated, according to the migration priorities of the plurality of hard disks to be migrated, reading data from the storage addresses corresponding to the hard disks to be migrated according to the height of the priorities, and sending the data to a target storage device through a network; and
and in response to the network bandwidth being idle, reading data from the storage addresses corresponding to the hard disk to be migrated in sequence according to the priority and sending the data to target storage equipment through a network.
6. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, which when executed by the processor, perform the steps of the method of any one of claims 1-4.
7. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method of any of claims 1-4.
CN202210135354.1A 2022-02-14 2022-02-14 Offline acceleration migration method, system, equipment and storage medium Active CN114489517B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210135354.1A CN114489517B (en) 2022-02-14 2022-02-14 Offline acceleration migration method, system, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210135354.1A CN114489517B (en) 2022-02-14 2022-02-14 Offline acceleration migration method, system, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN114489517A CN114489517A (en) 2022-05-13
CN114489517B true CN114489517B (en) 2023-09-08

Family

ID=81479910

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210135354.1A Active CN114489517B (en) 2022-02-14 2022-02-14 Offline acceleration migration method, system, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114489517B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116340295B (en) * 2023-05-25 2023-09-12 北京长河数智科技有限责任公司 Data migration method and device based on multi-type data sources

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113448688A (en) * 2021-06-29 2021-09-28 华云数据控股集团有限公司 Method, system and computer medium for automatically correcting network configuration in cloud migration scene
WO2021259094A1 (en) * 2020-06-23 2021-12-30 中兴通讯股份有限公司 Method and apparatus for migration of virtual machine across cloud platform, and storage medium and electronic apparatus
CN113986119A (en) * 2021-09-30 2022-01-28 贵州电网有限责任公司 Simplified data reading method and system for accelerating cloud migration speed

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112835817A (en) * 2017-09-05 2021-05-25 华为技术有限公司 Data migration method and system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021259094A1 (en) * 2020-06-23 2021-12-30 中兴通讯股份有限公司 Method and apparatus for migration of virtual machine across cloud platform, and storage medium and electronic apparatus
CN113448688A (en) * 2021-06-29 2021-09-28 华云数据控股集团有限公司 Method, system and computer medium for automatically correcting network configuration in cloud migration scene
CN113986119A (en) * 2021-09-30 2022-01-28 贵州电网有限责任公司 Simplified data reading method and system for accelerating cloud migration speed

Also Published As

Publication number Publication date
CN114489517A (en) 2022-05-13

Similar Documents

Publication Publication Date Title
US20060047926A1 (en) Managing multiple snapshot copies of data
US7461201B2 (en) Storage control method and system for performing backup and/or restoration
JP2019101703A (en) Storage system and control software arrangement method
JP4532423B2 (en) Server system
JP2010097533A (en) Application migration and power consumption optimization in partitioned computer system
US20100131728A1 (en) Computer-readable recording medium storing data migration program, data migration method, and data migration apparatus
US11693684B2 (en) Information processing system and information processing method
EP1637987A2 (en) Operation environment associating data migration method
US10235282B2 (en) Computer system, computer, and method to manage allocation of virtual and physical memory areas
US9875057B2 (en) Method of live migration
JPH10283210A (en) Virtual machine movement control system between virtual machine systems
CN113296880A (en) Container-based application management method and device
CN103902233B (en) On-line storage moving method, device and the server of virtual machine system
CN112181736A (en) Distributed storage system and configuration method thereof
CN110119308B (en) System for managing large-scale container applications
JP2016115253A (en) Information processing device, memory management method and memory management program
CN114489517B (en) Offline acceleration migration method, system, equipment and storage medium
WO2018158808A1 (en) Information system, management program, and program exchanging method for information system
CN108475201A (en) A kind of data capture method in virtual machine start-up course and cloud computing system
JP6028415B2 (en) Data migration control device, method and system for virtual server environment
US11675545B2 (en) Distributed storage system and storage control method
CN107832097B (en) Data loading method and device
JP2005018509A (en) Storage system, backup system and backup method
EP2703990A2 (en) Information processing apparatus, computer program, and area release control method
US10509567B2 (en) System and method for migrating storage while in use

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant