CN114281267A - Data migration method and device between distributed storage systems - Google Patents
Data migration method and device between distributed storage systems Download PDFInfo
- Publication number
- CN114281267A CN114281267A CN202111653977.XA CN202111653977A CN114281267A CN 114281267 A CN114281267 A CN 114281267A CN 202111653977 A CN202111653977 A CN 202111653977A CN 114281267 A CN114281267 A CN 114281267A
- Authority
- CN
- China
- Prior art keywords
- data
- distributed storage
- storage system
- disk
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 67
- 238000013508 migration Methods 0.000 title claims abstract description 65
- 230000005012 migration Effects 0.000 title claims abstract description 65
- 230000008569 process Effects 0.000 claims abstract description 38
- 238000011084 recovery Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 abstract description 25
- 230000008676 import Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 239000004744 fabric Substances 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 4
- 238000009795 derivation Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000008570 general process Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and a device for data migration between distributed storage systems, wherein the method comprises the following steps: a first data migration process, the first data migration process comprising: the client sends a request to the metadata node to acquire metadata information stored in the metadata node, and sends a broadcast message to the first distributed storage system to acquire data related information stored in a disk on each target server; the client sends the writing request and the writing position information to corresponding data nodes; the writing position information comprises a corresponding target server, a corresponding disk on the corresponding target server and an offset on the corresponding disk; and the data node writes the data block file into a corresponding position in the first distributed storage system according to the writing position information, and returns a writing completion message to the client after the writing is completed. The invention can realize high-concurrency data transmission.
Description
Technical Field
One or more embodiments of the present disclosure relate to the field of distributed storage technologies, and in particular, to a method and an apparatus for migrating data between distributed storage systems.
Background
For a conventional distributed storage system, a file system used by a client has very powerful functions and complex index logic, such as a file system ext4, and only a part of functions in the file system, for example, simple operations such as reading and writing, are needed in a data migration process between the distributed storage systems, so that the file system may generate some unnecessary index overhead to affect the read-write performance of data.
At present, no uniform solution exists for the problem of data migration between distributed file systems. Now, data migration between distributed storage systems is consistent with a common data access mode, referring to fig. 1, a client accesses a metadata server (i.e., NameNode), obtains distribution of data blocks, reads data from each data node (i.e., DataNode), and stores the data locally. The whole process needs to consume a large amount of time and depends on the data reading and writing performance of the file system of the Client side. The conventional method cannot guarantee the reliability of data migration and high-speed transmission of data.
Disclosure of Invention
One or more embodiments of the present specification describe a method and apparatus for data migration between distributed storage systems.
According to a first aspect, a method for data migration between distributed storage systems is provided, where the method is applied to data migration between a first distributed storage system and a second distributed storage system, the first distributed storage system includes at least one target server, each target server is provided with at least one disk, and the second distributed storage system includes metadata nodes and at least one data node; the method comprises the following steps: the migration direction is a first data migration process from the second distributed storage system to the first distributed storage system, wherein the first data migration process includes:
the client sends a request to the metadata node to acquire metadata information stored in the metadata node, and sends a broadcast message to the first distributed storage system to acquire data related information stored in a disk on each target server; the metadata information comprises the identification of each data block file stored on each data node;
the client distributes writing position information for the data block file in the data node according to the metadata information and the data related information, and sends a writing request and the writing position information to the corresponding data node; the writing position information comprises a corresponding target server, a corresponding disk on the corresponding target server and an offset on the corresponding disk;
and the data node writes the data block file into a corresponding position in the first distributed storage system according to the writing position information, and returns a writing completion message to the client after the writing is completed.
According to a second aspect, a data migration apparatus between distributed storage systems is provided, where the apparatus is applied to data migration between a first distributed storage system and a second distributed storage system, the first distributed storage system includes at least one target server, each target server is provided with at least one disk, and the second distributed storage system includes metadata nodes and at least one data node; the device comprises a client, a first data migration module and a second data migration module, wherein the client is used for executing a first data migration process with a migration direction from the second distributed storage system to the first distributed storage system; the client is specifically configured to: sending a request to the metadata node to obtain metadata information stored in the metadata node, and sending a broadcast message to the first distributed storage system to obtain data related information stored in a disk on each target server; distributing writing position information for data block files in the data nodes according to the metadata information and the data related information, sending a writing request and the writing position information to the corresponding data nodes, so that the data nodes write the data block files to corresponding positions in the first distributed storage system according to the writing position information, and returning a writing completion message to the client after the writing is completed;
the metadata information comprises the identification of each data block file stored on each data node; the writing position information includes a corresponding target server, a corresponding disk on the corresponding target server, and an offset on the corresponding disk.
In the data migration process from the second distributed storage system to the first distributed storage system, the client sends a request to the metadata node to acquire metadata information, and sends a broadcast message to the first distributed storage system to acquire data related information; and then distributing writing position information for the data block file in the data node according to the metadata information and the data related information, and sending a writing request and the writing position information to the corresponding data node, so that the data node directly writes the data block file into the corresponding position in the first distributed storage system according to the writing position information. Because the data transmission between a plurality of data nodes and a plurality of disks can be realized, high-concurrency data transmission is realized, and the transmission rate can be greatly improved.
Drawings
In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present specification, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic diagram of data migration between distributed storage systems in the prior art;
fig. 2 is a system architecture diagram corresponding to a data migration method between distributed storage systems according to an embodiment of the present invention;
FIG. 3 is a general flow diagram of data export from an HDFS cluster to an HSXport cluster in an embodiment of the present invention;
FIG. 4 is a flowchart illustrating exporting data from an HDFS cluster to an HSXport cluster according to an embodiment of the present invention;
FIG. 5 is a general flow diagram of data export from an HSXport cluster to an HDFS cluster in an embodiment of the present invention;
FIG. 6 is a flowchart illustrating exporting data from an HSXport cluster to an HDFS cluster according to an embodiment of the present invention;
FIG. 7 is a flow chart of initializing an HSXport cluster in an embodiment of the present invention;
FIG. 8 is a schematic illustration of information stored on a disk in an embodiment of the present invention;
fig. 9 is a schematic diagram of a correspondence relationship of data blocks between a plurality of data volume portions and one file in the embodiment of the present invention.
Detailed Description
The scheme provided by the specification is described below with reference to the accompanying drawings.
The scenarios in which data is migrated in different distributed storage systems include data import, data export, data transfer, and the like. Wherein:
data import: mass data generated by a large scientific instrument or a vehicle-mounted system needs to be imported into a distributed storage system at one time. The data volume of large scientific instruments, such as wind tunnel experiments, generated once can reach TB level, and the accumulated data of a vehicle-mounted system is hundreds of GB.
And (4) data export, namely, data accumulated by the Internet enterprises through methods of crawler, log analysis and the like are generated at a low speed, but the accumulated data is large in quantity. This data is slow to export to other systems for backup or analysis.
Data transmission: even though the network speed has been increased much now, the data transmission between cities still takes a long time. Taking 4TB data as an example, when the transmission rate reaches 5MB/s, it still takes about 10 days for the transmission to complete.
Aiming at various application scenes, the embodiment of the invention provides a data migration method among distributed storage systems. The method is applied to data migration between a first distributed storage system and a second distributed storage system, wherein the first distributed storage system comprises at least one target server, and each target server is provided with at least one disk; the second distributed storage system comprises metadata nodes and at least one data node.
Referring to fig. 2, a system architecture corresponding to the method provided by the embodiment of the present invention, specifically, a client, an HDFS cluster, and an Import/Export cluster, which are connected through an RDMA-supporting network. The HDFS cluster is a second distributed storage system composed of a NameNode and a DataNode, the Import/Export cluster is a first distributed storage system formed by a plurality of target servers, a disk SSD is arranged on each target server, and the cluster can also be called an HSXport cluster.
The method provided by the embodiment of the invention comprises the following steps: a first data migration process, wherein a migration direction of the first data migration process is from the second distributed storage system to the first distributed storage system.
Wherein the first data migration process comprises:
s1, the client sends a request to the metadata node to obtain the metadata information stored in the metadata node, and sends a broadcast message to the first distributed storage system to obtain the data related information stored in the disk on each target server; the metadata information comprises the identification of each data block file stored on each data node;
s2, the client distributes writing position information for the data block file in the data node according to the metadata information and the data related information, and sends a writing request and the writing position information to the corresponding data node; the writing position information comprises a corresponding target server, a corresponding disk on the corresponding target server and an offset on the corresponding disk;
and S3, the data node writes the data block file into a corresponding position in the first distributed storage system according to the writing position information, and returns a writing completion message to the client after the writing is completed.
Referring to fig. 3, a schematic flow chart of a first data migration process is shown, that is, data is exported from an HDFS cluster to an HSXport cluster, that is, data is exported from the HDFS cluster to a storage medium, that is, a disk, in the HSXport cluster that supports NVMe over fabric. The HDFS cluster and the HSXport cluster are connected via an RDMA enabled network. The Client is responsible for carrying out data communication among the clusters, writing position information of a data block file stored in a data node in the HDFS cluster at present is calculated, then the writing position information is sent to the DataNodes when the Client communicates with the DataNodes, each DataNode directly writes own node data into a corresponding position of a corresponding disk in a corresponding target server according to the writing position information by using an NVMe over fabric technology, high-concurrency high-speed derivation of the data is realized, and high-concurrency high-speed derivation can be realized because a plurality of data nodes can simultaneously write data into a plurality of disks.
Fig. 4 is a detailed flowchart of the first data migration process. The sequential steps of exporting data from an HDFS cluster to an HSXport cluster are depicted in the flowchart. The Client is responsible for establishing the connection between the HDFS cluster and the HSXport cluster, but data transmission is many-to-many in the data transmission process, so that high concurrency is realized, the bandwidth of a single node is prevented from becoming a bottleneck in the large-capacity data transmission process, and high-speed data import and export operations are realized. The process of exporting data to the HSXport cluster:
(1) the Client sends a message to the NameNode in the Hadoop cluster, and the NameNode returns required information to the Client after receiving the message. The specific message includes metadata information, where the metadata information includes an identifier of each data block file stored in each data node, that is, it can be known which data block files are stored on each data node according to the metadata information.
(2) The Client sends broadcast messages to each target server in the HSXport cluster, and the target servers in the HSXport cluster return required messages to the Client. The specific message includes data related information, and the data related information includes cluster information and the like.
(3) The Client allocation data block file is to be written in the location information in the HSXport cluster, i.e. the above-mentioned writing location information. The write location information includes which target server is written, which disk on the target server, and what location on the disk.
(4) And the Client informs each data node of performing writing operation, so that the data node writes the data block file stored by the data node into a corresponding position in the HSXport cluster. Specifically, each DataNode uses NVMeoverfabric technology to write many-to-many files, and transmits corresponding blk files to the designated position of the HSXport cluster, where many-to-many means that multiple data nodes correspond to multiple disks.
(5) And each DataNode node sends a message to the Client after the data transmission is finished, and informs the Client that the data transmission is finished.
In specific implementation, the method provided in the embodiment of the present invention may further include: a second data migration process, wherein a migration direction of the second data migration process is from the first distributed storage system to the second distributed storage system.
Wherein the second data migration process may include:
s4, the client acquires the data related information from each disk in the first distributed storage system, determines the storage position of each data block file according to the data related information, takes the storage position as reading position information, acquires a corresponding writable data node list from the metadata node aiming at each data block file, allocates a data block file to be read for each data node in the writable data node list, and sends the reading position information of the allocated data block file to the data node;
and S5, the data node reads the data block file from the first distributed storage system according to the reading position information, stores the data block file into the node of the data node, and returns a reading completion message to the client after the reading is completed.
That is, the client acquires data-related information, which is some descriptive description or the like of the data information on the disk, from each disk on each target server. After the number information is acquired, the information of which data blocks are stored in each disk, the specific positions of the data blocks in the disk and the like can be known, and the specific position information is used as the reading position information. Meanwhile, the client side can also obtain a writable data node list from the metadata nodes, the writable data node list comprises data nodes capable of performing writing operation, and the data nodes have free space for storing data block files. After obtaining the writable data node list, the client may allocate a corresponding data block file to each writable data node, and then send the corresponding data block file and the read location information in the disk to the writable data node, where the writable data node may perform data read operation.
Fig. 5 is a schematic flow chart of importing data from an HSXport cluster to an HDFS cluster according to an embodiment of the present invention, and describes a schematic flow of importing data from a storage medium (i.e., a disk) supporting an NVMe over fabric into an HDFS cluster. The HDFS cluster and the HSXport cluster are connected via an RDMA enabled network. The Client is responsible for data communication among the clusters, the Client obtains a writable data node list through communication with the NameNode, and the Client informs the position information of a blk file to be written in the HSXport cluster in the communication with the DataNode node. Each DataNode directly writes the blk file at the corresponding position in the corresponding disk on the corresponding target server in the HSXport cluster into the node by using the NVMe over fabric technology, thereby realizing data migration between a plurality of data nodes and a plurality of disks and realizing high-concurrency and high-speed import.
As shown in fig. 6, a sequence step of importing data from an HSXport cluster to an HDFS diversity cluster in the embodiment of the present invention is described, and a Client is responsible for establishing a connection between the HDFS cluster and the HSXport cluster, and performs many-to-many data transmission during data transmission, so as to implement high concurrency, avoid bandwidth of a single node becoming a bottleneck during large-capacity data transmission, and implement high-speed data import and export operations. The process of importing data into the HDFS cluster is as follows:
(1) the Client sends a file uploading request to a NameNode in the Hadoop cluster, the NameNode returns files and corresponding metadata information, and the Client can know which data block files are currently stored in which data node according to the metadata information;
(2) the Client sends a request to the NameNode aiming at a first blk file in the HSXport cluster, and the NameNode returns a writable data node list;
(3) the Client acquires the data related information from each disk in the first distributed storage system to determine the storage position of each blk file, uses the storage position as reading position information, then allocates blk files to be read for each data node in the writable data node list, and sends the reading position information of the blk files to be read by each data node to the data node;
(4) each DataNode node writes many-to-many files by using NVMeoverfabric technology, and transmits the corresponding blk file to the node from the designated position of the HSXport cluster; and each DataNode node sends a message to the Client after the data transmission is finished, and informs the Client that the data transmission is finished.
In specific implementation, before executing the first or second data migration process, the method provided in the embodiment of the present invention may further include: and initializing the first distributed storage system, wherein the initialization process is a process of writing data related information into a disk.
Wherein the initialization process comprises: traversing each disk on each target server, and if preset zone bit information is stored in a preset position of one disk, adding the disk into the first distributed storage system; and determining corresponding data related information for each disk added into the first distributed storage system, formatting each disk, and writing the data related information into the corresponding disk after formatting is completed.
It can be understood that a plurality of disks can be arranged on one target server, some disks have preset zone bit information at preset positions, some disks have no preset zone bit information at the preset positions, and only the disks with the preset zone bit information at the preset positions can be formatted and then added into the first distributed storage system. That is, not all the disks on the target server belong to the first distributed storage system, and only the disk having the predetermined flag bit information at the predetermined position belongs to the first distributed storage system. Formatting is performed for all disks added to the first distributed storage system, and after formatting, data-related information may be written to the disks.
Referring to fig. 7, the above initialization process, i.e. the flow chart of initializing the HSXport cluster, i.e. how to construct the nodes meeting the requirement as an HSXport cluster, is described in detail. The HSXport cluster bottom layer uses NVMe over Fabric technology-enabled storage media and connects with the Hadoop cluster over an RDMA-enabled network (e.g., InfiniBand network). The general process comprises the following steps:
(1) firstly, traversing each disk on each target server, and acquiring whether preset zone bit information exists at a preset position of the disk or not by traversing information on the disk, if so, determining that the disk can be formatted, and adding the disk into a disk list of a cluster;
(2) calculating cluster deployment information, namely actually calculating the capacity, available space and other information of a disk to obtain data related information;
(3) and formatting each disk in the disk list, and writing data related information into the disk after formatting.
In a specific implementation, after initialization, referring to fig. 8, each disk in the first distributed storage system may include: the data related information comprises cluster information and file information; wherein:
the cluster information comprises preset zone bit information, a disk list formed by disks added into the first distributed storage system and free space of each data volume part;
the file information comprises a corresponding relation between at least one file and a corresponding data block file, and the corresponding relation of one file comprises each data block file forming the file and a disk where each data block file is located;
the data information includes at least one data volume portion, each data volume portion for storing at least one data block file.
Understandably, the HSXport cluster is for more efficient storage and reading and writing of block data files in the HDFS cluster. The information in the magnetic disk mainly comprises two parts: data-related information and data information, and the data-related information includes two parts: cluster information (i.e., HSXportInfo), and file information (i.e., FileInfo), and the data information includes a plurality of data volume parts (i.e., HSXportVolume).
The cluster information (i.e. HSXportInfo) stores the relevant information of the cluster, including magic (i.e. preset flag bit information), and when the position of the disk is detected to have corresponding information, it means that the disk belongs to the cluster, and the disk can be formatted and subjected to other operations such as reading, writing, deleting and the like; the cluster information also includes disk _ list information, which records the SN serial number of the hard disk belonging to the HSXport cluster, and is used for operations such as information reading, cluster data space planning, and the like. The cluster information may also include a free space for recording each data volume portion, and the information may be recorded in RAIDInfo, which facilitates searching during write-delete operation.
The File information (i.e., File Info) records a correspondence between one File and a plurality of blk files, and usually, one File corresponds to a plurality of blk files in different hard disks, and in addition, directory structure information such as a File path is also recorded.
The data volume portion (i.e., HSXportVolume) mainly stores blk files, and the portion is the data itself.
Referring to fig. 9, a mapping diagram between HSXport volume and File is described, where the mapping diagram describes a correspondence relationship between data blocks between multiple data volume portions and a File, and it can be seen that: a single file typically consists of multiple blk files; different blk files of a single file may be distributed across different hsxportvolumes.
In a specific implementation, the cluster information further includes a Redundant Array of Independent Disks (RAID) type, where the RAID type is stored in RAIDInfo, meaning what RAID type is used to ensure data reliability, for example, RAID 5. Correspondingly, at least one data volume part of the data information comprises an operation block file, and the operation block file is a file obtained by performing preset processing on at least one data block file; and the operation block file is used for carrying out data recovery on the lost block according to the type of the independent redundant disk array when the data block file is lost.
The operation block files are p1 and p2 in fig. 9, and since p1 and p2 are obtained by operating a plurality of data blocks, when one data block is lost, recovery can be performed, specifically, data recovery is performed on the lost block according to the RAID5 type. That is, when a single blk file loss occurs, information of the damaged data block can be recovered using the remaining normal block corresponding data information.
It can be understood that the embodiment of the present invention provides a high-concurrency data importing/exporting method suitable for a distributed storage system, which is a new data transmission mechanism, and after data no longer passes through a local file system of a client, the client acquires relevant information and then notifies each data node to directly perform a write-in or read-out operation with respect to a disk, thereby implementing high-concurrency processing of multiple nodes and multiple disks.
The first distributed storage system may implement a required function through an interface provided by a user space file system Fuse (i.e., filesystem in userspace). When the client controls data transmission between the first distributed storage system and the second distributed storage system, compared with the traditional distributed file system Ext4, the client has no redundant function and index logic and only aims at efficient storage and reading of block data in the distributed storage system.
The HSXport is a storage system for storing block data files only, and cannot store files or directories other than blk. And when the disk is formatted, the disk is logically partitioned according to the index relation designed by the HSXport, and the disk information is stored in the super block. The Fuse operates the blk file as a whole when reading or writing the file, and the client determines whether the file is already present when writing or reading the file, for example, additional writing is available when writing, and data is directly read when reading the file.
It can be understood that, in the embodiment of the present invention, the data node is used to directly write into the disk, which reduces unnecessary overhead, and is particularly applied to the transmission of data between the disk and the data node on the HSXport cluster. When data is transmitted between a disk of the HSXport cluster and a DataNode, the DataNode can directly write the data into a corresponding position in the disk by using NVMe over Fabric technology, or read the data from the corresponding position of the disk, so as to realize the import and export of the data. Wherein, the position of the data on the disk is uniformly managed by the HSXport.
In the method provided by the embodiment of the invention, in the data migration process from the second distributed storage system to the first distributed storage system, the client sends a request to the metadata node to acquire metadata information, and sends a broadcast message to the first distributed storage system to acquire data related information; and then distributing writing position information for the data block file in the data node according to the metadata information and the data related information, and sending a writing request and the writing position information to the corresponding data node, so that the data node directly writes the data block file into the corresponding position in the first distributed storage system according to the writing position information. Because the data transmission between a plurality of data nodes and a plurality of disks can be realized, high-concurrency data transmission is realized, and the transmission rate can be greatly improved.
In a second aspect, an embodiment of the present invention provides a data migration apparatus between distributed storage systems, where the apparatus is applied to data migration between a first distributed storage system and a second distributed storage system, where the first distributed storage system includes at least one target server, and each target server is provided with at least one disk; the second distributed storage system comprises metadata nodes and at least one data node;
the data migration device comprises a client, wherein the client is used for executing a first data migration process, and the migration direction of the first data migration process is from the second distributed storage system to the first distributed storage system; the client is specifically configured to: sending a request to the metadata node to obtain metadata information stored in the metadata node, and sending a broadcast message to the first distributed storage system to obtain data related information stored in a disk on each target server; distributing writing position information for data block files in the data nodes according to the metadata information and the data related information, sending a writing request and the writing position information to the corresponding data nodes, so that the data nodes write the data block files to corresponding positions in the first distributed storage system according to the writing position information, and returning a writing completion message to the client after the writing is completed; the metadata information comprises the identification of each data block file stored on each data node; the writing position information includes a corresponding target server, a corresponding disk on the corresponding target server, and an offset on the corresponding disk.
In some embodiments, the client is further configured to perform a second data migration process, wherein a migration direction of the second data migration process is from the first distributed storage system to the second distributed storage system; wherein the client is specifically configured to: the data related information is obtained from each disk in the first distributed storage system, the storage position of each data block file is determined according to the data related information, the storage position is used as reading position information, a corresponding writable data node list is obtained from the metadata node aiming at each data block file, the data block file to be read is distributed to each data node in the writable data node list, the reading position information of the distributed data block file is sent to the data node, so that the data node reads the data block file from the first distributed storage system according to the reading position information and stores the data block file into the self node, and a message of reading completion is returned to the client after the reading is completed.
In some embodiments, the client, prior to performing the learning migration process, is further to: initializing the first distributed storage system; the client is specifically configured to: traversing each disk on each target server, and if preset zone bit information is stored in a preset position of one disk, adding the disk into the first distributed storage system; and determining corresponding data related information for each disk added into the first distributed storage system, formatting each disk, and writing the data related information into the corresponding disk after formatting is completed.
In some embodiments, after initialization, each disk in the first distributed storage system comprises: the data related information comprises cluster information and file information; wherein:
the cluster information comprises preset zone bit information, a disk list formed by disks added into the first distributed storage system and free space of each data volume part;
the file information comprises a corresponding relation between at least one file and a corresponding data block file, and the corresponding relation of one file comprises each data block file forming the file and a disk where each data block file is located;
the data information includes at least one data volume portion, each data volume portion for storing at least one data block file.
In some embodiments, the cluster information further includes a redundant array of independent disks type; at least one data volume part of the data information comprises an operation block file, wherein the operation block file is obtained by performing preset processing on at least one data block file; and the operation block file is used for carrying out data recovery on the lost block according to the type of the independent redundant disk array when the data block file is lost.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
Those skilled in the art will recognize that, in one or at least one of the examples described above, the functions described in this disclosure may be implemented in hardware, software, hardware, or any combination thereof. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium.
The above-mentioned embodiments, objects, technical solutions and advantages of the present invention are further described in detail, it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made on the basis of the technical solutions of the present invention should be included in the scope of the present invention.
Claims (10)
1. The method is applied to data migration between a first distributed storage system and a second distributed storage system, wherein the first distributed storage system comprises at least one target server, each target server is provided with at least one disk, and the second distributed storage system comprises metadata nodes and at least one data node; the method comprises the following steps: the migration direction is a first data migration process from the second distributed storage system to the first distributed storage system, wherein the first data migration process includes:
the client sends a request to the metadata node to acquire metadata information stored in the metadata node, and sends a broadcast message to the first distributed storage system to acquire data related information stored in a disk on each target server; the metadata information comprises the identification of each data block file stored on each data node;
the client distributes writing position information for the data block file in the data node according to the metadata information and the data related information, and sends a writing request and the writing position information to the corresponding data node; the writing position information comprises a corresponding target server, a corresponding disk on the corresponding target server and an offset on the corresponding disk;
and the data node writes the data block file into a corresponding position in the first distributed storage system according to the writing position information, and returns a writing completion message to the client after the writing is completed.
2. The method of claim 1,
the method further comprises the following steps: the migration direction is a second data migration process from the first distributed storage system to the second distributed storage system, wherein the second data migration process includes:
the client acquires the data related information from each disk in the first distributed storage system, determines the storage position of each data block file according to the data related information, takes the storage position as reading position information, acquires a corresponding writable data node list from the metadata node aiming at each data block file, allocates a data block file to be read for each data node in the writable data node list, and sends the reading position information of the allocated data block file to the data node;
and the data node reads the data block file from the first distributed storage system according to the reading position information, stores the data block file into the node of the data node, and returns a reading completion message to the client after the reading is completed.
3. The method according to claim 1 or 2,
before performing the learning migration process, the method further comprises: initializing the first distributed storage system, the initialization process comprising:
traversing each disk on each target server, and if preset zone bit information is stored in a preset position of one disk, adding the disk into the first distributed storage system;
and determining corresponding data related information for each disk added into the first distributed storage system, formatting each disk, and writing the data related information into the corresponding disk after formatting is completed.
4. The method of claim 3, wherein after initializing, each disk in the first distributed storage system comprises: the data related information comprises cluster information and file information; wherein:
the cluster information comprises preset zone bit information, a disk list formed by disks added into the first distributed storage system and free space of each data volume part;
the file information comprises a corresponding relation between at least one file and a corresponding data block file, and the corresponding relation of one file comprises each data block file forming the file and a disk where each data block file is located;
the data information includes at least one data volume portion, each data volume portion for storing at least one data block file.
5. The method of claim 4, wherein the cluster information further comprises a redundant array of independent disks type; at least one data volume part of the data information comprises an operation block file, wherein the operation block file is obtained by performing preset processing on at least one data block file; and the operation block file is used for carrying out data recovery on the lost block according to the type of the independent redundant disk array when the data block file is lost.
6. The device is applied to data migration between a first distributed storage system and a second distributed storage system, wherein the first distributed storage system comprises at least one target server, each target server is provided with at least one disk, and the second distributed storage system comprises metadata nodes and at least one data node; the device comprises a client, a first data migration module and a second data migration module, wherein the client is used for executing a first data migration process with a migration direction from the second distributed storage system to the first distributed storage system; the client is specifically configured to: sending a request to the metadata node to obtain metadata information stored in the metadata node, and sending a broadcast message to the first distributed storage system to obtain data related information stored in a disk on each target server; distributing writing position information for data block files in the data nodes according to the metadata information and the data related information, sending a writing request and the writing position information to the corresponding data nodes, so that the data nodes write the data block files to corresponding positions in the first distributed storage system according to the writing position information, and returning a writing completion message to the client after the writing is completed;
the metadata information comprises the identification of each data block file stored on each data node; the writing position information includes a corresponding target server, a corresponding disk on the corresponding target server, and an offset on the corresponding disk.
7. The apparatus of claim 6,
the client is further used for executing a second data migration process with the migration direction from the first distributed storage system to the second distributed storage system; wherein the client is specifically configured to: the data related information is obtained from each disk in the first distributed storage system, the storage position of each data block file is determined according to the data related information, the storage position is used as reading position information, a corresponding writable data node list is obtained from the metadata node aiming at each data block file, the data block file to be read is distributed to each data node in the writable data node list, the reading position information of the distributed data block file is sent to the data node, so that the data node reads the data block file from the first distributed storage system according to the reading position information and stores the data block file into the self node, and a message of reading completion is returned to the client after the reading is completed.
8. The apparatus according to claim 6 or 7,
the client, before performing the learning migration process, is further configured to: initializing the first distributed storage system; the client is specifically configured to: traversing each disk on each target server, and if preset zone bit information is stored in a preset position of one disk, adding the disk into the first distributed storage system; and determining corresponding data related information for each disk added into the first distributed storage system, formatting each disk, and writing the data related information into the corresponding disk after formatting is completed.
9. The apparatus of claim 8, wherein after initializing, each disk in the first distributed storage system comprises: the data related information comprises cluster information and file information; wherein:
the cluster information comprises preset zone bit information, a disk list formed by disks added into the first distributed storage system and free space of each data volume part;
the file information comprises a corresponding relation between at least one file and a corresponding data block file, and the corresponding relation of one file comprises each data block file forming the file and a disk where each data block file is located;
the data information includes at least one data volume portion, each data volume portion for storing at least one data block file.
10. The apparatus of claim 9, wherein the cluster information further comprises a redundant array of independent disks type; at least one data volume part of the data information comprises an operation block file, wherein the operation block file is obtained by performing preset processing on at least one data block file; and the operation block file is used for carrying out data recovery on the lost block according to the type of the independent redundant disk array when the data block file is lost.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111653977.XA CN114281267B (en) | 2021-12-30 | 2021-12-30 | Data migration method and device between distributed storage systems |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111653977.XA CN114281267B (en) | 2021-12-30 | 2021-12-30 | Data migration method and device between distributed storage systems |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114281267A true CN114281267A (en) | 2022-04-05 |
| CN114281267B CN114281267B (en) | 2024-04-26 |
Family
ID=80879123
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111653977.XA Active CN114281267B (en) | 2021-12-30 | 2021-12-30 | Data migration method and device between distributed storage systems |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114281267B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150324371A1 (en) * | 2013-12-31 | 2015-11-12 | Huawei Technologies Co., Ltd. | Data Processing Method and Device in Distributed File Storage System |
| CN105630808A (en) * | 2014-10-31 | 2016-06-01 | 北京奇虎科技有限公司 | Distributed file system based file reading and writing method and node server |
| US20180285263A1 (en) * | 2017-04-02 | 2018-10-04 | Sas Institute Inc. | Techniques for Reading From and Writing to Distributed Data Stores |
| CN111078121A (en) * | 2018-10-18 | 2020-04-28 | 深信服科技股份有限公司 | Data migration method, system and related components of distributed storage system |
| CN111444157A (en) * | 2019-01-16 | 2020-07-24 | 阿里巴巴集团控股有限公司 | Distributed file system and data access method |
| CN113467722A (en) * | 2021-07-26 | 2021-10-01 | 浙江大华技术股份有限公司 | Data migration method and device of distributed storage system |
-
2021
- 2021-12-30 CN CN202111653977.XA patent/CN114281267B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150324371A1 (en) * | 2013-12-31 | 2015-11-12 | Huawei Technologies Co., Ltd. | Data Processing Method and Device in Distributed File Storage System |
| CN105630808A (en) * | 2014-10-31 | 2016-06-01 | 北京奇虎科技有限公司 | Distributed file system based file reading and writing method and node server |
| US20180285263A1 (en) * | 2017-04-02 | 2018-10-04 | Sas Institute Inc. | Techniques for Reading From and Writing to Distributed Data Stores |
| CN111078121A (en) * | 2018-10-18 | 2020-04-28 | 深信服科技股份有限公司 | Data migration method, system and related components of distributed storage system |
| CN111444157A (en) * | 2019-01-16 | 2020-07-24 | 阿里巴巴集团控股有限公司 | Distributed file system and data access method |
| CN113467722A (en) * | 2021-07-26 | 2021-10-01 | 浙江大华技术股份有限公司 | Data migration method and device of distributed storage system |
Non-Patent Citations (1)
| Title |
|---|
| 王伟;: "分布式文件存储在跨局址数据备份和数据迁移中的设计与应用", 数字通信世界, no. 05, 1 May 2018 (2018-05-01) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114281267B (en) | 2024-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110554834B (en) | File system data access method and file system | |
| CN106407040B (en) | A kind of duplicating remote data method and system | |
| CN105144142B (en) | Use the information storage object of the multiple dimensions of object | |
| US6571351B1 (en) | Tightly coupled secondary storage system and file system | |
| US8560879B1 (en) | Data recovery for failed memory device of memory device array | |
| CN106951375B (en) | Method and device for deleting snapshot volume in storage system | |
| CN103761190B (en) | Data processing method and apparatus | |
| CN107832423B (en) | File reading and writing method for distributed file system | |
| CN111177143B (en) | Key value data storage method and device, storage medium and electronic equipment | |
| CN108108476A (en) | The method of work of highly reliable distributed information log system | |
| US11385823B2 (en) | Method, electronic device and computer program product for rebuilding disk array | |
| CN111309245B (en) | Hierarchical storage writing method and device, reading method and device and system | |
| CN110147203A (en) | A kind of file management method, device, electronic equipment and storage medium | |
| CN103514222B (en) | Storage method, management method, memory management unit and the system of virtual machine image | |
| US20170277451A1 (en) | Method to limit impact of partial media failure of disk drive and detect/report the loss of data for objects due to partial failure of media | |
| CN110134551B (en) | Continuous data protection method and device | |
| CN113885809B (en) | Data management system and method | |
| CN111435286B (en) | Data storage method, device and system | |
| CN113377292B (en) | Single machine storage engine | |
| US20090292882A1 (en) | Storage area network server with parallel processing cache and access method thereof | |
| CN109582235B (en) | Management metadata storage method and device | |
| CN114281267B (en) | Data migration method and device between distributed storage systems | |
| CN111913664B (en) | Data writing method and device | |
| CN114327283B (en) | Data migration method in distributed storage system and distributed storage system | |
| CN111752913B (en) | Data recovery method, medium, computer equipment and device of distributed system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |