CN113821176A - Data migration processing method and device and storage medium - Google Patents
Data migration processing method and device and storage medium Download PDFInfo
- Publication number
- CN113821176A CN113821176A CN202111154480.3A CN202111154480A CN113821176A CN 113821176 A CN113821176 A CN 113821176A CN 202111154480 A CN202111154480 A CN 202111154480A CN 113821176 A CN113821176 A CN 113821176A
- Authority
- CN
- China
- Prior art keywords
- data
- storage unit
- fault
- file
- migration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1471—Saving, restoring, recovering or retrying involving logging of persistent data for recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a processing method for data migration, which comprises the following steps: detecting the states of a plurality of DN data nodes, and when a storage unit A in a DN data node A in the DN data nodes fails, restoring a fault file to a data recovery position according to erasure information of a block file of the fault file in the storage unit A; wherein, the data recovery position is determined according to a preset recovery strategy; and if the fault of the fault unit A is recovered, migrating the file recovered to the data recovery position back to a data migration position to complete the migration of all the data, wherein the data migration position is set as the fault unit A. In the invention, when data migration is carried out, the data migration is carried out according to the position with the highest reliability of each initially distributed block file, and writing operation can be carried out in the migration process, so that the maximum reliability of the stored data is greatly improved while the load balance of the storage system is ensured.
Description
Technical Field
The invention belongs to the technical field of computers, and particularly relates to a data migration processing method, a data migration processing device and a storage medium.
Background
When the storage system is in use, the storage unit is inevitably failed, and when the storage unit is failed, the storage system can restore the data of the failed disk. During the period from the recovery of the data of the failed disk to the replacement of the failed disk by the new disk, the data of the failed disk is inevitably recovered to other data nodes (servers storing data, hereinafter abbreviated as DN) or other storage units of the present DN, which causes the data of the whole storage system to become unbalanced between the DN and between the storage units and to reduce the reliability.
The conventional data migration method at present basically determines an original server of migrated data, then determines a target server of migration and correspondingly conforms to the migrated data, but the migration process cannot or needs to adopt time delay to modify the data being migrated, and the balance and reliability of the migrated data are not guaranteed, which has certain disadvantages more or less.
Disclosure of Invention
In view of the above drawbacks of the prior art, the present invention provides a data migration processing method, apparatus and storage medium to solve the problem that it is not possible or necessary to modify the data being migrated by using a delay in the data migration process.
In order to achieve the above objects and other related objects, the present invention provides a data migration processing method, which is applied to an erasure correction based distributed storage system; the erasure correction distributed storage system comprises a plurality of DN data nodes and a management node ms for managing the DN data nodes; each DN data node comprises a plurality of storage units, and the processing method for data migration comprises the following steps:
detecting the state of a plurality of DN data nodes;
when a storage unit A in a DN data node A in a plurality of DN data nodes fails, restoring the fault file to a data restoration position according to erasure information of a block file of the fault file in the storage unit A so as to finish restoration of all data; the data recovery positions are determined according to a preset recovery strategy, the data recovery positions comprise a first fault recovery position or/and a second fault recovery position, the first fault recovery position is a storage unit other than the storage unit A in the DN data node A, and the second fault recovery position is a storage unit in one of other DN data nodes other than the DN data node A in the DN data nodes; the erasure correcting information is obtained from a redis database by responding to the fault message through a management node ms, and the erasure correcting information is stored in the redis database when the block file is established;
and if the fault of the storage unit A is recovered, migrating the file recovered to the data recovery position back to a data migration position to complete the migration of all the data, wherein the data migration position is set as the fault unit A.
Optionally, the preset recovery policy is determined according to at least one of the following:
load balancing, reliability, and the remaining storage space of each storage unit in the DN data node a and the remaining storage space of each storage unit in other DN data nodes except the DN data node a.
Optionally, if the failure of the storage unit a is recovered, when the failed file is recovered to the storage unit a, the failed file is recovered to the location information of the file at the time of creation.
Optionally, the creating method of the block file includes:
the management node ms calculates the number of large files which can be created according to the capacity of the storage unit;
and sequentially creating the block file under each large file according to the erasure ratio.
Optionally, the position determination of the block file at the time of creation satisfies the following condition: load balancing or/and reliability is highest.
Optionally, during the data migration, if there is a write operation on the data being migrated, the data is written to the migrated block and the migrated block at the same time.
Optionally, the erasure information includes: and deleting the same group of block file information and DN IP information.
In order to achieve the above objects and other related objects, the present invention provides a data migration processing method, which is applied to an erasure correction based distributed storage system; the erasure correction distributed storage system comprises a plurality of DN data nodes and a management node ms for managing the DN data nodes; each DN data node comprises a plurality of storage units, and the processing method for data migration comprises the following steps:
when a storage unit A in a DN data node A in a plurality of DN data nodes fails, the DN data node A sends a failure message to the management node ms;
the management node ms responds to the fault message, and acquires erasure information of the block file of the fault file in the storage unit A from a redis database; the erasure information is stored in the redis database when a block file is created;
the management node ms determines a data recovery position according to a preset recovery strategy so as to complete the recovery of all data; the data recovery positions comprise a first fault recovery position or/and a second fault recovery position, the first fault recovery position is a storage unit other than the storage unit A in the DN data node A, and the second fault recovery position is a storage unit in one of the other DN data nodes other than the DN data node A in the DN data nodes;
the storage unit A receives the erasure correction information and restores all files needing to be restored to the data restoration position according to the erasure correction information so as to finish restoration of all data;
and if the fault of the storage unit A is recovered, migrating the file recovered to the data recovery position back to a data migration position to complete the migration of all the data, wherein the data migration position is set as the fault unit A.
In order to achieve the above objects and other related objects, the present invention provides a processing apparatus for data migration, which is applied to an erasure-based distributed storage system; the erasure correction distributed storage system comprises a plurality of DN data nodes and a management node ms for managing the DN data nodes; each DN data node includes a plurality of storage units, and the processing device for data migration includes:
the state detection module is used for detecting the states of the DN data nodes;
the data recovery module is used for recovering a fault file in a storage unit A to a data recovery position according to erasure information of a block file of the fault file in the storage unit A when the storage unit A in the DN data nodes A in the plurality of DN data nodes fails so as to complete recovery of all data; the data recovery positions are determined according to a preset recovery strategy, the data recovery positions comprise a first fault recovery position or/and a second fault recovery position, the first fault recovery position is a storage unit other than the storage unit A in the DN data node A, and the second fault recovery position is a storage unit in one of other DN data nodes other than the DN data node A in the DN data nodes; the erasure correcting information responds to the fault message through a management node ms and is acquired from a redis database, and the erasure correcting information is stored in the redis database when the block file is created;
and the data migration module is used for migrating the file restored to the data restoration position back to the data migration position after the failure of the storage unit A is recovered, so as to complete the migration of all the data, wherein the data migration position is set as the failure unit A.
To achieve the above and other related objects, the present invention further provides a storage medium storing a computer program, which when executed by a processor performs the processing method of data migration.
As described above, the processing method, apparatus and storage medium for data migration according to the present invention have the following advantages:
according to the data migration processing method, during data migration, data is restored according to the position with the highest reliability of each block file which is initially distributed, and writing operation can be performed during the migration process, so that the maximum reliability of stored data is greatly improved while the load balance of a storage system is ensured.
Drawings
FIG. 1 is a flow chart of a data migration processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a data migration processing method according to another embodiment of the present invention;
fig. 3 is a schematic block diagram of a data migration processing apparatus according to an embodiment of the present invention.
Detailed Description
The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention, and the components related to the present invention are only shown in the drawings rather than drawn according to the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.
As shown in fig. 1, an embodiment of the present application provides a data migration processing method, which is applied to an erasure correction based distributed storage system; wherein the erasure correcting distributed storage system comprises a plurality of DN (DataNode, DN) data nodes and a management node ms for managing the DN data nodes; each DN data node comprises a plurality of storage units, and the processing method for data migration comprises the following steps:
s100, detecting the states of a plurality of DN data nodes;
specifically, when a storage unit a in the DN data node a fails, the DN data node a sends a failure message to the management node ms, and the failure message may be received by the receiving module.
After receiving the fault message, the receiving module sends the fault message to the management node ms, and a fault occurs through the management node ms storage unit a.
S101, when a storage unit A in a DN data node A in a plurality of DN data nodes fails, restoring a fault file to a data restoration position according to erasure information of a block file of the fault file in the storage unit A so as to finish restoration of all data; the data recovery positions are determined according to a preset recovery strategy, the data recovery positions comprise a first fault recovery position or/and a second fault recovery position, the first fault recovery position is a storage unit other than the storage unit A in the DN data node A, and the second fault recovery position is a storage unit in one of other DN data nodes other than the DN data node A in the DN data nodes; the erasure correcting information is obtained from a redis database by responding to the fault message through a management node ms, and the erasure correcting information is stored in the redis database when the block file is established;
the failure of the storage unit a means that the storage unit a is not detected in the system, and does not mean that the storage unit a is damaged.
Specifically, the management node ms will respond to the failure message, and trigger data recovery to perform data recovery on the failed file according to the block information of the failed file in the storage unit a when the block file is created, so as to recover the failed file to the data recovery position. At this time, the original information of the block file is recorded in a redis database, which may be specifically a recovered completion table in the redis database. The data recovery positions are determined according to a preset recovery strategy, the data recovery positions comprise a first fault recovery position or/and a second fault recovery position, the first fault recovery position is a storage unit of the DN data node A except the storage unit A, and the second fault recovery position is a storage unit of one of the DN data nodes except the DN data node A. The first fault recovery location is a location other than location a in DN data node a, for example, one or more of location B, location C, location D, etc. in DN data node a; the second fault recovery bit is set as a storage unit in another DN data node other than the DN data node a among the DN data nodes, for example, one or more of the DN data node B, DN data node C, DN data node D among the DN data nodes, and the like.
In one embodiment, the data recovery location is determined according to a preset recovery policy. Specifically, the preset recovery policy is determined according to at least one of the following: load balancing, reliability, and the remaining storage space of each storage unit in the DN data node a and the remaining storage space of each storage unit in other DN data nodes except the DN data node a. It is understood that in this embodiment, the recovery policy may determine the recovery policy according to load balancing, reliability, and remaining storage space of each storage unit in the DN data node a, and remaining storage space of each storage unit in other DN data nodes except the DN data node a, that is, whether data is recovered to the storage unit B, the storage unit C, or the storage unit D of the DN data node a; either the data is restored to DN data node B, DN data node C, or DN data node D.
In the data recovery process, a DN data node where the storage unit a is located may receive erasure information of block files of all files (fault files) to be recovered below the storage unit a, where the erasure information is sent by the management node ms, where the erasure information includes block file information and DN IP information in the same group, and the DN IP information is used to indicate a number of the DN data node. The erasure correction information is obtained from a redis database by responding to the fault message through a management node ms, and the erasure correction information is stored in the redis database when the block file is created, and specifically may be an FB table (file and block information table) in the redis database. And the DN data node A reads the block file information of the erasure same group, and writes the fault file in the storage unit A into other storage units except the storage unit A of the DN data node A.
S103, if the fault of the storage unit A is recovered, migrating the file recovered to the data recovery position back to a data migration position to complete the migration of all data, wherein the data migration position is set as the fault unit A.
Take the example of the data migration bit being the storage location in the DN data node a. After the recovery of the fault file of the storage unit A is completed on the storage unit C, the migration patrol thread can scan the recovered table at regular time, if the storage unit B of the storage unit A is replaced to be online, the file recorded in the recovered recovery completion table of the storage unit A is recovered on the storage unit C, and the file is migrated back to the storage unit B until the recovered recovery completion table is empty until all data is migrated.
In one embodiment, when the failure of the storage unit a is recovered, the failed file is recovered to the location information of the file at the time of creation when the failed file is recovered to the storage unit a. When determining the position information of the file at the time of creation, the following conditions need to be satisfied: load balancing and reliability are highest.
In one embodiment, during data processing, if there is a write operation to the data being migrated, the data is written to the migrated and migrated blocks simultaneously. The client is allowed to write the data being migrated, and when the client writes the data into the migrated block, the client can write the data into the migrated and migrated blcok at the same time, so that the synchronization consistency of the two data is ensured.
In an embodiment, the block file creating method includes: the management node ms calculates the number of large files which can be created according to the capacity of the storage unit; and sequentially creating the block file under each large file according to the erasure ratio. Specifically, when creating a Block file, the number of large files that can be created is calculated in advance according to the capacity allocated by the storage unit, a Block (data Block corresponding to DN metadata) under each large file is created in sequence according to the erasure correction ratio, and erasure correction information of the Block file of each large file under the storage unit is stored in an FB table (file and Block information table) of a redis database. If any file is failed to be created in the process of creating the lun, the file needs to be created again, and the load balance and the best reliability of distributing all files below the lun at the initial position of the storage system are ensured. Wherein a large file is defined as a file whose size equals or exceeds a set threshold.
The invention is further illustrated by the following specific example
1. After the storage system is deployed, the stability of the whole cluster can be ensured, and the stability comprises DN data nodes being online and storage units having no fault, so that when a Lun (logical storage unit) is created, each block file is distributed on different storage units of different DN data nodes with optimal reliability; in the creating process, a management node ms calculates the number of large files that can be created according to the capacity of luns, and then creates blocks under each large file in sequence, the creation of the Block files is sent to a selected DN data node according to a policy of load balance and highest reliability, the DN data node returns the ID of a storage unit where each successfully created Block file is located to the management node ms, the management node ms records the information of each Block file in an FB table (file and Block information table) of redis, the FB contains the information of all blocks under each large file, and the FB table takes the large file as a unit, for example, the table structure of a certain 1G large file is as follows:
DNID-DISKID-BLOCKSTATUS-BLOCKFULLSTATUS
the DN ID information of the block file, the ID information of the storage unit, the state of the block and the full or not full state of the block are sequentially displayed.
If any file is failed to be created in the process of creating the Lun, the file needs to be created again, and the load balance and the best reliability of distributing all the files below the Lun at the initial position of the storage system are ensured.
2. After normal data is written in, when a storage unit A with a certain DN1 data node in the erasure correction distributed storage system is offline, a DN1 data node where the storage unit A is located informs a management node ms of a fault message of the storage unit A, the management node ms inquires information of all Block files below the storage unit A with the fault, selects a recovered DN data node from a plurality of DN data nodes according to a load balancing and high-reliability strategy, and sends the recovered DN data node to the selected DN data node for erasure correction data recovery until all data recovery of the storage unit A is completed.
3. When a management node ms starts to recover a block file, recording file information of the block at the creation time (the most original position) to a Recovered table of redis, wherein the Recovered table takes fileID + obj ID + block ID as key, and DNID + DISKID of the block file as value until all data of a storage unit A are Recovered;
4. after all the data of the storage unit A are recovered to the storage unit B, the management node ms updates DNIDs and DISKIDs in FBs of all the block files in the storage unit A to information corresponding to the storage unit B;
5. the management node ms can pre-start a migration function thread in the starting process, and when the execution of the step 4 is finished, block file information is scanned from a Recovered table of the redis; the scanned Block is DstBlock to be migrated, the Block to be migrated is restored on the storage unit B, the file ID information of the Block is unchanged, namely the key of FB: the FileID + obj ID + BlockID is unchanged, the ID information and DNID information of the storage unit of Srcblock to be migrated, that is, the information related to the storage unit B, can be directly queried by key, and the Srcblock and DstBlock information are recorded in the Migration table (Migration table) of redis.
6. When a storage unit B of a replacement storage unit A is online and recovered data is online in a storage unit C, a management node ms scans a Migration table Migration of redis, recorded SrcBlock information comprises DNID of SrcBlock, storage unit ID, file information FileID + ObjID + Block ID, and the recorded SrcBlock information is sent to a DstDN where the DstBlock is located, the DstDN reads corresponding SrcBlock data from the SrcDN according to the SrcBlock information, and writes the data into the DstDN
7. When the migration starts, DstDN sends block information to be migrated to SrcDN, and SrcDN records in MigratList of memory linked list
8. And after the Migration is finished, the DstBlock sends a message of successful Migration to a management node ms, the management node ms sends the message of successful Migration to SrcDN to delete SrcBlock data, block members corresponding to a Recovered table and a Migration table of redis are deleted, and meanwhile FB information is updated to be DstBlock information.
If data which is being migrated is written in the migration process, namely, a client requests to write SrcBlock, in order to guarantee the latest principle of the migrated data, SrcDN returns DstBlock information recorded in a MigratList memory linked list to the client, and the client initiates a rewrite request to the DstDN again to guarantee that the block data is the latest after the data are migrated. Because the client writing is the covering random writing and the writing position cannot be determined, the DstDN needs to judge the version number of the data written by the client (marking the writing sequence of the data at the same position) during writing, if the version number is higher than the writing sequence of the data at the same position, the data is directly covered and written, otherwise, the data is not updated, and the data on the disk is ensured to be up to date.
As shown in fig. 2, an embodiment of the present invention provides a data migration processing method, which is applied to an erasure correction based distributed storage system; the erasure correction distributed storage system comprises a plurality of DN data nodes and a management node ms for managing the DN data nodes; each DN data node comprises a plurality of storage units, and the processing method for data migration comprises the following steps:
s200, when a storage unit A in a DN data node A in a plurality of DN data nodes fails, the DN data node A sends a failure message to the management node ms; the failure of the storage unit a means that the storage unit a is not detected in the system, and does not mean that the storage unit a is damaged.
Specifically, when a storage unit a in the DN data node a fails, the DN data node a sends a failure message to the management node ms, and the failure message may be received by the receiving module.
After receiving the fault message, the receiving module sends the fault message to the management node ms, and a fault occurs through the management node ms storage unit a.
S201, the management node ms responds to the fault message, and acquires erasure correction information of a block file of the fault file in the storage unit A from a redis database; the erasure information is stored in the redis database when a block file is created;
specifically, the management node ms will respond to the failure message, and trigger data recovery to perform data recovery on the failed file according to the block information of the failed file in the storage unit a when the block file is created, so as to recover the failed file to the data recovery position. At this time, the original information of the block file is recorded in a redis database, which may be specifically a recovered completion table in the redis database.
S202, the management node ms determines a data recovery position according to a preset recovery strategy to complete recovery of all data; the data recovery positions comprise a first fault recovery position or/and a second fault recovery position, the first fault recovery position is a storage unit other than the storage unit A in the DN data node A, and the second fault recovery position is a storage unit in one of the other DN data nodes other than the DN data node A in the DN data nodes;
the first fault recovery location is a location other than location a in DN data node a, for example, one or more of location B, location C, location D, etc. in DN data node a; the second fault recovery bit is set as a storage unit in another DN data node other than the DN data node a among the DN data nodes, for example, one or more of the DN data node B, DN data node C, DN data node D among the DN data nodes, and the like.
In one embodiment, the data recovery location is determined according to a preset recovery policy. Specifically, the preset recovery policy is determined according to at least one of the following: load balancing, reliability, and the remaining storage space of each storage unit in the DN data node a and the remaining storage space of each storage unit in other DN data nodes except the DN data node a. It is understood that in this embodiment, the recovery policy may determine the recovery policy according to load balancing, reliability, and remaining storage space of each storage unit in the DN data node a, and remaining storage space of each storage unit in other DN data nodes except the DN data node a, that is, whether data is recovered to the storage unit B, the storage unit C, or the storage unit D of the DN data node a; either the data is restored to DN data node B, DN data node C, or DN data node D.
S203, the storage unit A receives the erasure correcting information and restores all files needing to be restored to the data restoring position according to the erasure correcting information;
in the data recovery process, a DN data node where the storage unit a is located may receive erasure information of block files of all files (fault files) to be recovered below the storage unit a, where the erasure information is sent by the management node ms, where the erasure information includes block file information and DN IP information in the same group, and the DN IP information is used to indicate a number of the DN data node. The erasure correction information is obtained from a redis database by responding to the fault message through a management node ms, and the erasure correction information is stored in the redis database when the block file is created, and specifically may be an FB table (file and block information table) in the redis database. And the DN data node A reads the block file information of the erasure same group, and writes the fault file in the storage unit A into other storage units except the storage unit A of the DN data node A.
S204, if the storage unit A is recovered due to a fault, migrating the file recovered to the data recovery position back to a data migration position to complete migration of all data, wherein the data migration position is set as the fault unit A.
Take the example of the data migration bit being the storage location in the DN data node a. After the recovery of the fault file of the storage unit A is completed on the storage unit C, the migration patrol thread can scan the recovered table at regular time, if the storage unit B of the storage unit A is replaced to be online, the file recorded in the recovered recovery completion table of the storage unit A is recovered on the storage unit C, and the file is migrated back to the storage unit B until the recovered recovery completion table is empty until all data is migrated.
Take the example of the data migration bit being the storage location in the DN data node a. After the recovery of the fault file of the storage unit A is completed on the storage unit C, the migration patrol thread can scan the recovered table at regular time, if the storage unit B of the storage unit A is replaced to be online, the file recorded in the recovered recovery completion table of the storage unit A is recovered on the storage unit C, and the file is migrated back to the storage unit B until the recovered recovery completion table is empty, and all data is migrated.
As shown in fig. 3, an embodiment of the present invention provides a processing apparatus for data migration, which is applied to an erasure-based distributed storage system; the erasure correction distributed storage system comprises a plurality of DN data nodes and a management node ms for managing the DN data nodes; each DN data node includes a plurality of storage units, and the processing device for data migration includes: a state detection module 300, a data recovery module 301, a data migration module 302,
a state detection module 300, configured to detect states of a plurality of DN data nodes;
the failure of the storage unit a means that the storage unit a is not detected in the system, and does not mean that the storage unit a is damaged.
The data recovery module 301 is configured to, when a storage unit a in a DN data node a in the multiple DN data nodes fails, recover a failed file in the storage unit a to a data recovery position according to erasure information of a block file of the failed file in the storage unit a, so as to complete recovery of all data; the data recovery positions are determined according to a preset recovery strategy, the data recovery positions comprise a first fault recovery position or/and a second fault recovery position, the first fault recovery position is a storage unit other than the storage unit A in the DN data node A, and the second fault recovery position is a storage unit in one of other DN data nodes other than the DN data node A in the DN data nodes; the erasure correcting information responds to the fault message through a management node ms and is acquired from a redis database, and the erasure correcting information is stored in the redis database when the block file is created;
and a data migration module 302, configured to, after the failure of the storage unit a is recovered, migrate the file recovered to the data recovery position back to a data migration position to complete migration of all data, where the data migration position is the failure unit a.
It should be noted that, because the embodiment of the apparatus portion and the embodiment of the method portion correspond to each other, please refer to the description of the embodiment of the method portion for the content of the embodiment of the apparatus portion, which is not repeated here.
The present invention also provides a storage medium storing a computer program, which when executed by a processor performs the aforementioned processing method of data migration.
The present invention also provides an apparatus comprising:
a memory for storing a computer program;
and the processor is used for executing the computer program stored in the memory so as to enable the equipment to execute the processing method of the data migration.
The Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may be an internal storage unit or an external storage device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital Card (SD), a Flash memory Card (Flash Card), and the like. Further, the memory may also include both an internal storage unit and an external storage device. The memory is used for storing the computer program and other programs and data. The memory may also be used to temporarily store data that has been or will be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may comprise any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a Random Access Memory (RAM), an electrical carrier signal, a telecommunications signal, a software distribution medium, etc.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.
Claims (10)
1. A processing method for data migration is characterized in that the processing method is applied to an erasure correction based distributed storage system; the erasure correction distributed storage system comprises a plurality of DN data nodes and a management node ms for managing the DN data nodes; each DN data node comprises a plurality of storage units, and the processing method for data migration comprises the following steps:
detecting the state of a plurality of DN data nodes;
when a storage unit A in a DN data node A in a plurality of DN data nodes fails, restoring the fault file to a data restoration position according to erasure information of a block file of the fault file in the storage unit A so as to finish restoration of all data; the data recovery positions are determined according to a preset recovery strategy, the data recovery positions comprise a first fault recovery position or/and a second fault recovery position, the first fault recovery position is a storage unit other than the storage unit A in the DN data node A, and the second fault recovery position is a storage unit in one of other DN data nodes other than the DN data node A in the DN data nodes; the erasure correcting information is obtained from a redis database by responding to the fault message through a management node ms, and the erasure correcting information is stored in the redis database when the block file is established;
and if the fault of the fault unit A is recovered, migrating the file recovered to the data recovery position back to a data migration position to complete the migration of all the data, wherein the data migration position is set as the fault unit A.
2. The method according to claim 1, wherein the predetermined recovery policy is determined according to at least one of the following:
load balancing, reliability, and the remaining storage space of each storage unit in the DN data node a and the remaining storage space of each storage unit in other DN data nodes except the DN data node a.
3. The method according to claim 1, wherein if the failure of the storage unit a is recovered, when the failed file is migrated to the storage unit a, the failed file is migrated to location information of the file at the time of creation.
4. The data migration processing method according to claim 3, wherein the block file creation method comprises:
the management node ms calculates the number of large files which can be created according to the capacity of the storage unit;
and sequentially creating the block file under each large file according to the erasure ratio.
5. The data migration processing method according to claim 1, wherein the determination of the location of the block file at the time of creation satisfies the following condition: load balancing or/and reliability is highest.
6. The data migration processing method according to claim 1, wherein in the data migration process, if there is a write operation on the data being migrated, the data is written to the migrated block and the migrated block at the same time.
7. The method for processing data migration according to claim 1, wherein the erasure information includes: and deleting the same group of block file information and DN IP information.
8. A processing method for data migration is characterized in that the processing method is applied to an erasure correction based distributed storage system; the erasure correction distributed storage system comprises a plurality of DN data nodes and a management node ms for managing the DN data nodes; each DN data node comprises a plurality of storage units, and the processing method for data migration comprises the following steps:
when a storage unit A in a DN data node A in a plurality of DN data nodes fails, the DN data node A sends a failure message to the management node ms;
the management node ms responds to the fault message, and acquires erasure information of the block file of the fault file in the storage unit A from a redis database; the erasure information is stored in the redis database when a block file is created;
the management node ms determines a data recovery position according to a preset recovery strategy so as to complete the recovery of all data; the data recovery positions comprise a first fault recovery position or/and a second fault recovery position, the first fault recovery position is a storage unit other than the storage unit A in the DN data node A, and the second fault recovery position is a storage unit in one of the other DN data nodes other than the DN data node A in the DN data nodes;
the storage unit A receives the erasure correction information and restores all files needing to be restored to the data restoration position according to the erasure correction information so as to finish restoration of all data;
and if the fault of the fault unit A is recovered, migrating the file recovered to the data recovery position back to a data migration position to complete the migration of all the data, wherein the data migration position is set as the fault unit A.
9. A processing device for data migration is characterized by being applied to an erasure correction based distributed storage system; the erasure correction distributed storage system comprises a plurality of DN data nodes and a management node ms for managing the DN data nodes; each DN data node includes a plurality of storage units, and the processing device for data migration includes:
the state detection module is used for detecting the states of the DN data nodes;
the data recovery module is used for recovering a fault file in a storage unit A to a data recovery position according to erasure information of a block file of the fault file in the storage unit A when the storage unit A in the DN data nodes A in the plurality of DN data nodes fails so as to complete recovery of all data; the data recovery positions are determined according to a preset recovery strategy, the data recovery positions comprise a first fault recovery position or/and a second fault recovery position, the first fault recovery position is a storage unit other than the storage unit A in the DN data node A, and the second fault recovery position is a storage unit in one of other DN data nodes other than the DN data node A in the DN data nodes; the erasure correcting information responds to the fault message through a management node ms and is acquired from a redis database, and the erasure correcting information is stored in the redis database when the block file is created;
and the data migration module is used for migrating the file restored to the data restoration position back to the data migration position after the failure of the storage unit A is recovered, so as to complete the migration of all the data, wherein the data migration position is set as the failure unit A.
10. A storage medium storing a computer program, wherein the computer program, when executed by a processor, performs a processing method of data migration according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111154480.3A CN113821176B (en) | 2021-09-29 | 2021-09-29 | Data migration processing method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111154480.3A CN113821176B (en) | 2021-09-29 | 2021-09-29 | Data migration processing method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113821176A true CN113821176A (en) | 2021-12-21 |
| CN113821176B CN113821176B (en) | 2023-07-21 |
Family
ID=78915925
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111154480.3A Active CN113821176B (en) | 2021-09-29 | 2021-09-29 | Data migration processing method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113821176B (en) |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064765A (en) * | 2012-12-28 | 2013-04-24 | 华为技术有限公司 | Method and device for data recovery and cluster storage system |
| CN103593347A (en) * | 2012-08-14 | 2014-02-19 | 中兴通讯股份有限公司 | Load balancing method and distributed database system |
| CN105335251A (en) * | 2015-09-23 | 2016-02-17 | 浪潮(北京)电子信息产业有限公司 | Fault recovery method and system |
| CN106372160A (en) * | 2016-08-31 | 2017-02-01 | 天津南大通用数据技术股份有限公司 | Distributive database and management method |
| CN106662983A (en) * | 2015-12-31 | 2017-05-10 | 华为技术有限公司 | Method, apparatus and system for data reconstruction in distributed storage system |
| CN107943421A (en) * | 2017-11-30 | 2018-04-20 | 成都华为技术有限公司 | A kind of subregion partitioning method and device based on distributed memory system |
| US20180329647A1 (en) * | 2017-05-12 | 2018-11-15 | International Business Machines Corporation | Distributed storage system virtual and storage data migration |
| US20190361775A1 (en) * | 2018-05-25 | 2019-11-28 | Microsoft Technology Licensing, Llc | Meta-copysets for fault-tolerant data storage |
| CN111064789A (en) * | 2019-12-18 | 2020-04-24 | 北京三快在线科技有限公司 | Data migration method and system |
| CN111078667A (en) * | 2019-12-12 | 2020-04-28 | 腾讯科技（深圳）有限公司 | Data migration method and related device |
| CN111324596A (en) * | 2020-03-06 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Data migration method and device for database cluster and electronic equipment |
| CN111625502A (en) * | 2020-05-28 | 2020-09-04 | 浙江大华技术股份有限公司 | Data reading method and device, storage medium and electronic device |
| CN111698330A (en) * | 2020-06-12 | 2020-09-22 | 北京金山云网络技术有限公司 | Data recovery method and device of storage cluster and server |
| CN111752892A (en) * | 2019-03-27 | 2020-10-09 | 北京京东尚科信息技术有限公司 | Distributed file system, method for implementing the same, management system, device, and medium |
| CN113254270A (en) * | 2021-05-28 | 2021-08-13 | 济南浪潮数据技术有限公司 | Self-recovery method, system and storage medium for storing cache hotspot data |
-
2021
- 2021-09-29 CN CN202111154480.3A patent/CN113821176B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103593347A (en) * | 2012-08-14 | 2014-02-19 | 中兴通讯股份有限公司 | Load balancing method and distributed database system |
| CN103064765A (en) * | 2012-12-28 | 2013-04-24 | 华为技术有限公司 | Method and device for data recovery and cluster storage system |
| CN105335251A (en) * | 2015-09-23 | 2016-02-17 | 浪潮(北京)电子信息产业有限公司 | Fault recovery method and system |
| CN106662983A (en) * | 2015-12-31 | 2017-05-10 | 华为技术有限公司 | Method, apparatus and system for data reconstruction in distributed storage system |
| CN106372160A (en) * | 2016-08-31 | 2017-02-01 | 天津南大通用数据技术股份有限公司 | Distributive database and management method |
| US20180329647A1 (en) * | 2017-05-12 | 2018-11-15 | International Business Machines Corporation | Distributed storage system virtual and storage data migration |
| CN107943421A (en) * | 2017-11-30 | 2018-04-20 | 成都华为技术有限公司 | A kind of subregion partitioning method and device based on distributed memory system |
| US20190361775A1 (en) * | 2018-05-25 | 2019-11-28 | Microsoft Technology Licensing, Llc | Meta-copysets for fault-tolerant data storage |
| CN111752892A (en) * | 2019-03-27 | 2020-10-09 | 北京京东尚科信息技术有限公司 | Distributed file system, method for implementing the same, management system, device, and medium |
| CN111078667A (en) * | 2019-12-12 | 2020-04-28 | 腾讯科技（深圳）有限公司 | Data migration method and related device |
| CN111064789A (en) * | 2019-12-18 | 2020-04-24 | 北京三快在线科技有限公司 | Data migration method and system |
| CN111324596A (en) * | 2020-03-06 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Data migration method and device for database cluster and electronic equipment |
| CN111625502A (en) * | 2020-05-28 | 2020-09-04 | 浙江大华技术股份有限公司 | Data reading method and device, storage medium and electronic device |
| CN111698330A (en) * | 2020-06-12 | 2020-09-22 | 北京金山云网络技术有限公司 | Data recovery method and device of storage cluster and server |
| CN113254270A (en) * | 2021-05-28 | 2021-08-13 | 济南浪潮数据技术有限公司 | Self-recovery method, system and storage medium for storing cache hotspot data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113821176B (en) | 2023-07-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7818302B2 (en) | System and method for performing file system checks on an active file system | |
| US8112463B2 (en) | File management method and storage system | |
| US7681001B2 (en) | Storage system | |
| US10387271B2 (en) | File system storage in cloud using data and metadata merkle trees | |
| US11093387B1 (en) | Garbage collection based on transmission object models | |
| US11726883B2 (en) | Fast and optimized restore using delta information | |
| US20210064413A1 (en) | Deploying a cloud instance of a user virtual machine | |
| US10628298B1 (en) | Resumable garbage collection | |
| US20200150897A1 (en) | Cloud edition and retrieve | |
| US11487701B2 (en) | Incremental access requests for portions of files from a cloud archival storage tier | |
| EP3474143B1 (en) | Method and apparatus for incremental recovery of data | |
| US11841953B2 (en) | Scanning a backup for vulnerabilities | |
| CN105493080B (en) | The method and apparatus of data de-duplication based on context-aware | |
| CN114416665B (en) | Method, device and medium for detecting and repairing data consistency | |
| CN109753381B (en) | Continuous data protection method based on object storage | |
| CN112800019A (en) | Data backup method and system based on Hadoop distributed file system | |
| CN115098447A (en) | File recovery method and device, electronic equipment and readable storage medium | |
| CN102609454B (en) | Replica management method for distributed file system | |
| US8312237B2 (en) | Automated relocation of in-use multi-site protected data storage | |
| US20200333970A1 (en) | Data de-duplication | |
| CN113821176A (en) | Data migration processing method and device and storage medium | |
| CN103645967A (en) | Read-only snapshot rollback method and device | |
| US11645333B1 (en) | Garbage collection integrated with physical file verification | |
| CN112596959A (en) | Distributed storage cluster data backup method and device | |
| CN113238891A (en) | Backup deletion method and system based on backup chain |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |