WO2016029744A1 - Procédé de récupération de métadonnées et dispositif associé - Google Patents

Procédé de récupération de métadonnées et dispositif associé Download PDF

Info

Publication number
WO2016029744A1
WO2016029744A1 PCT/CN2015/083226 CN2015083226W WO2016029744A1 WO 2016029744 A1 WO2016029744 A1 WO 2016029744A1 CN 2015083226 W CN2015083226 W CN 2015083226W WO 2016029744 A1 WO2016029744 A1 WO 2016029744A1
Authority
WO
WIPO (PCT)
Prior art keywords
metadata
target file
information
data
data block
Prior art date
Application number
PCT/CN2015/083226
Other languages
English (en)
Chinese (zh)
Inventor
刘洋
张克星
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2016029744A1 publication Critical patent/WO2016029744A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/40—Support for services or applications

Definitions

  • the present invention relates to the field of communications, and in particular, to a metadata recovery method and related apparatus.
  • Hadoop is a widely distributed distributed system architecture in the communication field. It contains a very important storage component in Hadoop, namely Hadoop Distributed File System (HDFS).
  • the HDFS includes a metadata node (NameNode, NN for short) and a data node (DataNode, DN for short).
  • the NN is mainly used to provide a metadata service for the file. In the service, the NN generates metadata of the file in the memory, and the metadata. Can represent the data structure of the file, and DN is mainly used to store the actual data of the file. In an actual application, when an NN is abnormal, the NN will have a problem of metadata loss, causing the NN to fail.
  • a timing thread is opened inside the NN, which periodically writes the metadata in the NN memory to the local disk or the remote storage device to form a specific file.
  • the NN loads the specific file during the restart process to recover the metadata of the previous period to achieve the memory state of the NN recovery to this period.
  • the metadata in the NN is periodically stored, for example, the metadata is stored every ten minutes, so that when the time when the NN is abnormal has not reached the storage time point of ten minutes, then the NN Metadata in the event of an exception is not stored, so this part of the metadata cannot be recovered. It can be seen that the metadata of the NN anomaly cannot be recovered in the above technical solution.
  • the invention provides a metadata recovery method and related device, which can recover metadata when an NN is abnormal.
  • the present invention provides a metadata recovery method, including:
  • the target file is divided into multiple data blocks, where the multiple data blocks are saved on the multiple DNs;
  • the acquiring the metadata of the target file from the NN, and generating the metadata recovery information according to the metadata of the target file includes:
  • the metadata of each data block includes information of each data block and information of a DN where each of the data blocks is located;
  • the acquiring, by the NN, the metadata of the multiple data blocks includes:
  • the received metadata includes at least metadata of the first data block and the second data block Metadata, where:
  • the generated metadata recovery information of the first data block includes:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the generated metadata recovery information of the second data block includes:
  • the present invention provides a metadata recovery method, including:
  • the metadata recovery information is sent to the NN, and the metadata recovery information is used by the NN to perform recovery of the metadata.
  • the target file is divided into multiple data blocks, and the metadata recovery information of the target file sent by the receiving client includes:
  • the receiving the metadata recovery information of the data block of the target file that is sent by the client includes:
  • the received metadata recovery information is at least The metadata recovery information of the first data block and the metadata recovery information of the second data block, wherein:
  • the metadata recovery information of the first data block includes:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the metadata recovery information of the second data block includes:
  • the present invention provides a metadata recovery method, including:
  • the target file is divided into multiple data blocks, where the multiple data blocks are saved on the multiple DNs;
  • And generating the metadata of the target file, and sending the metadata to the client includes:
  • Receiving the metadata recovery information sent by the at least one of the plurality of DNs, and recovering the metadata according to the metadata recovery information including:
  • the sending, by the client, the metadata of the multiple data blocks includes:
  • the multiple data blocks include at least a first data block and a second Data block, where:
  • the metadata recovery information of the first data block includes:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the metadata recovery information of the second data block includes:
  • the present invention provides a client, including: a receiving unit, an obtaining unit, a generating unit, and a sending unit, where:
  • the receiving unit is configured to receive a write operation request for the target file, where the target file is saved on multiple DNs;
  • the acquiring unit is configured to acquire metadata of the target file from an NN, where metadata of the target file includes information of the multiple DNs;
  • the generating unit is configured to generate metadata recovery information according to metadata of the target file
  • the sending unit is configured to send the metadata recovery information to at least one of the plurality of DNs, where the metadata recovery information is used to be saved by the at least one DN, and
  • the NN is sent to the NN for recovery of the metadata after an abnormality occurs.
  • the target file is divided into multiple data blocks, where the multiple data blocks are saved on the multiple DNs;
  • the obtaining unit is configured to acquire metadata of the plurality of data blocks from the NN, and the metadata of each data block includes information of each data block and information of a DN where each of the data blocks is located;
  • the generating unit is configured to generate metadata recovery information of each of the data blocks according to the metadata of each of the data blocks.
  • the acquiring unit is configured to sequentially receive metadata of each data block that is sent by the NN; Alternatively, the obtaining unit is configured to receive metadata of all data blocks that are sent by the NN according to a write operation of the target file.
  • the received metadata includes at least metadata of the first data block And the metadata of the second data block, where:
  • the generated metadata recovery information of the first data block includes:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the generated metadata recovery information of the second data block includes:
  • the present invention provides a DN, including: a receiving unit and a sending unit, where:
  • the receiving unit is configured to receive metadata recovery information of the target file sent by the client, and save the metadata recovery information, where the metadata recovery information is that the client receives a write operation on the target file.
  • the metadata of the target file obtained from the NN is based on Metadata recovery information generated by the metadata.
  • the sending unit is configured to send the metadata recovery information to the NN after an abnormality occurs in the NN, where the metadata recovery information is used by the NN to perform recovery of the metadata.
  • the target file is divided into multiple data blocks
  • the receiving unit is configured to receive metadata recovery of the data block of the target file that is sent by the client Information
  • the metadata recovery information of the data block is metadata recovery information generated by the client according to metadata of the data block.
  • the receiving unit is configured to receive, in sequence, multiple data blocks of the target file that are sent by the client endpoint The metadata recovery information is used; or the receiving unit is configured to receive metadata recovery information of the plurality of data blocks of the target file that are sent by the client at one time.
  • the received metadata recovery information includes at least metadata recovery information of the first data block. And metadata recovery information for the second data block, wherein:
  • the metadata recovery information of the first data block includes:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the metadata recovery information of the second data block includes:
  • the present invention provides an NN, including: a receiving unit, a generating unit, and a restoring unit, where:
  • the receiving unit is configured to receive a write operation request for the target file sent by the client, where the target file is saved on multiple DNs;
  • the generating unit is configured to generate metadata of the target file, and send the data to the client The metadata, wherein the metadata includes information of the plurality of DNs;
  • the recovery unit is configured to receive metadata recovery information sent by at least one of the plurality of DNs when an abnormality occurs on the local end, and recover the metadata according to the metadata recovery information, where
  • the metadata recovery information is metadata recovery information generated by the client according to the metadata, and sent by the client to the at least one DN for saving.
  • the target file is divided into multiple data blocks, where the multiple data blocks are saved on the multiple DNs;
  • the generating unit is configured to generate metadata of the plurality of data blocks, and send metadata of the plurality of data blocks to the client, where metadata of each data block includes information and information of each data block Information about the DN where each data block is located;
  • the recovery unit is configured to receive metadata recovery information of the plurality of data blocks sent by at least one of the plurality of DNs when an abnormality occurs on the local end, and recover the metadata according to the plurality of data blocks.
  • the information recovers the metadata of the target file.
  • the generating unit is configured to generate metadata of the multiple data blocks, and send the Metadata of a plurality of data blocks; or the generating unit is configured to generate metadata of the plurality of data blocks, and send metadata of all data blocks to the client at a time according to a write operation request of the target file .
  • the multiple data blocks include at least a first data block and a second data block, where :
  • the metadata recovery information of the first data block includes:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the metadata recovery information of the second data block includes:
  • receiving a write operation request for the target file acquiring metadata of the target file from the NN, generating metadata recovery information according to the metadata of the target file, and then performing at least one of the plurality of DNs A DN sends the metadata recovery information.
  • the metadata recovery information can be used for saving by the at least one DN and sent to the NN for recovery of the metadata after an exception occurs in the NN.
  • the metadata when the NN is abnormal can be restored.
  • FIG. 1 is a schematic diagram of an optional metadata provided by an embodiment of the present invention.
  • FIG. 2 is a schematic flowchart of a metadata recovery method according to an embodiment of the present invention.
  • FIG. 3 is a schematic flowchart of another metadata recovery method according to an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of an optional data information provided by an embodiment of the present invention.
  • FIG. 5 is a schematic flowchart diagram of another metadata recovery method according to an embodiment of the present invention.
  • FIG. 6 is a schematic flowchart diagram of another metadata recovery method according to an embodiment of the present invention.
  • FIG. 7 is a schematic flowchart diagram of another metadata recovery method according to an embodiment of the present invention.
  • FIG. 8 is a schematic structural diagram of a client according to an embodiment of the present disclosure.
  • FIG. 9 is a schematic structural diagram of a DN according to an embodiment of the present invention.
  • FIG. 10 is a schematic structural diagram of a NN according to an embodiment of the present disclosure.
  • FIG. 11 is a schematic structural diagram of another metadata recovery device according to an embodiment of the present invention.
  • FIG. 12 is a schematic structural diagram of a DN device according to an embodiment of the present disclosure.
  • FIG. 13 is a schematic structural diagram of an NN device according to an embodiment of the present invention.
  • the metadata may be data for indicating a file data structure, where the metadata of the file may specifically represent version information of the HDFS system where the file is located and attribute information of the file, for example, metadata of the file.
  • the version information of the included HDFS system may include one or more of the following:
  • the version information of the HDFS system for example, imgVersion
  • the namespace identifier for example, namespaceID
  • the number of files of the HDFS system for example, numFiles
  • the time stamp of the HDFS system for example, genStamp
  • the attribute information of the file included in the metadata of the file may include one or more of the following:
  • the path of the file (for example: Path), the number of copies of the file (for example: replicas), the access time of the file (for example: atime), the size of each data block of the file (for example: blockSize), and the number of blocks of the data block of the file ( For example: numBlock), limit information for the space occupied by the file (for example: nsQuota and dsQuota) and user permission information (for example: username, group, and perm).
  • Path The path of the file
  • the number of copies of the file for example: replicas
  • the access time of the file for example: atime
  • the size of each data block of the file for example: blockSize
  • the number of blocks of the data block of the file For example: numBlock
  • limit information for the space occupied by the file for example: nsQuota and dsQuota
  • user permission information for example: username, group, and perm
  • the metadata of the file may further include metadata of each data block, where the metadata of the data block includes attribute information of the file to which the data block belongs and attribute information of the data block;
  • the attribute information of the data block may include one or more of the following:
  • the data block identifier for example: BlockId
  • the amount of data stored in the data block for example: numBytes
  • the time stamp of the data block for example: genStamp
  • the metadata of the file can be as shown in FIG. 1, and the metadata of the file can include the one in FIG. Three lines of data, where the first line represents the version information of the HDFS system, the second line represents the attribute information of the file, and the third line represents the metadata of the data block. That is, the metadata of the file may include version information of the HDFS system, attribute information of the file, and metadata of the data block.
  • the scenario to which the embodiment of the present invention is applied may be a single device, that is, the client including the HDFS, the at least one NN, and the at least one DN; or the scenario applied by the embodiment of the present invention may be a device cluster, that is, a client of the HDFS.
  • At least one NN and at least one DN are deployed on multiple devices.
  • the device may be a network side device, for example, a base station, a server, a controller, etc., and the device may also be a user side device, such as a computer, a tablet computer, a mobile phone, an e-reader, a remote controller, a notebook computer, and a car.
  • a smart device with network capabilities such as devices, network TVs, and wearable devices.
  • FIG. 2 is a schematic flowchart of a metadata recovery method according to an embodiment of the present invention. As shown in FIG. 2, the method includes the following steps:
  • the foregoing target file is saved on multiple DNs, and may be in response to the foregoing write operation request, and the target file is saved on multiple DNs, wherein the time of the save process is not limited, for example, after step 204, or in step Before 204.
  • the local end may apply to the NN for multiple DNs for saving the target file, so that the NN may allocate the NN for the target file, and generate metadata of the target file. And return the metadata of the target file to the above end.
  • the information of the plurality of DNs may be path information in which the DN stores the target file, for example, the path of the file in the attribute information of the file introduced above.
  • the metadata recovery information may include metadata of the target file, for example, including attribute information of the target file.
  • the attribute information may include one or more of the following:
  • Version information of the HDFS system file name, space limit information occupied by the file, user authority information, each data block size of the file, and the number of blocks of the data block of the file.
  • the file name may also carry path information of the file.
  • the foregoing method can be applied to a client of the HDFS, that is, the client can implement the foregoing method.
  • receiving a write operation request for the target file acquiring metadata of the target file from the NN, generating metadata recovery information according to the metadata of the target file, and then performing at least one of the plurality of DNs A DN sends the metadata recovery information.
  • the metadata recovery information can be used for saving by the at least one DN and sent to the NN for recovery of the metadata after an exception occurs in the NN.
  • the metadata when the NN is abnormal can be restored.
  • FIG. 3 is a schematic flowchart of another metadata recovery method according to an embodiment of the present invention. As shown in FIG. 3, the method includes the following steps:
  • the data write operation on the target file may be performed in units of data blocks, that is, an object file may be divided into multiple data blocks, wherein the size of each data block is Can be the same.
  • each data block can be 64 megabytes (MB) or 128 MB.
  • the size of the foregoing data block may be the storage space size of the data block, but in actual applications, the data amount of the file stored in each data block of one file may be different, for example, one file contains 150 MB of data, and each The size of the data block is 64MB, then the first data block of the file can store 64MB of data of the file, and the second data block can store 64MB of data of the file, and the third data block can store the data. 22MB of data for the file.
  • Metadata of the plurality of data blocks from the NN and metadata of each data block includes information of each data block and information of a DN where each of the data blocks is located.
  • the local end may apply to the NN for multiple data blocks for saving the target file, and save multiple DNs of the data block.
  • the NN can allocate a data block for the object file, and the DN of the data block, and generate metadata for each data block, and return the metadata of each data block to the above end.
  • the information of the foregoing data block may be attribute information of the data block, for example, a data block identifier and a time stamp of the data block.
  • the information of the plurality of DNs may be path information of each DN save data block, for example, the path of the file in the attribute information of the file described above.
  • the metadata recovery information of the data block may include metadata of the data block, for example, including attribute information of the target file.
  • the NN performs recovery of the metadata of the target file.
  • the obtaining the metadata of the multiple data blocks from the NN may include:
  • the metadata of each of the data blocks sent by the NN is sequentially received.
  • the local end may apply for a data block to the NN each time, so that the NN allocates a data block to the target file, and generates metadata of the data block, and returns the metadata of the data block to the local end.
  • the local end requests another data block from the NN, and the NN allocates another data block for the target file, and generates metadata of the data block. And return the metadata of the data block to the above end.
  • step 303 may be that the local end acquires the metadata recovery information of the data block for each time, and then generates the metadata recovery information of the data block to the at least one DN.
  • the obtaining the metadata of the multiple data blocks from the NN may include:
  • the local end may apply for multiple data blocks to the NN at one time, so that the NN allocates multiple data blocks for the target file, and generates metadata of the data blocks, and returns the metadata of the data blocks to the local end.
  • step 303 may be to generate metadata recovery information of a plurality of data blocks at a time, and step 304 further transmits metadata recovery information of the plurality of data blocks to the at least one DN.
  • the received metadata includes at least metadata of the first data block and metadata of the second data block, that is, the target file is at least divided into the first data block and the second data block.
  • the metadata recovery information of the generated first data block may include:
  • Attribute information of the object file and location information of the first data block in the object file are Attribute information of the object file and location information of the first data block in the object file.
  • the attribute information of the target file may include:
  • the version information of the HDFS system the file name, the number of copies of the file, the size of each block of the file, the limit information of the space occupied by the file, and the user authority information.
  • the location information of the first data block in the target file may be a data block number of the target file of the first data block.
  • the first data block may be a data block with a sequence number of 1 of the target file, where The data block with the sequence number 1 may be one of the earliest data blocks in the target file; the location information of the first data block in the target file may include:
  • the metadata recovery information of the generated second data block may include:
  • the location information of the second data block in the target file may include:
  • the metadata recovery information of the second data block may further include version information and a file name of the HDFS system, so that it can be clearly known that the metadata recovery information of the second data block is a data block for recovering which file, and the The version information of the file's HDFS system.
  • the metadata recovery information of the first data block may include:
  • the version information of the HDFS system the file name, the number of copies of the file, the size of each data block of the file, the restriction information of the space occupied by the file, the user authority information, the data block number, and the file end block information.
  • the metadata recovery information of the first data block may include:
  • Version information file name, data block number, and file end block information of the HDFS system.
  • the file end data block information may be represented by whether the current data block is an end block.
  • the file end data block information in the metadata recovery information of the first data block may be represented by a false (false);
  • the file end data block information in the metadata recovery information of the first data block may be a true (true) table.
  • the metadata recovery information of the first data block and the file end data block information in the metadata recovery information of the second data block may be false ( False) indicates that in this embodiment, only the file end block information corresponding to the last data block of the target file is represented by true (true), and the rest are represented by false (false), so that the data block can be ended according to the file.
  • the information can determine the number of data blocks included in the target file.
  • the metadata of each data block in a file may include the attribute information of the file, and the attribute information of the file is the same, for example, the restriction information of the space occupied by the file, the user authority information, the size information and the copy of the data block.
  • the number of information is the same, so that only the information needs to be written in the metadata recovery information of one of the data blocks, thereby saving the storage space of other DNs.
  • Information can also include:
  • Version information of the HDFS system file name, number of copies of the file, access time of the file, number of blocks of the data block of the file, limit information of the space occupied by the file, user permission information, data block identifier, size of the data block, and data block Timestamp.
  • the method may be an application request for sending a data block to the NN, and after receiving the request, the NN may allocate a data block for the target file, and store the data.
  • the DN of the data block may further include the following steps:
  • the foregoing method can be applied to a client of the HDFS, that is, the client can implement the foregoing method.
  • FIG. 5 is a schematic flowchart of another metadata recovery method according to an embodiment of the present invention. As shown in FIG. 5, the method includes the following steps:
  • the target file of the target file is saved on multiple DNs, and the metadata includes information of the plurality of DNs.
  • step 501 may be to store the metadata recovery information locally, or store the metadata recovery information on the third aspect storage device.
  • the NN After the NN is abnormal, send the metadata recovery information to the NN, where the metadata recovery information is used by the NN to perform recovery of the metadata.
  • the method can be applied to the DN of the HDFS.
  • the metadata recovery information of the target file sent by the client is received, and the metadata recovery information is saved; after the NN is abnormal, the metadata recovery information is sent to the NN, and the metadata recovery is performed. Information is used by the NN to perform recovery of the metadata. Thereby, it is possible to restore the metadata when the NN is abnormal.
  • FIG. 6 is a schematic flowchart of another metadata recovery method according to an embodiment of the present invention. As shown in FIG. 6, the method includes the following steps:
  • the object file is divided into a plurality of data blocks, and the metadata of each of the data blocks includes information of each data block and information of a DN of each of the data blocks.
  • the metadata recovery information received in step 601 may be metadata recovery information of one or more data blocks.
  • the target file is divided into a plurality of data blocks, wherein the DN performing the method may be metadata recovery information for storing one or more of the data blocks.
  • step 601 is to save the foregoing metadata recovery information in a specific file, for example, in a file named: blk_blockId_genstamp.blockmeta, where the name of the file may include an identifier of the data block, and the The file name may also include a time stamp of the data block.
  • the DN may receive the identifier of the data sent by the client and the time stamp of the data block while receiving the metadata recovery information of the data block.
  • the file may also save the data of the target file written by the client in the above data block to another specific file, for example, in a file named: blk_blockId.
  • the method may also save the CRC for storing the metadata in the specific file of the metadata recovery information, that is, the metadata.
  • the recovery information and CRC are stored in the same specific file.
  • the method can also store the CRC in a specific file dedicated to storing the CRC.
  • the two specific files may be set before receiving the data information, for example, when the write request sent by the client is received, the two specific files are generated. Since step 601 stores the above data information in the form of a file, this It can implement data warehouse workers (for example: Hive) or Spark functions. Spark is an open source cluster computing system based on memory computing, which aims to make data analysis faster.
  • the metadata recovery information may also be saved in a predefined data format, for example, in a HBase data format, where HBase is a distributed, column-oriented open source database.
  • the metadata recovery information when the metadata recovery information is obtained, the metadata recovery information may also be saved in the memory, so that the metadata recovery information may be sent more quickly.
  • the metadata recovery information of the memory can be directly uploaded to the NN.
  • the locally stored metadata recovery information may be scanned, and the metadata recovery information is uploaded to the NN.
  • the DN when the DN is restarted, the DN receives a notification message sent by the NN, and the notification message is used to request to send the data information, so that the message notification message can be returned to return the metadata recovery information.
  • the receiving, by the receiving, the metadata recovery information of the data block of the target file sent by the client includes:
  • the received metadata recovery information may include at least metadata recovery information of the first data block and metadata recovery information of the second data block, where:
  • the metadata recovery information of the first data block may include:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the metadata recovery information of the second data block may include:
  • the metadata recovery information of all the data blocks of the target file may be sent to the NN.
  • the DN receives the notification message of the NN abnormal restart, the DN scans the metadata recovery information of all the data blocks in the local or the memory in response to the notification message, and uploads the metadata recovery information of all the data blocks to the NN.
  • the foregoing method may be applied to the DN of the HDFS, that is, the DN may implement the foregoing method.
  • the metadata recovery information of the data block of the target file sent by the client is received, and the metadata recovery information is saved, and after the abnormality occurs in the NN, the element is sent to the NN.
  • Data recovery information the metadata recovery information is used by the NN to perform recovery of the metadata.
  • FIG. 7 is a schematic flowchart of another metadata recovery method according to an embodiment of the present invention. As shown in FIG. 7, the method includes the following steps:
  • step 702 after receiving the foregoing write operation request, the multiple DNs are allocated to the target file, and the metadata of the target file is generated.
  • the metadata recovery information sent by the at least one DN can be received, so that the metadata of the target file can be recovered.
  • the foregoing target file may be divided into multiple data blocks, where the multiple data blocks are saved on the multiple DNs;
  • generating the metadata of the target file and sending the metadata to the client may include:
  • the receiving the metadata recovery information sent by the at least one of the plurality of DNs, and recovering the metadata according to the metadata recovery information may include:
  • sending the metadata of the multiple data blocks to the client includes:
  • the foregoing multiple data blocks may include at least a first data block and a second data block, where:
  • the metadata recovery information of the first data block may include:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the metadata recovery information of the second data block may include:
  • the above method can be applied to the NN in the HDFS.
  • the receiving client sends a write operation request to the target file, generates metadata of the target file, and sends the metadata to the client; Metadata recovery information transmitted by at least one of the plurality of DNs, and recovering the metadata according to the metadata recovery information.
  • the device embodiment of the present invention is used to perform the method implemented in Embodiments 1 to 5 of the method of the present invention.
  • Embodiments 1 to 5 of the method of the present invention For the convenience of description, only parts related to the embodiment of the present invention are shown, and the specific technical details are not disclosed. Please refer to Embodiment 1, Example 2, Embodiment 3, Embodiment 4 and Embodiment 5 of the present invention.
  • FIG. 8 is a schematic structural diagram of a client according to an embodiment of the present invention. As shown in FIG. 8, the method includes: a receiving unit 81, an obtaining unit 82, a generating unit 83, and a sending unit 84, where:
  • the receiving unit 81 is configured to receive a write operation request for the target file, where the target file is saved on multiple DNs.
  • the obtaining unit 82 is configured to acquire metadata of the target file from an NN, where metadata of the target file includes information of the plurality of DNs.
  • the generating unit 83 is configured to generate a metadata recovery signal according to the metadata of the target file. interest.
  • the sending unit 84 is configured to send the metadata recovery information to at least one of the plurality of DNs, where the metadata recovery information is used to be saved by the at least one DN, and appears in the NN The exception is sent to the NN for recovery of the metadata.
  • the target file is divided into a plurality of data blocks, and the plurality of data blocks are saved on the plurality of DNs;
  • the obtaining unit 82 may be configured to acquire metadata of the plurality of data blocks from the NN, where metadata of each data block includes information of each data block and information of a DN where each of the data blocks is located;
  • the generating unit 83 may be configured to generate metadata recovery information of each of the data blocks according to the metadata of each of the data blocks.
  • metadata recovery information for generating each data block may be implemented such that the NN restores the metadata of the target file according to the metadata recovery information of the plurality of data blocks.
  • the obtaining unit 82 may be configured to sequentially receive metadata of each of the data blocks sent by the NN; or the acquiring unit 82 may be configured to receive the NN according to the target file.
  • the received metadata may include at least metadata of the first data block and metadata of the second data block, where:
  • the generated metadata recovery information of the first data block may include:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the generated metadata recovery information of the second data block may include:
  • the client may be an HDFS client.
  • receiving a write operation request for the target file acquiring metadata of the target file from the NN, generating metadata recovery information according to the metadata of the target file, and then performing at least one of the plurality of DNs A DN sends the metadata recovery information.
  • the metadata recovery information can be used for saving by the at least one DN and sent to the NN for recovery of the metadata after an exception occurs in the NN.
  • the metadata when the NN is abnormal can be restored.
  • FIG. 9 is a schematic structural diagram of a DN according to an embodiment of the present invention. As shown in FIG. 9, the method includes: a receiving unit 91 and a sending unit 92, where:
  • the receiving unit 91 is configured to receive metadata recovery information of the target file sent by the client, and save the metadata recovery information, where the metadata recovery information is that the client receives a write operation request for the target file.
  • the metadata recovery information is that the client receives a write operation request for the target file.
  • the sending unit 92 is configured to send the metadata recovery information to the NN after the NN is abnormal, and the metadata recovery information is used by the NN to perform recovery of the metadata.
  • the target file is divided into a plurality of data blocks
  • the receiving unit 91 may be configured to receive metadata recovery information of the data block of the target file sent by the client, where the data
  • the metadata recovery information of the block is metadata recovery information generated by the client according to the metadata of the data block.
  • the receiving unit 91 may be configured to sequentially receive metadata recovery information of multiple data blocks of the target file sent by the client endpoint; or the receiving unit 91 may be configured to receive the client for one time. Metadata recovery information of a plurality of data blocks of the target file transmitted.
  • the received metadata recovery information may include at least metadata recovery information of the first data block and metadata recovery information of the second data block, where:
  • the metadata recovery information of the first data block may include:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the metadata recovery information of the second data block may include:
  • the node can be the DN of the HDFS.
  • the metadata recovery information of the target file sent by the client is received, and the metadata recovery information is saved; after the NN is abnormal, the metadata recovery information is sent to the NN, and the metadata recovery is performed. Information is used by the NN to perform recovery of the metadata. Thereby, it is possible to restore the metadata when the NN is abnormal.
  • FIG. 10 is a schematic structural diagram of a NN according to an embodiment of the present invention. As shown in FIG. 10, the method includes: a receiving unit 101, a generating unit 102, and a restoring unit 103, where:
  • the receiving unit 101 is configured to receive a write operation request for the target file sent by the client, where the target file is saved on multiple DNs.
  • the generating unit 102 is configured to generate metadata of the target file, and send the metadata to the client, where the metadata includes information of the multiple DNs.
  • the recovery unit 103 is configured to receive metadata recovery information sent by at least one of the plurality of DNs when an abnormality occurs on the local end, and recover the metadata according to the metadata recovery information, where The metadata recovery information is generated by the client according to the metadata, and is sent by the client to the at least one DN for saving metadata recovery information.
  • the target file is divided into a plurality of data blocks, and the plurality of data blocks are saved on the plurality of DNs;
  • the generating unit 102 may be configured to generate metadata of the plurality of data blocks, and send metadata of the plurality of data blocks to the client, where metadata of each data block includes each data block Information and information about the DN of each of the data blocks;
  • the recovery unit 103 may be configured to receive metadata recovery information of the plurality of data blocks sent by at least one of the plurality of DNs when an abnormality occurs on the local end, and according to the elements of the multiple data blocks.
  • the data recovery information restores the metadata of the target file.
  • the generating unit 102 may be configured to generate metadata of the multiple data blocks, and sequentially send metadata of the multiple data blocks to the client; or, the generating unit 102 may be configured to generate Metadata of the plurality of data blocks, and transmitting metadata of all data blocks to the client at a time according to a write operation request of the target file.
  • the foregoing multiple data blocks may include at least a first data block and a second data block, where:
  • the metadata recovery information of the first data block may include:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the metadata recovery information of the second data block may include:
  • the foregoing node may be an NN in the HDFS.
  • the receiving client sends a write operation request to the target file, generates metadata of the target file, and sends the metadata to the client; Metadata recovery information transmitted by at least one of the plurality of DNs, and recovering the metadata according to the metadata recovery information.
  • FIG. 11 is a schematic structural diagram of another metadata recovery device according to an embodiment of the present invention, including at least one processor 1102 (for example, a CPU), at least one network interface 1105 or other communication interface, a memory 1106, and At least one communication bus 1103 for implementing this Connection communication between these devices.
  • the processor 1102 is configured to execute executable modules, such as computer programs, stored in the memory 1106.
  • the memory 1106 may include a high speed random access memory (RAM), and may also include a non-volatile memory such as at least one disk memory.
  • the communication connection between the system gateway and at least one other network element is implemented by at least one network interface 1105 (which may be wired or wireless), and may use an Internet, a wide area network, a local network, a metropolitan area network, or the like.
  • the memory 1106 stores a program 11061 that can be executed by the processor 1102.
  • This program includes:
  • the object file may be divided into a plurality of data blocks, the plurality of data blocks may be saved on the plurality of DNs; and the processor 1102 performs the acquisition of the target file from the NN.
  • the metadata, the program for generating the metadata recovery information according to the metadata of the target file may include:
  • the metadata of each data block includes information of each data block and information of a DN where each of the data blocks is located;
  • the metadata executed by the processor 1102 is used to acquire the metadata of the multiple data blocks from the NN.
  • the program can include:
  • the received metadata may include at least metadata of the first data block and metadata of the second data block, where:
  • the generated metadata recovery information of the first data block may include:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the generated metadata recovery information of the second data block may include:
  • the device may be a device to which the client of the HDFS belongs.
  • the metadata recovery information can be used to be saved by the at least one DN, and after the NN has an abnormality, it is sent to the NN to recover the metadata, so that the metadata when the NN is abnormal can be restored.
  • FIG. 12 is a schematic structural diagram of a DN device according to an embodiment of the present invention, including at least one processor 1202 (for example, a CPU), at least one network interface 1205 or other communication interface, a memory 1206, and at least one communication. Bus 1203 is used to implement connection communication between these devices.
  • the processor 1202 is configured to execute executable modules, such as computer programs, stored in the memory 1206.
  • the memory 1206 may include a high speed random access memory (RAM), and may also include a non-volatile memory such as at least one disk memory.
  • RAM high speed random access memory
  • Through at least one network interface 1205 realizes the communication connection between the system gateway and at least one other network element, and can use the Internet, a wide area network, a local network, a metropolitan area network, and the like.
  • the memory 1206 stores a program 12061 that can be executed by the processor 1202.
  • the program includes:
  • the metadata recovery information is sent to the NN, and the metadata recovery information is used by the NN to perform recovery of the metadata.
  • the target file is divided into a plurality of data blocks
  • the program executed by the processor 1202 to receive the metadata recovery information of the target file sent by the client may include:
  • the program that is executed by the processor 1202 to receive the metadata recovery information of the data block of the target file that is sent by the client may include:
  • the received metadata recovery information may include at least metadata recovery information of the first data block and metadata recovery information of the second data block, where:
  • the metadata recovery information of the first data block may include:
  • the metadata recovery information of the second data block may include:
  • the device of the local node may be the device to which the DN of the HDFS belongs.
  • the metadata recovery information of the target file sent by the client is received, and the metadata recovery information is saved; after the NN is abnormal, the metadata recovery information is sent to the NN, and the metadata recovery is performed. Information is used by the NN to perform recovery of the metadata. Thereby, it is possible to restore the metadata when the NN is abnormal.
  • FIG. 13 is a schematic structural diagram of an NN device according to an embodiment of the present invention, including at least one processor 1302 (eg, a CPU), at least one network interface 1305 or other communication interface, a memory 1306, and at least one communication.
  • a bus 1303 is used to implement connection communication between these devices.
  • the processor 1302 is configured to execute executable modules, such as computer programs, stored in the memory 1306.
  • the memory 1306 may include a high speed random access memory (RAM) and may also include a non-volatile memory such as at least one disk memory.
  • the communication connection between the system gateway and at least one other network element is implemented by at least one network interface 1305 (which may be wired or wireless), and may use an Internet, a wide area network, a local network, a metropolitan area network, or the like.
  • the memory 1306 stores a program 13061 that can be executed by the processor 1302. This program includes:
  • the object file is divided into a plurality of data blocks, the plurality of data blocks are saved on the plurality of DNs; the processor 1302 performs metadata generation of the object file, and Sending the metadata to the client may include:
  • the program executed by the processor 1302, which receives the metadata recovery information sent by the at least one of the plurality of DNs, and recovers the metadata according to the metadata recovery information, may include:
  • the program executed by the processor 1302 to send the metadata of the multiple data blocks to the client may include:
  • the foregoing multiple data blocks may include at least a first data block and a second data block, where:
  • the metadata recovery information of the first data block may include:
  • Attribute information of the target file and location information of the first data block in the target file are Attribute information of the target file and location information of the first data block in the target file
  • the metadata recovery information of the second data block may include:
  • the foregoing node device may be a device to which the NN in the HDFS belongs.
  • the receiving client sends a write operation request to the target file; generating metadata of the target file, and sending the metadata to the client; and when the local end is abnormal, receiving the multiple Metadata recovery information transmitted by at least one of the DNs, and the metadata is restored according to the metadata recovery information.
  • the metadata when the NN is abnormal can be restored.
  • the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

Abstract

L'invention concerne un procédé de récupération de métadonnées et un dispositif associé. Le procédé peut comporter les étapes consistant à: recevoir une demande d'opération d'écriture relative à un fichier de destination, le fichier de destination étant sauvegardé dans des nœuds de données multiples; acquérir des métadonnées du fichier de destination en provenance d'un nœud de nom, les métadonnées du fichier de destination contenant des informations concernant les nœuds de données multiples; générer des informations de récupération de métadonnées d'après les métadonnées du fichier de destination; et envoyer les informations de récupération de métadonnées à au moins un nœud de données parmi les nœuds de données multiples, les informations de récupération de métadonnées étant utilisées en vue d'être sauvegardées par le ou les nœuds de données, et étant envoyées au nœud de nom pour récupérer les métadonnées lorsqu'une anomalie affecte le nœud de nom. Les modes de réalisation de la présente invention sont capables de récupérer des métadonnées lorsqu'une anomalie affecte un NN.
PCT/CN2015/083226 2014-08-27 2015-07-02 Procédé de récupération de métadonnées et dispositif associé WO2016029744A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201410427894.2 2014-08-27
CN201410427894.2A CN104202387B (zh) 2014-08-27 2014-08-27 一种元数据恢复方法及相关装置

Publications (1)

Publication Number Publication Date
WO2016029744A1 true WO2016029744A1 (fr) 2016-03-03

Family

ID=52087612

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2015/083226 WO2016029744A1 (fr) 2014-08-27 2015-07-02 Procédé de récupération de métadonnées et dispositif associé

Country Status (2)

Country Link
CN (1) CN104202387B (fr)
WO (1) WO2016029744A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112947864A (zh) * 2021-03-29 2021-06-11 南方电网数字电网研究院有限公司 元数据的存储方法、装置、设备和存储介质

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104202387B (zh) * 2014-08-27 2017-11-24 华为技术有限公司 一种元数据恢复方法及相关装置
CN107562368B (zh) * 2016-06-30 2019-11-22 杭州海康威视数字技术股份有限公司 一种数据处理方法及装置
CN108241555B (zh) * 2016-12-26 2022-03-01 阿里巴巴集团控股有限公司 一种分布式数据库的备份、恢复方法、装置和服务器
CN109947730B (zh) * 2017-07-25 2024-02-02 中兴通讯股份有限公司 元数据恢复方法、装置、分布式文件系统及可读存储介质
CN109426587B (zh) * 2017-08-25 2020-08-28 杭州海康威视数字技术股份有限公司 一种数据恢复方法及装置
CN108390771B (zh) * 2018-01-25 2021-04-16 中国银联股份有限公司 一种网络拓扑重建方法和装置
CN111984460B (zh) * 2019-05-21 2023-11-24 华为云计算技术有限公司 元数据的恢复方法及装置
CN115185723B (zh) * 2022-06-02 2023-06-02 广州市玄武无线科技股份有限公司 一种Region元信息修复方法、系统、装置及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030220943A1 (en) * 2002-05-23 2003-11-27 International Business Machines Corporation Recovery of a single metadata controller failure in a storage area network environment
CN102024044A (zh) * 2010-12-08 2011-04-20 华为技术有限公司 分布式文件系统
CN102843403A (zh) * 2011-06-23 2012-12-26 盛大计算机（上海）有限公司 基于分布式文件系统的文件处理方法、系统及客户端
CN103076994A (zh) * 2012-12-31 2013-05-01 北京中科大洋科技发展股份有限公司 一种san共享文件系统中实现离线写文件的方法
CN104202387A (zh) * 2014-08-27 2014-12-10 华为技术有限公司 一种元数据恢复方法及相关装置

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100587692C (zh) * 2007-01-26 2010-02-03 华中科技大学 一种提高元数据服务可靠性的方法及其系统
CN101539873B (zh) * 2009-04-15 2011-02-09 成都市华为赛门铁克科技有限公司 数据恢复的方法、数据节点及分布式文件系统
CN101986276B (zh) * 2010-10-21 2013-07-10 华为数字技术（成都）有限公司 文件存储方法、文件恢复方法、系统及服务器
JP5611889B2 (ja) * 2011-05-17 2014-10-22 株式会社東芝 データ転送装置、データ送信システムおよびデータ送信方法
CN103067418A (zh) * 2011-10-19 2013-04-24 北大方正集团有限公司 一种元数据重建方法、设备及系统
CN103761162B (zh) * 2014-01-11 2016-12-07 深圳清华大学研究院 分布式文件系统的数据备份方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030220943A1 (en) * 2002-05-23 2003-11-27 International Business Machines Corporation Recovery of a single metadata controller failure in a storage area network environment
CN102024044A (zh) * 2010-12-08 2011-04-20 华为技术有限公司 分布式文件系统
CN102843403A (zh) * 2011-06-23 2012-12-26 盛大计算机（上海）有限公司 基于分布式文件系统的文件处理方法、系统及客户端
CN103076994A (zh) * 2012-12-31 2013-05-01 北京中科大洋科技发展股份有限公司 一种san共享文件系统中实现离线写文件的方法
CN104202387A (zh) * 2014-08-27 2014-12-10 华为技术有限公司 一种元数据恢复方法及相关装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112947864A (zh) * 2021-03-29 2021-06-11 南方电网数字电网研究院有限公司 元数据的存储方法、装置、设备和存储介质
CN112947864B (zh) * 2021-03-29 2024-03-08 南方电网数字平台科技(广东)有限公司 元数据的存储方法、装置、设备和存储介质

Also Published As

Publication number Publication date
CN104202387A (zh) 2014-12-10
CN104202387B (zh) 2017-11-24

Similar Documents

Publication Publication Date Title
WO2016029744A1 (fr) Procédé de récupération de métadonnées et dispositif associé
KR102153804B1 (ko) 데이터 동기화 방법, 장치, 및 시스템
US11221995B2 (en) Data replication from a cloud-based storage resource
TWI710897B (zh) 資料處理方法和流計算系統
US9779153B2 (en) Data transfer between storage systems using data fingerprints
CN107528872B (zh) 一种数据恢复方法、装置及云存储系统
WO2019075978A1 (fr) Procédé et appareil de transmission de données, dispositif informatique et support d'informations
WO2019144761A1 (fr) Procédé de synchronisation de données, système distribué, et dispositif
US20130238552A1 (en) Systems and methods for synchronizing files in a networked communication system
JP6264666B2 (ja) データ格納方法、データストレージ装置、及びストレージデバイス
WO2016180055A1 (fr) Procédé, dispositif et système de stockage et de lecture de données
CA2959999C (fr) Synchronisation de donnees reparties et resolution de conflits
CN107153644B (zh) 一种数据同步方法及装置
WO2020151181A1 (fr) Procédé et appareil de mise à jour de données inter-plateformes sur la base d'une chaîne de blocs et dispositif informatique
US20160241441A1 (en) Method and apparatus for changing configurations
CN109460438B (zh) 消息数据存储方法、装置、计算机设备和存储介质
CN105450682B (zh) 一种用于数据同步保存、向客户端同步数据的方法、装置和系统
CN109739435B (zh) 文件存储和更新方法及装置
CN111338834B (zh) 数据存储方法和装置
CN109254998B (zh) 数据管理方法、物联网设备、数据库服务器及系统
CN103716384A (zh) 跨数据中心实现云存储数据同步的方法和装置
CN106817387B (zh) 一种数据同步方法、装置和系统
CN106873902B (zh) 一种文件存储系统、数据调度方法及数据节点
CN107516048B (zh) 一种控制分布式文件系统中文件访问的方法与设备
WO2017005116A1 (fr) Procédé de traitement de fichier intermédiaire, clients, serveurs et système

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15836410

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15836410

Country of ref document: EP

Kind code of ref document: A1