Connect public, paid and private patent data with Google Patents Public Datasets

Metadata recovery method and relevant device

Info

Publication number
WO2016029744A1
WO2016029744A1 PCT/CN2015/083226 CN2015083226W WO2016029744A1 WO 2016029744 A1 WO2016029744 A1 WO 2016029744A1 CN 2015083226 W CN2015083226 W CN 2015083226W WO 2016029744 A1 WO2016029744 A1 WO 2016029744A1
Authority
WO
Grant status
Application
Patent type
Prior art keywords
metadata
recovery
file
data
target
Prior art date
Application number
PCT/CN2015/083226
Other languages
French (fr)
Chinese (zh)
Inventor
刘洋
张克星
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L29/00Arrangements, apparatus, circuits or systems, not covered by a single one of groups H04L1/00 - H04L27/00 contains provisionally no documents
    • H04L29/02Communication control; Communication processing contains provisionally no documents
    • H04L29/06Communication control; Communication processing contains provisionally no documents characterised by a protocol
    • H04L29/08Transmission control procedure, e.g. data link level control procedure

Abstract

Disclosed are a metadata recovery method and a relevant device. The method can comprise: receiving a write operation request for a target file, wherein the target file is saved in multiple data nodes; acquiring metadata of the target file from a namenode, the metadata of the target file containing information about the multiple data nodes; generating metadata recovery information according to the metadata of the target file; and sending the metadata recovery information to at least one data node of the multiple data nodes, the metadata recovery information being used for being saved by the at least one data node, and being sent to the namenode for recovering the metadata when abnormality occurs to the namenode. The embodiments of the present invention can recover metadata when abnormality occurs to an NN.

Description

一种元数据恢复方法及相关装置 A meta data restoration method and related apparatus 技术领域 FIELD

本发明涉及通信领域,尤其涉及一种元数据恢复方法及相关装置。 The present invention relates to the field of communications, particularly to a method and related meta data recovery apparatus.

背景技术 Background technique

Hadoop是目前通信领域中应用十分广泛的分布式系统架构,在Hadoop中包含一个非常重要的存储组件,即Hadoop分布式文件系统(Hadoop Distributed File System,简称HDFS)。 Hadoop is currently in the field of communications is widely distributed system architecture, contains a very important component stored in Hadoop, that Hadoop Distributed File System (Hadoop Distributed File System, referred to HDFS). 其中,HDFS包括元数据节点(NameNode,简称NN)和数据节点(DataNode,简称DN),NN主要用来提供文件的元数据服务,该服务中NN在内存会生成该文件的元数据,元数据可以表示该文件的数据结构,而DN主要用来存储文件的实际数据。 Wherein, the HDFS including metadata node (the NameNode, referred to as NN) and a data node (DataNodes, referred to as DN), NN mainly used to provide service metadata file, the generated metadata file in memory will be the service NN, metadata It may represent a data structure of the file, and the actual data DN is mainly used to store files. 在实际应用中当NN出现异常时,这样该NN就会出现元数据丢失的问题,以导致该NN失效。 When abnormal NN, NN so that there will be metadata loss problems in practical applications, to cause the failure of NN.

其中,目前主要通过如下技术解决上述问题: Among them, mainly to solve the above problems by the following technologies:

在NN内部开启一个定时线程,该线程定期将NN内存中的元数据写到本地磁盘或者远端存储设备形成一个特定文件。 Inside thread opening timing NN, the NN thread periodically metadata written to the local memory or a remote disk storage devices forming a specific file. 当NN出现异常后,NN在重新启动过程中加载该特定文件,以恢复之前的某一时段的元数据,以实现NN恢复到这一时段的内存状态。 When abnormal NN, NN load the specific files during restart, metadata for a period of time prior to recovery, to enable the memory to restore the state of the NN period.

但上述技术方案中,由于是定期的存储NN内的元数据,例如:每隔十分钟存储一次元数据,这样当NN出现异常的时间点还未到十分钟一次的存储时间点时,那么NN内的出现异常时的元数据不会存储,从而这部分元数据就无法恢复。 However, the above technical solution, since the metadata stored in the NN periodically, for example: every ten minutes metadata stored once, so that the time point when the abnormality has not occurred NN to ten minutes during a storage time point, then the NN metadata will not appear in the store at the time of abnormal, so this part of the metadata can not be recovered. 可见,上述技术方案中无法恢复NN异常时的元数据。 Visible above technical solution can not be restored when the metadata NN exception.

发明内容 SUMMARY

本发明提供了一种元数据恢复方法及相关装置,可以恢复NN异常时的元数据。 The present invention provides a method of recovering data and metadata related apparatus, can be restored when the metadata NN abnormality.

第一方面,本发明提供一种元数据恢复方法,包括: A first aspect, the present invention provides a method of recovering data element, comprising:

接收对目标文件的写操作请求,其中,所述目标文件保存在多个DN上; Receiving a write request for a target file, wherein said plurality of target files stored on the DN;

从NN获取所述目标文件的元数据,所述目标文件的元数据包含所述多个DN的信息; Acquiring the target file from the metadata NN, the target file metadata contains information of the plurality of the DN;

根据所述目标文件的元数据生成元数据恢复信息; Generate metadata according to the metadata recovery information for the target file;

向所述多个DN中的至少一个DN发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 DN direction in at least one of said plurality of transmitting the metadata DN restoration information, said restoration information metadata for the at least one DN is saved, and then transmitted to the abnormality occurs in the NN NN recovery of the metadata.

在第一方面的第一种可能的实现方式中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上; In a first possible implementation manner of the first aspect, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

则,所述从NN获取所述目标文件的元数据,根据所述目标文件的元数据生成元数据恢复信息包括: Then, the metadata acquiring the target file from the NN, the resume information including metadata of the target file generates metadata from:

从所述NN获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息; Acquiring metadata of the plurality of data blocks from the NN, the metadata of each data block contains information DN information and each data block of said each data block is located;

根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 The metadata of each data block, each data block generates the metadata recovery information.

结合第一方面的第一种可能的实现方式,在第二种可能实现方式中,所述从所述NN获取所述多个数据块的元数据包括: Binding a first possible implementation of the first aspect, in a second possible implementation manner, the obtaining of the plurality of data blocks from the NN metadata comprises:

依次接收所述NN发送的所述每个数据块的元数据;或者,接收所述NN根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 NN sequentially receiving said transmitted the metadata of each data block; or NN receiving the request metadata disposable all data blocks transmitted according to a write operation of the target file.

结合第一方面的第一种可能的实现方式或者第二种可能的实现方式,在第三种可能实现方式中,所述接收的元数据至少包括第一数据块的元数据和第二数据块的元数据,其中: First possible implementation manner of the first aspect or the second possible implementation manner, in a third possible implementation, the received metadata comprises metadata of at least a first data block and second data block metadata, wherein:

所述生成的所述第一数据块的元数据恢复信息包括: The generated metadata first data block recovery information comprising:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述生成的所述第二数据块的元数据恢复信息包括: Generating metadata for the second data block of the recovery information comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

第三方面,本发明提供一种元数据恢复方法,包括: A third aspect, the present invention provides a method of recovering data element, comprising:

接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从NN获取的所述目标文件的元数据,并根据所述元数据生成的元数据恢复信息.; Receiving the target files sent by the client metadata recovery information and storing the recovery information metadata, wherein the metadata information is restored upon receiving a client request for a write target file acquired from NN the target file metadata, and restoring metadata information to the metadata generated.;

在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 After the NN abnormal, sending the information to the meta data recovery NN, metadata information is the NN Recovery Recovery for the metadata.

在第二方面的第一种可能的实现方式中,所述目标文件被划分为多个数据块,所述接收客户端发送的目标文件的元数据恢复信息,包括: A first possible implementation of the second aspect, the target file is divided into a plurality of data blocks, the target receives the metadata file recovery information sent by the client, comprising:

接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,所述数据块的元数据恢复信息为所述客户端根据该数据块的元数据生成的元数据恢复信息。 Receiving the data blocks of the target file is sent by the client metadata recovery information, metadata information of the data block to recover the information according to the recovery client metadata in the metadata generated by the data block.

结合第二方面的第一种可能的实现方式,在第二方面的第二种可能的实现方式中,所述接收所述客户端发送的所述目标文件的数据块的元数据恢复信息包括: Binding a first possible implementation of the second aspect, the second aspect of the second possible implementation manner, the receiving metadata data blocks of the target client sends the file recovery information comprising:

依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Metadata plurality of data blocks sequentially receives the target file sent by the customer endpoint recovery information; or metadata of a plurality of data blocks of the target client receives the file recovery information transmitted disposable.

结合第二方面的第一种可能的实现方式或者第二种可能的实现方式中,在第二方面的第三种可能的实现方式中,所述接收的元数据恢复信息至少包 括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: Binding a first possible implementation of the second aspect or the second possible implementation manner, in a third possible implementation of the second aspect, the recovery of the received metadata information includes at least a first data block restoring metadata metadata information and the second resume information data block, wherein:

所述第一数据块的元数据恢复信息包括: The metadata information of first data recovery block comprises:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述第二数据块的元数据恢复信息包括: The metadata information of the second block of data recovery comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

第三方面,本发明提供一种元数据恢复方法,包括: A third aspect, the present invention provides a method of recovering data element, comprising:

接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个DN上; Target file is sent by a client requests a write operation, wherein the plurality of target files stored on the DN;

生成所述目标文件的元数据,并向所述客户端发送所述元数据,其中,所述元数据包含所述多个DN的信息; Generating a metadata of the target file, the client sends to the metadata, wherein the metadata information comprises a plurality of DN;

当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至所述至少一个DN进行保存的元数据恢复信息。 When an exception occurs this end, a plurality of receiving metadata of at least one DN DN recover transmitted information, and restoring the restoration information metadata according to the metadata, wherein the metadata information is the recovery client-generated based on the metadata sent by the client to at least one of said stored metadata DN recovery information.

在第三方面的第一种可能的实现方式中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上; In a first possible implementation manner of the third aspect, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

则所述生成所述目标文件的元数据,并向所述客户端发送所述元数据包括: The metadata is generated for the target file, the client sends to the metadata comprises:

生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息; Generating a plurality of data blocks of metadata, and metadata for the client sends the plurality of data blocks, each data block metadata comprises location information of each data block and each data block of the the DN information;

所述接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,包括: The receiving the at least one of the plurality DN DN metadata transmitted recovery information, and restoring the restoration information metadata according to the metadata, comprising:

接收所述多个DN中的至少一个DN发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 Receiving the plurality of data blocks in at least one of the plurality of DN DN recover transmitted information metadata, and restoring the restoration information file meta data of the target data blocks of the plurality of metadata.

结合第三方面的第一种可能的实现方式,在第三方面的第二种可能的实现方式中,所述向所述客户端发送所述多个数据块的元数据包括: Binding a first possible implementation of the third aspect, a second possible implementation manner of the third aspect, the transmitting client to the plurality of data blocks of the metadata comprises:

依次向所述客户端发送所述多个数据块的元数据;或者,根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Sequentially transmitted to the client metadata of the plurality of data blocks; or, according to a write request for a one-time operation of the target file metadata for all transmission data block to the client.

结合第三方面的第一种可能的实现方式或者第二种可能的实现方式,在第三方面的第三种可能的实现方式中,所述多个数据块至少包括第一数据块和第二数据块,其中: Binding a first possible implementation of the third aspect or the second possible implementation manner, a third possible implementation manner of the third aspect, the plurality of data blocks include at least a first and a second data block data block, wherein:

所述第一数据块的元数据恢复信息包括: The metadata information of first data recovery block comprises:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述第二数据块的元数据恢复信息包括: The metadata information of the second block of data recovery comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

第四方面,本发明提供一种客户端,包括:接收单元、获取单元、生成单元和发送单元,其中: A fourth aspect, the present invention provides a client, comprising: a receiving unit, an acquisition unit, generation unit, and a sending unit, wherein:

所述接收单元,用于接收对目标文件的写操作请求,其中,所述目标文件保存在多个DN上; The receiving unit, for receiving a write request for a target file, wherein said plurality of target files stored on the DN;

所述获取单元,用于从NN获取所述目标文件的元数据,所述目标文件的元数据包含所述多个DN的信息; The obtaining unit is configured to obtain the target file from the metadata NN, the target file metadata contains information of the plurality of the DN;

所述生成单元,用于根据所述目标文件的元数据生成元数据恢复信息; The generating means for generating metadata according to the metadata recovery information for the target file;

所述发送单元,用于向所述多个DN中的至少一个DN发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个DN进行保存,并在所 述NN出现异常后被发送到所述NN进行所述元数据的恢复。 The sending unit, configured to DN of said plurality of transmitting at least one of the metadata DN restoration information, said restoration information metadata for the at least one DN is saved, and abnormality occurs in the NN NN then sent to the recovery of the metadata.

在第四方面的第一种可能的实现方式中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上; A first possible implementation of the fourth aspect, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

则,所述获取单元用于从所述NN获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息; Then, the obtaining unit configured to obtain metadata of the plurality of data blocks from the NN, the metadata of each data block contains information DN information and each data block of said each data block is located;

所述生成单元用于根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 The metadata generating unit for each data block, each data block generates the metadata recovery information.

结合第四方面的第一种可能的实现方式,在第四方面的第二种可能的实现方式中,所述获取单元用于依次接收所述NN发送的所述每个数据块的元数据;或者,所述获取单元用于接收所述NN根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 Binding a first possible implementation of the fourth aspect, the fourth aspect of a second possible implementation, the metadata acquiring unit for sequentially receiving the data sent by the NN for each data block; Alternatively, the acquiring unit is configured to receive the metadata NN all data blocks transmitted according to the request for a one-time writing operation of the target file.

结合第四方面的第一种可能的实现方式或者第二种可能的实现方式,在第四方面的第三种可能的实现方式中,所述接收的元数据至少包括第一数据块的元数据和第二数据块的元数据,其中: First possible implementation manner of the fourth aspect or the second possible implementation manner, a third possible implementation of the fourth aspect, the received metadata comprises metadata of at least a first data block metadata and the second data block, wherein:

所述生成的所述第一数据块的元数据恢复信息包括: The generated metadata first data block recovery information comprising:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述生成的所述第二数据块的元数据恢复信息包括: Generating metadata for the second data block of the recovery information comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

第五方面,本发明提供一种DN,包括:接收单元和发送单元,其中: A fifth aspect, the present invention provides the DN, comprising: a receiving unit and a sending unit, wherein:

所述接收单元,用于接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从NN获取的所述目标文件的元数据,并根据 所述元数据生成的元数据恢复信息.; The receiving unit, for receiving metadata target file recovery information sent by the client, and storing the recovery information metadata, wherein the metadata information is restored in the client receiving a write operation of the target file when requested, the metadata of the target file acquired from the NN, and recovery information according to the metadata generated by the metadata.;

所述发送单元,用于在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 The sending unit, configured to, after the NN abnormal, sending the information to the meta data recovery NN, metadata information is the NN Recovery Recovery for the metadata.

在第五方面的第一种可能的实现方式中,所述目标文件被划分为多个数据块,所述接收单元用于接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,所述数据块的元数据恢复信息为所述客户端根据该数据块的元数据生成的元数据恢复信息。 The first possible implementation of the fifth aspect, the target file is divided into a plurality of data blocks, said data blocks of the target file receiving unit for receiving the transmitted client metadata retriever information, metadata information of the data block to recover the information according to the recovery client metadata in the metadata generated by the data block.

结合第五方面的第一种可能的实现方式,在第五方面的第二种可能的实现方式中,所述接收单元用于依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,所述接收单元用于接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 First possible implementation manner of the fifth aspect, a second possible implementation of the fifth aspect, the plurality of received data blocks for the target file unit for sequentially receiving said transmitted customer endpoint recovery information metadata; or the unit receiving metadata for receiving said plurality of data blocks sent by the client disposable target file recovery information.

结合第五方面的第一种可能的实现方式或者第二种可能的实现方式,在第三种可能的实现方式中,所述接收的元数据恢复信息至少包括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: First possible implementation manner of the fifth aspect or the second possible implementation manner, in a third possible implementation manner, the recovery information received metadata comprises metadata of at least the first data block recovery information and the second data block metadata recovery information, wherein:

所述第一数据块的元数据恢复信息包括: The metadata information of first data recovery block comprises:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述第二数据块的元数据恢复信息包括: The metadata information of the second block of data recovery comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

第六方面,本发明提供一种NN,包括:接收单元、生成单元和恢复单元,其中: A sixth aspect, the present invention provides a NN, comprising: a receiving unit, a generating unit and a restoring unit, wherein:

所述接收单元,用于接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个DN上; The receiving unit is configured to write the target file operation request sent by a client, wherein the plurality of target files stored on the DN;

所述生成单元,用于生成所述目标文件的元数据,并向所述客户端发送 所述元数据,其中,所述元数据包含所述多个DN的信息; The generating unit generating metadata for the target file, the client sends to the metadata, wherein the metadata information comprises a plurality of DN;

所述恢复单元,用于当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至所述至少一个DN进行保存的元数据恢复信息。 The restoring unit configured to, when abnormality occurs this end, a plurality of receiving metadata of at least one DN DN recover transmitted information, and restoring the restoration information metadata according to the metadata, wherein said metadata is information of the client is generated based on the metadata sent by the client to at least one of said stored metadata DN recovery information.

在第六方面的第一种可能的实现方式中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上; The first possible implementation of the sixth aspect, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

所述生成单元用于生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息; The metadata generating means for generating the plurality of data blocks, the client sends to the metadata of the plurality of data blocks, each data block of metadata containing information for each data block and the DN said information of each data block is located;

所述恢复单元用于当本端出现异常时,接收所述多个DN中的至少一个DN发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 The restoring unit configured to, when abnormality occurs this end, the metadata for the plurality of received data blocks of said at least one of the plurality DN DN recover transmitted information, and restoring metadata to the plurality of data blocks information restoring metadata of the target file.

结合第六方面的第一种可能的实现方式,在第二种可能的实现方式中,所述生成单元用于生成所述多个数据块的元数据,并依次向所述客户端发送所述多个数据块的元数据;或者,所述生成单元用于生成所述多个数据块的元数据,并根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Binding a first possible implementation of the sixth aspect, in a second possible implementation manner, the generating means for generating the plurality of data blocks of metadata, and sequentially transmits to the client the metadata plurality of data blocks; Alternatively, the generating means for generating a plurality of data blocks of the metadata, and requests all the data blocks sent at once to the client according to the write operation of the target file metadata .

结合第六方面的第一种可能的实现方式或者第二种可能的实现方式,在第三种可能的实现方式中,所述多个数据块至少包括第一数据块和第二数据块,其中: Binding a first possible implementation of the sixth aspect or the second possible implementation manner, in a third possible implementation manner, the plurality of data blocks include at least a first data block and second data block, wherein :

所述第一数据块的元数据恢复信息包括: The metadata information of first data recovery block comprises:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述第二数据块的元数据恢复信息包括: The metadata information of the second block of data recovery comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

上述技术方案中,接收对目标文件的写操作请求,再从NN获取所述目标文件的元数据,根据所述目标文件的元数据生成元数据恢复信息,再向所述多个DN中的至少一个DN发送所述元数据恢复信息。 In the above technical solution, receiving a request for a write operation of the target file, then the target file to obtain metadata from NN, generate metadata according to the metadata recovery information for the target file, again at least a plurality of DN DN a metadata transmitting said recovery information. 这样该元数据恢复信息就可以用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 Resume information so that the metadata can be used to save the at least one DN, and then sent to the NN abnormal recover the metadata present in the NN. 从而可以恢复NN出异常时的元数据。 NN thereby restore the metadata when an abnormality.

附图说明 BRIEF DESCRIPTION

为了更清楚地说明本发明实施例或现有技术中的技术方案,下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本发明的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。 In order to more clearly illustrate the technical solutions in the embodiments or the prior art embodiment of the present invention, the accompanying drawings briefly described embodiments or the prior art needed to be used in describing the embodiments. Apparently, the drawings in the following description only Some embodiments of the present invention are embodiments, those of ordinary skill in the art is concerned, without creative efforts, can derive from these drawings other drawings.

图1是本发明实施例提供的一种可选的元数据的示意图; FIG 1 is a schematic view of an alternative embodiment of metadata to an embodiment of the present invention;

图2是本发明实施例提供的一种元数据恢复方法的流程示意图; FIG 2 is a schematic flow diagram of a method for the recovery of metadata according to an embodiment of the present invention;

图3是本发明实施例提供的另一种元数据恢复方法的流程示意图; 3 is a schematic flow diagram of another method for recovery of metadata according to an embodiment of the present invention;

图4是本发明实施例提供的一种可选的数据信息的示意图; FIG 4 is a schematic view of an alternative embodiment of data provided by the present invention;

图5是本发明实施例提供的另一种元数据恢复方法的流程示意图; FIG 5 is a schematic flow diagram of another embodiment of metadata to provide recovery method of the present invention;

图6是本发明实施例提供的另一种元数据恢复方法的流程示意图; FIG 6 is a flow diagram of another method of recovery of metadata according to an embodiment of the present invention;

图7是本发明实施例提供的另一种元数据恢复方法的流程示意图; FIG. 7 is a flow diagram of another method of recovery of metadata according to an embodiment of the present invention;

图8是本发明实施例提供的一种客户端的结构示意图; FIG 8 is a block diagram of a client according to an embodiment of the present invention, end;

图9是本发明实施例提供的一种DN的结构示意图; 9 is a schematic structure of a DN to an embodiment of the present invention;

图10是本发明实施例提供的一种NN的结构示意图; FIG 10 is a configuration diagram of a NN to an embodiment of the present invention;

图11是本发明实施例提供的另一种元数据恢复设备的结构示意图; FIG 11 is a schematic structural diagram of another embodiment of metadata to provide recovery apparatus embodiment of the present invention;

图12是本发明实施例提供的一种DN设备的结构示意图; FIG 12 is a configuration diagram of a DN apparatus according to an embodiment of the present invention;

图13是本发明实施例提供的一种NN设备的结构示意图。 FIG 13 is a configuration diagram of a NN apparatus according to an embodiment of the present invention.

具体实施方式 detailed description

下面将结合本发明实施例中的附图,对本发明实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本发明一部分实施例,而不是全部的实施例。 Below in conjunction with the accompanying drawings of the present invention in embodiments, the technical solutions in the embodiments of the present invention are clearly and completely described, obviously, the described embodiments are merely part of embodiments of the present invention rather than all embodiments. 基于本发明中的实施例,本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例,都属于本发明保护的范围。 Based on the embodiments of the present invention, those of ordinary skill in the art to make all other embodiments without creative work obtained by, it falls within the scope of the present invention.

本发明实施例中,元数据可以是用于表示文件数据结构的数据,其中,文件的元数据具体可以表示该文件所在的HDFS系统的版本信息和该文件的属性信息,例如:文件的元数据包括的HDFS系统的版本信息可以包括如下一项或者多项: Embodiments of the present invention, the metadata may be used to represent the data structure of the file data, wherein the metadata file attribute information may represent a specific version of the file system information HDFS and the file is located, for example: metadata file HDFS version information included in the system may include one or more of the following:

HDFS系统的版本信息(例如:imgVersion)、名称空间标识(例如:namespaceID)、HDFS系统的文件个数(例如:numFiles)和HDFS系统的时间戳(例如:genStamp); HDFS system version information (e.g.: imgVersion), name space identifier (e.g.: namespaceID), the number of HDFS file system (e.g.: numFiles) HDFS systems and timestamp (e.g.: genStamp);

文件的元数据包括的该文件的属性信息可以包括如下一项或者多项: Attribute information of the file comprises a metadata file may include one or more of the following:

文件的路径(例如:Path)、文件的副本数(例如:replicas)、文件的访问时间(例如:atime)、文件的每个数据块大小(例如:blockSize)、文件的数据块的块数(例如:numBlock)、文件占据的空间的限制信息(例如:nsQuota和dsQuota)和用户权限信息(例如:username、group和perm)。 Path of the file (for example: Path), the number of copies of the file (e.g.: replicas), file access time (e.g.: atime), each data block size of the file (for example: blockSize), the number of data blocks of a file ( For example: numBlock), limiting the space occupied by the file information (for example: nsQuota and dsQuota) and user privileges information (for example: username, group and perm).

另外,当文件包括多个数据块时,文件的元数据还可以包括各数据块的元数据,其中,数据块的元数据包括该数据块所属的文件的属性信息和该数据块的属性信息;其中,数据块的属性信息可以包括如下一项或者多项: Further, when the file includes multiple data blocks, the metadata file can also include metadata for each data block, wherein the metadata includes attribute information of data blocks of the data block belongs attribute information file and the data block; wherein the attribute information of the data block may include one or more of:

数据块标识(例如:BlockId)、数据块存储的数据量(例如:numBytes)和数据块的时间戳(例如:genStamp)。 Data block identifier (eg: BlockId), the amount of data stored in the data block (e.g.: numBytes) and timestamp data block (e.g.: genStamp). 另外,文件的属性信息上面已经介绍,此处不重复说明。 In addition, the file attribute information above has been introduced, will not be repeated here.

例如:文件的元数据可以如图1所示,文件的元数据可以包括图1中的 三行数据,其中,第一行表示HDFS系统的版本信息,第二行表示文件的属性信息,第三行表示数据块的元数据。 For example: the metadata file can be shown in Figure 1, the metadata file may include data of three lines in FIG. 1, wherein, the first row indicates the version information HDFS system, the second row represents attribute information of the file, the third row represents the metadata of the data block. 即文件的元数据可以包括HDFS系统的版本信息、文件的属性信息和数据块的元数据。 I.e. the metadata file information may include a version HDFS system, metadata and attribute information of the file data blocks.

另外,本发明实施例应用的场景可以是单个设备,即该设备上包括HDFS的客户端、至少一个NN和至少一个DN;或者本发明实施例应用的场景可以是设备集群,即HDFS的客户端、至少一个NN和至少一个DN部署在多个设备上。 Further, the scenario of the application embodiment of the present invention may be a single apparatus, i.e. including HDFS client on the device, at least a NN and at least one of the DN; or scenario used in Examples of the present invention may be a device cluster, i.e. HDFS client , and at least one of the at least NN DN deployed on a plurality of devices. 其中,上述设备可以是网络侧设备,例如:基站、服务器、控制器等设备,上述设备还可以是用户侧设备,例如:计算机、平板电脑、手机、电子阅读器、遥控器、笔记本电脑、车载设备、网络电视、可穿戴设备等具有网络功能的智能设备。 Wherein the network side device may be a device, such as: base stations, servers, controllers and other equipment, said apparatus may also be user-side equipment, for example: a computer, a tablet computer, mobile phone, e-readers, remote controls, laptop computers, car equipment, network television, a wearable device such as a network-enabled smart devices.

请参阅图2,图2是本发明实施例提供的一种元数据恢复方法的流程示意图,如图2所示,包括以下步骤: Please refer to FIG. 2, FIG. 2 is a metadata according to an embodiment of the present invention is a method to restore flow diagram shown in Figure 2, comprising the steps of:

201、接收对目标文件的写操作请求,其中,所述目标文件保存在多个DN上。 201, receives a write request for a target file operation, wherein the plurality of target files stored on DN.

上述目标文件保存在多个DN上可以是响应上述写操作请求,将上述目标文件保存在多个DN上,其中,该保存过程的时间不作限定,例如:可以是在步骤204之后,或者在步骤204之前。 Above file is stored on the plurality of target DN may be in response to said write request, to store the files on the plurality of target DN, wherein the time of the save process is not limited, for example: after step 204 may be, or at step before 204.

202、从NN获取所述目标文件的元数据,所述目标文件的元数据包含所述多个DN的信息。 202, the metadata acquiring the target file from the NN, the metadata contains information for the target file of the plurality of DN.

可选的,当接收到上述写操作请求时,本端就可以向NN申请用于保存目标文件的多个DN,从而NN就可以为该目标文件分配NN,以及生成该目标文件的元数据,并将该目标文件的元数据返回到上述本端。 Alternatively, when receiving the write request, the terminal may apply to the NN for storing a plurality of object files the DN, so NN NN can be allocated for the object file, and the metadata is generated in the object file, and returns the metadata file to the target of the present terminal. 另外,上述多个DN的信息可以是各DN保存上述目标文件的路径信息,例如:上述介绍的文件的属性信息中的文件的路径。 Further, the information may be a plurality DN store the file path information of each target DN, for example: the file path attribute information described in the file.

203、根据所述目标文件的元数据生成元数据恢复信息。 203, recovery information generating metadata according to the metadata for the target file.

可选的,上述元数据恢复信息可以包括该目标文件的元数据,例如:包括目标文件的属性信息。 Optionally, the metadata information may include metadata recovery of the target file, for example: a target file attribute information. 其中,该属性信息可以包括该目标文件的如下一项或者多项: Wherein the attribute information may include one or more of the target file:

HDFS系统的版本信息、文件名、文件占据的空间的限制信息、用户权限信息、文件的每个数据块大小、文件的数据块的块数。 The number of data blocks HDFS version information systems, data block size for each file name, limit the space occupied by the file information, user rights information, documents, and files.

其中,上述文件名还可以携带有文件的路径信息。 Wherein the file name can also carry the file path information.

204、向所述多个DN中的至少一个DN发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 204, of the plurality of restoration DN DN transmitting at least one of the metadata information to the metadata information is used to restore at least one of said DN is saved, and after the abnormality occurs in the transmission to the NN NN said restore the metadata.

可选的,上述方法可以应用于HDFS的客户端(Client),即该客户端可以实现上述方法。 Optionally, the method may be applied HDFS client (Client), that is, the client may implement the method described above.

上述技术方案中,接收对目标文件的写操作请求,再从NN获取所述目标文件的元数据,根据所述目标文件的元数据生成元数据恢复信息,再向所述多个DN中的至少一个DN发送所述元数据恢复信息。 In the above technical solution, receiving a request for a write operation of the target file, then the target file to obtain metadata from NN, generate metadata according to the metadata recovery information for the target file, again at least a plurality of DN DN a metadata transmitting said recovery information. 这样该元数据恢复信息就可以用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 Resume information so that the metadata can be used to save the at least one DN, and then sent to the NN abnormal recover the metadata present in the NN. 从而可以恢复NN出异常时的元数据。 NN thereby restore the metadata when an abnormality.

请参阅图3,图3是本发明实施例提供的另一种元数据恢复方法的流程示意图,如图3所示,包括以下步骤: Refer to FIG. 3, FIG. 3 is a flowchart of another method for recovery of metadata according to an embodiment of the present invention, a schematic diagram, shown in Figure 3, comprising the steps of:

301、接收对目标文件的写操作请求,其中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上。 301, receives a write request for a target file operation, wherein the target file is divided into a plurality of data blocks, said plurality of data blocks are stored in the plurality of DN.

可选的,对目标文件进行数据写操作可以是以数据块为单位进行写操作,即一个目标文件可以被划分为多个数据块中,其中,每个数据块的大小 可以是相同。 Alternatively, the target file may be in a data write operation in units of blocks of data write operation, i.e. a target file may be divided into a plurality of data blocks, wherein the size of each data block may be the same. 例如:每个数据块可以是64兆(MB)或者128MB等。 For example: Each data block may be 64 megabytes (MB) 128MB or the like. 另外,上述数据块的大小可以是数据块的存储空间大小,但在实际应用中一个文件的各数据块存储的该文件的数据量可以是不同的,例如:一个文件包含150MB数据,而每个数据块的大小为64MB,那么该文件的第一个数据块就可以存储该文件的64MB数据,而第二个数据块就可以存储该文件的64MB数据,而第三个数据块就可以存储该文件的22MB数据。 Further, a size of the data block may be a block of storage space, but in practical applications the amount of data of each data block of a file stored in the file may be different, for example: a 150MB file contains data, and each data block size is 64MB, then the first data block of the file data can be stored in the 64MB file, and the second data block can be stored data of the 64MB file, and a third block of data may be stored 22MB data file.

302、从所述NN获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息。 302, the metadata acquiring the plurality of data blocks from the NN, the metadata of each data block contains information DN information and each data block of said each data block is located.

可选的,当接收到上述写操作请求时,本端就可以向NN申请用于保存目标文件的多个数据块,以及保存数据块的多个DN。 Alternatively, when receiving the write request, the terminal may apply to a plurality of data blocks stored NN object file, and a plurality of data blocks stored DN. 这样从而NN就可以为该目标文件分配数据块,以及数据块所在的DN,以及生成每个数据块的元数据,并将每个数据块的元数据返回到上述本端。 NN DN so that this data block can be allocated for the target file, and the data block is located, and metadata is generated for each data block, and returning metadata of each data block according to said end. 另外,上述数据块的信息可以是数据块的属性信息,例如:数据块标识和数据块的时间戳。 Further, information of the data block may be the attribute information of the data block, for example: timestamp data block identifier and the data block. 上述多个DN的信息可以是各DN保存数据块的路径信息,例如:上述介绍的文件的属性信息中的文件的路径。 DN said plurality of information may be stored in each data block DN path information, for example: the file path attribute information described in the file.

303、根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 303, the metadata of each data block, generates the metadata of each data block recovery information.

可选的,上述数据块的元数据恢复信息可以包括该数据块的元数据,例如:包括目标文件的属性信息。 Alternatively, said data block metadata information may include metadata recovery of the data block, for example: a target file attribute information.

304、向所述多个DN中的至少一个DN发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述目标文件的元数据的恢复。 304, of the plurality of restoration DN DN transmitting at least one of the metadata information to the metadata information is used to restore at least one of said DN is saved, and after the abnormality occurs in the transmission to the NN NN said metadata recovery of the target file.

由于上述元数据恢复信息被一个或者多个DN保存,这样可以防止单个DN出现异常后,导致该元数据恢复信息丢失。 Since the meta data recovery information stored by one or more DN, which can prevent the abnormal single DN, resulting in the loss recovery information metadata.

可选的,上述从所述NN获取所述多个数据块的元数据可以包括: Optionally, the obtaining of the plurality of data blocks from the NN metadata may include:

依次接收所述NN发送的所述每个数据块的元数据。 Sequentially receiving the metadata of the NN for each data block transmitted.

例如:本端可以是每次向NN申请一个数据块,这样NN就会为目标文件分配一个数据块,以及生成该数据块的元数据,并将该数据块的元数据返回到上述本端。 For example: This end may apply each time a block of data to the NN, NN so that a data block is allocated for the object file, and a metadata generating block data, and returns the metadata to the data block of the present terminal. 当上述数据块被写完时,而目标文件还没有被写完,本端再向NN申请另一个数据块,NN就会为目标文件分配另一个数据块,以及生成该数据块的元数据,并将该数据块的元数据返回到上述本端。 When said data block is written, and the target file has not been finished, the end of the data block again to another application NN, NN another block of data is allocated for the object file, and generating metadata of the data block, and returns the metadata to the data block of the present terminal. 当上述数据块被写完时,而目标文件还没有被写完,继续向NN申请数据块,直到目标文件被写完。 When said data block is written, and the target file has not been finished, NN continues to apply data block, until the target file is written. 从而步骤303可以是本端每获取一个数据块的元数据,再生成该数据块的元数据恢复信息,步骤304再将该数据块的元数据恢复信息发送至上述至少一个DN。 Step 303 may be present such that each end obtain metadata of a data block, the metadata re-generation of the data block restoration information, then in step 304 the metadata of the data block information to recover said at least one DN.

可选的,上述从所述NN获取所述多个数据块的元数据可以包括: Optionally, the obtaining of the plurality of data blocks from the NN metadata may include:

接收所述NN根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 NN metadata receiving the request for all data blocks transmitted on a one-time write operation according to the target file.

例如:本端可以是一次性向NN申请多个数据块,这样NN就会为目标文件分配多个数据块,以及生成这些数据块的元数据,并将这些数据块的元数据返回到上述本端。 For example: This end may be disposable to apply a plurality of data blocks NN, NN will be assigned such a plurality of data blocks for the target file, and the metadata is generated in the data blocks, and returns the metadata to the data blocks of the present terminal . 从而步骤303可以是一次性生成多个数据块的元数据恢复信息,步骤304再将这多个数据块的元数据恢复信息发送至上述至少一个DN。 Thus step 303 may be generated once metadata for the plurality of blocks of data recovery information, then in step 304 the plurality of data blocks of metadata information to recover said at least one DN.

可选的,上述接收的元数据至少包括第一数据块的元数据和第二数据块的元数据,即目标文件至少被划分为第一数据块和第二数据块。 Alternatively, the metadata including at least the received metadata of the first data block and second data block of metadata, i.e., the target file is divided into at least a first data block and second data block.

其中,上述生成的所述第一数据块的元数据恢复信息可以包括: Wherein said generated metadata is the first data block restoration information may include:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息。 Attribute information of the target file in the first data block and the location information of the target file.

其中,目标文件的属性信息可以包括: Among them, the target file attribute information may include:

HDFS系统的版本信息、文件名、文件的副本数、文件的每个数据块大小、文件占据的空间的限制信息和用户权限信息。 HDFS version information systems, information and restrict user rights information file name, file number of copies of each data block size of the file, the file space occupied.

上述第一数据块在所述目标文件中的位置信息可以是指,第一数据块的目标文件的数据块序号,例如:第一数据块可以是目标文件的序号为1的数据块,其中,序号为1的数据块可以是目标文件中时间戳最早的一个数据块;上述第一数据块在所述目标文件中的位置信息可以包括: Said first data block in the target document may refer to the location information, the target file block number of the first data block, for example: a first data block number of the target file may be a data block 1, wherein, 1 is a block number may be the target of a file timestamp of the oldest data block; the first data block in the target file location information may include:

数据块序号和文件结束数据块信息。 End block number and the file block information.

上述生成的所述第二数据块的元数据恢复信息可以包括: The metadata generated by the above-described second data block may resume information comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

上述第二数据块在所述目标文件中的位置信息可以包括: The second data block in the target file location information may include:

数据块序号和文件结束数据块信息。 End block number and the file block information.

另外,第二数据块的元数据恢复信息还可以包括HDFS系统的版本信息和文件名,这样就可以清楚地知道第二数据块的元数据恢复信息是用于恢复哪个文件的数据块,以及该文件的HDFS系统的版本信息。 Further, meta data of the second data block further recovery information may include a version information and the file name HDFS system, so that the metadata will be clear to recover the second data block is a data block for which information file recovery, and version information HDFS file system.

例如:如图4a所示,第一数据块的元数据恢复信息可以包括: For example: As shown in FIG. 4a, metadata of the first data block may resume information comprising:

HDFS系统的版本信息、文件名、文件的副本数、文件的每个数据块大小、文件占据的空间的限制信息、用户权限信息、数据块序号和文件结束数据块信息。 HDFS system version information, restrict information, user rights information, block number and file name, file number of copies of each data block size of the file, the file space occupied by the end of the data block information.

如图4b所示,第一数据块的元数据恢复信息可以包括: As shown in FIG. 4b, metadata of the first data block may resume information comprising:

HDFS系统的版本信息、文件名、数据块序号和文件结束数据块信息。 HDFS system version information, file name, block number and end of file data block information.

其中,文件结束数据块信息可以是用当前数据块是否为结束Block的信息表示,例如:上述第一数据块的元数据恢复信息中的文件结束数据块信息可以用假(false)表示;当上述目标文件只包括两个数据块时,那么,上述第一数据块的元数据恢复信息中的文件结束数据块信息可以用真(true)表 示;当上述目标文件包括三个或者三个以上的数据块时,上述第一数据块的元数据恢复信息和上述第二数据块的元数据恢复信息中的文件结束数据块信息可以用假(false)表示;即该实施方式中,只有目标文件的最后一个数据块对应的文件结束数据块信息才用真(true)表示,其余的都用假(false)表示,从而可以根据文件结束数据块信息可以确定目标文件包括的数据块的个数。 Wherein the end of the file block information may be information whether the end is indicated by Block current data block, for example: metadata of the first data block restoration information file data block end can be represented by false information (false); when said object file includes only two data block, then the meta data of the first data block end information file recovery data block information can be true (true) indicates; when said target file includes three or more than three data when the block, the metadata of the first data block recovery information and metadata of the second data block restoration information file data block end can be represented by false information (false); i.e. this embodiment, the final target file only a data block corresponding to the end of file block information represented only by true (true), the rest are represented by a false (false), number of data blocks so that the data block comprises information to determine the target file according to the file can end.

另外,由于一个文件内各数据块的元数据都可以包括文件的属性信息,而文件的属性信息是相同的,例如:文件占据的空间的限制信息、用户权限信息、数据块的大小信息和副本数等信息是相同的,这样就只需要在其中一个数据块的元数据恢复信息中写入这些信息即可,从而可以节省其他DN的存储空间。 Further, since the metadata for each data block within a file may include a file attribute information, and file attribute information are the same, for example: restriction information, user rights information, size information, and a copy of the file space occupied by the data block the number of other information is the same, so that only need to write the information to a metadata block which resume information, thus saving storage space other DN.

另外,上述仅举例了所述第一数据块和第二数据块的元数据恢复信息的多种可选的实施方式,本实施例中对此不作限定,例如:第一数据块的元数据恢复信息也可以包括: Further, the above example only the metadata of the first data block and second data block information recovery many alternative embodiments, this embodiment is not limited to the present embodiment, for example: a first data block of the metadata recovery information may also include:

HDFS系统的版本信息、文件名、文件的副本数、文件的访问时间、文件的数据块的块数、文件占据的空间的限制信息、用户权限信息、数据块标识、数据块的大小和数据块时间戳。 The number of data blocks version information HDFS systems, file names, access times the number of copies the file, file, file, restrict information, user rights information file space occupied by the data block identifier, the size of the data blocks and data blocks timestamp.

可选的,本方法中每获取NN发送的数据块的元数据之前,都可以是向NN发送数据块的申请请求,NN接收到该请求后,就可以为目标文件分配数据块,以及存储该数据块的DN。 Alternatively, the present process obtain metadata before each data block transmitted NN, may be sent to the application request data block NN, after the NN receives the request, it is possible that the target file allocation data blocks, and storing the DN data block. 另外,在步骤301之前,所述方法还可以包括以下步骤: Further, prior to the step 301, the method may further comprise the step of:

创建目标文件,并生成该目标文件的概要信息。 Create the target file, and generate summary information of the target file. 例如:生成该文件的文件名等信息。 For example: to generate the file name and other information.

可选的,上述方法可以应用于HDFS的客户端(Client),即该客户端可以实现上述方法。 Optionally, the method may be applied HDFS client (Client), that is, the client may implement the method described above.

上述技术方案中,接收对目标文件的写操作请求,从NN获取多个数据块的元数据,再根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息,从而可以向所述多个DN中的至少一个DN发送所述元数据恢复信息。 In the above technical solution, the target receives the write request for a file operation, metadata is acquired from a plurality of data blocks NN, then the meta data of said each data block, generates the metadata of each data block recovery information, DN can be at least one transmit the metadata of the plurality of restoration information to the DN. 这样元数据恢复信息可以用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 Such metadata information may be used to recover at least one of said DN is saved, and then transmitted to the NN exception recovery of the metadata occurs in the NN. 从而当NN出现异常后,可以及时恢复NN在出现异常时生成的元数据。 When such abnormal NN, NN restoring metadata may be generated when an abnormality occurs in a timely manner.

请参阅图5,图5是本发明实施例提供的另一种元数据恢复方法的流程示意图,如图5所示,包括以下步骤: Refer to FIG. 5, FIG. 5 is a flowchart of another method for recovery of metadata according to an embodiment of the present invention, a schematic diagram, shown in Figure 5, comprising the steps of:

501、接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从NN获取的所述目标文件的元数据,并根据所述元数据生成的元数据恢复信息.。 501, a target metadata file sent by a client of the recovery information and storing the recovery information metadata, wherein the metadata information is restored upon receiving a client request for a write operation of the target file from the NN acquiring metadata of the target file, and recover the information according to the metadata generated by the metadata ..

其中,上述目标文件所述目标文件保存在多个DN上,且上述元数据包括所述多个DN的信息。 Wherein the target file on the target file stored in a plurality of DN, and said metadata comprises information of the plurality of DN.

可选的,步骤501可以是将元数据恢复信息存储在本地,或者将元数据恢复信息存储第三方面存储设备上。 Alternatively, step 501 may restore the metadata information is stored locally, or to restore the metadata stored on the information storage device of the third aspect.

502、在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 502, after the NN abnormal, sending the information to the meta data recovery NN, metadata information is the NN Recovery Recovery for the metadata.

可选的,本方法可以应用于HDFS的DN。 Optionally, the method may be applied to the HDFS DN.

上述技术方案中,接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息;在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 In the above technical solution, the target metadata file sent by a client of the recovery information, and store the meta data recovery information; after abnormality occurs in the NN, sending the metadata information to recover the NN, the metadata retriever NN said information is used for recovery of the metadata. 从而可以实现恢复NN出异常时的元数据。 Thereby restoring metadata may be achieved when the abnormality NN.

请参阅图6,图6是本发明实施例提供的另一种元数据恢复方法的流程示意图,如图6所示,包括以下步骤: Refer to FIG. 6, FIG. 6 is a flowchart of another method for recovery of metadata according to an embodiment of the present invention, a schematic diagram, shown in Figure 6, comprising the steps of:

601、接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,所述客户端根据该数据块的元数据生成的元数据恢复信息.。 601, the metadata of the data block receiving the target file recovery information sent by the client, and storing the recovery information metadata, wherein the metadata information is restored in the client receiving the target document when a write operation request, the client restoration information metadata according to the metadata generated by the data block ..

其中,上述目标文件被划分为多个数据块,且上述每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息。 Wherein the target file is divided into a plurality of data blocks, and said metadata of each data block contains information DN information and each data block of said each data block is located.

可选的,步骤601接收到元数据恢复信息可以是一个或者多个数据块的元数据恢复信息。 Optionally, the step 601 is received metadata or restoring metadata information may be a plurality of data blocks recovery information. 例如:目标文件被划分为多个数据块,其中,执行本方法的DN可以是保存其中一个或者多个数据块的元数据恢复信息。 For example: the target file is divided into a plurality of data blocks, wherein the method is performed DN may be stored in which one or more metadata blocks of data recovery information.

可选的,步骤601可以是将上述元数据恢复信息保存在一个特定文件内,例如:保存在命名为:blk_blockId_genstamp.blockmeta的文件内,其中,该文件的名称可以包括数据块的标识,且该文件名称还可以包括数据块的时间戳,例如:DN在接收到数据块的元数据恢复信息的同时,还可以接收到客户端发送的数据的标识和数据块的时间戳。 Alternatively, step 601 may restore data to the meta information stored in a particular file, for example: name stored in: the blk_blockId_genstamp.blockmeta file, wherein the file name may include an identification data block, and the file names may further include a time stamp of a data block, for example: DN while receiving the metadata of the data block recovery information, may also receive a timestamp and the data block identification data sent by the client. 另外,该文件内还可以将客户端在上述数据块中写入的该目标文件的数据保存至另一个特定的文件内,例如:保存在命名为:blk_blockId的文件内。 Further, in the document can also save data of the client writing the target file to another particular within said data block file, for example: name stored in: the blk_blockId file. 当客户端还写入了上述数据块的循环冗余校验码(Cyclic Redundancy Check,CRC),本方法还可以将该CRC保存用于保存上述元数据恢复信息的特定文件内,即将上述元数据恢复信息和CRC存储在同一个特定文件内,当然,本方法还可以将CRC存储在一个专门用于存储CRC的特定文件内。 When the client also writes the cyclic redundancy check code to said data block (Cyclic Redundancy Check, CRC), the present method can save the CRC for a particular file storing the meta-data recovery information, i.e. the meta data resume information and the CRC is stored in a particular file in the same, of course, the present method may be stored in a special CRC for a particular file is stored CRC. 另外,上述两个特定文件可以是接收到上述数据信息之前设置好的,例如:接收到客户端发送的写请求时,生成上述两个特定文件。 Further, the two may be provided good specific file before receiving the data information, for example: when receiving the write request sent by the client, generating the two specific file. 由于步骤601将上述数据信息以文件的形式进行存储,这 样可以实现数据仓库工(例如:Hive)功能,或者Spark功能,其中,Spark是一个基于内存计算的开源的集群计算系统,目的是让数据分析更加快速。 Since steps 601 outputs the data information stored in the form file, which can implement a data warehouse workers (e.g.: Hive) function, or Spark function, wherein, Spark is an open source cluster computing system memory-based computing, to let the data analysis more quickly.

另外,上述元数据恢复信息还可以是预定义的数据格式进行保存,例如:以HBase的数据格式进行保存,其中,HBase是一个分布式的、面向列的开源数据库。 Further, the metadata information may also be restored predefined data storage format, for example: to save HBase data format, wherein, HBase is a distributed, open column-oriented database.

可选的,在获取到上述元数据恢复信息时,还可以在内存中保存上述元数据恢复信息,这样可以更快速地发送该元数据恢复信息。 Alternatively, when the acquired metadata to said recovery information, the meta data may also be stored in the memory recovery information, which can more quickly transmit the metadata recovery information. 可以直接将内存的元数据恢复信息上传到NN。 Metadata can be directly uploaded to the memory recovery information NN. 另外,当实现本方法的DN重新启动后,可以扫描本地存储的元数据恢复信息,再将该元数据恢复信息上传给NN。 Further, the method when implemented DN restart, can scan locally stored recovery information metadata, and then restore the metadata information uploaded to the NN.

602、在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述目标文件的元数据的恢复。 602, after the NN abnormal, sending the information to the meta data recovery NN, meta data recovery is the restoration information for the target file NN metadata.

例如:DN在NN重新启动时,会接收到该NN发送的通知消息,该通知消息用于请求发送上述数据信息,这样就可以响应消息通知消息,以返回上述元数据恢复信息。 For example: DN NN when restarted, receiving the notification message sent by the NN, the notification message for requesting transmission of said data information, so that response message notification message, to return the meta data recovery information.

可选的,所述接收所述客户端发送的所述目标文件的数据块的元数据恢复信息包括: Metadata of the data blocks in the target file Optionally, the client receives the information sent by the recovery comprises:

依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Metadata plurality of data blocks sequentially receives the target file sent by the customer endpoint recovery information; or metadata of a plurality of data blocks of the target client receives the file recovery information transmitted disposable.

可选的,所述接收的元数据恢复信息至少可以包括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: Optionally, the recovery information received metadata may include at least a first data block of metadata and metadata information restoring the second data block recovery information, wherein:

所述第一数据块的元数据恢复信息可以包括: The metadata of the first data block may resume information comprising:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述第二数据块的元数据恢复信息可以包括: The metadata of the second data block may resume information comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

另外,本实施例中,当所述生成的元数据出现异常后,可以是向NN发送目标文件的所有数据块的元数据恢复信息。 In the present embodiment, when the abnormality occurs to generate metadata, may be sent to the target file metadata for all data blocks NN recovery information. 例如:当DN接收到NN异常重启的通知消息后,响应该通知消息扫描本地或者内存中所有数据块的元数据恢复信息,并向NN上传所有数据块的元数据恢复信息。 For example: After receiving the notification message NN DN abnormal restart, in response to the notification message scanning or local metadata for all data blocks in the memory recovery information, upload metadata and all data blocks in the recovery information NN.

可选的,上述方法可以应用于HDFS的DN,即该DN可以实现上述方法。 Optionally, the method may be applied to the HDFS DN, DN i.e., the above-described method can be implemented.

上述技术方案中,接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,并保存所述元数据恢复信息,在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 After the above technical solution, the metadata of the data block receiving the target file recovery information sent by the client, and store the meta data recovery information, abnormality occurs in the NN, the NN transmitted to the element data recovery information, metadata information is the NN recovery recovery for the metadata. 从而可以及时恢复NN在出现异常时生成的元数据。 Thereby restoring metadata NN generated when an exception occurs in a timely manner.

请参阅图7,图7是本发明实施例提供的另一种元数据恢复方法的流程示意图,如图7所示,包括以下步骤: Please refer to FIG. 7, FIG. 7 is a flowchart of another method for recovery of metadata according to an embodiment of the present invention, a schematic diagram, shown in Figure 7, comprising the steps of:

701、接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个DN上。 701, sent by a client of the target file write request, wherein the target file is stored on a plurality of DN.

702、生成所述目标文件的元数据,并向所述客户端发送所述元数据,其中,所述元数据包含所述多个DN的信息。 702, metadata is generated for the target file, and sends the metadata to the client, wherein the metadata information comprises a plurality of said DN.

可选的,步骤702可以是接收到上述写操作请求后,为上述目标文件分配所述多个DN,再生成上述目标文件的元数据。 Alternatively, step 702 may be Upon reception of the write operation request, allocating the plurality DN above object file, and then generates the meta data of the target file.

703、当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至 所述至少一个DN进行保存的元数据恢复信息。 703, when an abnormality occurs this end, a plurality of receiving metadata of at least one DN DN recover transmitted information, and restoring the restoration information metadata according to the metadata, wherein the metadata information is restored the client-generated based on the metadata sent by the client to at least one of said stored metadata DN recovery information.

这样当本端,即NN出现异常时,可以接收到上述至少一个DN发送的元数据恢复信息,从而可以恢复出上述目标文件的元数据。 Thus, when the present terminal, i.e. NN abnormal, can be received at least one metadata DN sent the recovery information, which can recover the data of the target file metadata.

可选的,上述目标文件可以被划分为多个数据块,所述多个数据块被保存在所述多个DN上; Optionally, the target file may be divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

则,上述生成所述目标文件的元数据,并向所述客户端发送所述元数据可以包括: Then, the generation of the metadata of the target file, the client sends to the metadata may include:

生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息。 Generating a plurality of data blocks of metadata, and metadata for the client sends the plurality of data blocks, each data block metadata comprises location information of each data block and each data block of the information DN.

上述接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,可以包括: At least one metadata DN for resuming the transmission of said plurality of receiving the DN information, and restoring the restoration information metadata according to the metadata, may include:

接收所述多个DN中的至少一个DN发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 Receiving the plurality of data blocks in at least one of the plurality of DN DN recover transmitted information metadata, and restoring the restoration information file meta data of the target data blocks of the plurality of metadata.

这样可以实现根据多个数据块的元数据恢复信息恢复目标文件的元数据。 This allows recovery information metadata according to the metadata of the plurality of blocks of data recovery target file.

可选的,向所述客户端发送所述多个数据块的元数据包括: Alternatively, the client sends the plurality of data blocks of the metadata comprises:

依次向所述客户端发送所述多个数据块的元数据;或者,根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Sequentially transmitted to the client metadata of the plurality of data blocks; or, according to a write request for a one-time operation of the target file metadata for all transmission data block to the client.

可选的,上述多个数据块至少可以包括第一数据块和第二数据块,其中: Optionally, the plurality of data blocks may include at least a first data block and second data block, wherein:

所述第一数据块的元数据恢复信息可以包括: The metadata of the first data block may resume information comprising:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述第二数据块的元数据恢复信息可以包括: The metadata of the second data block may resume information comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

需要说明的是,本实施例中针对元数据和元数据恢复信息的定义可以参考图2或者图3所示的实施例,此处不作重复说明。 Incidentally, in the embodiment defined resume information metadata and the metadata of the present embodiment can be shown with reference to FIG. 2 or FIG. 3 embodiment, description will not be repeated here.

可选的,上述方法可以应用于HDFS内的NN。 Optionally, the method may be applied in the NN HDFS.

上述技术方案中,接收客户端发送的对目标文件进行写操作请求,生成所述目标文件的元数据,并向所述客户端发送所述元数据;这样当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据。 The above technical solution, the receiver sent by the client metadata write request, generating the target file is the target file, the client sends to the metadata; so that when an abnormality occurs this end, the receiving metadata for at least one of the plurality of DN DN recover transmitted information, and restoring the restoration information metadata according to the metadata. 从而可以恢复NN出异常时的元数据。 NN thereby restore the metadata when an abnormality.

下面为本发明装置实施例,本发明装置实施例用于执行本发明方法实施例一至五实现的方法,为了便于说明,仅示出了与本发明实施例相关的部分,具体技术细节未揭示的,请参照本发明实施例一、实施例二、实施例三、实施例四和实施例五。 The following embodiments of the present invention apparatus embodiment, the present embodiment of the invention apparatus for performing the method of the present invention is a method implemented in first to fifth embodiments embodiment, for convenience of explanation, only a portion related to the embodiment of the present invention, the technical details are not disclosed Please refer to an embodiment of the present invention, according to the second embodiment, third embodiment, fourth embodiment and fifth embodiment.

请参阅图8,图8是本发明实施例提供的一种客户端的结构示意图,如图8所示,包括:接收单元81、获取单元82、生成单元83和发送单元84,其中: Refer to FIG. 8, FIG. 8 is a structural diagram of a client according to an embodiment of the present invention ends, as shown in FIG. 8, comprising: a receiving unit 81, an acquisition unit 82, a transmission unit generation unit 83 and 84, wherein:

所述接收单元81,用于接收对目标文件的写操作请求,其中,所述目标文件保存在多个DN上。 The receiving unit 81, for receiving a write request for a target file, wherein said plurality of target files stored on DN.

所述获取单元82,用于从NN获取所述目标文件的元数据,所述目标文件的元数据包含所述多个DN的信息。 The obtaining unit 82, configured to obtain the target file from the metadata NN, the target file metadata contains a plurality of information DN.

所述生成单元83,用于根据所述目标文件的元数据生成元数据恢复信 息。 The generating unit 83 for generating metadata according to the metadata recovery information for the target file.

所述发送单元84,用于向所述多个DN中的至少一个DN发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 The sending unit 84, configured to DN of said plurality of transmitting at least one of the metadata DN restoration information, said restoration information metadata for the at least one DN is saved, and appears in the NN abnormal then sent to the NN to recover the metadata.

在另一个实施例中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上; In another embodiment, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

则,获取单元82可以用于从所述NN获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息; The obtaining unit 82 may be configured to obtain metadata of the plurality of data blocks from the NN, the metadata of each data block contains information DN information and each data block of said each data block is located;

生成单元83可以用于根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 Unit 83 may be used for generating the metadata of each data block, each data block generates the metadata recovery information.

该实施方式中,可以实现生成每个数据块的元数据恢复信息,以使NN根据多个数据块的元数据恢复信息恢复目标文件的元数据。 This embodiment may be implemented to generate metadata for each block of data recovery information, so that recovery information metadata according to the metadata NN plurality of blocks of data recovery target file.

可选的,所述获取单元82可以用于依次接收所述NN发送的所述每个数据块的元数据;或者,所述获取单元82可以用于接收所述NN根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 Alternatively, the acquiring unit 82 may be configured to sequentially receive the metadata transmitted from the NN for each data block; or the acquisition of the NN receives the target file according to the write unit 82 may be used operation requests metadata for all data block to be transmitted at once.

可选的,上述接收的元数据至少可以包括第一数据块的元数据和第二数据块的元数据,其中: Alternatively, the received metadata may include metadata, at least a first data block and second data block metadata, wherein:

所述生成的所述第一数据块的元数据恢复信息可以包括: The generated metadata to restore the first data block information may include:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述生成的所述第二数据块的元数据恢复信息可以包括: Generating metadata for the second data block of the recovery information may include:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

可选的,上述客户端可以是HDFS的客户端(Client)。 Optionally, the client may be HDFS client (Client).

上述技术方案中,接收对目标文件的写操作请求,再从NN获取所述目标文件的元数据,根据所述目标文件的元数据生成元数据恢复信息,再向所述多个DN中的至少一个DN发送所述元数据恢复信息。 In the above technical solution, receiving a request for a write operation of the target file, then the target file to obtain metadata from NN, generate metadata according to the metadata recovery information for the target file, again at least a plurality of DN DN a metadata transmitting said recovery information. 这样该元数据恢复信息就可以用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 Resume information so that the metadata can be used to save the at least one DN, and then sent to the NN abnormal recover the metadata present in the NN. 从而可以恢复NN出异常时的元数据。 NN thereby restore the metadata when an abnormality.

请参阅图9,图9是本发明实施例提供的一种DN的结构示意图,如图9所示,包括:接收单元91和发送单元92,其中: Please refer to FIG. 9, FIG. 9 is a structural diagram of a DN according to an embodiment of the present invention, shown in Figure 9, comprising: a receiving unit 91 and a sending unit 92, wherein:

接收单元91,用于接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从NN获取的所述目标文件的元数据,并根据所述元数据生成的元数据恢复信息.。 The receiving unit 91, a target metadata file sent by a client of the recovery information and storing the recovery information metadata, wherein the metadata information is restored in the client receiving a request for a write target file when, NN metadata acquired from the target file, and recover the information according to the metadata generated by the metadata ..

发送单元92,用于在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 A sending unit 92, configured to, after the NN abnormal, sending the information to the meta data recovery NN, restoring metadata information is the NN for the metadata recovery.

在另一个实施例中,所述目标文件被划分为多个数据块,所述接收单元91可以用于接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,所述数据块的元数据恢复信息为所述客户端根据该数据块的元数据生成的元数据恢复信息。 In another embodiment, the target file is divided into a plurality of data blocks, the receiving unit 91 may be the metadata for the data block receiving the target file recovery information sent by the client, the data restoring metadata block information to the client restoration information metadata according to the metadata generated by the data block.

可选的,接收单元91可以用于依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,所述接收单元91可以用于接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Metadata plurality of data blocks of the target file Alternatively, the receiving unit 91 may receive the customer endpoint for sequentially transmitting recovery information; Alternatively, the receiving unit 91 may be configured to receive the one-time client metadata plurality of data blocks transmitted by the target file recovery information.

可选的,上述接收的元数据恢复信息至少可以包括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: Alternatively, the received metadata information restoring metadata may include at least a first data block of the second resume information and metadata information recovery data block, wherein:

所述第一数据块的元数据恢复信息可以包括: The metadata of the first data block may resume information comprising:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述第二数据块的元数据恢复信息可以包括: The metadata of the second data block may resume information comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

可选的,本节点可以是HDFS的DN。 Alternatively, this may be a node of HDFS DN.

上述技术方案中,接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息;在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 In the above technical solution, the target metadata file sent by a client of the recovery information, and store the meta data recovery information; after abnormality occurs in the NN, sending the metadata information to recover the NN, the metadata retriever NN said information is used for recovery of the metadata. 从而可以实现恢复NN出异常时的元数据。 Thereby restoring metadata may be achieved when the abnormality NN.

请参阅图10,图10是本发明实施例提供的一种NN的结构示意图,如图10所示,包括:接收单元101、生成单元102和恢复单元103,其中: Please refer to FIG. 10, FIG. 10 is a configuration diagram of a NN to an embodiment of the present invention, shown in Figure 10, comprising: a receiving unit 101, generating unit 102 and a recovery unit 103, wherein:

所述接收单元101,用于接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个DN上。 The receiving unit 101 for receiving the target files sent by the client write request, wherein the target file is stored on a plurality of DN.

所述生成单元102,用于生成所述目标文件的元数据,并向所述客户端发送所述元数据,其中,所述元数据包含所述多个DN的信息。 The generating unit 102, generating metadata for the target file, and sends the metadata to the client, wherein the metadata information comprises a plurality of said DN.

所述恢复单元103,用于当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至所述至少一个DN进行保存的元数据恢复信息。 The restoring unit 103, configured to, when abnormality occurs this end, a plurality of receiving metadata of at least one DN DN recover transmitted information, and restoring the restoration information metadata according to the metadata, wherein the said recovery information is metadata generated by the client based on the metadata sent by the client to at least one of said stored metadata DN recovery information.

在另一个实施例中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上; In another embodiment, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

所述生成单元102可以用于生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的 信息和所述每个数据块所在的DN的信息; The generating unit 102 may be used to generate the plurality of data blocks of metadata, and metadata for the client sends the plurality of data blocks, each data block metadata information of each data block contains DN information and each data block is located;

所述恢复单元103可以用于当本端出现异常时,接收所述多个DN中的至少一个DN发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 The recovery unit 103 may be used when an abnormality occurs this end, the metadata for the plurality of received data blocks of said at least one of the plurality DN DN recover transmitted information, and metadata in accordance with said plurality of data blocks recovery data recovery information of the target file metadata.

可选的,生成单元102可以用于生成所述多个数据块的元数据,并依次向所述客户端发送所述多个数据块的元数据;或者,所述生成单元102可以用于生成所述多个数据块的元数据,并根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Alternatively, unit 102 may generate a plurality of data blocks for generating the metadata, the metadata and sequentially transmits the plurality of data blocks to the client; Alternatively, the generating unit 102 may be used to generate metadata of the plurality of data blocks, and a one-time request to the target file according to the write operation of all data blocks transmits the metadata to the client.

可选的,上述多个数据块至少可以包括第一数据块和第二数据块,其中: Optionally, the plurality of data blocks may include at least a first data block and second data block, wherein:

所述第一数据块的元数据恢复信息可以包括: The metadata of the first data block may resume information comprising:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述第二数据块的元数据恢复信息可以包括: The metadata of the second data block may resume information comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

可选的,上述节点可以是HDFS内的NN。 Optionally, the node may be within the NN HDFS.

上述技术方案中,接收客户端发送的对目标文件进行写操作请求,生成所述目标文件的元数据,并向所述客户端发送所述元数据;这样当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据。 The above technical solution, the receiver sent by the client metadata write request, generating the target file is the target file, the client sends to the metadata; so that when an abnormality occurs this end, the receiving metadata for at least one of the plurality of DN DN recover transmitted information, and restoring the restoration information metadata according to the metadata. 从而可以恢复NN出异常时的元数据。 NN thereby restore the metadata when an abnormality.

请参阅图11,图11是本发明实施例提供的另一种元数据恢复设备的结构示意图,包括至少一个处理器1102(例如CPU),至少一个网络接口1105或者其他通信接口,存储器1106,和至少一个通信总线1103,用于实现这 些装置之间的连接通信。 Referring to FIG 11, FIG 11 is another schematic diagram of metadata configuration embodiment provided recovery apparatus embodiment of the present invention, comprising at least one processor 1102 (e.g., the CPU), at least one network interface, or other communications interfaces 1105, memory 1106, and the at least one communication bus 1103 for enabling communication between these devices is connected. 处理器1102用于执行存储器1106中存储的可执行模块,例如计算机程序。 Processor 1102 for performing a memory 1106 stores an executable module, such as a computer program. 存储器1106可能包含高速随机存取存储器(RAM:Random Access Memory),也可能还包括非不稳定的存储器(non-volatile memory),例如至少一个磁盘存储器。 Memory 1106 may include high speed random access memory (RAM: Random Access Memory), and may also include non-volatile memory (non-volatile memory), for example, at least one disk memory. 通过至少一个网络接口1105(可以是有线或者无线)实现该系统网关与至少一个其他网元之间的通信连接,可以使用互联网,广域网,本地网,城域网等。 Via at least one network interface 1105 (which may be wired or wireless) a communication connection between the gateway system with at least one other network element can use the Internet, a wide area network, local area network, metropolitan area network.

在一些实施方式中,存储器1106存储了程序11061,程序11061可以被处理器1102执行,这个程序包括: In some embodiments, the memory 1106 stores a program 11061, 11061 may be a program executed by the processor 1102, the program comprising:

接收对目标文件的写操作请求,其中,所述目标文件保存在多个DN上; Receiving a write request for a target file, wherein said plurality of target files stored on the DN;

从NN获取所述目标文件的元数据,所述目标文件的元数据包含所述多个DN的信息; Acquiring the target file from the metadata NN, the target file metadata contains information of the plurality of the DN;

根据所述目标文件的元数据生成元数据恢复信息; Generate metadata according to the metadata recovery information for the target file;

向所述多个DN中的至少一个DN发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 DN direction in at least one of said plurality of transmitting the metadata DN restoration information, said restoration information metadata for the at least one DN is saved, and then transmitted to the abnormality occurs in the NN NN recovery of the metadata.

在另一个实施例中,所述目标文件可以被划分为多个数据块,所述多个数据块可以被保存在所述多个DN上;处理器1102执行的从NN获取所述目标文件的元数据,根据所述目标文件的元数据生成元数据恢复信息的程序,可以包括: In another embodiment, the target file may be divided into a plurality of data blocks, said plurality of data blocks may be stored on the plurality of the DN; the target file acquired from the processor 1102 performs the NN metadata, metadata is generated recovery program metadata information for the target file may include:

从所述NN获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息; Acquiring metadata of the plurality of data blocks from the NN, the metadata of each data block contains information DN information and each data block of said each data block is located;

根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 The metadata of each data block, each data block generates the metadata recovery information.

可选的,处理器1102执行的从所述NN获取所述多个数据块的元数据 的程序,可以包括: Optionally, acquiring the plurality of program data blocks from the NN metadata processor 1102 performs, comprising:

依次接收所述NN发送的所述每个数据块的元数据;或者,接收所述NN根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 NN sequentially receiving said transmitted the metadata of each data block; or NN receiving the request metadata disposable all data blocks transmitted according to a write operation of the target file.

可选的,所述接收的元数据至少可以包括第一数据块的元数据和第二数据块的元数据,其中: Optionally, the received metadata may include at least the metadata of the first data block and second data block metadata, wherein:

所述生成的所述第一数据块的元数据恢复信息可以包括: The generated metadata to restore the first data block information may include:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述生成的所述第二数据块的元数据恢复信息可以包括: Generating metadata for the second data block of the recovery information may include:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

可选的,上述设备可以是HDFS的客户端(Client)所属的设备。 Optionally, the device may be HDFS client (Client) device belongs.

上述技术方案中,接收对目标文件的写操作请求;从NN获取所述目标文件的元数据;根据所述目标文件的元数据生成元数据恢复信息;向所述多个DN中的至少一个DN发送所述元数据恢复信息。 In the above technical solution, receiving a request for a write operation of the target file; obtaining the target file from the metadata NN; Recovery information element data of the target file according to generate metadata; DN to at least one of said plurality of DN transmitting the metadata recovery information. 这样元数据恢复信息可以用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复,从而可以恢复NN出异常时的元数据。 Such metadata information may be used to recover at least one of said DN is saved, and then transmitted to the NN abnormality recovery occurs in the metadata of the NN, thereby restoring metadata NN when an abnormality.

请参阅图12,图12是本发明实施例提供的一种DN设备的结构示意图,包括至少一个处理器1202(例如CPU),至少一个网络接口1205或者其他通信接口,存储器1206,和至少一个通信总线1203,用于实现这些装置之间的连接通信。 Please refer to FIG. 12, FIG. 12 is a structural diagram of a DN apparatus according to an embodiment of the present invention comprises at least a processor 1202 (e.g., the CPU), at least one network interface, or other communications interfaces 1205, memory 1206, and at least one communication bus 1203 for implementing communications connections between these devices. 处理器1202用于执行存储器1206中存储的可执行模块,例如计算机程序。 Processor 1202 for performing a memory 1206 stores an executable module, such as a computer program. 存储器1206可能包含高速随机存取存储器(RAM:Random Access Memory),也可能还包括非不稳定的存储器(non-volatile memory),例如至少一个磁盘存储器。 Memory 1206 may include high speed random access memory (RAM: Random Access Memory), and may also include non-volatile memory (non-volatile memory), for example, at least one disk memory. 通过至少一个网络接口1205(可以是有线或者无 线)实现该系统网关与至少一个其他网元之间的通信连接,可以使用互联网,广域网,本地网,城域网等。 Via at least one network interface 1205 (which may be wired or wireless) to achieve a communication connection between the gateway system with at least one other network element can use the Internet, a wide area network, local area network, metropolitan area network.

在一些实施方式中,存储器1206存储了程序12061,程序12061可以被处理器1202执行,这个程序包括: In some embodiments, the memory 1206 stores a program 12061, 12061 may be a program executed by the processor 1202, the program comprising:

接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从NN获取的所述目标文件的元数据,并根据所述元数据生成的元数据恢复信息.; Receiving the target files sent by the client metadata recovery information and storing the recovery information metadata, wherein the metadata information is restored upon receiving a client request for a write target file acquired from NN the target file metadata, and restoring metadata information to the metadata generated.;

在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 After the NN abnormal, sending the information to the meta data recovery NN, metadata information is the NN Recovery Recovery for the metadata.

在另一个实施例中,所述目标文件被划分为多个数据块,处理器1202执行的接收客户端发送的目标文件的元数据恢复信息的程序,可以包括: In another embodiment, the target file is divided into a plurality of data blocks, the target metadata file from a client processor 1202 performs transmission of information recovery program may include:

接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,所述数据块的元数据恢复信息为所述客户端根据该数据块的元数据生成的元数据恢复信息。 Receiving the data blocks of the target file is sent by the client metadata recovery information, metadata information of the data block to recover the information according to the recovery client metadata in the metadata generated by the data block.

可选的,处理器1202执行的接收所述客户端发送的所述目标文件的数据块的元数据恢复信息的程序,可以包括: Alternatively, the program metadata of the data block receiving the target file to the client processor 1202 performs transmit resume information may include:

依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Metadata plurality of data blocks sequentially receives the target file sent by the customer endpoint recovery information; or metadata of a plurality of data blocks of the target client receives the file recovery information transmitted disposable.

可选的,所述接收的元数据恢复信息至少可以包括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: Optionally, the recovery information received metadata may include at least a first data block of metadata and metadata information restoring the second data block recovery information, wherein:

所述第一数据块的元数据恢复信息可以包括: The metadata of the first data block may resume information comprising:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置 信息; Attribute information of the target file in the first data block and the location information of the target file;

所述第二数据块的元数据恢复信息可以包括: The metadata of the second data block may resume information comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

可选的,本节点设备可以是HDFS的DN所属的设备。 Alternatively, the present device may be a device node DN of HDFS belongs.

上述技术方案中,接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息;在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 In the above technical solution, the target metadata file sent by a client of the recovery information, and store the meta data recovery information; after abnormality occurs in the NN, sending the metadata information to recover the NN, the metadata retriever NN said information is used for recovery of the metadata. 从而可以实现恢复NN出异常时的元数据。 Thereby restoring metadata may be achieved when the abnormality NN.

请参阅图13,图13是本发明实施例提供的一种NN设备的结构示意图,包括至少一个处理器1302(例如CPU),至少一个网络接口1305或者其他通信接口,存储器1306,和至少一个通信总线1303,用于实现这些装置之间的连接通信。 Refer to FIG. 13, FIG. 13 is a structural diagram of a NN apparatus according to an embodiment of the present invention comprises at least a processor 1302 (e.g., the CPU), at least one network interface, or other communications interfaces 1305, memory 1306, and at least one communication bus 1303 for implementing communications connections between these devices. 处理器1302用于执行存储器1306中存储的可执行模块,例如计算机程序。 Processor 1302 for performing a memory 1306 stores an executable module, such as a computer program. 存储器1306可能包含高速随机存取存储器(RAM:Random Access Memory),也可能还包括非不稳定的存储器(non-volatile memory),例如至少一个磁盘存储器。 Memory 1306 may include high speed random access memory (RAM: Random Access Memory), and may also include non-volatile memory (non-volatile memory), for example, at least one disk memory. 通过至少一个网络接口1305(可以是有线或者无线)实现该系统网关与至少一个其他网元之间的通信连接,可以使用互联网,广域网,本地网,城域网等。 Via at least one network interface 1305 (which may be wired or wireless) a communication connection between the gateway system with at least one other network element can use the Internet, a wide area network, local area network, metropolitan area network.

在一些实施方式中,存储器1306存储了程序13061,程序13061可以被处理器1302执行,这个程序包括: In some embodiments, the memory 1306 stores a program 13061, 13061 may be a program executed by the processor 1302, the program comprising:

接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个DN上; Target file is sent by a client requests a write operation, wherein the plurality of target files stored on the DN;

生成所述目标文件的元数据,并向所述客户端发送所述元数据,其中,所述元数据包含所述多个DN的信息; Generating a metadata of the target file, the client sends to the metadata, wherein the metadata information comprises a plurality of DN;

当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至所述至少一个DN进行保存的元数据恢复信息。 When an exception occurs this end, a plurality of receiving metadata of at least one DN DN recover transmitted information, and restoring the restoration information metadata according to the metadata, wherein the metadata information is the recovery client-generated based on the metadata sent by the client to at least one of said stored metadata DN recovery information.

在另一个实施例中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上;处理器1302执行的生成所述目标文件的元数据,并向所述客户端发送所述元数据,可以包括: Generating a metadata of the target file execution processor 1302, and; in another embodiment, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored in the plurality of DN sends the metadata to the client, may include:

生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息。 Generating a plurality of data blocks of metadata, and metadata for the client sends the plurality of data blocks, each data block metadata comprises location information of each data block and each data block of the information DN.

处理器1302执行的接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据的程序,可以包括: Receiving said plurality of program meta data DN processor 1302 performs at least a DN recover transmitted information, and restoring the restoration information metadata according to the metadata, may include:

接收所述多个DN中的至少一个DN发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 Receiving the plurality of data blocks in at least one of the plurality of DN DN recover transmitted information metadata, and restoring the restoration information file meta data of the target data blocks of the plurality of metadata.

可选的,处理器1302执行的向所述客户端发送所述多个数据块的元数据的程序,可以包括: Optionally, the processor 1302 executing program sending the plurality of data blocks of metadata to the client, may include:

依次向所述客户端发送所述多个数据块的元数据;或者,根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Sequentially transmitted to the client metadata of the plurality of data blocks; or, according to a write request for a one-time operation of the target file metadata for all transmission data block to the client.

可选的,上述多个数据块至少可以包括第一数据块和第二数据块,其中: Optionally, the plurality of data blocks may include at least a first data block and second data block, wherein:

所述第一数据块的元数据恢复信息可以包括: The metadata of the first data block may resume information comprising:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;

所述第二数据块的元数据恢复信息可以包括: The metadata of the second data block may resume information comprising:

所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.

可选的,上述节点设备可以是HDFS内的NN所属的设备。 Optionally, the device may be a device node NN belongs within the HDFS.

上述技术方案中,接收客户端发送的对目标文件进行写操作请求;生成所述目标文件的元数据,并向所述客户端发送所述元数据;当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据。 In the above technical solution, the target file is sent by a client requests a write operation; metadata generating the target file, the client sends to the metadata; when an abnormality occurs this end, the multi-receiver metadata a DN of at least one transmitted DN recovery information, and restoring the restoration information metadata according to the metadata. 从而可以恢复NN出异常时的元数据。 NN thereby restore the metadata when an abnormality.

本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程,是可以通过计算机程序来指令相关的硬件来完成,所述的程序可存储于一计算机可读取存储介质中,该程序在执行时,可包括如上述各方法的实施例的流程。 Those of ordinary skill in the art may understand that the above-described method embodiments all or part of the processes may be related hardware instructed by a computer program, the program may be stored in a computer readable storage medium, the program when executed, the processes of the foregoing method embodiments. 其中,所述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory,ROM)或随机存取存储器(Random Access Memory,简称RAM)等。 Wherein the storage medium may be a magnetic disk, an optical disk, read-only memory (Read-Only Memory, ROM) or random access memory (Random Access Memory, referred to as RAM).

以上所揭露的仅为本发明较佳实施例而已,当然不能以此来限定本发明之权利范围,因此依本发明权利要求所作的等同变化,仍属本发明所涵盖的范围。 Preferred embodiments disclosed above are merely embodiments of the present invention, it, of course, not intended to limit the scope of the invention, therefore equivalent changes made under this invention as claimed in claim still covered by the scope of the present invention.

Claims (24)

  1. 一种元数据恢复方法,其特征在于,包括: A meta data recovery method comprising:
    接收对目标文件的写操作请求,其中,所述目标文件保存在多个数据节点上; Receiving a write request for a target file, wherein the target file is stored on a plurality of data nodes;
    从元数据节点获取所述目标文件的元数据,所述目标文件的元数据包含所述多个数据节点的信息; Metadata node acquires the target file from the metadata, the target file metadata contains information of the plurality of data nodes;
    根据所述目标文件的元数据生成元数据恢复信息; Generate metadata according to the metadata recovery information for the target file;
    向所述多个数据节点中的至少一个数据节点发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个数据节点进行保存,并在所述元数据节点出现异常后被发送到所述元数据节点进行所述元数据的恢复。 Sending the metadata information to recover at least one data node of the plurality of data nodes, the metadata for the at least one data recovery information of the node is saved, and abnormal after the metadata node transmitting the metadata node to recover the metadata.
  2. 如权利要求1所述的方法,其特征在于,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个数据节点上; The method according to claim 1, wherein the target file is divided into a plurality of data blocks, said plurality of data blocks are stored in the plurality of data nodes;
    则,所述从元数据节点获取所述目标文件的元数据,根据所述目标文件的元数据生成元数据恢复信息包括: Then, the metadata acquiring the target file from the metadata node, the resume information including metadata of the target file generates metadata from:
    从所述元数据节点获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的数据节点的信息; Acquiring said plurality of data blocks of metadata from the metadata node, the metadata of each data block contains information data node information of said each data block and each data block is located;
    根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 The metadata of each data block, each data block generates the metadata recovery information.
  3. 如权利要求2所述的方法,其特征在于,所述从所述元数据节点获取所述多个数据块的元数据包括: The method according to claim 2, wherein said acquiring said plurality of data blocks of metadata from the metadata node comprises:
    依次接收所述元数据节点发送的所述每个数据块的元数据;或者,接收所述元数据节点根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 The metadata node sequentially transmits the received metadata of each data block; or the metadata node receives a request-time metadata transmission of all the data blocks according to the write operation of the target file.
  4. 如权利要求2或3所述的方法,其特征在于,所述接收的元数据至少包括第一数据块的元数据和第二数据块的元数据,其中: The method of claim 2 or claim 3, wherein the received metadata comprises metadata in the metadata of at least the first data block and second data block, wherein:
    所述生成的所述第一数据块的元数据恢复信息包括: The generated metadata first data block recovery information comprising:
    所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;
    所述生成的所述第二数据块的元数据恢复信息包括: Generating metadata for the second data block of the recovery information comprising:
    所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.
  5. 一种元数据恢复方法,其特征在于,包括: A meta data recovery method comprising:
    接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从元数据节点获取的所述目标文件的元数据,并根据所述元数据生成的元数据恢复信息.; Receiving metadata target file recovery information sent by the client, and storing the recovery information metadata, wherein the metadata information is restored upon receiving a client request for a write operation of the target file from the metadata node acquiring metadata of the target file, and recover the information according to the metadata generated by the metadata.;
    在所述元数据节点出现异常后,向所述元数据节点发送所述元数据恢复信息,元数据恢复信息被所述元数据节点用于进行所述元数据的恢复。 After an exception occurs the metadata node sends the metadata information to recover the metadata node, metadata information restored by the restoration metadata node for the metadata.
  6. 如权利要求5所述的方法,其特征在于,所述目标文件被划分为多个数据块,所述接收客户端发送的目标文件的元数据恢复信息,包括: The method as claimed in claim 5, wherein the target file is divided into a plurality of data blocks, the target receives the metadata file recovery information sent by the client, comprising:
    接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,所述数据块的元数据恢复信息为所述客户端根据该数据块的元数据生成的元数据恢复信息。 Receiving the data blocks of the target file is sent by the client metadata recovery information, metadata information of the data block to recover the information according to the recovery client metadata in the metadata generated by the data block.
  7. 如权利要求6所述的方法,其特征在于,所述接收所述客户端发送的所述目标文件的数据块的元数据恢复信息包括: The method according to claim 6, wherein the metadata of the data block of the target file receiving the resume information sent by the client comprises:
    依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Metadata plurality of data blocks sequentially receives the target file sent by the customer endpoint recovery information; or metadata of a plurality of data blocks of the target client receives the file recovery information transmitted disposable.
  8. 如权利要求6或7所述的方法,其特征在于,所述接收的元数据恢复信息至少包括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: The method of claim 6 or claim 7, wherein said recovery information received metadata comprises at least a first data block of metadata and metadata information restoring the second data block recovery information, wherein:
    所述第一数据块的元数据恢复信息包括: The metadata information of first data recovery block comprises:
    所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;
    所述第二数据块的元数据恢复信息包括: The metadata information of the second block of data recovery comprising:
    所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.
  9. 一种元数据恢复方法,其特征在于,包括: A meta data recovery method comprising:
    接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个数据节点上; Write request to the target file is sent by a client, wherein the target file is stored on a plurality of data nodes;
    生成所述目标文件的元数据,并向所述客户端发送所述元数据,其中,所述元数据包含所述多个数据节点的信息; Generating a metadata of the target file, the client sends to the metadata, wherein the metadata information comprises a plurality of data nodes;
    当本端出现异常时,接收所述多个数据节点中的至少一个数据节点发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至所述至少一个数据节点进行保存的元数据恢复信息。 When this abnormal end occurs, the metadata of the at least one data transmitting node of said plurality of receiving nodes in the data recovery information, and restoring the restoration information metadata according to the metadata, wherein the metadata information is restored the client-generated based on the metadata sent by the client to the at least one data node metadata stored in the recovery information.
  10. 如权利要求9所述的方法,其特征在于,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个数据节点上; The method according to claim 9, wherein the target file is divided into a plurality of data blocks, said plurality of data blocks are stored in the plurality of data nodes;
    则所述生成所述目标文件的元数据,并向所述客户端发送所述元数据包括: The metadata is generated for the target file, the client sends to the metadata comprises:
    生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的数据节点的信息; Generating a plurality of data blocks of metadata, and metadata for the client sends the plurality of data blocks, each data block metadata comprises location information of each data block and each data block of the information of the data node;
    所述接收所述多个数据节点中的至少一个数据节点发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,包括: Receiving said plurality of said data node metadata node sending at least one data recovery information, and restoring the restoration information metadata according to the metadata, comprising:
    接收所述多个数据节点中的至少一个数据节点发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 Metadata of the plurality of data blocks of said plurality of data receiving nodes node sending at least one data recovery information, and restoring the restoration information file meta data of the target data blocks of the plurality of metadata.
  11. 如权利要求10所述的方法,其特征在于,所述向所述客户端发送 所述多个数据块的元数据包括: The method according to claim 10, wherein said transmitting metadata to the client of the plurality of data blocks comprising:
    依次向所述客户端发送所述多个数据块的元数据;或者,根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Sequentially transmitted to the client metadata of the plurality of data blocks; or, according to a write request for a one-time operation of the target file metadata for all transmission data block to the client.
  12. 如权利要求10或11所述的方法,其特征在于,所述多个数据块至少包括第一数据块和第二数据块,其中: The method of claim 10 or claim 11, wherein said plurality of data blocks include at least a first data block and second data block, wherein:
    所述第一数据块的元数据恢复信息包括: The metadata information of first data recovery block comprises:
    所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;
    所述第二数据块的元数据恢复信息包括: The metadata information of the second block of data recovery comprising:
    所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.
  13. 一种客户端,其特征在于,包括:接收单元、获取单元、生成单元和发送单元,其中: A client, characterized by comprising: a receiving unit, an acquisition unit, generation unit, and a sending unit, wherein:
    所述接收单元,用于接收对目标文件的写操作请求,其中,所述目标文件保存在多个数据节点上; The receiving unit, for receiving a write request for a target file, wherein the target file is stored on a plurality of data nodes;
    所述获取单元,用于从元数据节点获取所述目标文件的元数据,所述目标文件的元数据包含所述多个数据节点的信息; The obtaining unit is configured to obtain the target file metadata from the metadata node, the target file metadata contains information of the plurality of data nodes;
    所述生成单元,用于根据所述目标文件的元数据生成元数据恢复信息; The generating means for generating metadata according to the metadata recovery information for the target file;
    所述发送单元,用于向所述多个数据节点中的至少一个数据节点发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个数据节点进行保存,并在所述元数据节点出现异常后被发送到所述元数据节点进行所述元数据的恢复。 The sending unit, for sending the metadata information to at least one recovery data node of the plurality of data nodes, the metadata for the at least one data recovery information of the node is saved, and the exception occurs after transmitting metadata node to the metadata node to recover the metadata.
  14. 如权利要求13所述的客户端,其特征在于,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个数据节点上; The client according to claim 13, wherein the target file is divided into a plurality of data blocks, said plurality of data blocks are stored in the plurality of data nodes;
    则,所述获取单元用于从所述元数据节点获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的数据节点的信息; Then, the acquisition unit for acquiring said plurality of data blocks of metadata from the metadata node, the metadata of each data block contains information data node of said each data block and each data block is located Information;
    所述生成单元用于根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 The metadata generating unit for each data block, each data block generates the metadata recovery information.
  15. 如权利要求14所述的客户端,其特征在于,所述获取单元用于依次接收所述元数据节点发送的所述每个数据块的元数据;或者,所述获取单元用于接收所述元数据节点根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 The client according to claim 14, wherein said means for acquiring metadata of said metadata node sequentially receives each transmitted data block; or the acquisition unit for receiving the request metadata metadata node all data blocks transmitted according to a one-time writing operation of the target file.
  16. 如权利要求14或15所述的客户端,其特征在于,所述接收的元数据至少包括第一数据块的元数据和第二数据块的元数据,其中: The client of claim 14 or claim 15, wherein the received metadata comprises metadata in the metadata of at least the first data block and second data block, wherein:
    所述生成的所述第一数据块的元数据恢复信息包括: The generated metadata first data block recovery information comprising:
    所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;
    所述生成的所述第二数据块的元数据恢复信息包括: Generating metadata for the second data block of the recovery information comprising:
    所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.
  17. 一种数据节点,其特征在于,包括:接收单元和发送单元,其中: A data node, characterized by comprising: a receiving unit and a sending unit, wherein:
    所述接收单元,用于接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从元数据节点获取的所述目标文件的元数据,并根据所述元数据生成的元数据恢复信息.; The receiving unit, for receiving metadata target file recovery information sent by the client, and storing the recovery information metadata, wherein the metadata information is restored in the client receiving a write operation of the target file when requested, the metadata of the target file acquired from the metadata node, and recover information according to the metadata generated by the metadata.;
    所述发送单元,用于在所述元数据节点出现异常后,向所述元数据节点发送所述元数据恢复信息,元数据恢复信息被所述元数据节点用于进行所述元数据的恢复。 The sending unit, configured to, after the metadata node abnormality occurs, sending the metadata information to recover the metadata node, metadata information restored by the restoration metadata node for the metadata .
  18. 如权利要求17所述的数据节点,其特征在于,所述目标文件被划分为多个数据块,所述接收单元用于接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,所述数据块的元数据恢复信息为所述客户端根据该数据块的元数据生成的元数据恢复信息。 Metadata of the data node according to claim 17, wherein the target file is divided into a plurality of data blocks, said data blocks of said receiving means for receiving the target files sent by the client recovery information, metadata information of the data block to recover the information according to the recovery client metadata in the metadata generated by the data block.
  19. 如权利要求18所述的数据节点,其特征在于,所述接收单元用于 依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,所述接收单元用于接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Data node according to claim 18, wherein said plurality of data blocks received metadata of the target file receiving unit for sequentially transmitting the customer endpoint recovery information; Alternatively, the receiving unit with a plurality of data blocks of a metadata file to the target client receiving the recovery information sent disposable.
  20. 如权利要求17或18所述的数据节点,其特征在于,所述接收的元数据恢复信息至少包括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: Data node 17 or claim 18, wherein said recovery information received metadata comprises at least a first data block of metadata and metadata information restoring the second data block recovery information, wherein:
    所述第一数据块的元数据恢复信息包括: The metadata information of first data recovery block comprises:
    所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;
    所述第二数据块的元数据恢复信息包括: The metadata information of the second block of data recovery comprising:
    所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.
  21. 一种元数据节点,其特征在于,包括:接收单元、生成单元和恢复单元,其中: A meta data nodes, wherein, comprising: a receiving unit, a generating unit and a restoring unit, wherein:
    所述接收单元,用于接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个数据节点上; The receiving unit is configured to write the target file operation request sent by a client, wherein the target file is stored on a plurality of data nodes;
    所述生成单元,用于生成所述目标文件的元数据,并向所述客户端发送所述元数据,其中,所述元数据包含所述多个数据节点的信息; The generating unit generating metadata for the target file, the client sends to the metadata, wherein the metadata information comprises a plurality of data nodes;
    所述恢复单元,用于当本端出现异常时,接收所述多个数据节点中的至少一个数据节点发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至所述至少一个数据节点进行保存的元数据恢复信息。 The restoring unit configured to, when abnormality occurs this end, the at least one data receiving metadata of the plurality of nodes is transmitting data recovery information, and restoring the restoration information metadata according to the metadata, wherein the restoring metadata generating information of the client based on the metadata sent by the client to the at least one data node metadata stored in the recovery information.
  22. 如权利要求21所述的元数据节点,其特征在于,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个数据节点上; Metadata node according to claim 21, wherein the target file is divided into a plurality of data blocks, said plurality of data blocks are stored in the plurality of data nodes;
    所述生成单元用于生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息 和所述每个数据块所在的数据节点的信息; The metadata generating means for generating the plurality of data blocks, the client sends to the metadata of the plurality of data blocks, each data block of metadata containing information for each data block and the said information data node of each data block is located;
    所述恢复单元用于当本端出现异常时,接收所述多个数据节点中的至少一个数据节点发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 The restoring unit configured to, when abnormality occurs this end, the metadata for the plurality of data blocks of the at least one data node of said plurality of receiving nodes in the data transmission resume information and the meta data of the plurality of blocks recovery data recovery information of the target file metadata.
  23. 如权利要求21所述的元数据节点,其特征在于,所述生成单元用于生成所述多个数据块的元数据,并依次向所述客户端发送所述多个数据块的元数据;或者,所述生成单元用于生成所述多个数据块的元数据,并根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Metadata node according to claim 21, wherein the metadata generating means for generating the plurality of data blocks, and sequentially transmits the metadata for the plurality of data blocks to the client; Alternatively, the generating means for generating a plurality of data blocks of the metadata, and requests the metadata for all one-time transmission data block to the client according to the write operation of the target file.
  24. 如权利要求10或11所述的元数据节点,其特征在于,所述多个数据块至少包括第一数据块和第二数据块,其中: Metadata node of claim 10 or claim 11, wherein said plurality of data blocks include at least a first data block and second data block, wherein:
    所述第一数据块的元数据恢复信息包括: The metadata information of first data recovery block comprises:
    所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file in the first data block and the location information of the target file;
    所述第二数据块的元数据恢复信息包括: The metadata information of the second block of data recovery comprising:
    所述第二数据块在所述目标文件中的位置信息。 Position information of the second data block in the target file.
PCT/CN2015/083226 2014-08-27 2015-07-02 Metadata recovery method and relevant device WO2016029744A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN 201410427894 CN104202387B (en) 2014-08-27 2014-08-27 A meta data restoration method and related apparatus
CN201410427894.2 2014-08-27

Publications (1)

Publication Number Publication Date
WO2016029744A1 true true WO2016029744A1 (en) 2016-03-03

Family

ID=52087612

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2015/083226 WO2016029744A1 (en) 2014-08-27 2015-07-02 Metadata recovery method and relevant device

Country Status (2)

Country Link
CN (1) CN104202387B (en)
WO (1) WO2016029744A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104202387B (en) * 2014-08-27 2017-11-24 华为技术有限公司 A meta data restoration method and related apparatus

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030220943A1 (en) * 2002-05-23 2003-11-27 International Business Machines Corporation Recovery of a single metadata controller failure in a storage area network environment
CN102024044A (en) * 2010-12-08 2011-04-20 华为技术有限公司 Distributed file system
CN102843403A (en) * 2011-06-23 2012-12-26 盛大计算机(上海)有限公司 File processing method based on distributed file system, system, and client
CN103076994A (en) * 2012-12-31 2013-05-01 北京中科大洋科技发展股份有限公司 Method for realizing off-line file writing in SAN (Storage Area Network) shared-file system
CN104202387A (en) * 2014-08-27 2014-12-10 华为技术有限公司 Metadata recovery method and related device

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100587692C (en) * 2007-01-26 2010-02-03 华中科技大学 Method and system for promoting metadata service reliability
CN101539873B (en) * 2009-04-15 2011-02-09 成都市华为赛门铁克科技有限公司 Data recovery method, data node and distributed file system
CN101986276B (en) * 2010-10-21 2013-07-10 华为数字技术(成都)有限公司 Methods and systems for storing and recovering files and server
JP5611889B2 (en) * 2011-05-17 2014-10-22 株式会社東芝 Data transfer apparatus, a data transmission system and data transmission method
CN103067418A (en) * 2011-10-19 2013-04-24 北大方正集团有限公司 Method, device and system for reconstructing metadata
CN103761162B (en) * 2014-01-11 2016-12-07 深圳清华大学研究院 Data backup method of distributed file system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030220943A1 (en) * 2002-05-23 2003-11-27 International Business Machines Corporation Recovery of a single metadata controller failure in a storage area network environment
CN102024044A (en) * 2010-12-08 2011-04-20 华为技术有限公司 Distributed file system
CN102843403A (en) * 2011-06-23 2012-12-26 盛大计算机(上海)有限公司 File processing method based on distributed file system, system, and client
CN103076994A (en) * 2012-12-31 2013-05-01 北京中科大洋科技发展股份有限公司 Method for realizing off-line file writing in SAN (Storage Area Network) shared-file system
CN104202387A (en) * 2014-08-27 2014-12-10 华为技术有限公司 Metadata recovery method and related device

Also Published As

Publication number Publication date Type
CN104202387B (en) 2017-11-24 grant
CN104202387A (en) 2014-12-10 application

Similar Documents

Publication Publication Date Title
US20080243847A1 (en) Separating central locking services from distributed data fulfillment services in a storage system
US8473775B1 (en) Locality based quorums
US20140279930A1 (en) Fast crash recovery for distributed database systems
US20110196828A1 (en) Method and System for Dynamically Replicating Data Within A Distributed Storage System
US9053167B1 (en) Storage device selection for database partition replicas
US8285689B2 (en) Distributed file system and data block consistency managing method thereof
US20150134796A1 (en) Dynamic partitioning techniques for data streams
US20150207844A1 (en) Content Item Synchronization By Block
US20120278429A1 (en) Cluster system, synchronization controlling method, server, and synchronization controlling program
CN201682522U (en) Conversation information storage system and application server
US20150032692A1 (en) Prioritizing content item synchronization based on sharing
US20150134626A1 (en) Partition-based data stream processing framework
JPH11249943A (en) Method and system for synchronous management of distributed data base
US20150234845A1 (en) Subpartitioning of a namespace region
US20140188868A1 (en) Predictive models of file access patterns by application and file type
US20150135255A1 (en) Client-configurable security options for data streams
CN103268318A (en) Distributed key value database system with strong consistency and read-write method thereof
US20110196882A1 (en) Operating On Objects Stored In A Distributed Database
US20140304384A1 (en) Uploading large content items
US20140181035A1 (en) Data management method and information processing apparatus
CN101699436A (en) Method, device and system for resource management
US20150019491A1 (en) Replication of Data Between Mirrored Data Sites
US20100228774A1 (en) Protocol Independent Mirroring
US9519664B1 (en) Index structure navigation using page versions for read-only nodes
CN101673289A (en) Method and device for constructing distributed file storage framework

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15836410

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase in:

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15836410

Country of ref document: EP

Kind code of ref document: A1