CN110278222A

CN110278222A - The method, system and relevant device of data management in distributed file storage system

Info

Publication number: CN110278222A
Application number: CN201810213670.XA
Authority: CN
Inventors: 金中良
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2018-03-15
Filing date: 2018-03-15
Publication date: 2019-09-24
Anticipated expiration: 2038-03-15
Also published as: CN110278222B

Abstract

The embodiment of the present application provides a kind of data managing method applied to distributed file storage system, system and relevant device, this method includes reading file data from file to be written to form n file data unit, redundant computation is carried out to the n file data unit and obtains m redundant data unit, and the n file data unit is respectively written into the n blocks of files on multiple back end of distributed file storage system, and the m redundant data unit is respectively written into m redundant block of multiple back end of the distributed file storage system, wherein n and m is positive integer；It is subsequent when there is blocks of files to occur abnormal, the content that redundant computation recover abnormal document block can be carried out to normal file block and redundant block.This method improves the utilization rate of memory space under the premise of ensureing distributed file storage system fault-tolerance.

Description

The method, system and relevant device of data management in distributed file storage system

Technical field

This application involves field of data storage, more particularly in distributed memory system data management method, system And relevant device.

Background technique

Traditional network store system stores all data using the storage server concentrated, and storage server becomes system The bottleneck and reliability of performance and the focus of safety are not able to satisfy the needs of Mass storage application.Distributed storage system System is that data dispersion is stored in more independent equipment.Distributed network storage system uses expansible system structure, Storage load is shared using more storage servers, positions storage information using location server, it not only increases system Reliability, availability and access efficiency are also easy to extend.

File is divided into multiple " blocks " (Block) storage to multiple back end by existing distributed memory system, and Multiple back end store the copy (backup or copy for described piece) of these blocks, and pass through metadata record file and these The corresponding relationship of block/copy corresponding relationship, block/copy and back end；There is copy in multiple back end due to each piece, When the damage of some back end, file access equipment can read from other back end and store on damage back end The content of block keeps operation layer unaffected to file access, improves the fault-tolerance (fault tolerance) of storage system.

This method has that storage space utilization is low, for example, by taking 2 copies of each blocks of files as an example, storage Space utilization rate=1/ (1+2)=33%, if copy is more, storage space utilization will be lower.

Summary of the invention

In consideration of it, it is necessary to provide a kind of data managing method, to promote the memory space of distributed file storage system Utilization rate.

In a first aspect, the embodiment of the present application provides a kind of data management side applied to distributed file storage system Method, the distributed file storage system includes multiple back end, this method comprises: reading number of files from file to be written N file data unit is formed accordingly, and the n file data unit includes the file data of the reading；To the n text Part data cell carries out redundant computation, obtains m redundant data unit；The n file data unit is respectively written into described In n blocks of files on multiple back end, and the m redundant data unit is respectively written into the multiple back end On m redundant block in；Wherein described n, m are positive integer.

This method can reduce redundant data, effectively promote the utilization rate of memory space.

In a possible scheme, the storage location of the n blocks of files is located in the multiple back end N back end on, the n file n file data unit being respectively written on the multiple back end It include: that concurrently the n file data unit is respectively written into the n blocks of files in block.

In a possible scheme, the storage location of the n blocks of files is located in the multiple back end N back end on, the storage location of the m redundant block is located at other m data in the multiple back end It is described the n file data unit to be respectively written into the n blocks of files and by the m redundant data in node It includes: that the n file data unit is concurrently respectively written into the n text that unit, which is respectively written into the m redundant block, The m redundant data unit is concurrently respectively written into the m redundant block by part block.

In a possible scheme, the file data that reads from the file to be written is to form n number of files Comprise determining that the file data amount read from the file to be written less than the corresponding number of n file data unit according to unit According to amount；In the file data end supplemental data of the reading to form the n file data unit.

In a possible scheme, the distributed file storage system further includes managing the multiple back end Management node, the method also includes: Xiang Suoshu management node sends the quantity of the supplemental data.

In a possible scheme, the distributed file storage system further includes managing the multiple back end Management node, it is described the n file data unit is respectively written into the n blocks of files before, the method also includes: from The management node obtains the storage location letter of the n blocks of files, the m blocks of files in the multiple back end Breath.

It is described to obtain the n blocks of files, the m blocks of files from the management node in a possible scheme Storage location information in the multiple back end includes: to send storage location request to the management node, described to deposit Store up unique identification of the location request containing the file to be written in the distributed file storage system；It is saved from the management Point receives storage location response, includes the n blocks of files in the multiple back end in the storage location response The storage location of storage location and the m redundant block in the multiple back end.

In a possible scheme, the storage location request is also comprising redundancy scheme mark or the n and m.

In a possible scheme, comprising the n blocks of files in the multiple data in the storage location response The storage location of storage location and the m redundant block in the multiple back end in node, the storage location are rung It include the metadata information of the file to be written in answering, the metadata information is identified to blocks of files to blocks of files comprising n and exists The corresponding relationship and m of storage location in the multiple back end are identified to redundant block in the multiple data section to redundant block The corresponding relationship of storage location in point.

In a possible scheme, the distributed file storage system further includes managing the multiple back end Management node, it is described to the n file data unit carry out redundant computation before, the method also includes: from the management Node obtains the corresponding redundant arithmetic of the redundant computation.

In a possible scheme, the distributed file storage system further includes managing the multiple back end Management node, the method also includes: Xiang Suoshu management node sends the corresponding redundant arithmetic of the redundant computation.

In a possible scheme, when m is 1, the redundant arithmetic includes parity arithmetic, alternatively, when m is 2 When, the redundant arithmetic includes the Q checking algorithm based on Galois Field.

In a possible scheme, the byte number that the file data unit and the redundant data unit are included is equal For 8 multiple.

Second aspect, the embodiment of the present application provide a kind of data management side applied to distributed file storage system Method, the distributed file storage system include multiple back end, and file to be restored is stored in the multiple back end N blocks of files and m redundant block, include f normal file block and n-f abnormal document block in the n blocks of files, it is described Method includes: to read a file data unit respectively from d normal file block, obtains d file data unit, the d It less than or equal to f, and reads a redundant data unit respectively from the m redundant block, obtains m redundant data list Member；Redundant computation is carried out to the d file data unit and the m redundant data unit, obtains n-f file data list Member；The n-f file data unit is restored to respectively in n-f abnormal document block in the n blocks of files；The n, M, f, d are positive integer.

This method guarantees that the fault-tolerance of the distributed file storage system can when failure blocks of files To restore the data of blocks of files of being out of order from normal file block and redundant file block.

It is described that the n-f file data unit is restored to the n blocks of files respectively in a possible scheme In n-f abnormal document block in include: to be created that new blocks of files in the storage location of the n-f abnormal document block respectively, Obtain n-f new blocks of files；The n-f file data unit is respectively written into the n-f new blocks of files.

In a possible scheme, it is described according to redundant arithmetic to the d file data unit and the m redundancy Data cell carries out redundant computation, before obtaining n-f file data unit, the method also includes: from the text to be restored The redundant arithmetic is obtained in the corresponding metadata information of part.

It is described before reading a file data unit respectively in d normal file block in a possible scheme, The method also includes: the storage location of the n blocks of files is obtained from the corresponding metadata information of the file to be restored With the storage location of the m blocks of files.

The third aspect, the embodiment of the present application provide a kind of data management side applied to distributed file storage system Method, the distributed file storage system include management node, multiple back end and data writing device, this method comprises: The data writing device sends storage location request to the management node, and the storage location request includes file to be written Unique identification in the distributed file storage system；The management node is according to the file to be written in the distribution Unique identification in formula document storage system determines n blocks of files and m redundant block depositing on the multiple back end Location information is stored up, and returns to storage location response to the data writing device, the storage location response includes the storage Location information, the n and m are positive integer；The data writing device reads file data from the file to be written with shape At first group of n file data unit, redundant computation is carried out to described first group n file data unit, obtains m redundant digit According to unit, described first group n file data unit is respectively written into the n blocks of files, and by the m redundant data Unit is respectively written into the m redundant block, wherein the n and m are positive integer.

In a possible scheme, when in the n blocks of files include f normal file block and n-f abnormal document When block, the method also includes: the management node reads a file data unit respectively from d normal file block, obtains To d file data unit, the d is less than or equal to f, and reads a redundant data respectively from the m redundant block Unit obtains m redundant data unit；The management node is to the d file data unit and the m redundant data list Member carries out redundant computation, obtains n-f file data unit；The management node distinguishes the n-f file data unit It is restored in n-f abnormal document block in the n blocks of files.

In a possible scheme, the distributed file storage system further includes data taking equipment, the method Further include: the data taking equipment reads a file data unit from the n data blocks of files respectively, obtains second N file data unit of group；The n file data unit is written the data taking equipment and created by the data taking equipment In the file built.

In a possible scheme, the data writing device reads file data from the file to be written with shape It include: that the data writing device determines the file read from the file to be written at first group of n file data unit Data volume be less than the corresponding data volume of n file data unit, the reading file data end supplemental data to be formed Described first group n file data unit；The quantity of the supplemental data is sent to the management node；The data reading is set It include: the data taking equipment in the standby file that the n file data unit is written to the data taking equipment creation The quantity that the supplemental data is obtained from the management node, it is literary from described second group n according to the quantity of the supplemental data Part data cell end removes the supplemental data, obtains remaining file data, the remaining file data is written to In the file of the creation.

In a possible scheme, when m is 1, the corresponding redundant arithmetic of the redundant computation includes that even-odd check is calculated Method, alternatively, the corresponding redundant arithmetic of the redundant computation includes the Q checking algorithm based on Galois Field when m is 2.

Fourth aspect, the embodiment of the present application provide a kind of client device, including processor and memory, in which: institute Memory is stated, for storing program instruction；The processor refers to for calling and executing the program stored in the memory It enables, so that the client device executes the data managing method of the first aspect.

5th aspect, the embodiment of the present application provides a kind of computer readable storage medium, including instruction, when it is being calculated When being run on machine, so that the computer executes the data managing method of the first aspect.

6th aspect, the embodiment of the present application provide a kind of management equipment, including processor and memory, in which: described Memory, for storing program instruction；The processor, for calling and executing the program instruction stored in the memory, So that the management equipment executes the data managing method of the second aspect.

7th aspect, the embodiment of the present application provides a kind of computer readable storage medium, including instruction, when it is being calculated When being run on machine, so that the computer executes the data managing method of the second aspect.

Eighth aspect, the embodiment of the present application provide a kind of distributed file storage system, comprising: such as the fourth aspect The client device；Management equipment as described in terms of the described 6th.

Detailed description of the invention

Fig. 1 is the architecture diagram of distributed file storage system provided by the embodiments of the present application；

Fig. 2 is the method flow diagram provided by the embodiments of the present application that file is written in distributed file storage system；

Fig. 3 is the method flow diagram provided by the embodiments of the present application that file is read in distributed file storage system；

Fig. 4 is the method flow diagram provided by the embodiments of the present application for restoring data in distributed file storage system；

Fig. 5 is the method flow provided by the embodiments of the present application that file metadata is managed in distributed file storage system Figure；

Fig. 6 A is the another method flow diagram provided by the embodiments of the present application that file is written in distributed file storage system；

Fig. 6 B is provided by the embodiments of the present application in distributed file storage system write-in file data and redundant data Schematic diagram；

Fig. 7 A is the another method flow diagram in distributed file storage system reading file applying for embodiment and providing；

Fig. 7 B is the schematic diagram provided by the embodiments of the present application that file data is read in distributed file storage system；

Fig. 8 A is the another method flow provided by the embodiments of the present application for restoring data in distributed file storage system Figure；

Fig. 8 B is the schematic diagram provided by the embodiments of the present application for restoring data in distributed file storage system

Fig. 9 is HDFS system architecture schematic diagram provided by the embodiments of the present application；

Figure 10 is the method flow diagram that file is written to HDFS system for HDFS client provided by the embodiments of the present application；

Figure 11 is the method flow diagram that NameNode provided by the embodiments of the present application restores data in HDFS；

Figure 12 is the hardware structure diagram of distributed file storage system equipment provided by the embodiments of the present application；

Figure 13 is the structural schematic diagram of data writing device provided by the embodiments of the present application；

Figure 14 is the structural schematic diagram of data taking equipment provided by the embodiments of the present application；

Figure 15 is the structural schematic diagram of management node provided by the embodiments of the present application.

Specific embodiment

Below in conjunction with the attached drawing in the embodiment of the present application, technical solutions in the embodiments of the present application is described.

Embodiment one

Fig. 1 is the architecture diagram of distributed file storage system provided by the embodiments of the present application, is set including data write-in Standby 101, data taking equipment 102, management node 103, back end 104, back end 105, back end 106, function It is described as follows respectively:

Data writing device 101: for the client device of the distributed file storage system, first is that for will be to be written Enter the file data read out in file 107 to be written on the back end of the distribution of management node 103, comprising: from text to be written Part 107 reads or copies file data unit, is written in the blocks of files of the back end of the distribution, and to the file Data cell carries out redundant computation and obtains redundant data unit, is written in the redundant block in the back end of the distribution；Two It is the metadata information that file 107 to be written is updated for triggering management node 103.It is subsequent that specific wiring method is detailed in the application Embodiment.Data writing device can be Hadoop distributed file system (Hadoop Distributed File System, HDFS the data writing device (HDFS Client) in).

Data taking equipment 102: it for the client device of the distributed file storage system, is used for according to management node The metadata information of 103 files provided, file data is read out from multiple back end and is written to newly created file 108, To restore or restore data 107 to be written, comprising: if the back end for storage file block works normally, directly It connects and reads file data from the blocks of files, newly created file 108 is written；If for storage file block back end, There is node failure in back end for storing redundant block, then it can be according to the redundant arithmetic, to normal data section Blocks of files and redundant block in point are calculated, and corresponding blocks of files or the content (phase of redundant block on fault data node are obtained When in recovering the data on fault data node), then again from All Files block (for fault data node, corresponding document Block be calculated) in read file data, be written newly created file 108.Specific read method is detailed in the application subsequent implementation Example.Data taking equipment is also possible to the HDFS Client in HDFS.

Management node 103: for the management equipment of the distributed file storage system, for for file distribute blocks of files and In the storage location of multiple back end (back end 104-106 etc.), (storage location may include data section to redundant block The store path of point identification and blocks of files on corresponding data node), management file metadata information, the metadata information It may include file name, size, position, attribute, creation time, modification time, the corresponding relationship of file to blocks of files, file Corresponding relationship etc. of the block to storage location.One distributed memory system can there are two management node (active/standby relationship), Fig. 1 In only show a management node, in actual deployment can there are two or it is more.Management node can be in HDFS NameNode。

Back end 104: for storing above-mentioned blocks of files or redundant block.Back end can be in HDFS DataNode。

Back end 105: similar with back end 104.

Back end 106: similar with back end 104.

The write-in file based on system shown in Figure 1 framework is further described below by embodiment and reads the technology of file Scheme.

Embodiment two

Fig. 2 is the method flow diagram provided by the embodiments of the present application that file is written in distributed file storage system, In, data writing device, which is written into file, (can be the local file or long-range of the data writing device self-management Device file) it is written in multiple back end of the distributed file storage system, specifically comprise the following steps:

Step 201: data writing device reads data from file to be written.

Specifically, the data writing device reads file data from file to be written to form n file data list Member, the n file data unit include the file data of the reading.

The data writing device first determines the n before reading file data.The data writing device can root Determine the n according to business demand (such as file read-write speed), for example, when needing file reading speed very fast, determine n value compared with Greatly, conversely, determining that n value is smaller.The data writing device can also be from the management node of the distributed file storage system It is middle to obtain the n.

Optionally, when the data writing device determines the file data amount read from the file to be written less than n It, can be in the file data end supplemental data of the reading to form the n when the corresponding data volume of a file data unit A file data unit, the quantity of supplemental data can be sent to the management node by the data writing device, with after an action of the bowels It is used when the data of removal addition when continuous data taking equipment reads data from the n blocks of files.

Step 202: the data writing device carries out redundant computation to the file data of reading.

Specifically, the data writing device carries out redundant computation to the n file data unit, obtains m redundancy Data cell.

The data writing device first determines corresponding redundant arithmetic and the m before carrying out the redundant computation.Institute Stating data writing device oneself can determine or select redundant arithmetic, can also be from the management of the distributed file management system Node obtains the redundant arithmetic and the m.When the data writing device oneself determines or select redundant arithmetic and redundant block It when quantity, needs the redundant arithmetic and the m being sent to the management node, so that subsequent access, maintenance are written It is used when file data.

Step 203: the file data of reading and calculated redundant data are written to data by the data writing device Node.

Specifically, the n file data unit is respectively written into the multiple back end by the data writing device On n blocks of files in, and the m redundancy m redundant data unit being respectively written on the multiple back end In block.The data writing device obtains the n blocks of files and institute from the management node of the distributed file storage system M redundant block is stated in the storage location of the multiple back end.

Optionally, the n blocks of files is located on n back end in the multiple back end, and the m Redundant block is located on the other m data node in the multiple back end, so that the n blocks of files and institute The read-write of file data can concurrently be carried out by stating m redundant block, and then the reading performance of file data can be improved.

Specifically, when m is 1, the corresponding redundant arithmetic of the redundant computation may include parity arithmetic, alternatively, When m is 2, the redundant arithmetic may include the Q checking algorithm based on Galois Field.

Specifically, the byte number that the file data unit and the redundant data unit are included all can be 8 times Number.

Embodiment two allows data writing device that one file is written to the multiple of distributed file storage system In n blocks of files and m redundant block on back end, file space utilisation is improved, and file reading property can be improved Energy.

Embodiment three

Fig. 3 is the method flow diagram provided by the embodiments of the present application that file is read in distributed file storage system, In, the file that is read out for being stored on the distributed file storage system is written to newly created file by data taking equipment It is described to be read out the corresponding n text of file in (the i.e. described data taking equipment is in the newly created file of Local or Remote equipment) Part block is stored in multiple back end of the distributed file storage system, and the n is positive integer, specifically includes following step It is rapid:

Step 301: the storage location of data taking equipment acquisition blocks of files.

Specifically, the data taking equipment from the management node obtain described in be read out the corresponding n file of file Storage location of the block in the multiple back end.It is described here to be read out file to refer to that its file data has been written into described In n blocks of files of multiple back end of distributed file storage system.

Specifically, the data taking equipment can send storage location request, the storage position to the management node It sets in request comprising the mark for being read out file；The data taking equipment receives storage location from the management node and rings It answers, includes the storage location information for being read out the corresponding n blocks of files of file in the storage location response.

Step 302: the data taking equipment reads file data from blocks of files.

Specifically, whom the data read and read a file data unit respectively from the n blocks of files, obtain x A file data unit, x are positive integer, and x is less than or equal to n.

Optionally, it is read out the quantity of the supplemental data of file described in the data taking equipment acquisition, is chased after according to described The quantity of addend evidence removes supplemental data from the n file data unit end, obtains remaining file data.

Step 303: the file data of reading is written in newly created file by the data taking equipment.

Specifically, the x file data unit is written in the newly created file data taking equipment.

Specifically, the byte number that the file data unit is included can be 8 multiple.

Embodiment three allows data taking equipment to read out file data from distributed file storage system, write-in Into newly created file.

Example IV

Fig. 4 is the method flow diagram provided by the embodiments of the present application for restoring data in distributed file storage system, Described in distributed file storage system include management node and multiple back end, stored in the multiple back end to Restore the n blocks of files and m redundant block of file, includes that f normal file block and n-f are abnormal literary in the n blocks of files Part block, includes the following steps:

Step 401: the management node reads the redundant data in file data and redundant block in normal file block.

Specifically, the management node reads one from d normal file block in the f normal file block respectively File data unit obtains d file data unit, and the d is less than or equal to f, and reads respectively from the m redundant block A redundant data unit is taken, m redundant data unit is obtained；Described n, m, f, d are positive integer.

Specifically, the management node can obtain the n from the corresponding metadata information of the file to be restored The storage location of the storage location of blocks of files and the m blocks of files, to carry out the reading of the file data and redundant data It takes.

Step 402: the management node carries out redundant computation to the file data and redundant data of reading.

Specifically, the management node carries out redundancy to the d file data unit and the m redundant data unit It calculates, obtains n-f file data unit.

Optionally, the management node can obtain the redundancy from the corresponding metadata information of the file to be restored Calculate corresponding redundant arithmetic.

Step 403: the management node restores abnormal document block according to the data being calculated.

Specifically, the n-f file data unit is restored in the n blocks of files by the management node respectively In n-f abnormal document block.

Specifically, the management node can be created that new text in the storage location of the n-f abnormal document block respectively Part block obtains n-f new blocks of files, then the n-f file data unit is respectively written into the n-f new blocks of files.

Specifically, the byte number that the file data unit and the redundant data unit are included is 8 multiple.

Example IV makes the file data being stored in distributed file storage system, when there is back end failure, In the case where leading to file lost block or damage, the blocks of files that redundant block recovers loss or damage can use.

Embodiment five

Fig. 5 is the method flow provided by the embodiments of the present application that file metadata is managed in distributed file storage system Figure, wherein management node includes the following steps: for managing multiple back end and metadata information

Step 501: management node obtains storage location request from file write device.

Specifically, the management node receives storage location request, the storage location request packet from data writing device Containing file to be written the distributed file storage system unique identification.

It optionally, can also include redundancy scheme parameter, the redundancy scheme parameter includes in the storage location request File number of blocks n, redundancy number of blocks m and redundant arithmetic, so that data writing device oneself can determine that file write-in is relevant Parameter.

Step 502: the storage location of management node the distribution blocks of files and redundant block.

Specifically, the management node is that the file to be written distributes n blocks of files in the multiple back end In storage location in the multiple back end of storage location and m blocks of files.

Specifically, the management node can be distributed the storage location of the n file respectively in the multiple data On n back end in node, the storage location of the m redundant block is distributed respectively in the multiple back end In addition on m data node.For example, it is assumed that the multiple back end specifically has C back end, the management node is in institute It states and determines n back end in C back end, be respectively used to store the n blocks of files, in remaining C-n data M data node is determined in node, is respectively used to storage redundant block.

Optionally, the management node can distribute a mark for each blocks of files respectively, and generate n blocks of files It is identified to the corresponding relationship of respective memory locations, and distributes a mark respectively for each redundant block, and generate m redundant block It is identified to the corresponding relationship of respective memory locations.

Optionally, the management node can be the document creation metadata information, by the redundant arithmetic, the n A blocks of files is identified to the corresponding relationship of respective memory locations and the m redundant block is identified to the corresponding of respective memory locations and closes System.

Step 503: the management node returns to storage location response to file write device.

Specifically, the management node returns to storage location and the institute of the n blocks of files to the file write device State the storage location of m redundant block.

Specifically, the management node can return to metadata information, the metadata letter to the file write device The storage location of storage location and the m redundant block block in breath comprising the n file.

The management node can also receive the quantity of supplemental data from the data writing device, and by the additional number According to quantity the metadata information is written.

Embodiment five allow distributed file management system management node be file to be written distribute blocks of files and Redundant block, and maintenance documentation block and the information such as redundant block storage location information and redundant arithmetic.

Embodiment six

Fig. 6 A is the another method flow diagram provided by the embodiments of the present application that file is written in distributed file storage system, Include the following steps:

Step 601: data writing device determines the redundancy scheme parameter of file.

Specifically, the data writing device determines that file to be written (will be written to the distributed document storage The file of system is the file in the Local or Remote equipment of the data writing device management) after, according to business demand (example Requirement such as business to file read-write speed, in the case where multiple blocks of files can be with concurrent reading and concurrent writing, file number of blocks is more, reads Writing rate is faster), the factors such as preset condition (information, user preference setting etc. is locally configured), determine the file to be written The redundancy scheme parameter for needing to use when the distributed file storage system is written, the redundancy scheme parameter may include text The information of part number of blocks n, redundancy number of blocks m, file data element length w and redundant arithmetic.Wherein, w is optional parameters, because It can arrange it within the scope of entire distributed file storage system and unify default value, such as 8.

N, m, w and redundant arithmetic will be further described in subsequent step.

Step 602: the data writing device sends file write request (file identification, redundancy scheme to management node Parameter).

Specifically, the data writing device sends file write request to management node, can be in said write request Include the file identification and the redundancy scheme parameter.The file identification is the file to be written in the distributed text Unique identification in part storage system, can be logical path (such as :/home/foo) or filename (such as: foo).

Specifically, said write request can be the request of creation file, be also possible to modify the request of file.

Optionally, the data writing device can identify corresponding with redundancy scheme parameter according to preset redundancy scheme Relationship (as shown in table 1, is stored in the data writing device and the addressable position of the management node), to the management Node sends redundancy scheme mark, to achieve the purpose that transmit redundancy scheme parameter to the management node.

Table 1- redundancy scheme number

Redundancy scheme mark	Redundancy scheme parameter
		RdntSchm1	N=4, m=1, w=8
RdntSchm2	N=8, m=2, w=16
		RdntSchm3	N=4, m=1, w=8
RdntSchm4	N=8, m=2, w=16

It is alternatively possible to arrange within the scope of the distributed file storage system, when in the file write request not When identifying comprising redundancy scheme parameter or redundancy scheme, then management node can be using the redundancy scheme parameter of default.

Optionally, the distributed file storage system can set corresponding redundancy scheme parameter for logical directories step by step (as shown in table 2), so as to arrange, when in the file write request do not include redundancy scheme parameter or redundancy scheme mark When knowledge, using the redundancy scheme parameter of the file parent directory.For example, when the data writing device is sent out to the management equipment It send comprising file identification "/home/A/asdf.dat " in file write request, does not include redundancy scheme parameter or redundancy scheme mark Know, then redundancy scheme of the management node by the redundancy scheme of "/home/A " as "/home/A/asdf.dat " (RdntSchm2, n=8, m=2, w=16).

The corresponding redundancy scheme parameter of table 2- logical directories

Logical directories	Redundancy scheme mark	Redundancy scheme parameter
			/home	RdntSchm1	N=4, m=1, w=8
/home/A	RdntSchm2	N=8, m=2, w=16
			/home/B	RdntSchm3	N=4, m=1, w=8
/home/B/BB	RdntSchm4	N=8, m=2, w=16

It is alternatively possible to arrange within the scope of the distributed file storage system, when in the redundancy scheme parameter not When comprising file data element length w, default value (such as 8) are unified using it.

Step 603: the management node is that the file distributes back end.

Specifically, the management node distributes multiple data according to the redundancy scheme parameter for the file to be written Node, such as n+m back end can be distributed for the file to be written, wherein n back end is for storing n file Block (wherein each back end stores a blocks of files), m data node is for storing m redundant block (wherein each data Node stores a redundant block), in this way, only one blocks of files is impacted when a data node failures, and It can be obtained from other normal file blocks and redundant block by redundant computation.The embodiment of the present application is mainly to distribute n+m For back end, it can improve file with parallel work-flow when file data is written or reads file data in this way and read The speed write, but be also not excluded for for multiple blocks of files being assigned to the same back end or multiple redundant blocks be assigned to it is same The possibility of back end.

Optionally, the management node can quantity according to the back end of system configuration and each back end Memory space service condition etc., select back end for the file.For example, the management node can choose remaining storage The most back end in space is as the blocks of files of the file and the memory node of redundant block.

Optionally, the management node can further be created that the n file on the back end distributed Block and m redundant block.

Step 604: the management node returns to file write-in response (metadata information) to the data writing device.

Specifically, the management node is the document creation metadata information to be written, and by the metadata information Response message is written by file and is sent to the data writing device, the metadata information may include the text to be written The file identification of part, the redundancy scheme parameter, blocks of files information list (contain n file block message, each blocks of files packet The corresponding relationship of the storage location of blocks of files is identified to containing blocks of files), redundant block information list (containing m redundancy block message, each Redundant block information includes the corresponding relationship that redundant block is identified to the storage location of redundant block), it specifically also may refer to table 3.It needs Illustrate, the blocks of files information list and the redundant block information list are ordered into metadata information, read and write Journey requires to carry out file data referring to listed sequence or reads file data, to ensure to read the file of file data and write-in Data are consistent.

Optionally, the management node can also be created that the n text in this step or before this step Part block and m redundant block, to need to further include the instruction of " blocks of files/redundant block has created " in file write-in response Information, to indicate that the data writing device need not create corresponding document block/redundant block again.

In addition, for the ease of subsequent access (reading, rewriting etc.), the management node also needs to save the metadata letter Breath can specifically be stored in the management node local, can also be stored in the accessible remote equipment of the management node On.

Table 3- metadata information

Step 605: the data writing device does write-in and prepares: n blocks of files of creation and m redundant block.

Specifically, after the data writing device receives the metadata information, start that the preparation before file data is written, Including establishing session according to the back end specified in the metadata information, with the metadata information, and notify respective counts Corresponding blocks of files or redundant block are created according to node.Creation blocks of files described here refers to according to metadata information, creates file The corresponding physical file of block, the creation redundant block are also referred to according to metadata information, are created that the corresponding physics text of redundant block Part.And it is subsequent data are written into blocks of files/redundant block, also refer to and write number into the corresponding physical file of blocks of files/redundant block According to subsequent to repeat no more.

By taking sample in table 2 as an example, the data writing device will respectively with back end 3, back end 7, back end 10, back end 12, back end 15 establish session, and trigger data node 3 create blocks of files BlockD1 (Make Path for/ The physical file of root/blkd_001), back end 7 create blocks of files BlockD2 (Make Path as/root/blkd_002 Physical file), back end 10 create blocks of files BlockD3 (Make Path as/root/blkd_003), back end 12 Creation blocks of files BlockD4 (Making Path as the physical file of/root/blkd_004), back end 15 create redundant block BlcokR (Makes Path as the physical file of/root/blkr_001).

Optionally, the n blocks of files and m redundant block, can also be in above-mentioned steps 603 or step 604 process by institute Management equipment creation is stated, then the data writing device can not include " blocks of files/superfluous in determining the file write-in response Remaining block has created " instruction information, notification data node create blocks of files/redundant block.

At this point, the storage location on corresponding data node has just been assigned in the blocks of files and redundant block that are newly created that, File data or redundant data are not included also temporarily.The subsequent data writing device will execute below step by circulation 606 arrive step 608, realize the file to be written being written to the distributed file storage system.

Step 606: the data writing device reads out file data grouping from file.

Specifically, the data writing device reads out file data grouping Di (for example, figure from the file to be written D1, D2 etc. in 6B), it include n file data unit in Di, i.e. Ui1, Ui2 ... Uin (for example, U11 in Fig. 6 B, U12 ... U1n etc.), each file data unit may include the data of 8 integral multiple (8*y, y are positive integer) a byte, such as Shown in Fig. 6 B, any other quantum byte also may include.The embodiment of the present application does not make the length of each file data unit It limits, subsequent embodiment is illustrated by taking 8*y byte as an example.

It is performed a plurality of times after this step (the subscript i of data grouping Di is stepped up), when the end for reading the file, Its remaining byte number may be not enough to constitute the complete file data grouping, i.e., last remaining byte quantity is few In n*8*y (if the length of every unit is set as 8*y byte), needs in the addition of the file data end of reading or add Data are to form complete file data grouping (reaching n*8*y byte).Assuming that last remainder bytes number is RemainByteCount (RemainByteCount < n*8*y) then needs to add or add AppendByteCount=(n*8* Y-RemainByteCount) a byte is with polishing or complete file data grouping of gathering.

When having addition or supplemental data, the data writing device can send the additional number to the management node According to quantity, so that the management node updates the quantity of the supplemental data into metadata information, referring specifically to step 609.The quantity of supplemental data bitwise can or not apply for that embodiment is not construed as limiting as unit of byte.

Step 607: the data writing device calculates redundant data packet.

Specifically, the data writing device carries out redundant computation to Di and obtains redundancy according to the redundancy scheme parameter Data grouping Ri (for example, R1, R2 etc. in Fig. 6 B), Ri include m redundant data unit, i.e. Uri1, Uri2 ... Urim.

For example, when using " n+1 redundancy scheme " (i.e. m=1, only one redundant data unit, n > 1), it can be using surprise Even parity check algorithm calculates redundant data unit, i.e. Uri1=Ui1XOR Ui2XOR ... Uin from n file data unit, Middle XOR indicates that exclusive or calculates, thus, it, will if any one node damages in the corresponding n+1 back end of the redundancy scheme A cell value in this n+1 unit of Ui1, Ui2 ... Uin and Uri1 is caused to lack or damage, at this moment, Ke Yicong In addition the value of the missing unit is calculated in n unit according to the parity arithmetic, for example, it is assumed that first node damage It is bad, cause Ui1 cell value to lack, then can calculate Ui1=Ui2XOR Ui3XOR ... Uin XOR Uril.

When for another example, using " N+2 redundancy scheme " (i.e. m=2), parity arithmetic can be respectively adopted and based on Galois The Q checking algorithm in domain (galois field) generates first redundant data unit Uri1 and the second redundant data unit Uri2.

Step 608: file data grouping and redundant data packet are written to blocks of files and superfluous by the data writing device In remaining block.

Specifically, the data writing device is by n file data unit (Ui1, Ui2 ... in data grouping Di Uin it is (so-called " concurrently to write in newly created n blocks of files (BlockD1, BlockD2 ... BlockDn)) to be written in parallel in Enter " referring to need not wait a file data unit to write writes another file data unit again, but by n file data unit Simultaneously, it is respectively written into n blocks of files of n back end, is mutually not to wait for, this, which is equivalent to, mentions file writing speed It is n times high)；The m redundant data unit (Uri1, Uri2 ... Urim) in redundancy packets Ri is written in parallel in m simultaneously (" being written in parallel in " meaning and above-mentioned by n file data in redundant block (BlockR1, BlockR2 ... BlockRm) It is identical that n blocks of files is written in unit).

After ensuring that data are written, follow-up data readout equipment can be from the n blocks of files and m redundant block just File content really is read out, needs to arrange the corresponding relationship of file data unit and blocks of files, redundant data unit and redundant block Rule, for example can arrange: the file data unit correspondence (from low byte to high byte) is first read hereof and writes first number It is believed that in the blocks of files/back end first occurred in the blocks of files information list of breath.Assuming that the reading order of file data unit For Ui1, Ui2 ... Uin, and appearance sequence of the blocks of files in the blocks of files information list in the metadata information is BlockD1, BlockD2 ... BlockDn, then can be by the corresponding write-in BlockD2 ... of corresponding write-in BlockD1, the Ui2 of Ui1 The corresponding write-in BlockDn of Uin.It is subsequent when reading file data, can according to identical corresponding relationship rule carry out number of files According to reading.The corresponding relationship of redundant data unit and redundant block is similar, repeats no more.

It, can be by n file data unit in D1 for the D1 in Fig. 6 B and by the calculated R1 of D1 by taking Fig. 6 B as an example It is written in parallel on the BlockD2 ... DataNoden on BlockD1, DataNode2 on DataNode1 respectively In BlockDn；M redundant data unit in R1 can be written in parallel to respectively BlockR1 on DataNode (n+1), In the BlockRm on BlockR2 ... DataNode (n+m) on DataNode (n+2).

Again shown in the table 1 for sample, then the data writing device can will be read out from file file data Unit be written in parallel to respectively BlockD1 (positioned at back end 3 /root/blkd_001), BlockD2 (be located at data section Point 7 /root/blkd_002), BlockD3 (positioned at back end 10 /root/blkd_003), BlockD4 (be located at data Node 12 /root/blkd_004), by calculated redundant data unit be written BlockR (positioned at back end 15/ Root/blkr_001 in).

According to step 607, redundant data unit is to calculate and obtain from multiple file data units, when multiple data cells and Have in multiple redundant data units some data cell missing when (due to place node failure), then can according to redundant arithmetic by The data cell (recovering the data cell of missing) is calculated in alternative document data cell/redundant data unit.And by It is alignment in the data cell quantity that each blocks of files, redundant block are included, is all (x+1) a unit, then as shown in Figure 6B When have node damage cause some blocks of files lack when, can repeatedly ((x+1) is secondary) execute data cell recovery operation, to reach To the purpose for restoring entire blocks of files, it is ensured that the fault-tolerant ability of storage system.

On this basis, this more makes the utilization rate of memory space get a promotion, and utilization rate formula is n/ (n+m), when m is 1, n is 4 is that space utilization rate reaches 80%, much larger than the storage space utilization (< 33%) of existing distributed memory system.

It optionally, is the efficiency for improving write-in file data, the data writing device can use caching mechanism toward text Data are written in part block or redundant block.For example, multiple file data units can be written to buffering by the data writing device File data units multiple in buffer area are written in corresponding blocks of files by Qu Zhong when waiting buffer areas full.For example, in Fig. 6 B In, file data unit U11, U12 ... U1n will be finally written in the BlockD1 on DataNode1, then the data Write device can divide n times, every time to BlockD1 1 file data unit of write-in, can also be first by U11, U12 ... U1n Buffer area (assuming that buffer area is sufficiently large) is put into, again by the session established with DataNode1, disposably by this n data sheet In member write-in BlockD1.The writing mode of redundant data unit is similar, repeats no more.

Step 609: the data writing device notifies the management node update metadata information that (file identification adds The quantity of data).

This step is optional step, depending on whether having additional number during circulation executes step 606~step 608 According to.

Assuming that there is supplemental data, then specifically, said write equipment can send metadata updates to the management node It requests, includes the quantity of the file identification and the supplemental data in the request.The management node is by the additional number According to quantity be written in the corresponding metadata information of the file identification.

Above-described embodiment six allows data writing device to read out file data from file to be written, calculate redundancy File data and redundant data write-in are located at multiple on multiple back end of distributed file storage system by data respectively In blocks of files and multiple redundant blocks, on the basis of ensuring fault-tolerance, the space of the distributed file storage system is improved The writing speed of utilization rate and file.

Embodiment seven

Fig. 7 A is the another method flow diagram in distributed file storage system reading file applying for embodiment and providing, packet Include following steps:

Step 701: data taking equipment sends file to management node and reads request (file identification).

Specifically, the data taking equipment sends file reading to the management node and requests, in the reading request Comprising file identification, the file identification is to be read out file in the unique identification of the distributed file storage system.

Step 702: the management node returns to file read response (metadata information) to the data taking equipment.

Specifically, the management node is according to the file identification, (is located locally or remotely from metadata memory space Equipment) in inquire corresponding metadata information, the data taking equipment is sent to by file read response.The member number It is believed that ceasing referring to the description in step 604.It at this time may include the quantity of supplemental data in metadata information, it will be in subsequent step It is used in rapid.

Step 703: the data taking equipment does reading and prepares: establishing session.

Specifically, the data taking equipment is according to the blocks of files information list in the metadata information, with each text Back end where part block establishes session.

Step 704: the data taking equipment is created that file.

Specifically, the data taking equipment is in its file storage (be located locally or remote equipment) the new text of creation Part obtains newly created file.At this point, the newly created file does not include file data also, the subsequent data taking equipment will 705~step 706 of below step is executed by circulation, reads out file from each blocks of files in the blocks of files information list Data cell, then be written in the newly created file.

Step 705: the data taking equipment reads n file data unit.

Specifically, the data taking equipment is read according to the blocks of files information list, from n back end N blocks of files in, concurrently respectively read a file data unit, obtain n file data unit.

By taking Fig. 7 B as an example, the data taking equipment will read file from the BlockD1 being located on DataNode1 respectively Data cell Ui1, file data unit Ui2 ... is read from the BlockD2 being located on DataNode2 from being located at File data unit Uin is read on BlockDn on DataNoden.

Optionally, to improve the efficiency for reading file data, the data taking equipment can use caching mechanism from text Data are read in part block.For example, in figure 7b, file data unit U11, U21 ... U (x+1) 1 will finally be read and is written Into the newly created file, then the data taking equipment can divide that (x+1) is secondary, from BlockD1 reads 1 text every time Part data cell disposably can also read local buffer (assuming that buffering from BlockD1 by U11, U21 ... U (x+1) 1 Area is sufficiently large), then U11, U21 ... U (x+1) 1 is read from buffer area, it is written into the newly created file.

Step 706: the file is written in n file data unit by the data taking equipment.

Specifically, the n file data unit is written in the newly created file by the data taking equipment. Used file data unit and blocks of files/node when the sequence of write-in wants reference data write device that file data is written Corresponding relationship rule.For example, it is assumed that the corresponding relationship rule that data writing device uses are as follows: the number of files first read hereof It corresponds to according to unit and first occurs in blocks of files/back end in the blocks of files information list for writing metadata information, then the data The file data unit that readout equipment should will be read from the blocks of files first occurred in the blocks of files information list of metadata information It is first written in the newly created file.By taking Fig. 7 B as an example, it is assumed that each blocks of files of blocks of files information list of metadata information goes out Existing sequence are as follows: the BlockDn on the BlockD2 ... DataNoden on BlockD1, DataNode2 on DataNode1, Then according to the corresponding relationship rule, the Ui1 read out in BlockD1 can be first written to the newly created file, secondly The Ui2 read in BlockD2 ... the Uin read in last BlockDn.

After the All Files data cell of All Files block has all read out and has been written to the newly created file, institute Stating newly created file is the file read out from the distributed file storage system.

Step 707: the additional data of the data taking equipment removal.

The non-optional step of this step.The data taking equipment can be according to the metadata information read in step 702 The quantity of supplemental data determines whether to remove additional data from end of file.If the supplemental data of metadata information Quantity is greater than zero, then can remove the data of respective numbers from end of file.For example, the quantity of supplemental data is 15 bytes, then Finally to remove 15 bytes, file content when obtaining being originally written from file.

Alternatively it is also possible in step 706, according to the quantity of the supplemental data in metadata information, identify in advance Supplemental data ignores the supplemental data, and the newly created file only is written in file data, thus without going through this step Supplemental data is removed at the newly created file end.

Above-described embodiment seven allows data taking equipment to read file from distributed file storage system, due to reading Access evidence can carry out parallel, the available very big promotion of the reading speed of file.

Embodiment eight

Fig. 8 A is the another method flow provided by the embodiments of the present application for restoring data in distributed file storage system Figure, includes the following steps:

Step 801: management node determines that fault data node has been restarted.

Specifically, management node determines one or more back end in distributed file storage system from failure Resume operation (such as heartbeat message by detecting faulty equipment), but the blocks of files on this or these back end need it is extensive Multiple (due to having damaged or having lost).

Step 802: the management node obtains the metadata information of file.

Specifically, the management node determines file involved in fault data node, obtains its metadata information, Middle redundancy scheme parameter includes n, m, w and redundant arithmetic, and n is file block quantity (and the storage blocks of files of file The quantity of back end), m is the redundancy number of blocks (and quantity of the back end of the storage redundant block) of file, and w is number According to the length (if the value lacks in metadata, default-length can be used) of unit.Assuming that d is to be out of order and restored to transport Row, but need to restore the quantity of the back end of data block, d < m, then it is normal for having n+m-d back end.

Step 803: the management node carries out recovery preparation: establishing session with normal data node.

Specifically, the management node carries out recovery preparation, including determines the n+m-d normal back end, And session is established with these back end.

Step 804: the management node carries out recovery preparation: establishing session with the back end restarted after failure, triggers Create blocks of files.

Specifically, the management node carries out recovery preparation, including the back end restarted after determining d failure, and with These back end establish session, on this basis, trigger the back end restarted after each failure be created that and damage or The corresponding blocks of files of the blocks of files of loss.At this point, the newly created blocks of files has allocated only memory space, there are no effective texts Number of packages evidence.Circulation is executed step 805~step 807 by the management node, to recover the newly created blocks of files.

Step 805: the management node reads file data unit/redundant data unit on normal data node.

Specifically, the management node reads out n+m-d file data from the n+m-d normal data node Unit/redundant data unit.

By taking Fig. 8 B as an example, it is assumed that d=1, back end DataNode2 be failure restarted back end, then respectively from Total n+m-1 in DataNode1, DataNode3 ... DataNoden and DataNode (n+1) ... DataNode (n+m) Read n+m-1 file data unit/redundant data unit in normal data node, the Uin that is respectively as follows: Ui1, Ui3 ... with Uri1……Urim。

Step 806: the management node calculates d data cell according to redundant arithmetic.

Specifically, the redundant arithmetic for including in metadata information of the management node according to the file is based on the n + m-d file data unit/redundant data units, are calculated d data cell.

Continue the example in step 805, it is assumed that m=1, redundant arithmetic are parity arithmetic, then can calculate Ui2= Ui1XOR Ui3……XOR Uin.Assuming that m=2, d=2, then can be respectively adopted parity arithmetic and based on Galois Field Q checking algorithm generate first data cell and second data cell.

Step 807: d back end is written in d data cell by the management node.

Specifically, d blocks of files, each data cell and file is written in the d data cell by the management node It is consistent when the corresponding relationship of block is with write-in file.

Above-described embodiment eight allows management node in file distribution document storage system according to the metadata of file Information recovers the damage blocks of files of file from the normal file block and redundant block of file, it is ensured that distributed document storage The fault-tolerance of system.

Below by taking HDFS framework as an example, scheme described in the above embodiments of the present application is further described.

Embodiment nine

Fig. 9 be HDFS system architecture schematic diagram provided by the embodiments of the present application, including HDFS Client 101, NameNode1 902、NameNode2 903、DataNode1 904、DataNode2 905、DataNode3 906、 DataNode4907, DataNode5 908, function is described as follows respectively:

HDFS Client 901: for the data writing device of HDFS, include data writing device 101 and data shown in Fig. 1 The function of readout equipment 102, it can in the case where NameNode1 902 or NameNode2 903 is assisted, file content is written to In DataNode1-DataNode5, file content can also be read out from DataNode1-DataNode5.

NameNode1 902: for the main management node of HDFS, the function comprising management node 103 shown in Fig. 1, for managing Manage the metadata information of file.

NameNode2 903: for the spare management node of HDFS.When NameNode1 failure, NameNode2 takes over it Work, improves the availability of system.

DataNode1 904: for the back end of HDFS, the function comprising back end 104 described in Fig. 1, for storing Blocks of files or redundant block.

DataNode2 905: similar with DataNode1 904.

DataNode3 906: similar with DataNode1 904.

DataNode4 907: similar with DataNode1 904.

DataNode5 908: similar with DataNode1 904.

5 DataNode are only shown shown in Fig. 9, it can actually be there are also more DataNode.

The technical solution of the write-in file based on system shown in Figure 9 framework is further described below by embodiment.

Embodiment ten

Figure 10 is the method flow diagram that file is written to HDFS system by HDFS Client provided by the embodiments of the present application, In, " foo " is that HDFS Client is locally written to the file in HDFS, and the mark or path in HDFS system will be "/home/foo ", HDFS Client are determined using 4+1 redundancy scheme (needing 4 blocks of files, 1 redundant block), HDFS Client be written into file f oo be written to HDFS become/process of home/foo specifically may include following steps:

Step 1001:HDFS Client sends document creation request to NameNode1, carries files-designated in the request Knowledge/home/foo and redundancy scheme parameter: 4+1 redundancy.

Specifically, HDFS Client sends document creation request to NameNode1, carry in the request file identification/ Home/foo and redundancy scheme parameter: 4 blocks of files and 1 redundant block.Due to there is no carrying redundant arithmetic, indicate using silent Recognize algorithm, such as parity arithmetic；And due to do not have instruction read file when file data unit length, then using default Value, such as 8 bytes.

About the length of file data unit, optionally, HDFS system may be arranged as other less than setting in HDFS The size (such as 64k) of the buffer area of the file read-write of standby (HDFS Client, DataNode etc.), in this way, from each data point Each data cell that group is read out can be written in parallel in each blocks of files of each back end, to improve The performance of file read-write.

Step 1002:NameNode1 selects back end.

Specifically, NameNode1 in multiple back end, selects DataNode1- in HDFS for file to be written DataNode5, wherein DataNode1-DataNode4 is used for storage file block, and DataNode5 is for storing redundant block.

Step 1003:NameNode1 returns to document creation response, carrying/home/ in the response to HDFS Client The metadata information of foo.

Specifically, NameNode1 is created that metadata information in its memory space, as described in Table 4

Table 4- metadata information sample

Optionally, NameNode1 can be that BlockD1- is respectively created out on DataNode1-DataNode5 BlockD4 and BlockR, thus, NameNode1 can include further " blocks of files/redundancy in document creation response Block has created " instruction information.

Step 1004:HDFS Client and DataNode1 establishes session, creates blocks of files BlockD1.

Specifically, HDFS Client believes after receiving the document creation response according to metadata wherein included Breath, establishes session with DataNode1, according to metadata information described in table 4, and notifies DataNode1 in its local "/root " mesh Lower storage location of creation physical file " blkd_001 " as the blocks of files for being identified as BlockD1 of record.

Optionally, HDFS Client can be in determining the document creation response not comprising " blocks of files/redundant block is When the instruction information of creation ", the corresponding physical file of notice DataNode1 creation BlockD1.

Step 1005:HDFS Client and DataNode2 establishes session, creates blocks of files BlockD2.

It is similar with step 1004.

Step 1006:HDFS Client and DataNode3 establishes session, creates blocks of files BlockD3.

It is similar with step 1004.

Step 1007:HDFS Client and DataNode4 establishes session, creates blocks of files BlockD4.

It is similar with step 1004.

Step 1008:HDFS Client and DataNode5 establishes session, creates redundant block BlockR.

Specifically, HDFS Client and DataNode5 establish session, according to metadata information described in table 4, notice DataNode5 creates file " blkr_001 " depositing as the redundant block for being identified as BlockR under its local "/root " catalogue Store up space.

After HDFS Client confirms that 4 blocks of files and 1 redundant block have created, circulation is executed into below step 1009- step 1014, until the reading data of local foo file finishes, to realize foo write-in HDFS (above-mentioned 4 files Block, 1 redundant block).

Step 1009:HDFS Client reads next grouping of local file foo, calculates redundant data unit.Such as Fruit file data is insufficient, can chase after and cut byte with polishing into grouping.

Specifically, HDFS Client is successively read down from file header (or low address) to end-of-file (or high address) direction One grouping (being assumed to be i-th of grouping), calculates redundant data packet (only 1 redundant data according to parity arithmetic Unit, subsequent to be directly known as calculating redundant data unit).According to the condition of step 1001 and the metadata information of table 4, grouping Length are as follows: 4*8=32 byte, wherein include the file data of 4 units, respectively Ui1, Ui2, Ui3, Ui4, then redundant digit According to unit Uri=Ui1XOR Ui2XOR Ui3XOR Ui4.According to the metadata information of table 4, the storage of this 4+1 storage unit Position is as shown in table 5.

Table 5- storage location corresponding relationship

Data cell	Ui1	Ui2	Ui3	Ui4	Uri
						Storage location	BlockD1	BlockD2	BlockD3	BlockD4	BlockR

If reading the end local file foo, remainder bytes are less than 32 bytes, then HDFS Client should supplemental data With polishing at a grouping (32 byte), and the quantity of record addition data, so that subsequent notification NameNode1 is by additional word Joint number is updated into/corresponding the metadata information of home/foo, referring specifically to step 1015.

File data unit Ui1 is written to BlockD1 in step 1010:HDFS Client.

After write-in, DataNode1 local file "/root/blkd_001 " will increase by 8 byte contents, i.e. Ui1 is included Content.

File data unit Ui2 is written to BlockD2 in step 1011:HDFS Client.

It is similar with step 1010.

File data unit Ui3 is written to BlockD3 in step 1012:HDFS Client.

It is similar with step 1010.

File data unit Ui4 is written to BlockD4 in step 1013:HDFS Client.

It is similar with step 1010.

Redundant data unit Uri is written to BlockDR in step 1014:HDFS Client.

It is similar with step 1010.

It should be noted that above-mentioned steps 1010- step 1015HDFS Client written document block/redundant block process can To execute parallel, this will greatly improve the writing speed of file f oo (for original 4 times).

Step 1015:HDFS Client is sent to NameNode updates metadata request, and the request includes file identification With the quantity of supplemental data.

Specifically, HDFS Client to NameNode1 send update metadata request, it is described request comprising file identification/ The quantity (being assumed to be 15) of home/foo and supplemental data, then the quantity of supplemental data is written to shown in table 4 by NameNode1 In metadata information, when so as to other subsequent HDFS Client reading/home/foo, additional byte can be removed accordingly, also Original goes out file.

Optionally, above-mentioned steps description in, HDFS Client in each cycle, Ui1, Ui2, Ui3, Ui4, Uri It is respectively written into the corresponding session of BlockD1, BlockD2, BlockD3, BlockD4 and BlockR.Since session has been beaten It opens, therefore data is directly written into the corresponding network Socket connection of session.In order to further enhance write performance, often Buffer area can be used in one session, and after can accumulating several bytes in this way, property is written again.After session is written in data, meeting It is transmitted to corresponding DataNode host, corresponding service can in time be eventually written data corresponding above Data Node host In the storage of host, before this write-in, buffer area can be used also with improving performance.

Above-described embodiment ten allows HDFS Client to improve the utilization rate of memory space to 4/ (4 for single file + 1) speed=80%, being written improves 4 times (4 blocks of files can be written in parallel to).Moreover, because different files can To use different redundancy scheme parameters, so that the file for reaching different service features can be using different storage strategies Effect.

Embodiment 11

Figure 11 is the method flow diagram that NameNode provided by the embodiments of the present application restores data in HDFS, wherein DataNode4 is the back end restarted after failure, wherein All Files block, including "/home/foo " described in embodiment ten Blocks of files BlockD4 on this node has lost, then the process that NameNode1 recovers BlockD4 specifically may include Following steps:

Step 1101:NameNode1 determines that malfunctioning node has been restarted.

Specifically, NameNode1 determines that DataNode4 is restarted from failure by approach such as heartbeat detections, but its In All Files block need to restore, the blocks of files BlockD4 including "/home/foo ".

Step 1102:NameNode1 obtains file/home/foo metadata information.

NameNode1 obtains the metadata information of file "/home/foo ", as shown in table 4.

Step 1103:NameNode1 and DataNode1 establish session.

Step 1104:NameNode1 and DataNode2 establish session.

Step 1105:NameNode1 and DataNode3 establish session.

Step 1106:NameNode1 and DataNode4 establish session, newly created file of laying equal stress on block BlockD4.

Detailed process is similar with step 1007.

Step 1107:NameNode1 and DataNode5 establish session.

After all interdependent node sessions of HDFS Client confirmation are normal, BlockD4 has been created, circulation is executed Below step 1108- step 1113 recovers BlockD4 to realize until the reading data such as BlockD1 are finished.

Step 1108:NameNode1 reads next file data unit from BlockD1.

Specifically, according to the metadata information of "/home/foo ", a data unit length is 8, then NameNode1 from BlockD1 reads out the file data unit of next 8 byte, is Ui1.Reading order can from the file header of BlockD1 (or Low address) it is read to end-of-file (or high address) direction sequence.

Step 1109:NameNode1 reads next file data unit from BlockD2.

Similarly, NameNode1 reads out the file data unit of 8 bytes from BlockD2, is Ui2.

Step 1110:NameNode1 reads next file data unit from BlockD3.

Similarly, NameNode1 reads out the file data unit of 8 bytes from BlockD2, is Ui3.

Step 1111:NameNode1 reads next redundant data unit from BlockR.

Similarly, NameNode1 reads out the redundant data unit of 8 bytes from BlockR, is Uri.

Step 1112:NameNode1 calculates a data cell.

Specifically, NameNode1 calculates Ui1, Ui2, Ui3 and Uri according to redundant arithmetic " parity arithmetic " Obtain a data cell Ui4=Ui1XOR Ui2XOR Ui3XOR Uri.

Calculated file data unit is written to BlockD4 in step 1113:NameNode1.

Specifically, BlockD4 write-in institute calculated file data unit Ui4 of the NameNode1 on DataNode4, BlockD4 will increase the byte of a data unit length (length of Uri) after write-in.

Embodiment 12

Figure 12 is the hardware structure diagram of distributed file storage system equipment provided by the embodiments of the present application.The application is all Data writing device, data taking equipment, management node etc. in embodiment, wherein the reading of the data writing device, data is set Standby 901 (the HDFS Client) that can be in Fig. 9, the management node can be 902 in Fig. 9, can use Figure 12 institute The general computer hardware shown comprising processor 1201, memory 1202, bus 1203, input equipment 1204, output are set Standby 1205 and network interface 1206.

Specifically, memory 1202 may include the computer storage in the form of volatibility and or nonvolatile memory Media, such as read-only memory and/or random access memory.Memory 1202 can store an operating system, application program, other Program module, executable code and program data.

Input equipment 1204 can be used for inputting information, and the equipment is operated and managed convenient for system manager Deng, such as default redundant arithmetic is set, input equipment 1204 such as keyboard or sensing equipment, such as mouse, trace ball, touch tablet, wheat Gram wind, control stick, game mat, satellite tv antenna, scanner or similar devices.These input equipments can pass through bus 1203 It is connected to processor 1201.

Output equipment 1205 can be used for output information, and the equipment is operated and managed convenient for system manager Deng, in addition to the monitor, output equipment 1205 can also be set for other periphery outputs it is each, such as loudspeaker and/or printing device, These output equipments can also be connected to processor 1201 by bus 1203.

The equipment (data writing device, data taking equipment, management node) can be connected by network interface 1206 Into network, such as it is connected to local area network (Local Area Network, LAN).Under networked environment, stored in the equipment Computer executed instructions can store in remote storage device, and be not limited to be locally stored.

When the processor 1201 in the equipment executes the executable code stored in memory 1202 or application program, When the equipment is data writing device, method step corresponding with data writing device in all of above embodiment can be executed Suddenly；When the equipment is data taking equipment, side corresponding with data taking equipment in all of above embodiment can be executed Method step；When the equipment is management node, method step corresponding with management node in all of above embodiment can be executed Suddenly；Specific implementation procedure is referring to above-described embodiment, and details are not described herein.

Embodiment 13

Figure 13 is the structural schematic diagram of data writing device provided by the embodiments of the present application, the data writing device packet It includes:

Data read module 1301, for reading file data from file to be written, specific implementation procedure is referring to above-mentioned reality Apply explanation the step of data writing device side into embodiment 11 of example one, such as step 201,606；

Redundant computation module 1302, the file data for reading to data read module 1301 carry out redundant computation and obtain To redundant data, specific implementation procedure is said the step of data writing device side referring to above-described embodiment one into embodiment 11 It is bright, such as step 202,606；

Data write. module 1303, file data and redundant computation module for reading data read module 1301 1302 calculated redundant datas are written in the back end of distributed file storage system, and specific implementation procedure is referring to above-mentioned Embodiment one illustrates the step of data writing device side into embodiment 11, such as step 203,608.

In the present embodiment, data writing device is presented in the form of functional module.Here " module " can refer to Application-specific integrated circuit (application-specific integrated circuit, ASIC), circuit execute one Or the processor and memory of multiple softwares or firmware program, integrated logic circuit and/or other above-mentioned function can be provided Device.In a simple embodiment, those skilled in the art is contemplated that data writing device can also use Figure 12 Shown in form, data read module 1301, redundant computation module 1302 and Data write. module 1303 can pass through Figure 12 In processor 1201 and memory 1202 realize.For example, redundant computation module 1302 carries out redundant computation to file data Function can be realized by executing the code stored in memory 1202 by processor 1201.

Embodiment 14

Figure 14 is the structural schematic diagram of data taking equipment provided by the embodiments of the present application, the data taking equipment packet It includes:

Position acquisition module 1401, for obtaining the storage location for being read out the blocks of files of file, specific implementation procedure ginseng See explanation the step of data taking equipment side into embodiment 11 of above-described embodiment one, such as step 701,702；

Data read module 1402, it is specific to execute for reading file data from described be read out in the blocks of files of file Process illustrates the step of data taking equipment side referring to above-described embodiment one into embodiment 11, such as step 705；.

Data write. module 1403, for the file data of the reading to be written to the text of data taking equipment creation In part, specific implementation procedure illustrates the step of data taking equipment side referring to above-described embodiment one into embodiment 11, such as walks Rapid 706 etc.；

In the present embodiment, data taking equipment is presented in the form of functional module.Here " module " can refer to Application-specific integrated circuit (application-specific integrated circuit, ASIC), circuit execute one Or the processor and memory of multiple softwares or firmware program, integrated logic circuit and/or other above-mentioned function can be provided Device.In a simple embodiment, those skilled in the art is contemplated that client can also be using shown in Figure 12 Form.Position acquisition module 1401, data read module 1402 and Data write. module 1403 can pass through the place in Figure 12 Device 1201 and memory 1202 are managed to realize.For example, the file data read out is written to newly by Data write. module 1403 The function of the file of creation can be realized by executing the code stored in memory 1202 by processor 1201.

Embodiment 15

Figure 15 is the structural schematic diagram of management node provided by the embodiments of the present application, and the management node includes:

Data read module 1501, for reading file data and redundant digit respectively from normal file block and redundant block According to specific implementation procedure illustrates the step of management node side referring to above-described embodiment one into embodiment 11, such as step 805 Deng；

Redundant computation module 1502, file data and redundant data for being read out to data read module 1501 Redundant computation is carried out, specific implementation procedure illustrates the step of management node side referring to above-described embodiment one into embodiment 11, Such as step 806；

Data recovery module 1503, the redundant data for being calculated according to redundant computation module 1502 carry out data Restore, specific implementation procedure illustrates the step of management node side referring to above-described embodiment one into embodiment 11, such as step 807 Deng.

Optionally, the management node further includes metadata management module 1504, for managing the metadata information of file (storage location of such as redundant block, redundant arithmetic) etc., specific implementation procedure is referring to above-described embodiment one into embodiment 11 The step of management node side, illustrates, such as step 701, step 702, step 801 and step 802；

In the present embodiment, management node is presented in the form of functional module.Here " module " can refer to specific Application integrated circuit (application-specific integrated circuit, ASIC), circuit execute one or more The processor and memory of a software or firmware program, integrated logic circuit and/or other device of above-mentioned function can be provided Part.In a simple embodiment, those skilled in the art is contemplated that client can also be using shape shown in Figure 12 Formula, data read module 1501, redundant computation module 1502 and data recovery module 1503 can pass through the processing in Figure 12 Device 1201 and memory 1202 are realized.For example, data recovery module 1502 is calculated superfluous according to redundant computation module 1502 Remainder can be by being executed the code stored in memory 1202 by processor 1201 come real according to the function of carrying out data recovery It is existing.

Those of ordinary skill in the art may be aware that list described in conjunction with the examples disclosed in the embodiments of the present disclosure Member and algorithm steps, can be realized with electronic hardware, computer software, or a combination of the two, in order to clearly demonstrate hardware With the interchangeability of software, each exemplary composition and step are generally described according to function in the above description.This A little functions are implemented in hardware or software actually, the specific application and design constraint depending on technical solution.Specially Industry technical staff can use different methods to achieve the described function each specific application, but this realization is not It is considered as beyond scope of the present application.

Those of ordinary skill in the art can be understood that for convenience of description and succinctly, foregoing description is The specific work process of system, device and unit, can refer to corresponding processes in the foregoing method embodiment, details are not described herein.

In several embodiments provided herein, it should be understood that disclosed systems, devices and methods, it can be with It realizes by another way.For example, the division of the unit, only a kind of logical function partition in actual implementation can be with There is other division mode, such as multiple units or components can be combined or can be integrated into another system or some spies Sign can be ignored, or not execute.In addition, shown or discussed mutual coupling, direct-coupling or communication connection can be with Be through some interfaces, the indirect coupling or communication connection of device or unit, be also possible to electricity, mechanical or other forms Connection.

The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple In network unit.Some or all of unit therein can be selected to realize the embodiment of the present application scheme according to the actual needs Purpose.

It, can also be in addition, each functional unit in each embodiment of the application can integrate in one processing unit It is that each unit physically exists alone, is also possible to two or more units and is integrated in one unit.It is above-mentioned integrated Unit both can take the form of hardware realization, can also realize in the form of software functional units.

If the integrated unit is realized in the form of SFU software functional unit and sells or use as independent product When, it can store in a computer readable storage medium.Based on this understanding, the technical solution of the application is substantially The all or part of the part that contributes to existing technology or the technical solution can be in the form of software products in other words It embodies, which is stored in a storage medium, including some instructions are used so that a computer Equipment (can be personal computer, server or the network equipment etc.) executes the complete of each embodiment the method for the application Portion or part steps.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory, random access memory, magnetic disk Or the various media that can store program code such as CD.

The above, the only specific embodiment of the application, but the protection scope of the application is not limited thereto, it is any Those familiar with the art within the technical scope of the present application, can readily occur in various equivalent modifications or replace It changes, these modifications or substitutions should all cover within the scope of protection of this application.Therefore, the protection scope of the application should be with right It is required that protection scope subject to.

Claims

1. a kind of data managing method applied to distributed file storage system, the distributed file storage system includes more A back end characterized by comprising

File data is read from file to be written to form n file data unit, the n file data unit includes institute State the file data of reading；

Redundant computation is carried out to the n file data unit, obtains m redundant data unit；

The n file data unit is respectively written into n blocks of files on the multiple back end, and by the m A redundant data unit is respectively written into m redundant block on the multiple back end；

Wherein described n, m are positive integer.

2. the method according to claim 1, wherein the storage location of the n blocks of files be located at it is described It is described that the n file data unit is respectively written into the multiple data section on n back end in multiple back end Include: in n blocks of files on point

Concurrently the n file data unit is respectively written into the n blocks of files.

3. the method according to claim 1, wherein the storage location of the n blocks of files be located at it is described On n back end in multiple back end, the storage location of the m redundant block is located at the multiple back end In in addition in m data node, it is described the n file data unit to be respectively written into the n blocks of files and by institute It states m redundant data unit and is respectively written into the m redundant block and include:

The n file data unit is concurrently respectively written into the n blocks of files, concurrently by the m redundant data Unit is respectively written into the m redundant block.

4. the method according to claim 1, wherein it is described from the file to be written read file data with Forming n file data unit includes:

Determine the file data amount read from the file to be written less than the corresponding data volume of n file data unit；

In the file data end supplemental data of the reading to form the n file data unit.

5. according to the method described in claim 4, it is characterized in that, the distributed file storage system further includes described in management The management node of multiple back end, the method also includes:

The quantity of the supplemental data is sent to the management node.

6. the method according to claim 1, wherein the distributed file storage system further includes described in management The management node of multiple back end, it is described the n file data unit is respectively written into the n blocks of files before, institute State method further include:

The storage position of the n blocks of files, the m blocks of files in the multiple back end is obtained from the management node Confidence breath.

7. according to the method described in claim 6, it is characterized in that, it is described from the management node obtain the n blocks of files, Storage location information of the m blocks of files in the multiple back end includes:

Storage location request is sent to the management node, the storage location request is comprising the file to be written at described point Unique identification in cloth document storage system；

Storage location response is received from the management node, includes the n blocks of files described in the storage location response The storage location of storage location and the m redundant block in the multiple back end in multiple back end.

8. the method according to the description of claim 7 is characterized in that the storage location request also comprising redundancy scheme mark or N and m described in person.

9. according to the method described in claim 7, comprising the n blocks of files in the multiple number in storage location response According to the storage location of storage location and the m redundant block in the multiple back end in node, it is characterised in that:

It include the metadata information of the file to be written in the storage location response, the metadata information includes n to text The corresponding relationship and m of storage location of the part block identification to blocks of files in the multiple back end are identified to redundancy to redundant block The corresponding relationship of storage location of the block in the multiple back end.

10. the method according to claim 1, wherein the distributed file storage system further includes management institute State the management node of multiple back end, it is described redundant computation is carried out to the n file data unit before, the method is also Include:

The corresponding redundant arithmetic of the redundant computation is obtained from the management node.

11. the method according to claim 1, wherein the distributed file storage system further includes management institute The management node for stating multiple back end, the method also includes:

The corresponding redundant arithmetic of the redundant computation is sent to the management node.

12. method described in -11 any one according to claim 1, which is characterized in that when m is 1, the redundant arithmetic packet Parity arithmetic is included, alternatively, the redundant arithmetic includes the Q checking algorithm based on Galois Field when m is 2.

13. method described in -11 any one according to claim 1, which is characterized in that the file data unit and described superfluous The byte number that remaining data cell is included is 8 multiple.

14. a kind of data managing method applied to distributed file storage system, the distributed file storage system include Multiple back end, which is characterized in that n blocks of files and the m that file to be restored is stored in the multiple back end are a superfluous Remaining block includes f normal file block and n-f abnormal document block in the n blocks of files, which comprises

Read a file data unit respectively from d normal file block, obtain d file data unit, the d be less than or It equal to f, and reads a redundant data unit respectively from the m redundant block, obtains m redundant data unit；

Redundant computation is carried out to the d file data unit and the m redundant data unit, obtains n-f file data Unit；

The n-f file data unit is restored to respectively in n-f abnormal document block in the n blocks of files；

Wherein described n, m, f, d are positive integer.

15. according to the method for claim 14, which is characterized in that described that n-f file data unit difference is extensive It is multiple to include: into n-f abnormal document block in the n blocks of files

New blocks of files is created that in the storage location of the n-f abnormal document block respectively, obtains n-f new blocks of files；

The n-f file data unit is respectively written into the n-f new blocks of files.

16. according to the method for claim 14, which is characterized in that when m is 1, the redundant arithmetic includes even-odd check Algorithm, alternatively, the redundant arithmetic includes the Q checking algorithm based on Galois Field when m is 2.

17. according to the method for claim 14, which is characterized in that it is described according to redundant arithmetic to the d file data Unit and the m redundant data unit carry out redundant computation, and before obtaining n-f file data unit, the method is also wrapped It includes:

The redundant arithmetic is obtained from the corresponding metadata information of the file to be restored.

18. according to the method for claim 14, which is characterized in that described to read one respectively from d normal file block Before file data unit, the method also includes:

Obtained from the corresponding metadata information of the file to be restored the n blocks of files storage location and the m text The storage location of part block.

19. according to full benefit require 14 described in method, which is characterized in that the file data unit and the redundant data unit The byte number for being included is 8 multiple.

20. a kind of data managing method applied to distributed file storage system, the distributed file storage system include Management node, multiple back end and data writing device, characterized by comprising:

The data writing device sends storage location request to the management node, and the storage location request is comprising to be written Unique identification of the file in the distributed file storage system；

The management node determines n according to unique identification of the file to be written in the distributed file storage system The storage location information of a blocks of files and m redundant block on the multiple back end, and returned to the data writing device Storage location response is returned, the storage location response includes the storage location information, and the n and m are positive integer；

The data writing device reads file data from the file to be written to form first group of n file data list Member carries out redundant computation to described first group n file data unit, m redundant data unit is obtained, by first group of n A file data unit is respectively written into the n blocks of files, and the m redundant data unit is respectively written into the m In redundant block；

Wherein described n, m are positive integer.

21. according to the method for claim 20, which is characterized in that when in the n blocks of files comprising f normal file block When with n-f abnormal document block, the method also includes:

The management node reads a file data unit respectively from d normal file block, obtains d file data list Member, and read a redundant data unit respectively from the m redundant block, obtain m redundant data unit；

The management node carries out redundant computation to the d file data unit and the m redundant data unit, obtains n- F file data unit；

The management node by the n-f file data unit be restored to respectively n-f in the n blocks of files it is abnormal literary In part block；

Wherein described f, d are positive integer, and f is less than n, and d is less than or equal to f.

22. according to the method for claim 20, which is characterized in that the distributed file storage system further includes that data are read Equipment out, the method also includes:

The data taking equipment reads a file data unit from the n data blocks of files respectively, obtains second group of n A file data unit；

The data taking equipment n file data unit is written in the file of the data taking equipment creation.

23. according to the method for claim 22, it is characterised in that:

The data writing device reads file data from the file to be written to form first group of n file data unit Include:

The data writing device determines the file data amount read from the file to be written less than n file data unit Corresponding data volume, in the file data end supplemental data of the reading to form described first group n file data unit； The quantity of the supplemental data is sent to the management node；

The n file data unit is written in the file that the data taking equipment creates and is wrapped by the data taking equipment It includes:

The data taking equipment obtains the quantity of the supplemental data from the management node, according to the number of the supplemental data Amount removes the supplemental data from described second group n file data unit end, obtains remaining file data, will be described surplus Remaining file data is written in the file of the creation.

24. according to method described in claim 20-23 any one, which is characterized in that when m is 1, the redundant computation pair The redundant arithmetic answered includes parity arithmetic, alternatively, the corresponding redundant arithmetic of the redundant computation includes being based on when m is 2 The Q checking algorithm of Galois Field.

25. according to method described in claim 20-23 any one, which is characterized in that the file data unit and described The byte number that redundant data unit is included is 8 multiple.

26. a kind of client device, which is characterized in that including processor and memory, in which:

The memory, for storing program instruction；

The processor, for calling and executing the program instruction stored in the memory, so that the client device is held Data managing method of the row as described in any one of claims 1 to 13.

27. a kind of computer readable storage medium, including instruction, when run on a computer, so that the computer is held Data managing method of the row as described in any one of claims 1 to 13.

28. a kind of management equipment, which is characterized in that including processor and memory, in which:

The memory, for storing program instruction；

The processor, for calling and executing the program instruction stored in the memory, so that the management equipment executes Data managing method as described in any one of claim 14 to 19.

29. a kind of computer readable storage medium, including instruction, when run on a computer, so that the computer is held Data managing method of the row as described in any one of claim 14 to 19.

30. a kind of distributed file storage system, including multiple back end, characterized by comprising:

Client device as claimed in claim 26；

Management equipment as claimed in claim 28.