CN107291876A - A kind of DDM method - Google Patents
A kind of DDM method Download PDFInfo
- Publication number
- CN107291876A CN107291876A CN201710461880.6A CN201710461880A CN107291876A CN 107291876 A CN107291876 A CN 107291876A CN 201710461880 A CN201710461880 A CN 201710461880A CN 107291876 A CN107291876 A CN 107291876A
- Authority
- CN
- China
- Prior art keywords
- metadata
- file
- data
- copy
- document
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
- G06F16/152—File search processing using file content signatures, e.g. hash values
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of DDM method, belong to distributed storage field.Metadata management function is distributed to multiple nodes by technical solution of the present invention, and set up mapping relations, in metadata tree structure design, select what common multiway tree combined with Hash table to realize structure, complete trails field is added in the data structure of metadata, when writing data by log write after new data；Get the storage location for obtaining file data after file metadata by mapping mode；File metadata copy and file data copy are supported simultaneously, when using file metadata copy or file data copy, using different mapping relations, ensure that copy is distributed to different nodes, in reading and writing of files metadata or file data, when primary storage end is failed, copy storage end can take over the work of primary storage end rapidly.Technical solution of the present invention can effectively improve the Performance And Reliability of DDM.
Description
Technical field
The invention belongs to Distributed Storage field, more particularly, to a kind of DDM method.
Background technology
DDM, refers to by network, is accessed in the way of similar local file system on other nodes
Data.
Accessing file actually it is corresponding be accessing file each block, obtain file metadata information after, it is necessary to after
The continuous block message for obtaining file.Traditional method is the mode based on block distribution table, i.e., a block distribution table, table are safeguarded in internal memory
In house the storage location of blocks of files, file block message is obtained by query block distribution table.
When accessing file, the metadata information of file is first got, respective file data are then accessed again.First number
According to the data for meaning description data, the information such as file size, position, creation time are generally comprised.Traditional distributed data pipe
When reason uses the mode, i.e. accessing file of individual node storage metadata, the node of storage metadata is accessed first, is obtained
To after file metadata, further according to metadata accessing file.The expenses such as internal memory, the calculating that so metadata management is brought will be by one
The node of individual feature is undertaken, and the resource and performance of system are limited.
Obtain metadata information when marquis, it is necessary to according to the filename of the corresponding file of the metadata, to position first number
According to position.The metadata generally tissue in the form of file tree, is so easy to search.In traditional file system, search
Metadata is needed since root node, according to title successively coupling path.When the directories deep of file system is excessive, this undoubtedly can
Bring larger performance cost.
When written document, in order to system stability, it is necessary to write daily record, record modification operation, for system fault
Recovery operation afterwards.Traditional file system, daily record is to separate with file data, that is, a file is used alone and is used for depositing
Daily record.It means that written document needs to carry out file input and output twice every time, this obviously can bring extra performance to hold
Pin.
The content of the invention
For the disadvantages described above or Improvement requirement of prior art, the invention provides a kind of DDM method,
Its object is to which the metadata management function of DDM is distributed into multiple nodes, and set up mapping relations；Simultaneously
Support file metadata copy and file data copy, it is ensured that copy is distributed to different nodes, in reading and writing of files metadata
Or during file data, when primary storage end is failed, copy storage end can take over the work of primary storage end rapidly, thus solve existing
The expenses such as internal memory, calculating that metadata management is brought will be undertaken by the node of a feature, and the resource and performance of system will be by
Limitation is the problem of data trnascription distribution reliability is inadequate.
To achieve the above object, it is described according to one aspect of the present invention there is provided a kind of DDM method
Method includes：
Set up the mapping relations of fileinfo and multiple nodes；In the multiple node, one of node storage should
The metadata of file, other each nodes are stored with the metadata copy of this document；
Set up the mapping relations of file metadata information and multiple nodes；In the multiple node, one of node
This document data are stored, other each nodes are stored with this document data trnascription；
The metadata tree structure uses the structure that common multiway tree is combined with Hash table, in the data structure of metadata
Middle addition complete trails field；
In written document, directly by daily record addition after file data.
Further, methods described also includes：
During reading and writing of files metadata, operation is written and read to file metadata, if success, enters to file metadata copy
Row simultaneously operating；If failure, operation is written and read to file metadata copy.
Further, methods described also includes：
During reading and writing of files data, operation is written and read to file data, if success, is synchronized to file data copy
Operation；If failure, operation is written and read to file data copy.
In general, by the contemplated above technical scheme of the present invention compared with prior art, it is special with following technology
Levy and beneficial effect：
(1) in metadata tree structure design, what the common multiway tree of selection was combined with Hash table realizes structure, on the one hand may be used
Quickly to be searched using Hash, memory access number of times is reduced；On the other hand it can be conveniently accomplished using tree structure to whole
The modification operation of catalogue file.Complete trails field is added in the data structure of metadata, after lookup, is directly contrasted complete
Pathname just can determine that whether the node found is consistent with demand nodes, and without the successively coupling path since root node；
(2) data write operation and its corresponding metadata journalizing are carried out simultaneously, write in same file.In order to
Daily record expense is reduced as far as possible, this logging schemes are designed, and needs the behavior of daily record twice carried out to reduce each data manipulation
For once, i.e., operation reduces the expense of a memory access every time, the efficiency of system is improved；
(3) the metadata management function of the DDM traditionally undertaken by individual node be dispersed to storage with
In multiple General Porcess Unit of directorial area, i.e., the expense such as internal memory, calculating that metadata management is brought will be by all general procedures
Unit shared, eliminates system resource that may be present and performance bottleneck, so as to improve the system expandability, works as system
After topology is determined, system can work independent of a certain Centroid；
(4) by the data access mode based on mapping, any file read-write request in theory can be no longer to first number
According to server location request, so as to reduce management metadata, access the expense that metadata is brought, any client or dimension
Shield end can quickly position any in file simply by virtue of the fileinfo such as filename or file bias internal address and system topological
Storage end where one byte；
(5) metadata and data management use master slave mode, and principal and subordinate's storage end retains metadata and data trnascription, works as master
When storage end fails, the work of primary storage end, effective lifting system availability and reliability can be taken over rapidly from storage end.
Brief description of the drawings
Fig. 1 is the schematic diagram of distributed meta-data management of the present invention；
Fig. 2 is the schematic diagram of metadata read-write of the present invention；
Fig. 3 is the schematic diagram of reading and writing data of the present invention；
Fig. 4 is the data structure schematic diagram of metadata of the present invention；
Fig. 5 is the schematic diagram of daily record location mode of the present invention.
Embodiment
In order to make the purpose , technical scheme and advantage of the present invention be clearer, it is right below in conjunction with drawings and Examples
The present invention is further elaborated.It should be appreciated that specific embodiment described herein is only to explain the present invention, not
For limiting the present invention.As long as in addition, technical characteristic involved in each embodiment of invention described below that
Not constituting conflict between this can just be mutually combined.
Some definition will be used in the embodiment of the present invention, explanation of nouns therein is as follows：
Storage end：It is used for the node of data storage in DDM；
As shown in figure 1, present invention aims at the metadata management function of DDM is distributed into multiple sections
Point, and set up mapping relations；
Its step is now discussed in detail with specific embodiment：
(1) design hash function 1 and hash function 2, using the first class catalogue name where file as input, storage end numbering
For output；
(2) for a certain file in DDM, the Hash that input file first class catalogue name is designed to previous step
In function, hash function 1 exports the metadata that corresponding storage end n1 houses this document, and hash function 2 exports corresponding deposit
Storage end n2 houses the metadata copy of this document；
(3) as shown in Fig. 2 reading the metadata of this document, metadata is obtained from n1；If it fails, then obtaining member from n2
Data trnascription；
(4) design hash function 3 and hash function 4, using file metadata numbering as input, storage end numbering is output；
(5) for a certain file in DDM, the Hash that input file metadata numbering is designed to previous step
In function, hash function 3 exports the data that corresponding storage end p1 houses this document, and the output of hash function 4 is corresponding to deposit
Storage end p2 houses the data trnascription of this document；
(6) if written document, then step 7 is performed；If reading file, then step 8 is performed；
(7) write request is sent to p1, if written in work(, then request is synchronized to p2 by p1；If writing failure, sent to p2
Write request；Perform step 9；
(8) read request is sent to p1；If reading failure, as shown in figure 3, sending read request to p2；
(9) file is closed, turn-off request is sent to n1, if turned off success, then request is synchronized to n2 by p1；If turned off
Failure, then send turn-off request to n2.
As shown in figure 4, adding complete trails field in the data structure of metadata, such as file C complete trails is "/A/
C ", then add field "/A/C " in the metadata structure of this document.When being searched, it is not necessary to from root "/" to text
Part catalogue "/A " arrive again file "/A/C ", but directly by complete trails "/A/C " carry out Hash, so as to navigate to file C's
Metadata.
As shown in figure 5, after written document completion, directly by daily record addition after file data, it is not necessary to open again
Other files, so operation reduces the expense of a memory access every time, improves the efficiency of system.
As it will be easily appreciated by one skilled in the art that the foregoing is merely illustrative of the preferred embodiments of the present invention, it is not used to
The limitation present invention, any modifications, equivalent substitutions and improvements made within the spirit and principles of the invention etc., it all should include
Within protection scope of the present invention.
Claims (3)
1. a kind of DDM method, it is characterised in that methods described includes：
Set up the mapping relations of fileinfo and multiple nodes；In the multiple node, one of node stores this document
Metadata, other each nodes are stored with the metadata copy of this document；
Set up the mapping relations of file metadata information and multiple nodes；In the multiple node, one of node storage
This document data, other each nodes are stored with this document data trnascription；
The metadata tree structure uses the structure that common multiway tree is combined with Hash table, adds in the data structure of metadata
Enter complete trails field；
In written document, directly by daily record addition after file data.
2. a kind of DDM method according to claim 1, it is characterised in that methods described also includes：
During reading and writing of files metadata, operation is written and read to file metadata, if success, file metadata copy is carried out same
Step operation；If failure, operation is written and read to file metadata copy.
3. a kind of DDM method according to claim 1, it is characterised in that methods described also includes：
During reading and writing of files data, operation is written and read to file data, if success, behaviour is synchronized to file data copy
Make；If failure, operation is written and read to file data copy.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710461880.6A CN107291876A (en) | 2017-06-19 | 2017-06-19 | A kind of DDM method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710461880.6A CN107291876A (en) | 2017-06-19 | 2017-06-19 | A kind of DDM method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107291876A true CN107291876A (en) | 2017-10-24 |
Family
ID=60096550
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710461880.6A Pending CN107291876A (en) | 2017-06-19 | 2017-06-19 | A kind of DDM method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107291876A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109271361A (en) * | 2018-08-13 | 2019-01-25 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Distributed storage method and system for massive small files |
| CN109726252A (en) * | 2018-12-29 | 2019-05-07 | 税友软件集团股份有限公司 | A kind of data managing method and relevant apparatus of distributed data collection |
| CN110825698A (en) * | 2019-11-07 | 2020-02-21 | 重庆紫光华山智安科技有限公司 | Metadata management method and related device |
| CN111078633A (en) * | 2019-10-17 | 2020-04-28 | 珠海格力电器股份有限公司 | Method and device for communicating devices in device cluster, storage medium and electronic device |
| CN111324305A (en) * | 2020-02-16 | 2020-06-23 | 西安奥卡云数据科技有限公司 | Data writing/reading method in distributed storage system |
| CN112260874A (en) * | 2020-10-23 | 2021-01-22 | 南京鹏云网络科技有限公司 | Management system and method based on distributed storage unit |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101188544A (en) * | 2007-12-04 | 2008-05-28 | 浙江大学 | File transfer method for distributed file server based on buffer |
| CN101520805A (en) * | 2009-03-25 | 2009-09-02 | 中兴通讯股份有限公司 | Distributed file system and file processing method thereof |
| CN101697168A (en) * | 2009-10-22 | 2010-04-21 | 中国科学技术大学 | Method and system for dynamically managing metadata of distributed file system |
-
2017
- 2017-06-19 CN CN201710461880.6A patent/CN107291876A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101188544A (en) * | 2007-12-04 | 2008-05-28 | 浙江大学 | File transfer method for distributed file server based on buffer |
| CN101520805A (en) * | 2009-03-25 | 2009-09-02 | 中兴通讯股份有限公司 | Distributed file system and file processing method thereof |
| US20110307534A1 (en) * | 2009-03-25 | 2011-12-15 | Zte Corporation | Distributed file system supporting data block dispatching and file processing method thereof |
| CN101697168A (en) * | 2009-10-22 | 2010-04-21 | 中国科学技术大学 | Method and system for dynamically managing metadata of distributed file system |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109271361A (en) * | 2018-08-13 | 2019-01-25 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Distributed storage method and system for massive small files |
| CN109271361B (en) * | 2018-08-13 | 2020-07-24 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Distributed storage method and system for massive small files |
| CN109726252A (en) * | 2018-12-29 | 2019-05-07 | 税友软件集团股份有限公司 | A kind of data managing method and relevant apparatus of distributed data collection |
| CN111078633A (en) * | 2019-10-17 | 2020-04-28 | 珠海格力电器股份有限公司 | Method and device for communicating devices in device cluster, storage medium and electronic device |
| CN111078633B (en) * | 2019-10-17 | 2023-09-12 | 珠海格力电器股份有限公司 | Method and device for communicating devices in device cluster, storage medium and electronic device |
| CN110825698A (en) * | 2019-11-07 | 2020-02-21 | 重庆紫光华山智安科技有限公司 | Metadata management method and related device |
| CN110825698B (en) * | 2019-11-07 | 2021-02-09 | 重庆紫光华山智安科技有限公司 | Metadata management method and related device |
| CN111324305A (en) * | 2020-02-16 | 2020-06-23 | 西安奥卡云数据科技有限公司 | Data writing/reading method in distributed storage system |
| CN111324305B (en) * | 2020-02-16 | 2021-02-02 | 西安奥卡云数据科技有限公司 | Data writing/reading method in distributed storage system |
| CN112260874A (en) * | 2020-10-23 | 2021-01-22 | 南京鹏云网络科技有限公司 | Management system and method based on distributed storage unit |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107291876A (en) | A kind of DDM method | |
| US8572136B2 (en) | Method and system for synchronizing a virtual file system at a computing device with a storage device | |
| Hupfeld et al. | The XtreemFS architecture—a case for object‐based file systems in Grids | |
| US8214334B2 (en) | Systems and methods for distributed system scanning | |
| CN104618482B (en) | Access method, server, conventional memory device, the system of cloud data | |
| CN102122285B (en) | Data cache system and data inquiry method | |
| WO2021169113A1 (en) | Data management method and apparatus, and computer device and storage medium | |
| US9547706B2 (en) | Using colocation hints to facilitate accessing a distributed data storage system | |
| CN103002027B (en) | Data-storage system and the method for tree directory structure is realized based on key-value pair system | |
| CN109684282B (en) | Method and device for constructing metadata cache | |
| US20090012932A1 (en) | Method and System For Data Storage And Management | |
| US20100161657A1 (en) | Metadata server and metadata management method | |
| CN104408111A (en) | Method and device for deleting duplicate data | |
| US20100036872A1 (en) | Data management method | |
| CN104281506A (en) | Data maintenance method and system for file system | |
| CN102821138A (en) | Metadata distributed storage method applicable to cloud storage system | |
| US20150149500A1 (en) | Multi-level lookup architecture to facilitate failure recovery | |
| CN109407975B (en) | Data writing method, computing node and distributed storage system | |
| CN109542861A (en) | File management method, device and system | |
| CN104184812A (en) | Multi-point data transmission method based on private cloud | |
| CN108427728A (en) | Management method, equipment and the computer-readable medium of metadata | |
| US20150169623A1 (en) | Distributed File System, File Access Method and Client Device | |
| US7730351B2 (en) | Per file dirty region logging | |
| CN105138275A (en) | Data sharing method for Lustre storage system | |
| CN104951475B (en) | Distributed file system and implementation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20171024 |
|
| RJ01 | Rejection of invention patent application after publication |