CN106777047A - A kind of metadata read method and its device for distributed system - Google Patents
A kind of metadata read method and its device for distributed system Download PDFInfo
- Publication number
- CN106777047A CN106777047A CN201611129527.XA CN201611129527A CN106777047A CN 106777047 A CN106777047 A CN 106777047A CN 201611129527 A CN201611129527 A CN 201611129527A CN 106777047 A CN106777047 A CN 106777047A
- Authority
- CN
- China
- Prior art keywords
- metadata
- file
- read
- node
- local node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The invention discloses a kind of metadata read method and its device for distributed system, including when a file is opened, judge whether cache documentary metadata in current local node, if so, carrying out File read operation according to the metadata of file；Otherwise, the metadata of file is read from metadata node by network and is cached to local node, File read operation is carried out according to the metadata of file；And choose whether to open the pre- read operation of metadata, if so, according to the metadata that predetermined number is read under catalogue of the preset rules from metadata node where the metadata of file, and in caching to local node.The present invention can improve caching rate of the metadata in local node, avoid the inter-network in continuous reading heap file to read the number of times of metadata as far as possible, save file read access time, improve file reading performance.
Description
Technical field
Technical field, more particularly to a kind of unit for distributed system are read the present invention relates to distributed system file
Method for reading data and its device.
Background technology
In distributed file system, consumer process and storage medium are often distributed in different nodes, due to working as user
, it is necessary to obtain the metadata of file first when opening file, if current meta data is not present in local node (i.e. local service
Device) in, then need across a network to go to obtain metadata on meta data server, increased file read latch.
Also, user sometimes may require that and continuously substantial amounts of file is read, if at this time each file is read
When taking equal across a network take acquisition metadata if, read metadata time can be taken in all files reading process it is larger
Ratio, so as to reduce the speed of file reading.
Therefore, how to provide a kind of file reading performance good metadata read method and its dress for distributed system
Put be those skilled in the art need at present solve problem.
The content of the invention
It is an object of the invention to provide a kind of metadata read method and its device for distributed system, it is possible to increase
Metadata avoids the inter-network in continuous reading heap file from reading the number of times of metadata as far as possible in the caching rate of local node,
File read access time is saved, file reading performance is improved.
In order to solve the above technical problems, the invention provides a kind of metadata read method for distributed system, bag
Include：
When a file is opened, whether the interior metadata for being cached with the file of current local node is judged, if so, according to institute
The metadata for stating file carries out File read operation；
Otherwise, the metadata of the file is read from metadata node by network and is cached to the local node
It is interior, carry out File read operation according to the metadata of the file；
And choose whether to open the pre- read operation of metadata, if so, according to preset rules described in the metadata node
The metadata of predetermined number is read under catalogue where the metadata of file, and is cached to the local node.
Preferably, under the catalogue according to preset rules described in the metadata node where the metadata of file
Also include before the metadata for reading predetermined number：
Preset rules when being pre-read from one conduct of selection is current in default multiple rules.
Preferably, the preset rules are specially：
Order according to file name is successively read.
Preferably, the preset rules are specially：
Order according to document time is successively read.
Preferably, the preset rules are specially：
Order according to file size is successively read.
Preferably, the predetermined number is 10~20.
In order to solve the above technical problems, present invention also offers a kind of metadata reading device for distributed system,
Including：
Caching determining module, for, judging whether be cached with the file in current local node when a file is opened
Metadata, if so, triggering file read module；Otherwise, trigger the first metadata read module and pre-read opening module；
The first metadata read module, the metadata for reading the file from metadata node by network
And cache to the local node, trigger the file read module；
It is described to pre-read opening module, for choosing whether to open the pre- read operation of metadata for user, if operation is opened, triggering
Second metadata read module；
The second metadata read module, for the unit according to preset rules file described in the metadata node
The metadata of predetermined number is read under catalogue where data, and is cached to the local node；
The file read module, File read operation is carried out for the metadata according to the file.
The invention provides a kind of metadata read method and its device for distributed system, file reading is being carried out
When, if the uncached metadata for having this document in local node, on the one hand by network reading metadata node memory storage
The metadata of this document, on the other hand chooses whether to open the pre- read operation of metadata, if opening, according to preset rules from first number
According to the metadata that predetermined number disposably is read in the catalogue where the metadata of this document in node, and cache to local section
Point.I.e. when user is continuously read out to heap file, by opening the pre- read operation of metadata, it is possible to increase metadata is at this
The caching rate of ground node, that is, improve the possibility of the metadata cache in local node of opened file, avoids as far as possible
In continuous reading heap file, inter-network reads the number of times of metadata, saves file read access time, improves file and reads property
Energy.
Brief description of the drawings
Technical scheme in order to illustrate more clearly the embodiments of the present invention, below will be to institute in prior art and embodiment
The accompanying drawing for needing to use is briefly described, it should be apparent that, drawings in the following description are only some implementations of the invention
Example, for those of ordinary skill in the art, on the premise of not paying creative work, can also obtain according to these accompanying drawings
Obtain other accompanying drawings.
A kind of flow chart of the process of metadata read method for distributed system that Fig. 1 is provided for the present invention；
A kind of structural representation of metadata reading device for distributed system that Fig. 2 is provided for the present invention.
Specific embodiment
Core of the invention is to provide a kind of metadata read method and its device for distributed system, it is possible to increase
Metadata avoids the inter-network in continuous reading heap file from reading the number of times of metadata as far as possible in the caching rate of local node,
File read access time is saved, file reading performance is improved.
To make the purpose, technical scheme and advantage of the embodiment of the present invention clearer, below in conjunction with the embodiment of the present invention
In accompanying drawing, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is
A part of embodiment of the present invention, rather than whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art
The every other embodiment obtained under the premise of creative work is not made, belongs to the scope of protection of the invention.
The invention provides a kind of metadata read method for distributed system, shown in Figure 1, Fig. 1 is this hair
A kind of flow chart of the process of the metadata read method for distributed system of bright offer；The method includes：
Step s101：When a file is opened, judge whether cache documentary metadata in current local node, if so,
Into step s103；Otherwise, into step s102；
Step s102：The metadata of file is read from metadata node by network and is cached to local node, entered
Enter step s103；And choose whether open the pre- read operation of metadata, if so, according to preset rules from metadata node file
The metadata of predetermined number is read under catalogue where metadata, and is cached to local node；
Step s103：Metadata according to file carries out File read operation.
It should be noted that in step s102, including asynchronous two processes for carrying out, i.e., once judging current local node
Interior uncached documentary metadata, then process the metadata and caching of file to local node can be read by network
It is interior, File read operation then is carried out according to the metadata, another process determines whether to open the pre- read operation of metadata, if
It is, according to the metadata that predetermined number is read under catalogue of the preset rules from metadata node where the metadata of file, and
In caching to local node.Two processes do not interfere with each other, and dividing without sequencing.
Wherein, the process for choosing whether to open the pre- read operation of metadata here is specifically, output display metadata pre-reads
Prompting, if receiving the open command of user input, opens the pre- read operation of metadata, if it is defeated to receive user to display interface
The refusal instruction for entering, then be not turned on；It is further known that, metadata here pre-read in prompting can chain be connected to and release the button and refusal
Button, user is clicked on after releasing the button, and display interface is generation open command, and after user clicks on refusal button, display interface is
Generation refusal instruction；Certainly, preferred scheme is these are only, user can also be represented unlatching or be refused by default shortcut,
Which kind of do not limited using the mode present invention specifically.
It is understood that when user needs continuous reading multiple file, to avoid multiple across a network from obtaining metadata
Lose time, then may be selected to open the pre- read operation of metadata, raising open the metadata cache of file in local node can
Can property；If user only reads one or less several files, even if each file is required to across a network and reads metadata
Have more than is needed many times, so when can select to be not turned on metadata to pre-read.Certainly, whether unlatching metadata is specifically selected to pre-read
Operation is relevant with user's request, and the present invention is not limited this.
In addition, predetermined number here is 10~20.Certainly, predetermined number can be according to actual conditions sets itself, this
Invention is not construed as limiting to this.
Preferably, read under the catalogue according to preset rules from metadata node where the metadata of file default
Also include before the metadata of number：
Preset rules when being pre-read from one conduct of selection is current in default multiple rules.
It is understood that several different preset rules can be set in a distributed system in advance, user opens every time
Open after metadata pre-reads, the metadata that the file that the metadata returned will read with oneself is pre-read to improve as far as possible overlaps
Rate, preset rules when can independently be selected currently to pre-read according to the characteristics of file to be read.
Wherein, preset rules here are specially：
Order according to file name is successively read.
Furthermore it is also possible to be：
Order according to document time is successively read.
Furthermore it is also possible to be：
Order according to file size is successively read.
Certainly, several preset rules of the above are common rule, only preferred scheme, and also dependent on needing, setting others are pre-
If regular.
The invention provides a kind of metadata read method for distributed system, when file reading is carried out, if this
The uncached metadata for having this document in ground node, then on the one hand by this document of network reading metadata node memory storage
Metadata, on the other hand chooses whether to open the pre- read operation of metadata, if opening, according to preset rules from metadata node
The metadata of predetermined number disposably is read in catalogue where the metadata of this document, and is cached to local node.Work as
When user is continuously read out to heap file, by opening the pre- read operation of metadata, it is possible to increase metadata is in local node
Caching rate, that is, improve the possibility of the metadata cache in local node of opened file, avoid as far as possible continuous
Inter-network reads the number of times of metadata when reading heap file, saves file read access time, improves file reading performance.
Shown in Figure 2 present invention also offers a kind of metadata reading device for distributed system, Fig. 2 is this
A kind of structural representation of the metadata reading device for distributed system for providing is provided.The device includes：
Caching determining module 1, for, judging whether cache documentary first number in current local node when a file is opened
According to if so, triggering file read module 4；Otherwise, trigger the first metadata read module 2 and pre-read opening module 3；
First metadata read module 2, for reading the metadata of file from metadata node by network and caching
To local node, triggering file read module 4；
Opening module 3 is pre-read, for choosing whether to open the pre- read operation of metadata for user, if operation is opened, triggering the
Binary data read module 5；
Second metadata read module 5, for according to preset rules from metadata node where the metadata of file
The metadata of predetermined number is read under catalogue, and is cached to local node；
File read module 4, File read operation is carried out for the metadata according to file.
The invention provides a kind of metadata reading device for distributed system, when file reading is carried out, if this
The uncached metadata for having this document in ground node, then on the one hand by this document of network reading metadata node memory storage
Metadata, on the other hand chooses whether to open the pre- read operation of metadata, if opening, according to preset rules from metadata node
The metadata of predetermined number disposably is read in catalogue where the metadata of this document, and is cached to local node.Work as
When user is continuously read out to heap file, by opening the pre- read operation of metadata, it is possible to increase metadata is in local node
Caching rate, that is, improve the possibility of the metadata cache in local node of opened file, avoid as far as possible continuous
Inter-network reads the number of times of metadata when reading heap file, saves file read access time, improves file reading performance.
It should be noted that in this manual, term " including ", "comprising" or its any other variant be intended to
Nonexcludability is included, so that process, method, article or equipment including a series of key elements not only will including those
Element, but also other key elements including being not expressly set out, or also include being this process, method, article or equipment
Intrinsic key element.In the absence of more restrictions, the key element limited by sentence "including a ...", it is not excluded that
Also there is other identical element in process, method, article or equipment including the key element.
The foregoing description of the disclosed embodiments, enables professional and technical personnel in the field to realize or uses the present invention.
Various modifications to these embodiments will be apparent for those skilled in the art, as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the present invention.Therefore, the present invention
The embodiments shown herein is not intended to be limited to, and is to fit to and principles disclosed herein and features of novelty phase one
The scope most wide for causing.
Claims (7)
1. a kind of metadata read method for distributed system, it is characterised in that including：
When a file is opened, whether the interior metadata for being cached with the file of current local node is judged, if so, according to the text
The metadata of part carries out File read operation；
Otherwise, the metadata of the file is read from metadata node by network and is cached to the local node, pressed
Metadata according to the file carries out File read operation；
And choose whether to open the pre- read operation of metadata, if so, according to preset rules file described in the metadata node
Metadata where catalogue under read the metadata of predetermined number, and cache in the local node.
2. method according to claim 1, it is characterised in that it is described according to preset rules from the metadata node institute
Also include before the metadata that predetermined number is read under catalogue where stating the metadata of file：
Preset rules when being pre-read from one conduct of selection is current in default multiple rules.
3. method according to claim 1 and 2, it is characterised in that the preset rules are specially：
Order according to file name is successively read.
4. method according to claim 1 and 2, it is characterised in that the preset rules are specially：
Order according to document time is successively read.
5. method according to claim 1 and 2, it is characterised in that the preset rules are specially：
Order according to file size is successively read.
6. method according to claim 1, it is characterised in that the predetermined number is 10~20.
7. a kind of metadata reading device for distributed system, it is characterised in that including：
Caching determining module, for when a file is opened, judging the interior first number for whether being cached with the file of current local node
According to if so, triggering file read module；Otherwise, trigger the first metadata read module and pre-read opening module；
The first metadata read module, for reading the metadata of the file from metadata node by network and delaying
Deposit to the local node, trigger the file read module；
It is described to pre-read opening module, for choosing whether to open the pre- read operation of metadata for user, if operation is opened, triggering second
Metadata read module；
The second metadata read module, for the metadata according to preset rules file described in the metadata node
The metadata of predetermined number is read under the catalogue at place, and is cached to the local node；
The file read module, File read operation is carried out for the metadata according to the file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611129527.XA CN106777047A (en) | 2016-12-09 | 2016-12-09 | A kind of metadata read method and its device for distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611129527.XA CN106777047A (en) | 2016-12-09 | 2016-12-09 | A kind of metadata read method and its device for distributed system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106777047A true CN106777047A (en) | 2017-05-31 |
Family
ID=58875083
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611129527.XA Pending CN106777047A (en) | 2016-12-09 | 2016-12-09 | A kind of metadata read method and its device for distributed system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106777047A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107291870A (en) * | 2017-06-15 | 2017-10-24 | 郑州云海信息技术有限公司 | Files in batch read method in a kind of distributed storage |
| CN107360245A (en) * | 2017-07-28 | 2017-11-17 | 郑州云海信息技术有限公司 | A kind of local cache method and device based on lease lock mechanism |
| CN107491545A (en) * | 2017-08-25 | 2017-12-19 | 郑州云海信息技术有限公司 | The catalogue read method and client of a kind of distributed memory system |
| CN107633102A (en) * | 2017-10-25 | 2018-01-26 | 郑州云海信息技术有限公司 | A kind of method, apparatus, system and equipment for reading metadata |
| CN109947720A (en) * | 2019-04-12 | 2019-06-28 | 苏州浪潮智能科技有限公司 | A kind of pre-reading method of files, device, equipment and readable storage medium storing program for executing |
| CN111399784A (en) * | 2020-06-03 | 2020-07-10 | 广东睿江云计算股份有限公司 | Pre-reading and pre-writing method and device for distributed storage |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080077633A1 (en) * | 2006-09-25 | 2008-03-27 | International Business Machines Corporation | Method for policy-based data placement when restoring files from off-line storage |
| CN103491152A (en) * | 2013-09-17 | 2014-01-01 | 华为数字技术（苏州）有限公司 | Metadata obtaining method, device and system in distributed file system |
| CN105141685A (en) * | 2015-08-18 | 2015-12-09 | 浪潮(北京)电子信息产业有限公司 | File read-write system and meta data memory thereof as well as method and device for reading and writing files |
| CN105573667A (en) * | 2015-12-10 | 2016-05-11 | 华为技术有限公司 | Data reading method and storage server |
-
2016
- 2016-12-09 CN CN201611129527.XA patent/CN106777047A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080077633A1 (en) * | 2006-09-25 | 2008-03-27 | International Business Machines Corporation | Method for policy-based data placement when restoring files from off-line storage |
| CN103491152A (en) * | 2013-09-17 | 2014-01-01 | 华为数字技术（苏州）有限公司 | Metadata obtaining method, device and system in distributed file system |
| CN105141685A (en) * | 2015-08-18 | 2015-12-09 | 浪潮(北京)电子信息产业有限公司 | File read-write system and meta data memory thereof as well as method and device for reading and writing files |
| CN105573667A (en) * | 2015-12-10 | 2016-05-11 | 华为技术有限公司 | Data reading method and storage server |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107291870A (en) * | 2017-06-15 | 2017-10-24 | 郑州云海信息技术有限公司 | Files in batch read method in a kind of distributed storage |
| CN107291870B (en) * | 2017-06-15 | 2021-03-09 | 苏州浪潮智能科技有限公司 | Method for reading files in distributed storage in batch |
| CN107360245A (en) * | 2017-07-28 | 2017-11-17 | 郑州云海信息技术有限公司 | A kind of local cache method and device based on lease lock mechanism |
| CN107360245B (en) * | 2017-07-28 | 2020-10-16 | 苏州浪潮智能科技有限公司 | Local caching method and device based on lease lock mechanism |
| CN107491545A (en) * | 2017-08-25 | 2017-12-19 | 郑州云海信息技术有限公司 | The catalogue read method and client of a kind of distributed memory system |
| CN107633102A (en) * | 2017-10-25 | 2018-01-26 | 郑州云海信息技术有限公司 | A kind of method, apparatus, system and equipment for reading metadata |
| CN109947720A (en) * | 2019-04-12 | 2019-06-28 | 苏州浪潮智能科技有限公司 | A kind of pre-reading method of files, device, equipment and readable storage medium storing program for executing |
| CN111399784A (en) * | 2020-06-03 | 2020-07-10 | 广东睿江云计算股份有限公司 | Pre-reading and pre-writing method and device for distributed storage |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106777047A (en) | A kind of metadata read method and its device for distributed system | |
| CN103713815B (en) | A kind of method, device and mobile terminal returning quickly to specified page | |
| US20120151380A1 (en) | Intelligent message processing | |
| CN106973376A (en) | Application program is selected to access the method and device of network | |
| US20190205056A1 (en) | Transparent data movement between a private cloud and storage ecosystem and another storage system | |
| CN104462516B (en) | A kind of information cuing method and mobile terminal | |
| CN103744618B (en) | Realize the method and system of the shared storage of team | |
| CN106909281A (en) | One kind application inner boundary changing method and device | |
| CN105094736A (en) | Display control method and device and electronic device | |
| CN103440271B (en) | The method and its device of history catalogue are shown in operating system | |
| CN107547748A (en) | A kind of picture management method, terminal and computer-readable recording medium | |
| CN107633102A (en) | A kind of method, apparatus, system and equipment for reading metadata | |
| CN106383752A (en) | Browser page abnormity recovery processing method and device | |
| CN103530386B (en) | The edit methods and browser of browsing device net page | |
| CN109697281A (en) | The online method, apparatus and electronic equipment for merging document | |
| US20140258823A1 (en) | Intelligent categorization of bookmarks | |
| CN103634470A (en) | Human-computer interaction prediction method based on terminal mobile data access network Qos | |
| CN103942226A (en) | Method and device for obtaining hot content | |
| CN103150400A (en) | MapReduce-framework-based data screening method | |
| CN105426038A (en) | Picture popularity algorithm based picture pre-loading method for power grid scheduling control system | |
| CN103530019B (en) | A kind of method and apparatus of switch window | |
| CN104063403B (en) | A kind of method and system of web page interlinkage management in pair of browser | |
| JP5627772B2 (en) | Accessing file systems using semantic indexing | |
| CN105426526B (en) | A kind of method and device that page info is chosen | |
| CN109189291A (en) | Menu managing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170531 |