CN107315547A - A kind of method and device for reading distributed meta data file - Google Patents
A kind of method and device for reading distributed meta data file Download PDFInfo
- Publication number
- CN107315547A CN107315547A CN201710585356.XA CN201710585356A CN107315547A CN 107315547 A CN107315547 A CN 107315547A CN 201710585356 A CN201710585356 A CN 201710585356A CN 107315547 A CN107315547 A CN 107315547A
- Authority
- CN
- China
- Prior art keywords
- catalogue
- burst
- file
- metadata node
- corresponding relation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of method for reading distributed meta data file, after createing directory, catalogue is split as multiple catalogue bursts and migrated to multiple metadata nodes, and need the file for writing catalogue to be stored in each catalogue burst, when reading file, according to the corresponding relation between the identification information of file and catalogue burst and catalogue burst and metadata node, it is final to determine the metadata node that file is stored, to read file, the frequency that metadata node where reducing focus catalogue to a certain extent is accessed frequently, reduce metadata node temperature, reduce the load pressure of metadata node, improve the performance of whole metadata management system.The invention also discloses a kind of device of the reading distributed meta data file with above-mentioned beneficial effect.
Description
Technical field
The present invention relates to metadata management field, more particularly to a kind of method and dress for reading distributed meta data file
Put.
Background technology
Data in any file system are divided into data and metadata.Data refer to the real data in ordinary file, member
Data are mainly the information of description data attribute, for supporting that storage location, historical data, resource lookup, file are recorded as indicated
The functions such as record.Metadata management has two ways, centralized management and distributed management, and distributed management refers to deposit metadata
It is placed on the arbitrary node of system and the migration of dynamic.Responsibility to metadata management is also distributed to each different node
On.
In the system of multiple metadata nodes, each metadata node has oneself responsible bibliographic structure, newly-built
Subdirectory belongs to the metadata node load belonging to parent directory.If the directory content visit capacity below some metadata node adds
Greatly, hot spot data is constituted, this metadata node load pressure will increase, and form focus metadata node.
In order to reduce the temperature of focus metadata node, a kind of the more commonly used method is each metadata section of personal monitoring
The temperature of point, once it is larger more than temperature, the higher subdirectory of temperature is migrated manually to idle metadata node.This mode
Shortcoming be, for minimum transition unit catalogue may the file that be stored itself be exactly a hot spot data；In addition, artificial
Monitoring and migrating catalogue has certain hysteresis quality.
The content of the invention
It is an object of the invention to provide a kind of method for reading distributed meta data file, the focus catalogue that is stored with is reduced
Metadata node visit capacity, reduce metadata node temperature, improve systematic function, the present invention also provides a kind of read
The device of distributed meta data file.
In order to solve the above technical problems, the present invention provides a kind of method for reading distributed meta data file, including：
The request for reading file is received, wherein reading the identification information of file needed for including in the request；According to institute
Identification information and the first corresponding relation are stated, it is determined that storing the catalogue burst of the file, first corresponding relation is each institute
State the corresponding relation between the identification information of catalogue burst and the file being stored in the catalogue burst；According to the mesh
Burst and the second corresponding relation are recorded, the metadata node where the catalogue burst is determined, second corresponding relation is establishment
After catalogue, the catalogue is split into multiple catalogue bursts, where each described catalogue burst and the catalogue burst
Corresponding relation between each metadata node；There is provided by the metadata node and read file response.
Wherein, include after the catalogue to be split into multiple catalogue bursts：
The file is stored to each described in catalogue burst, and the file size in each described catalogue burst
In the first preset range, first preset range is determined according to the quantity for being stored in the file total in the catalogue.
Wherein, it is described according to the identification information and the first corresponding relation, it is determined that storing the catalogue fragment packets of the file
Include：
According to the identification information of the file, determine to store the catalogue burst of the file by hash algorithm.
Wherein, it is described the catalogue is split into multiple catalogue bursts to include：
The catalogue is split as the catalogue burst of the quantity for the metadata node number integer times, and each institute
The quantity for stating the catalogue burst in metadata node is identical, and records between the catalogue burst and place metadata node
Corresponding relation.
Wherein, also include before the request for reading file is received：
The each metadata node of detection exceedes predetermined threshold value with the presence or absence of the size of the file of storage, if
It is that will then store the quantity of documents to exceed the catalogue burst in the metadata node of predetermined threshold value, migrates to depositing
The size for storing up the file is no more than in the metadata node of predetermined threshold value, and records the catalogue burst and place member number
According to the corresponding relation between node.
Present invention also offers a kind of device for reading distributed meta data file, including：
Request module is received, the request of file is read for receiving, wherein reading file needed for including in the request
Identification information；
Computing module, for according to the identification information and the first corresponding relation, it is determined that storing the catalogue point of the file
Piece, first corresponding relation is the mark letter of each described catalogue burst and the file being stored in the catalogue burst
Corresponding relation between breath；
Searching modul, for according to the catalogue burst and the second corresponding relation, determining the member where the catalogue burst
Back end, second corresponding relation is that after createing directory, the catalogue is split into multiple catalogue bursts, each institute
State the corresponding relation between each metadata node where catalogue burst and the catalogue burst；
Respond module, file response is read for being provided by the metadata node.
Wherein, the computing module includes：
First arithmetic element, for splitting into the catalogue after multiple catalogue bursts, the file is deposited
Storage to the quantity of documents each described in catalogue burst, and in each described catalogue burst in the first preset range, institute
The first preset range is stated to be determined according to the quantity for being stored in the file total in the catalogue.
Wherein, the computing module includes：
Second arithmetic element, for the filename identification information according to the file, storage institute is determined by hash algorithm
State the catalogue burst of file.
Wherein, the searching modul includes：
First migration units, for the catalogue to be split as into quantity described in the metadata node number integer times
The quantity of the catalogue burst in catalogue burst, and each metadata node is identical, and record the catalogue burst and
Corresponding relation between the metadata node of place.
Wherein, the searching modul includes：
Second migration units, for before the request for reading file is received, whether each metadata node of detection
Quantity in the presence of the file of storage exceedes predetermined threshold value, if it is, the quantity of documents will be stored more than predetermined threshold value
The metadata node in the catalogue burst, migrate to storing the quantity of the file no more than described in predetermined threshold value
In metadata node, and record the corresponding relation between the catalogue burst and place metadata node.
The method and device provided by the present invention for reading distributed meta data file, after createing directory, catalogue is torn open
It is divided into multiple catalogue bursts that can be migrated, and needs the file for writing catalogue to be stored in each catalogue burst, then for depositing
The minimum split cells for storing up meta data file is catalogue burst, and each catalogue burst is migrated to multiple metadata nodes, then respectively
A part for file in the catalogue that is all only stored with individual metadata node, when reading data, can believe according to the mark of file
The corresponding relation between corresponding relation and catalogue burst and metadata node between breath and catalogue burst finds file phase
The metadata node answered, it is to avoid during using catalogue as minimum split cells, the file in some focus catalogues is stored in together
In one metadata node, and read any one file in focus catalogue and be required for accessing same metadata node, so that
Cause metadata node hot issue formed by the increase of metadata node visit capacity.
The present invention split cells minimum by refining data storage, when will read data to the access of metadata node by
Concentrate the access to same metadata node to be changed into the scattered access to multiple metadata nodes, avoid to a certain extent
Some metadata node due to storage file it is excessive formed by hot issue, reach reduction metadata node temperature purpose,
The load pressure of metadata node is reduced, the performance of whole system is improved.
Brief description of the drawings
, below will be to embodiment or existing for the clearer explanation embodiment of the present invention or the technical scheme of prior art
The accompanying drawing used required in technology description is briefly described, it should be apparent that, drawings in the following description are only this hair
Some bright embodiments, for those of ordinary skill in the art, on the premise of not paying creative work, can be with root
Other accompanying drawings are obtained according to these accompanying drawings.
The flow chart of the embodiment for the reading distributed meta data file that Fig. 1 provides for the present invention；
Fig. 2 is the invention provides a kind of flow chart of the embodiment creatied directory；
Fig. 3 is the invention provides a kind of flow chart of embodiment of storage file；
A kind of specific embodiment party that the file and target burst that Fig. 4 provides for the present invention are distributed in each metadata node
Formula schematic diagram；
Fig. 5 is the structured flowchart provided in an embodiment of the present invention for reading distributed meta data file device.
Embodiment
For in distributed file system, the request of metadata accounts for more than the 50% of all requests of file system, so first
The quality of the request processing of data directly affects the performance of whole file system, and wherein some metadata node visit capacity compared with
Greatly, focus metadata node will be formed, is to influence one of major reason of whole file system performance.Some metadata node
The larger main cause of visit capacity is a main original for having focus catalogue, forming focus catalogue for the metadata node memory storage
Because being to contain substantial amounts of file under the catalogue.Because the file in each catalogue is according to certain classification storage, if more
Change the catalogue of storage file, can be made troubles to the management of file, be that this does not change what file was stored the invention provides a kind of
On the basis of catalogue, the scheme of the temperature of focus metadata node is reduced.
In order that those skilled in the art more fully understand the present invention program, with reference to the accompanying drawings and detailed description
The present invention is described in further detail.Obviously, described embodiment is only a part of embodiment of the invention, rather than
Whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art are not making creative work premise
Lower obtained every other embodiment, belongs to the scope of protection of the invention.
The flow chart of the embodiment for the reading distributed meta data file that the present invention is provided is as shown in figure 1, the party
Method can include：
Step S101：Receive the request for reading file.
The identification information of file is read needed for including in the request, identification information can be the file for needing to read file
The identification information of name or other forms, as long as by identification information with regard to that can judge which file is required reading
The file taken, the concrete form of identification information is not the essential features of the present invention.
Step S102：According to the identification information and the first corresponding relation, it is determined that storing the catalogue burst of the file.
Step S103：According to the catalogue burst and the second corresponding relation, the metadata where the catalogue burst is determined
Node.
It should be noted that in the present invention storage file catalogue, create the catalogue after, the catalogue is split into
File in multiple catalogue bursts, catalogue is stored in each catalogue burst, each catalogue burst and is stored in catalogue point
Corresponding relation between the identification information of piece file is the first corresponding relation；And multiple catalogue bursts are respectively distributed to multiple members
In back end, the corresponding relation between each catalogue burst and the metadata node at place is the second corresponding relation.
It can thus be appreciated that the file in catalogue is also stored in the described catalogue, but the storage of file finally is affiliated each
Individual metadata node, even if the catalogue is focus catalogue, but is not but some metadata of central access when reading data
Node, so as to reduce the temperature of metadata node, and in the file during user reads the catalogue, be still to what user showed
Complete catalogue.
But this is not the unique technical scheme for realizing present invention problem, for example, it can also create multiple subdirectories and use
With storage file, and multiple subdirectories are belonged into same type of catalogue, the file that the catalogue is included also belongs to each yuan
Back end, this scheme can also reach the purpose of reduction metadata node temperature, but when reading data, give user's exhibition
What is shown is not complete catalogue, so that the catalogue of storage file excessively cumbersome use feeling is caused to user, so above-mentioned
Embodiment is a kind of present invention more preferred embodiment, also has other similar technical schemes, does not do go to live in the household of one's in-laws on getting married one by one herein
State.
Step S104：There is provided by the metadata node and read file response.
It is determined that after the particular location for the file that needs are read, corresponding metadata node directly provides the user the text of correlation
Part content, then read file success.
Based on above-described embodiment, before reading data, the invention provides a kind of specific embodiment party creatied directory
The flow chart of formula is as shown in Fig. 2 this method can include：
Step S201：Create directory.
The catalogue may belong to any one metadata node, usually gives tacit consent to the catalogue and belongs to some specific metadata
Node.
Step S202：Catalogue is split into multiple catalogue bursts.
The quantity of the catalogue burst can store the quantity of file total in the catalogue to determine as needed, can also
Determined according to the quantity of metadata node, or directly provide that each catalogue splits into the catalogue burst of fixed qty, this is all
The realization of technical solution of the present invention is not influenceed.
And in order to preferably solve the hot issue of metadata node, can be according to the quantity of catalogue burst to each catalogue
The quantity of file and the quantity of each metadata node in-list burst do certain constraint in burst, but this is not this hair
Bright essential features.
Step S203：Each catalogue Cell round Robin is moved into each different metadata node.
To each catalogue burst each metadata node distribution, can using poll migration by the way of or
Each catalogue burst metadata node specifically to be migrated first is determined, then migrate simultaneously, finally ensures each catalogue point
Piece all disperses to migrate to each metadata node, and similar embodiment, will not be repeated here with this.
Step S204：Record the corresponding relation between the metadata node where each catalogue burst and catalogue burst.
The corresponding relation is recorded, first number at the catalogue burst specific place of storage file is determined when being both for storage file
It is also for the specific metadata node of storage file can be found when reading data according to node.
Step S205：Directory creating is completed.
After the completion of directory creating, you can to store the file that user needs to store into catalogue.
Based on above-described embodiment, the invention provides a kind of flow chart such as Fig. 3 institutes of embodiment of storage file
State, this method can include：
Step S301：Receive the file of the catalogue to be written of user's transmission.
Step S302：According to the filename of file by hash algorithm, the catalogue burst of storage file is determined.
It should be noted that being used as identification information using filename herein, it is not excluded that use the mark of other forms
Information, the concrete form of identification information is not the essential features of the present invention.
In addition, the computing mode of hash algorithm can be used by determining the concrete mode of the catalogue burst of storage file, may be used also
The file in catalogue is stored in each catalogue burst or otherwise determines that file is specifically stored in the way of poll
In which catalogue burst, technical scheme can be realized, and is a kind of tool of the present invention using hash algorithm
The embodiment of body.
But, after the burst that storage file is determined using hash algorithm, when reading data, it is necessarily required to calculate using hash
The catalogue burst that specifically stores of file read required for method inquiry, each described catalogue burst and is stored in the catalogue burst
The filename of corresponding relation namely each catalogue burst and stored file between the identification information of the interior file it
Between meet the operation rules of hash computings.
Step S303：The corresponding relation between metadata node according to where each catalogue burst and catalogue burst, really
Determine the metadata node where the catalogue burst of storage file.
Step S304：File is write in corresponding metadata node.
File is write after corresponding metadata node, what metadata node just can be according to the reading file of user please
Ask, provide a user corresponding file content.
It should be noted that the above-mentioned embodiment for createing directory and writing data is not unique embodiment party of the invention
Formula, for example, it is above-mentioned be after catalogue burst is migrated to each metadata node, then to perform the behaviour of write-in file in embodiment
Make, but if being carried out writing file operation after catalogue splits into each burst, file write after each catalogue burst,
Each catalogue burst is distributed to each metadata node again, after even createing directory, just stored file to catalogue institute
Metadata node, then enter the operation such as the fractionation of column catalogue and the migration of catalogue burst, specific embodiment may omit
There is difference, but can realize that the present invention's is dispersed in each metadata node by the file in catalogue, to reduce metadata section
The purpose of point temperature, as long as so can realize before user sends and reads file, the file being stored in catalogue is dispersed in respectively
In individual metadata node, to ensure that user will not frequently access the technical side of some metadata node when reading the catalogue file
Case, belongs to the technical scheme in protection scope of the present invention.
Based on above-described embodiment, it is contemplated that be stored in some catalogue point if on what the file in catalogue was excessively concentrated
In piece, cause the file amount of metadata node memory storage where the burst excessive, and cause metadata node to still have one
What is provided in fixed hot issue, the present invention can be with a kind of further improved embodiment, the party to above-described embodiment
Method can specifically include：
The file is stored to each described in catalogue burst, and the quantity of documents in each described catalogue burst
In the first preset range；First preset range is determined according to the quantity for being stored in the file total in the catalogue.
, can be according to catalogue in order to avoid what the file being stored in catalogue was excessively concentrated is stored in some catalogue burst
In the quantity of total file adjust in each catalogue burst in a preset range of storage file quantity.
Illustrate for a specific example, for example, the file that be stored in catalogue is 10,000,000 file, and mesh
The quantity for marking burst is 5, then the first preset range can be 1,800,000 to 2,200,000, then each target burst memory storage text
The quantity of part is substantially all in 2,000,000 or so interval, it is ensured that the file amount in each target burst is generally average
Distribution.
It should be noted that this is a kind of more preferred embodiment of the present invention.For example, for equally to deposit
The file that the file in catalogue is 10,000,000 is stored up, target burst is 5, and the quantity of documents of each target burst memory storage does not surpass
3,000,000 are crossed, is also avoided that file is excessively centrally stored in some target burst, but file may also be caused centrally stored
In certain 2 to 3 catalogue burst, the problem of metadata node temperature can only being reduced to a certain extent, so in the present invention compared with
To be preferred embodiment, the size of the file amount of each target burst memory storage is substantially the same, but this is not to realize
The essential features of the present invention.
Based on above-described embodiment, to realize that the size of storage file amount in each catalogue burst is substantially the same, can take
Using hash algorithm to each catalogue in the embodiment of a variety of catalogue bursts for determining storage file, such as above-mentioned embodiment
The file of burst storage is allocated, because hash algorithm has balance, can be assigned in each catalogue burst
The size of file amount is substantially the same.
Again because the writing mode and reading manner of file are corresponding to a certain extent, if taking hash algorithm
Write-in data are carried out, are necessarily to be also required to use hash algorithm reading data, conversely, being calculated when reading data using hash
Method, then it is known that be also to use hash algorithm that file is write into each catalogue burst when writing data, it is thus of the invention
The a kind of of offer determines the specific embodiment of the catalogue burst of storage file according to the identification information of file when reading data,
It can include：
According to the identification information of the file, determine to store the catalogue burst of the file by hash algorithm.
Based on above-described embodiment, except the file amount size for needing to consider to store in each catalogue burst, in addition it is also necessary to consider
Some metadata node is distributed in what catalogue each catalogue burst was excessively concentrated, some metadata node can be equally caused
The problem of temperature is of a relatively high.
One kind for this present invention the fractionation catalogue burst provided and the metadata node for distributing each catalogue burst is specific
Embodiment, this method can include：
The catalogue is split as the catalogue burst of the quantity for the metadata node number integer times, and each institute
The quantity for stating the catalogue burst in metadata node is identical, and records between the catalogue burst and place metadata node
Corresponding relation.
Catalogue split target burst quantity can it is identical with the quantity of metadata node can also be metadata node
Integral multiple, this assigns to each metadata node for each target burst and provides premise, so as to avoid target burst from excessively collecting
In distribution some metadata node in the problem of.
But if the quantity of each target burst memory storage file is randomly assigned, in each metadata node
The quantity distribution of target burst should be all that principle is carried out with how much body phases of the quantity of each metadata node memory storage file
Distribution.If the number of the quantity of documents of each target burst memory storage is substantially the same, each target burst assigns to each yuan
Back end could cause the number of each metadata node memory storage quantity of documents to be substantially the same.Than more typical example, such as
Shown in Fig. 4, a kind of specific embodiment party that the file and target burst that Fig. 4 provides for the present invention are distributed in each metadata node
Formula schematic diagram.File total amount is to include a target burst in 10,000,000, each metadata node in catalogue in this method, and
The quantity of storage file is about all 2,000,000.
But above-described embodiment is that one kind of the present invention compares preferred embodiment, for example, in each target burst
On the basis of how much quantity of documents is substantially the same, it can distribute each target burst corresponding metadata node in the way of poll,
In this way as a result, target burst quantity in each metadata node at most difference one, also can be in certain journey
Above mentioned problem is avoided on degree, also similar embodiment is numerous to list herein.
It should be noted that it is that the present invention is more excellent that the size of the amount of each metadata node memory storage file, which is substantially the same,
The embodiment of choosing, as long as the hot issue of metadata node will not be caused, each metadata node memory storage quantity of documents
How many realizations that technical solution of the present invention is nor affected on a certain distance.
As it was previously stated, in order to avoid the quantity of each metadata node memory storage file has a certain distance, and
The problem of metadata node stroke focus caused, present invention also offers another specific embodiment, this method can be wrapped
Include：
Before the request for reading file is received, the file of each metadata node of detection with the presence or absence of storage
Quantity exceed predetermined threshold value, if it is, exceeding the quantity of documents is stored in the metadata node of predetermined threshold value
The catalogue burst, migrate to storing the quantity of the file no more than in the metadata node of predetermined threshold value, and remember
Record the corresponding relation between the catalogue burst and place metadata node.
, can be by partial list in the metadata node when the file amount for detecting some metadata node storages is larger
Burst is migrated to the less metadata node of storage file amount, until the quantity of the file amount of each metadata node memory storage
Default scope is no more than, the preset range can be determined according to quantity of documents total in catalogue is stored in.
It should be noted that the present invention is to reach that the file in catalogue is scattered to be stored in each yuan of number by will be stored in
According to, so as to avoid the purpose of some metadata node formation focus, and file can not be concentrated excessively is stored in some in node
Even in a few catalogue burst, and each catalogue burst can not excessively concentrate be present in some or a few yuan
In back end, it is for reaching the purpose, then if eventually through detecting and migrating target burst, can also reach this
One purpose, then can not be to target burst in the number of the quantity of documents of target burst memory storage and each metadata node
Quantity carry out special restriction, if on the contrary, above-mentioned purpose can be reached with particular determination in excessively, then can not also enter
The above-mentioned detection program of row, sum it up, the file for enabling to be stored in catalogue disperses to be stored in each metadata node
It is interior, it is to avoid the embodiment of some metadata node formation focus, is all in the protection domain for belong to technical solution of the present invention.
Reading distributed meta data file device provided in an embodiment of the present invention is introduced below, it is described below to read
Take distributed meta data file device can be mutually to should refer to above-described reading distributed meta data document method.
Fig. 5 is the structured flowchart provided in an embodiment of the present invention for reading distributed meta data file device, and reference picture 5 is read
Distributed meta data file device can include：
Request module 100 is received, the request of file is read for receiving, wherein reading text needed for including in the request
The identification information of part；
Computing module 200, for according to the identification information and the first corresponding relation, it is determined that storing the catalogue of the file
Burst, first corresponding relation is the mark of each described catalogue burst and the file being stored in the catalogue burst
Corresponding relation between information；
Searching modul 300, where for according to the catalogue burst and the second corresponding relation, determining the catalogue burst
Metadata node, second corresponding relation is that after createing directory, the catalogue is split into multiple catalogue bursts, each
The corresponding relation between each metadata node where the catalogue burst and the catalogue burst；
Respond module 400, file response is read for being provided by the metadata node.
It is preferred that, the computing module 200 includes：
First arithmetic element, for splitting into the catalogue after multiple catalogue bursts, the file is deposited
Storage to the quantity of documents each described in catalogue burst, and in each described catalogue burst in the first preset range, institute
The first preset range is stated to be determined according to the quantity for being stored in the file total in the catalogue.
It is preferred that, the computing module 200 includes：
Second arithmetic element, for the identification information according to the file, determines to store the file by hash algorithm
The catalogue burst
It is preferred that, the searching modul 300 includes：
First migration units, for the catalogue to be split as into quantity described in the metadata node number integer times
The quantity of the catalogue burst in catalogue burst, and each metadata node is identical, and record the catalogue burst and
Corresponding relation between the metadata node of place.
It is preferred that, the searching modul 300 includes：
Second migration units, for before the request for reading file is received, whether each metadata node of detection
Quantity in the presence of the file of storage exceedes predetermined threshold value, if it is, the quantity of documents will be stored more than predetermined threshold value
The metadata node in the catalogue burst, migrate to storing the quantity of the file no more than described in predetermined threshold value
In metadata node, and record the corresponding relation between the catalogue burst and place metadata node.
The device of the reading distributed meta data file of the present embodiment is used to realize foregoing reading distributed meta data text
The method of part, therefore it is distributed to read the visible reading hereinbefore of embodiment in the device of distributed meta data file
The embodiment part of the method for meta data file, for example, receiving request module 100, computing module 200, searching modul 300 is rung
Answer module 400, be respectively used to realize in the method for above-mentioned reading distributed meta data file step S101, S102, S103 and
S104, so, its embodiment is referred to the description of corresponding various pieces embodiment, will not be repeated here.
The embodiment of each in this specification is described by the way of progressive, what each embodiment was stressed be with it is other
Between the difference of embodiment, each embodiment same or similar part mutually referring to.For being filled disclosed in embodiment
For putting, because it is corresponded to the method disclosed in Example, so description is fairly simple, related part is referring to method part
Explanation.
Professional further appreciates that, with reference to the unit of each example of the embodiments described herein description
And algorithm steps, can be realized with electronic hardware, computer software or the combination of the two, in order to clearly demonstrate hardware and
The interchangeability of software, generally describes the composition and step of each example according to function in the above description.These
Function is performed with hardware or software mode actually, depending on the application-specific and design constraint of technical scheme.Specialty
Technical staff can realize described function to each specific application using distinct methods, but this realization should not
Think beyond the scope of this invention.
Directly it can be held with reference to the step of the method or algorithm that the embodiments described herein is described with hardware, processor
Capable software module, or the two combination are implemented.Software module can be placed in random access memory (RAM), internal memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In any other form of storage medium well known in field.
Reading distributed meta data document method provided by the present invention and device are described in detail above.This
Apply specific case in text to be set forth the principle and embodiment of the present invention, the explanation of above example is only intended to
Help to understand method and its core concept of the invention.It should be pointed out that for those skilled in the art,
Without departing from the principles of the invention, some improvement and modification can also be carried out to the present invention, these are improved and modification also falls
Enter in the protection domain of the claims in the present invention.
Claims (10)
1. a kind of method for reading distributed meta data file, it is characterised in that including：
The request for reading file is received, wherein including the required identification information for reading file in the request；
According to the identification information and the first corresponding relation, it is determined that storing the catalogue burst of the file, first correspondence is closed
It is the corresponding relation between the identification information for catalogue burst each described and the file being stored in the catalogue burst；
According to the catalogue burst and the second corresponding relation, the metadata node where the catalogue burst, described second are determined
The catalogue is after createing directory, is split into multiple catalogue bursts by corresponding relation, each described catalogue burst and described
The corresponding relation between each metadata node where catalogue burst；
There is provided by the metadata node and read file response.
2. according to the method described in claim 1, it is characterised in that by the catalogue split into multiple catalogue bursts it
After include：
The file is stored to the quantity of documents each described in catalogue burst, and in each described catalogue burst
In one preset range, first preset range is determined according to quantity of documents total in the catalogue is stored in.
3. method according to claim 2, it is characterised in that described according to the identification information and the first corresponding relation,
It is determined that storing the catalogue burst of the file includes：
According to the identification information of the file, determine to store the catalogue burst of the file by hash algorithm.
4. method according to claim 3, it is characterised in that described that the catalogue is split into multiple catalogue fragment packets
Include：
The catalogue is split into the catalogue burst of the quantity for the metadata node number integer times, and each member
The quantity of the catalogue burst in back end is identical, and records pair between the catalogue burst and place metadata node
It should be related to.
5. method according to claim 3, it is characterised in that also include before the request for reading file is received：
Whether the size of the file of each metadata node storage of detection exceedes predetermined threshold value, if it is, will deposit
Store up quantity of documents exceed predetermined threshold value the metadata node in the catalogue burst, migrate to storage file size not
More than in the metadata node of predetermined threshold value, and record the correspondence pass between the catalogue burst and place metadata node
System.
6. a kind of device for reading distributed meta data file, it is characterised in that including：
Request module is received, the request of file is read for receiving, wherein including the required mark for reading file in the request
Information；
Computing module, for according to the identification information and the first corresponding relation, it is determined that storing the catalogue burst of the file, institute
State the first corresponding relation for each described catalogue burst and the identification information of the file being stored in the catalogue burst it
Between corresponding relation；
Searching modul, for according to the catalogue burst and the second corresponding relation, determining the metadata where the catalogue burst
Node, second corresponding relation is that after createing directory, the catalogue is split into multiple catalogue bursts, each described mesh
The corresponding relation between each metadata node where record burst and the catalogue burst；
Respond module, file response is read for being provided by the metadata node.
7. device according to claim 6, it is characterised in that the computing module includes：
First arithmetic element, for the catalogue being split into after multiple catalogue bursts, by the file store to
In each described catalogue burst, and the quantity of documents in each described catalogue burst is in the first preset range, and described
One preset range is determined according to quantity of documents total in the catalogue is stored in.
8. device according to claim 7, it is characterised in that the computing module includes：
Second arithmetic element, for the identification information according to the file, determines to store the institute of the file by hash algorithm
State catalogue burst.
9. device according to claim 8, it is characterised in that the searching modul includes：
First migration units, for the catalogue to be split into the catalogue of the quantity for the metadata node number integer times
The quantity of the catalogue burst in burst, and each metadata node is identical, and records the catalogue burst and place
Corresponding relation between metadata node.
10. device according to claim 8, it is characterised in that the searching modul includes：
Second migration units, the institute for before the request for reading file is received, detecting each metadata node storage
Whether the size for stating file exceedes predetermined threshold value, if it is, the member of the quantity of documents more than predetermined threshold value will be stored
The catalogue burst in back end, migrates to the size of storage file and is no more than the metadata node of predetermined threshold value
It is interior, and record the corresponding relation between the catalogue burst and place metadata node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710585356.XA CN107315547A (en) | 2017-07-18 | 2017-07-18 | A kind of method and device for reading distributed meta data file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710585356.XA CN107315547A (en) | 2017-07-18 | 2017-07-18 | A kind of method and device for reading distributed meta data file |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107315547A true CN107315547A (en) | 2017-11-03 |
Family
ID=60178277
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710585356.XA Pending CN107315547A (en) | 2017-07-18 | 2017-07-18 | A kind of method and device for reading distributed meta data file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107315547A (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108509275A (en) * | 2018-03-29 | 2018-09-07 | 新华三技术有限公司 | A kind of catalogue moving method and metadata load-balancing method |
| CN109002503A (en) * | 2018-06-29 | 2018-12-14 | 郑州云海信息技术有限公司 | A kind of metadata read method, device, equipment and readable storage medium storing program for executing |
| CN109445694A (en) * | 2018-10-19 | 2019-03-08 | 郑州云海信息技术有限公司 | A kind of distributed memory system separated from meta-data method and apparatus |
| CN110008174A (en) * | 2019-04-11 | 2019-07-12 | 苏州浪潮智能科技有限公司 | A kind of catalogue deriving method, system and relevant apparatus |
| CN111061681A (en) * | 2019-11-15 | 2020-04-24 | 浪潮电子信息产业股份有限公司 | Method and device for partitioning directory based on case insensitivity and storage medium |
| CN111352938A (en) * | 2020-02-26 | 2020-06-30 | 深圳市网心科技有限公司 | Data processing method, computer device and storage medium |
| CN111475108A (en) * | 2020-03-20 | 2020-07-31 | 平安国际智慧城市科技股份有限公司 | Distributed storage method, computer equipment and computer readable storage medium |
| CN111638853A (en) * | 2020-05-08 | 2020-09-08 | 杭州海康威视系统技术有限公司 | Data storage method and device, storage cluster, gateway equipment and main equipment |
| CN111666257A (en) * | 2020-06-03 | 2020-09-15 | 中国建设银行股份有限公司 | File fragment storage method, device, equipment and storage medium |
| CN112100129A (en) * | 2020-09-14 | 2020-12-18 | 北京金山云网络技术有限公司 | Data access method, data storage method, data access device and file storage system |
| CN112667149A (en) * | 2020-12-04 | 2021-04-16 | 北京浪潮数据技术有限公司 | Data heat sensing method, device, equipment and medium |
| CN112988062A (en) * | 2021-01-28 | 2021-06-18 | 腾讯科技（深圳）有限公司 | Metadata reading limiting method and device, electronic equipment and medium |
| CN113703688A (en) * | 2021-09-20 | 2021-11-26 | 河南锦誉网络科技有限公司 | Distributed storage node load adjustment method based on big data and file heat |
| CN115033187A (en) * | 2022-08-10 | 2022-09-09 | 蓝深远望科技股份有限公司 | Big data based analysis management method |
| CN117478669A (en) * | 2023-11-10 | 2024-01-30 | 公安部交通管理科学研究所 | Method for efficiently exchanging data under cross-physical network environment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104580471A (en) * | 2015-01-12 | 2015-04-29 | 浪潮电子信息产业股份有限公司 | Method for improving metadata cluster performance |
| CN104735110A (en) * | 2013-12-23 | 2015-06-24 | 中国电信股份有限公司 | Metadata management method and system |
| CN105701209A (en) * | 2016-01-13 | 2016-06-22 | 广西师范大学 | Load balancing method for improving parallel connection performance on big data |
| CN106599102A (en) * | 2016-11-29 | 2017-04-26 | 郑州云海信息技术有限公司 | Metadata performance improvement method based on catalogue splitting mechanism |
| CN106909472A (en) * | 2015-12-22 | 2017-06-30 | 中国移动通信集团河北有限公司 | The input/output load regulation method and device of a kind of distributed file system |
-
2017
- 2017-07-18 CN CN201710585356.XA patent/CN107315547A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104735110A (en) * | 2013-12-23 | 2015-06-24 | 中国电信股份有限公司 | Metadata management method and system |
| CN104580471A (en) * | 2015-01-12 | 2015-04-29 | 浪潮电子信息产业股份有限公司 | Method for improving metadata cluster performance |
| CN106909472A (en) * | 2015-12-22 | 2017-06-30 | 中国移动通信集团河北有限公司 | The input/output load regulation method and device of a kind of distributed file system |
| CN105701209A (en) * | 2016-01-13 | 2016-06-22 | 广西师范大学 | Load balancing method for improving parallel connection performance on big data |
| CN106599102A (en) * | 2016-11-29 | 2017-04-26 | 郑州云海信息技术有限公司 | Metadata performance improvement method based on catalogue splitting mechanism |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108509275B (en) * | 2018-03-29 | 2019-11-12 | 新华三技术有限公司 | A kind of catalogue moving method and metadata load-balancing method |
| CN108509275A (en) * | 2018-03-29 | 2018-09-07 | 新华三技术有限公司 | A kind of catalogue moving method and metadata load-balancing method |
| CN109002503A (en) * | 2018-06-29 | 2018-12-14 | 郑州云海信息技术有限公司 | A kind of metadata read method, device, equipment and readable storage medium storing program for executing |
| CN109445694B (en) * | 2018-10-19 | 2022-02-18 | 郑州云海信息技术有限公司 | Metadata separation method and device for distributed storage system |
| CN109445694A (en) * | 2018-10-19 | 2019-03-08 | 郑州云海信息技术有限公司 | A kind of distributed memory system separated from meta-data method and apparatus |
| CN110008174A (en) * | 2019-04-11 | 2019-07-12 | 苏州浪潮智能科技有限公司 | A kind of catalogue deriving method, system and relevant apparatus |
| CN111061681A (en) * | 2019-11-15 | 2020-04-24 | 浪潮电子信息产业股份有限公司 | Method and device for partitioning directory based on case insensitivity and storage medium |
| CN111352938B (en) * | 2020-02-26 | 2023-07-18 | 深圳市迅雷网络技术有限公司 | Data processing method, computer device and storage medium |
| CN111352938A (en) * | 2020-02-26 | 2020-06-30 | 深圳市网心科技有限公司 | Data processing method, computer device and storage medium |
| CN111475108A (en) * | 2020-03-20 | 2020-07-31 | 平安国际智慧城市科技股份有限公司 | Distributed storage method, computer equipment and computer readable storage medium |
| CN111475108B (en) * | 2020-03-20 | 2023-11-28 | 深圳赛安特技术服务有限公司 | Distributed storage method, computer equipment and computer readable storage medium |
| CN111638853A (en) * | 2020-05-08 | 2020-09-08 | 杭州海康威视系统技术有限公司 | Data storage method and device, storage cluster, gateway equipment and main equipment |
| CN111666257A (en) * | 2020-06-03 | 2020-09-15 | 中国建设银行股份有限公司 | File fragment storage method, device, equipment and storage medium |
| CN111666257B (en) * | 2020-06-03 | 2024-03-19 | 中国建设银行股份有限公司 | Method, device, equipment and storage medium for file fragment storage |
| CN112100129A (en) * | 2020-09-14 | 2020-12-18 | 北京金山云网络技术有限公司 | Data access method, data storage method, data access device and file storage system |
| CN112667149A (en) * | 2020-12-04 | 2021-04-16 | 北京浪潮数据技术有限公司 | Data heat sensing method, device, equipment and medium |
| CN112667149B (en) * | 2020-12-04 | 2023-12-29 | 北京浪潮数据技术有限公司 | Data heat sensing method, device, equipment and medium |
| CN112988062A (en) * | 2021-01-28 | 2021-06-18 | 腾讯科技（深圳）有限公司 | Metadata reading limiting method and device, electronic equipment and medium |
| CN112988062B (en) * | 2021-01-28 | 2023-02-14 | 腾讯科技（深圳）有限公司 | Metadata reading limiting method and device, electronic equipment and medium |
| CN113703688B (en) * | 2021-09-20 | 2024-03-15 | 安徽丰合佳行信息技术有限公司 | Distributed storage node load adjustment method based on big data and file heat |
| CN113703688A (en) * | 2021-09-20 | 2021-11-26 | 河南锦誉网络科技有限公司 | Distributed storage node load adjustment method based on big data and file heat |
| CN115033187B (en) * | 2022-08-10 | 2022-11-08 | 蓝深远望科技股份有限公司 | Big data based analysis management method |
| CN115033187A (en) * | 2022-08-10 | 2022-09-09 | 蓝深远望科技股份有限公司 | Big data based analysis management method |
| CN117478669A (en) * | 2023-11-10 | 2024-01-30 | 公安部交通管理科学研究所 | Method for efficiently exchanging data under cross-physical network environment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107315547A (en) | A kind of method and device for reading distributed meta data file | |
| CN104679661B (en) | hybrid storage control method and hybrid storage system | |
| CN104408091B (en) | The date storage method and system of distributed file system | |
| CN104850358B (en) | A kind of magneto-optic electricity mixing storage system and its data acquisition and storage method | |
| CN102855294B (en) | Intelligent hash data layout method, cluster storage system and method thereof | |
| CN107302561B (en) | A kind of hot spot data Replica placement method in cloud storage system | |
| CN106446001B (en) | A kind of method and system of the storage file in computer storage medium | |
| CN106406759B (en) | Data storage method and device | |
| CN104536903B (en) | A kind of mixing storage method and system stored classifiedly by data attribute | |
| CN103838853B (en) | Mixed file system based on different storage media | |
| CN103593436A (en) | File merging method and device | |
| CN104899297A (en) | Hybrid index structure with storage perception | |
| CN106775446A (en) | Based on the distributed file system small documents access method that solid state hard disc accelerates | |
| CN103064639A (en) | Method and device for storing data | |
| CN103488685B (en) | Fragmented-file storage method based on distributed storage system | |
| CN105981033B (en) | Placement Strategy is distributed into set of segments | |
| CN102253985B (en) | File system data management method and system | |
| CN106648939A (en) | Method and equipment for inter-process resource sharing | |
| CN109522283A (en) | A kind of data de-duplication method and system | |
| CN108052541A (en) | The realization of file system based on multi-level page-table bibliographic structure, access method, terminal | |
| CN107609140A (en) | A kind of method and device of distributive catalogue of document system file access | |
| CN109471843A (en) | A kind of metadata cache method, system and relevant apparatus | |
| CN102541969B (en) | File protection method and system based on file allocation table (FAT) file system, and memory | |
| CN107340975A (en) | A kind of method and device of file storage | |
| CN102122284B (en) | Compound document storage and read-write method and compound document storage and read-write device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20171103 |
|
| RJ01 | Rejection of invention patent application after publication |