CN103457963B - The method and distributed memory system of storage file - Google Patents
The method and distributed memory system of storage file Download PDFInfo
- Publication number
- CN103457963B CN103457963B CN201210170213.XA CN201210170213A CN103457963B CN 103457963 B CN103457963 B CN 103457963B CN 201210170213 A CN201210170213 A CN 201210170213A CN 103457963 B CN103457963 B CN 103457963B
- Authority
- CN
- China
- Prior art keywords
- storage server
- storage
- server group
- file
- stored
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of method of storage file,Apply in a distributed memory system,Obtain the first level of the first file to be stored from the first client,A first level storage server group corresponding with the first level is determined in the M rank storage server group set from the distributed memory system,A preset strategy is based on again,A first storage server group is determined from the first level storage server group,A preset rules are based on again,A first server is determined from the first storage server group,The first file to be stored of the first level is obtained again,And by first file to be stored storage to the first server,Wherein,Can be synchronized to first file to be stored on each server in other servers of the first storage server group by the first server.
Description
Technical field
Method and distributed document the present invention relates to file field of storage, more particularly to a kind of storage file store system
System.
Background technology
The design of existing distributed file storage system is typically based on following principle：CAP is theoretical:One distribution is deposited
Storage system can not possibly simultaneously meet uniformity, availability and subregion tolerance these three demands, be at best able to meet two simultaneously,
Wherein, C：In distributed environment, the data of multiple spot are consistent to Consistency；A：Each operation of Availability is total
Can it is determined that time return, i.e., system is all at any time available；P：Tolerance of network Partition (subregions
Tolerance) in the case where there is network partition (such as suspension), the system of separation also can normally be run.
Existing distributed memory system, primary demand during subregion tolerance, therefore, two kinds of selections of only CP and AP.CP
Pattern is used to ensure to be distributed in the uniformity of different node datas on network；Ap mode mainly realizes " final consistency " to ensure
Availability and subregion tolerance.
But present inventor is during inventive technique scheme in realizing the embodiment of the present application, above-mentioned technology is found extremely
There is following technical problem less：
One：Existing distributed memory system does not have to the file to be stored and storage during storage file
Server group makes a distinction, but some files are important, and some files for user in the file to be stored
It is hardly important, causes the existing distributed memory system file to be stored can not be made a distinction into storage.
Secondly：Existing distributed memory system is general to be divided using uniformity hash algorithm when storage file
The storage file is stored with a first storage server group, once the first storage server group breaks down, just
All data Cun Chudao first and the first storage server group up time of the first storage server group can be stored
In the second connected storage server group of pin, in this way, the data stored in may result in the second storage server group increase
And the storage pressure increase of the second storage server group, not only promote each storage clothes in the distributed memory system
The data stored in business device group are unbalanced, it is also possible to causing loss of data.
Thirdly:Existing distributed memory system is to record all storage file institutes in place by Tracker (distributor)
Put and path, in causing Tracker substantial amounts of memory space is occupied, once the data stored in Tracker reach necessarily
Quantity when, can facilitate the operating efficiency of Tracker reduces.
It four:The existing distributed memory system general file all to storing deposited again after burst treatment
Storage, if user wants to check this document that can be completed for each burst of this document by the distributed memory system
Afterwards, user can just check this document, but when storing small documents in distributed memory system, can reduce distributed memory system
Performance.
The content of the invention
The embodiment of the present application is right for realizing by providing the method and distributed file storage system of a kind of storage file
Storage file and storage server group carry out classification treatment, realize the storage file storage of different stage to the storage of different stage
In server group, and the file of same rank can be only put into the storage server group of same rank.
The embodiment of the present application provides a kind of method of storage file, applies in a distributed memory system, the side
Method includes：
Obtain the first level of the first file to be stored from the first client；
The M rank storage server group set in the distributed memory system is obtained, including：Based on storage server
The quantity of server in group, determines the M rank storage server group；Wherein, the M is the integer not less than 1；
A first level storage clothes corresponding with the first level are determined from the M rank storage server group
Business device group, wherein, the first level storage server group includes at least one storage server group；
The weight of each storage server group of the first level storage server group is obtained, the weight takes with storage
The available memory space parameter correspondence of device group of being engaged in；
Based on the size of the weight, first storage server is determined from the first level storage server group
Group；Wherein, the first storage server group includes at least one server；
First file to be stored is assigned in the first storage server group；
Based on a preset rules, a first server is determined from the first storage server group；
The first file to be stored of the first level is obtained, and first file to be stored storage is arrived described first
In server, wherein, first file to be stored can be synchronized to first storage server by the first server
On each server in other servers of group.
Optionally, the first level for obtaining the first file to be stored from the first client, specifically includes：Obtain
The related parameter information of first file to be stored；Based on the parameter information, the of first file to be stored is determined
One rank.
Optionally, it is described to obtain the M rank storage server group set in the distributed memory system, specific bag
Include：Obtain the quantity of server in each storage server group in the distributed system；Based on described each storage server
The quantity of server in group, it is determined that the rank of each storage server；Based on the rank of each storage server, obtain
Obtain the M rank storage server group set in the distributed memory system.
Optionally, the weight of each the storage server group for obtaining the first level storage server group, specifically
Including：Obtain the available memory space parameter of each storage server group of the first level storage server group；Based on institute
Available memory space parameter is stated to determine the weight of each storage server group.
Optionally, after by first file to be stored storage to the first server, methods described is also wrapped
Include：The filename of first file to be stored is generated, and the filename is transferred to first client, for described
First client is stored, wherein, the filename includes that the first server position and the first server belong to
The first storage server group position.
Optionally, after first file to be stored is obtained, methods described also includes：It is to be stored by described first
File is stored as an entirety.
Optionally, it is described to be based on a preset rules, a first server is determined from the first storage server group,
Specially：A second server is randomly choosed in the first storage server group, the second server is defined as
The first server.
Based on the above method, the embodiment of the present application additionally provides a kind of storage system of distributed document, described
System includes：M rank storage server group, wherein, each rank storage server in the M rank storage server group
Group includes at least one storage server group, and one storage server group includes at least one server, and M is not less than 1
Integer；
Distributor, is connected with each server group in the M rank storage server group, for obtaining from the first visitor
The first level of first file to be stored at family end, determines one and described first from the M rank storage server group
The corresponding first level storage server group of rank, the distributor is additionally provided with Weight Acquisition module, for obtaining described
The available storage of the weight of each storage server group of one rank storage server group, the weight and storage server group
Spatial parameter is related, and based on the size of the weight, the distributor determines one from the first level storage server group
Individual first storage server group；Wherein, the first storage server group includes at least one server, and described first is waited to deposit
Storage file is assigned in the first storage server group, and based on a preset rules, from the first storage server group
Determine a first server；
Memory, the first file to be stored for obtaining the first level, and first file to be stored is deposited
In storing up the first server, wherein, first file to be stored can be synchronized to described by the first server
On each server of one storage server group.
Optionally, the system is additionally provided with storage file rank determination module, is arranged in first client, uses
In the related parameter information of acquisition first file to be stored, and based on the parameter information, determine that described first is to be stored
The first level of file.
Optionally, the system also includes server rank setup module, obtains each storage in the distributed system
The quantity of server in server group, and the quantity based on server in described each storage server group, it is determined that it is described each
The rank of storage server group, based on the rank of each storage server group, obtains the M rank storage server
Group.
Optionally, the Weight Acquisition module also includes memory space acquisition submodule, for obtaining the first level
The available memory space parameter of each storage server group of storage server group, the weight of each storage server group is
Determined based on the available memory space parameter.
Optionally, the system also includes：Filename generation unit, arrives described by first file to be stored storage
After first server, for generating the filename of first file to be stored, and the filename is transferred to described first
Client is stored, wherein, the filename include that the first server position and the first server belong to the
The position of one storage server group.
Optionally, the system also includes：Storage file arranges unit, after first file to be stored is obtained,
For being stored first file to be stored as an entirety.
Optionally, the distributor is additionally provided with selecting module, for being selected at random in the first storage server group
A second server is selected, the second server is defined as the first server.
One or more technical schemes provided in the embodiment of the present application, at least have the following technical effect that or advantage：
One, due to the embodiment of the present application to each the storage server group in distributed memory system and each treat
Storage file is all classified, and the file to be stored of different stage is corresponded into different stage storage server group, in this way,
Existing distributed system is efficiently solved because not made a distinction to file to be stored and storage server group, so as to cause not
The file to be stored can be made a distinction the problem of storage, realized the storage file storage of different stage to different stage
Storage server group in, and the file of same rank can be only put into the storage server group of same rank.
Secondly, be each the storage server group by obtaining the first level storage server group due to the present invention
Memory space determines the weight of each storage server group, then based on the weight, from the first level storage server
Determine a first storage server group in group to store first file to be stored, therefore, it is possible to effectively solve described point
The data stored in each storage server group in cloth storage system are unbalanced and lose the problem of data, so as to realize described
The data balancing distribution stored in each storage server group in distributed memory system, and reduce loss of data.
Thirdly, due to the filename of each storage file of distributed memory system of the present invention be storage to correspondence client
, and the filename includes the corresponding server position and its path where described each storage file, thus it is possible to have
Effect solves the storage file name and takes the substantial amounts of memory space of distributor and reduce the operating efficiency of the distributor,
Promote the distributor only to be parsed to filename, improve the operating efficiency of the distributor.
Four, because first file to be stored is not carried out burst by the present invention, but directly described first is waited to deposit
Storage file stored as an entirety, therefore, it is possible to effectively solve existing distributed memory system reduce storage file when,
Need multiple bursts to re-assembly the problem of complexity and expense together, realize when carrying out small documents and storing, carry
The storage performance of distributed system of the present invention high.
Brief description of the drawings
Fig. 1 is the flow chart of the method for storage file in the embodiment of the present application；
Fig. 2 is the system architecture figure of distributed memory system in the embodiment of the present application.
Specific embodiment
The embodiment of the present application is solved existing by providing the method and distributed file storage system of a kind of storage file
Because not made a distinction to file to be stored and storage server group, causing can not will be described to be stored for distributed memory system
File makes a distinction the problem of storage, realizes the storage file storage of different stage to the storage server group of different stage
In, and the file of same rank can be only put into the storage server group of same rank.
The technical scheme of the embodiment of the present application is to solve above-mentioned differentiation storage problem, and general thought is as follows：
It is classified by by each storage server group in distributed memory system, and first treating to the first client
Storage file is also carried out classification, based on the rank of first file to be stored, the storage from the distributed memory system
A first level storage server group corresponding with the rank of first file to be stored is determined in the rank of server group；
A preset strategy is based on again, and a first storage server group is determined from the first level storage server group；It is based on again
One preset rules, select a first server to store the described first text to be stored from the first storage server group
Part, and first file to be stored is synchronized to each server in the first storage server group in other servers
On.
Above-mentioned first file to be stored when being stored, be based on first file to be stored rank come
This is stored in the storage server group of which rank to determine first file to be stored, is realized and is deposited the file of different stage
In storing up the storage server group of different stage, and the file of same rank can only be put into the storage server group of same rank
In.
In order to be better understood from above-mentioned technical proposal, below in conjunction with Figure of description and specific embodiment to upper
Technical scheme is stated to be described in detail.
The invention provides a kind of method of storage file, apply in a distributed memory system, as shown in figure 1, bag
Include following steps：
S101, the first level for obtaining the first file to be stored from the first client：
Specifically, will be to the service of the distributed memory system when first client needs data storage
Device sends one first storage request, and the first storage request bag contains the size and described first of first file to be stored
The information such as the first level of file to be stored；After the server receives the first storage request, described the is parsed
As soon as storage request, can obtain the first level of the first file to be stored from the first client.
Specifically, the first level of first file to be stored is determined by first client.Described
When first client is stored first file to be stored, it is to be stored that first client can obtain described first
The related parameter information of file；The parameter information is based on again, determines the first level of first file to be stored.
In implementing, first client can be analyzed to the content of first file to be stored, be based on
Advance setting, determines the importance of the business and first file to be stored belonging to first file to be stored, such as
This, just obtains the content of first file to be stored, the business of first file to be stored and described first waits to deposit
Store up the related parameter information such as importance of file.Such as：If it is some landscape that customer end A needs the content of the file B of storage
The fragmented information such as picture or exquisite wallpaper, based on advance setting, it may be determined that the file B is personal business, and important
Property is relatively low；If customer end A needs the personal information, base such as name, age and the telephone number that the content of the file C of storage is client
In advance setting, it may be determined that the file C is individual privacy business, and importance highest；If customer end A needs storage
The content of file D is the working document of client, based on advance setting, it may be determined that the file D is personal work business, and
Importance is higher；If it is the information of the job placement of client that customer end A needs the content of the file E of storage, set based on advance
Put, it may be determined that the file E is personal work business, and importance is high.In this way, based in first file to be stored
Hold and the advance setting, can necessarily determine the business and first text to be stored of first file to be stored
The importance of part.
Table 1
| File to be stored | B | C | D | E |
| Rank | Fourth level | First level | Second level | Third level |
In implementing, first client obtains the related parameter of first file to be stored and believes by analysis
After breath, the first level of first file to be stored is determined based on the parameter information.The embodiment of the present application treats each
Storage file determines a corresponding level, and the corresponding level of each file to be stored is based on described each file to be stored
Related parameter determines, such as：By taking above-mentioned steps S102 given examples as an example, as shown in table 1, if customer end A needs storage
The content of file B be fragmented information, and it is personal business that the file B is determined based on advance setting, and importance
It is relatively low, in this way, determining that the file B is fourth level；If it is personal information that customer end A needs the content of the file C of storage, with
And the file C is determined for individual privacy business based on advance setting, and importance highest, in this way, determining the file C
It is first level；If customer end A needs the content of the file F of storage for the working document of client, and based on advance setting
The file D is determined for personal work business, and importance is higher, in this way, determining that the file D is second level；If objective
It is the information of the job placement of client that family end A needs the content of the E of storage, and the file is determined based on advance setting
E is personal work business, and importance is high, in this way, determining that the file E is third level.Certainly, in implementation process, can
Determine the first level of first file to be stored to set different ranks according to different situations.Here, just no longer
Schematically illustrate one by one, but either use any mode, as long as being equal to technical scheme in the embodiment of the present application, belong to this patent
Within claims.
S102, M rank storage server group for obtaining setting in the distributed memory system, wherein, the M is not for
Integer less than 1：
Specifically, the embodiment of the present application has carried out point each storage server group in the distributed memory system
Level, so as to need to be determined the quantity of the rank of storage server group in the distributed memory system, so as to determine by
First file to be stored is stored in the storage server group of appropriate level, wherein, the storage server group bag of each rank
At least one storage server group is included, each storage server group includes at least one server.
Table 2
| Storage server group | The quantity of server | The rank of storage server group |
| a | 3 | Fourth level |
| b | 3 | Fourth level |
| c | 5 | Third level |
| d | 5 | Third level |
| e | 8 | Second level |
| f | 8 | Second level |
| g | 12 | First level |
| h | 12 | First level |
In implementing, it is necessary in obtaining each storage server group server quantity, based on the quantity, it is determined that
The rank of each storage server group, such as, as described in table 2, due to server in a storage servers group and b storage service groups
Quantity all be 3, therefore, a storage servers group and b storage service groups belong to fourth level storage server group；Together
Reason, the c storage servers group and d storage service groups belong to third level storage server group；The e storage servers group and
F storage service groups belong to second level storage server group；The g storage servers group and h storage service groups belong to the first order
Other storage server group.Knowable to the content of above-mentioned table 1, the quantity of server is more in the storage server group, storage clothes
The rank of device group of being engaged in is also higher；The quantity of server is fewer in the storage server group, the rank of the storage server group
It is lower.Equally, according to the quantity of server in the storage server group, it is also possible to which the rank to the storage server group is entered
Row reversely setting, belongs to fourth level storage server group, by institute than g storage servers group as will be described and h storage service groups
State a storage servers group and b storage service groups belong to first level storage server group.Here, just no longer schematically illustrate one by one, but
Any mode is either used, as long as being equal to technical scheme in the embodiment of the present application, patent claims protection is belonged to
Within the scope of.
Additionally, the storage server group that the embodiment of the present application can also will include varying number is arranged on same rank and deposits
In storage server group, such as：The storage server group that 1 to 4 servers will be included is set to fourth level storage server
Group.
S103, one first level corresponding with the first level of determination from the M rank storage server group
Storage server group：
Specifically, it is the first level based on first file to be stored, from the M rank storage server group
Determine a first level storage server group corresponding with the first level, wherein, the first level storage server
Group includes at least one storage server group.In the embodiment of the present application, by the rank of first file to be stored with the M
The rank of rank storage server group is mapped, for the storage file of same rank can only to be stored into depositing for same rank
In storage server group, and by the storage file storage of different stage to the storage server group of different stage.
In implementing, based on the first level, from the M rank storage server group determine one with it is described
The corresponding first level storage server group of first level, such as：As shown in Table 1 and Table 2, it is described based on the file B
Four ranks, and first level storage server group corresponding with the fourth level is fourth level storage server group, so that
Determine the fourth level storage service group to store the file B；Similarly, the file C correspondences first level storage
Server group；The file D correspondences second level storage server group；The file E correspondences third level storage
Server group.Certainly, the difference of the grade setting according to the file to be stored, and the rank of the storage server group sets
Fixed difference, then the file to be stored is also different in the corresponding relation of the storage server group.Here, just no longer one one action
Example, but any mode is either used, as long as being equal to technical scheme in the embodiment of the present application, belonging to this patent right will
Ask within protection domain.
S104, based on a preset strategy, from the first level storage server group determine first storage service
Device group, wherein, the first storage server group includes at least one server：
Specifically, after determining the first level storage server group of first file to be stored, due to
One rank storage server group includes at least one storage server group, accordingly, it would be desirable to a preset strategy is based on, from described first
A first storage server group is determined in rank storage server group, for promoting the first level storage server group in
The data distribution that each storage server group is preserved is balanced, wherein, the preset strategy uses following methods：
The first step：Obtain the weight of each storage server group of the first level storage server group；
Specifically, the acquisition methods of the weight of each storage server group, comprise the following steps：
Memory space to each storage server group detects that each storage server group can described in acquisition
Using memory space parameter, the free memory parameter includes the appearance of the free memory of each storage server group
Amount, the weight of each storage server group is determined based on the free memory parameter.
Table 3
| First level storage server group | Free memory | Weight |
| g | 100G | 100/ (100+300)=0.25 |
| h | 300G | 300/ (100+300)=0.75 |
In implementing, the free memory parameter based on each storage server group is deposited determining each
The weight of server group is stored up, such as：By taking first level storage server group as an example, as described in Table 3, first level storage server
Group includes g storage servers group and h storage server groups, wherein, the available of Servers-all is deposited in the g storage servers group
Storage space is 100G, and the free memory of Servers-all is 300G in the h storage servers group.In this way, the g storages
The weight of server group is 0.25 and the weight of the h storage servers group is 0.75.
Second step, based on the weight, determines first storage service from the first level storage server group
Device group：
Specifically, it is exactly size according to weight, first file to be stored is assigned to corresponding described first
In storage server group, for promoting in the first level storage server group data balancing point in each storage server group
Match somebody with somebody.
During implementing, such as：By taking the file C as an example, the file C should be stored in the first level and deposit
In storage server group, in this way, during the file C will store the g storage servers group or the h storage servers group,
It can be seen from the table 3, the weight of the g storage servers group is 0.25 and the weight of the h storage servers group is 0.75,
In this way, 1~25 numeral can be distributed to the g storage servers group by the distributed memory system, and by 26~100 numeral
The h storage servers group is distributed to, then, the distributed memory system can randomly choose one inside 1~100 numeral
Individual numeral, if the numeral of the selection is in the range of 1~25, by file C storages to the g storage servers group
In, otherwise, then by file C storages to the h storage servers group.So, the file C storages to the g are caused
Storage server group has 25% probability, and stores the probability for having 75% to the h storage servers group, with the first order
The data stored in other storage server group are more, the number stored in the g storage servers group and the h storage servers group
According to will be more and more balanced.
S105, based on a preset rules, in the first storage server group determine a first server：
Specifically, because each storage server group includes at least one server, accordingly, it would be desirable to be based on a default rule
Then, a first server is determined from described each storage server group, for storing first file to be stored.
The rule is general by the way of being randomly assigned, such as：By file C storages to the g storage servers
During group, a server is then randomly choosed in 12 servers in the g storage servers group to store described first
File to be stored.Certainly, the rule can also be taken using uniformity Kazakhstan property algorithm or other algorithms from the described first storage
Determine a first server in business device group.
S106, based on first file to be stored and the first server, by first file to be stored storage
To in the first server：
Specifically will, once it is determined that the first server that first file to be stored is stored, will obtain institute
The store path and address information of first server are stated, it is to be stored by described first based on the store path and address information
File is stored in the first server.
S107, in first file to be stored storage to after the first server, the first server can be by
First file to be stored is synchronized on each server of the first storage server group：
Specifically, by each storage service of the storage belonging to the data syn-chronization to be stored to the server in server
In device group, in this way, promoting to carry out redundancy backup to the data to be stored.Such as, the file C is storage to the first order
In other storage server group, and the number of the server of each storage server group is in the first level storage server group
12；And during the file B can only store the fourth level storage server group, but the fourth level storage service
The server number of each storage server group is 3 in device group, in this way, not only can be by more redundancy backups the need for differentiation
File carry out more redundancy backups, to prevent loss of data, also can use will distinguish the file for not needing more redundancy backups
The quantity of backup is reduced, to save memory space.
Additionally, after by first file to be stored storage to the first server, generating described first to be stored
The filename of file, and the filename is transferred to first client is stored, wherein, the filename includes institute
State the position of the first storage server group of first server position and first server ownership.In this way, the distribution
Storage system need not just store all storages to the position of the storage file of the distributed memory system, save the distribution
The memory space of formula storage system, only carries out parsing the path that can be obtained by the storage file and storage position to filename
Put, there is provided operating efficiency；In addition, when there is a client to need to download first file to be stored, and described first treats
When first server belonging to storage file breaks down or is unreachable, can be by the filename of first file to be stored
Selection is organized other servers and is downloaded together.
Furthermore, the embodiment of the present application is by first file to be stored after first file to be stored is obtained
Stored as an entirety, because what general the Internet, applications were commonly stored is small documents, by described first
File to be stored as an entirety, eliminate needs after the first file to be stored burst again by multiple bursts again
The complexity and expense for fitting together, so as to improve the performance of the distributed memory system.
On this basis, the embodiment of the present application additionally provides a kind of storage system of distributed document, as shown in Fig. 2 bag
Include：
Taken by the storage of level1 rank storage server groups and level2 rank storage server groups and other ranks
Business device group collectively constitutes the M storage server group of rank, wherein, each rank is deposited in the M rank storage server group
Storage server group includes at least one storage server group, and one storage server group includes at least one server, and M is
Integer not less than 1；
Distributor (track1), is connected, with each server group in the M rank storage server group for being come
From the first level of the first file to be stored of the first client, from the M rank storage server group determine one with
The corresponding first level storage server group of the first level, then based on a preset strategy, stored from the first level and taken
Determine a first storage server group in business device group, and based on a preset rules, from the first storage server group really
A fixed first server；
Memory (server is equivalent to a memory), the first text to be stored for obtaining the first level
Part, and first file to be stored is stored in the first server, wherein, the first server can be described
First file to be stored is synchronized on each server of the first storage server group.
If shown in Fig. 2,3 storage server groups are provided with the level1 ranks storage server group, wherein,
Group storage servers group includes 2 servers (server), the number based on server in the group storage servers group
Measure to determine that the group storage servers group belongs to the level1 ranks storage server group, i.e., described M rank storage
The rank of each storage server is determined based on its quantity of server in each storage server group in server group.
Wherein, the system is additionally provided with storage file rank determination module, is arranged in first client, is used for
The related parameter information of first file to be stored is obtained, and based on the parameter information, determines the described first text to be stored
The first level of part, wherein, the parameter information includes business belonging to first file to be stored, described first to be stored
The content of the importance of file and first file to be stored.
Wherein, the distributor is additionally provided with Weight Acquisition module, for obtaining the first level storage server group
Each storage server group weight, from least first level storage server group determine first storage service
Device group is determined based on the weight.And the Weight Acquisition module also includes memory space acquisition submodule, for obtaining
Take the available memory space parameter of each storage server group of the first level storage server group, described each storage clothes
The weight of device group of being engaged in is determined based on the available memory space parameter.In this way, can promote to wait to deposit by described first
Storage file has bigger probability storage to the depositing with more free memories in the storage server group of same rank
In storage server group, promote the distribution of the data storage in the storage server group of the same rank more balanced.
The distributed system also includes：Filename generation unit, arrives described by first file to be stored storage
After first server, for generating the filename of first file to be stored, and the filename is transferred to described first
Client is stored, wherein, the filename includes that the first server position and the first server belong to
The first storage server group position.Such as：After the server (server) stores a storage file, it is determined that described
The filename of storage file, the filename include position of the server (server) in the distributed system and
The path of the server (server), client is returned to by the filename, and client stores the filename.In this way,
When the client needs to read the file, the filename of the storage file is transferred to distributor (track1), point
Orchestration (track1) is parsed to the filename, determines the position and path of the storage file storage, and based on institute
Rheme is put and path, obtains the storage file, promotes the distributor (track1) only to turn into one to filename
Resolver, therefore the distributor (track1) meeting running speed is faster.
The system also includes：Storage file arranges unit, after first file to be stored is obtained, for by institute
State the first file to be stored to be stored as an entirety, the storage of burst do not carried out to first file to be stored,
But each described first file to be stored is stored as an entirety, because general the Internet, applications are usual
Storage is small documents (such as picture), this not burst, eliminates the complexity that needs that multiple pieces are fitted together again and opens
Pin, actually increases performance.
Wherein, the distributor is additionally provided with selecting module, for being randomly choosed in the first storage server group
One second server, the first server is defined as by the second server.The selecting module can also be by one
Cause property hash algorithm or other algorithms to select a second server from the first storage server group, and by described the
Two servers are defined as the first server.
As shown in Fig. 2 additionally, the distributor (track1) has also carried out Hot Spare, once the distributor
(track1) go wrong, the distributor (track2) may proceed to offer service, for ensuring the distributed memory system
Stability.
Furthermore, the distributor (track1) is additionally provided with control module, can dynamically be added and be deleted with command mode
Except server and dynamic addition and deletion server group, in this way, being reduced to one in the available memory space of the server
When determining degree or the server due to other reasonses it is non-serviceable when, for by the server from described point
Deleted in cloth system, then add other servers and add the distributed system, for ensuring the distributed system in,
The data of server are constant.
Technical scheme in above-mentioned the embodiment of the present application, at least has the following technical effect that or advantage：
1st, due to the embodiment of the present application to each the storage server group in distributed memory system and each wait to deposit
Storage file is all classified, and the file to be stored of different stage is corresponded into different stage storage server group, in this way, having
Effect solves existing distributed system because not made a distinction to file to be stored and storage server group, so as to cause can not
The file to be stored is made a distinction the problem of storage, is realized and different stage is arrived into the storage file storage of different stage
In storage server group, and the file of same rank can be only put into the storage server group of same rank.
2nd, because the present invention is by obtaining depositing for each storage server group of the first level storage server group
Space is stored up to determine the weight of each storage server group, then based on the weight, from the first level storage server group
It is middle to determine a first storage server group to store first file to be stored, therefore, it is possible to effectively solve the distribution
The data stored in each storage server group in formula storage system are unbalanced and lose the problem of data, so as to realize described point
The data balancing distribution stored in each storage server group in cloth storage system, and reduce loss of data.
3rd, because the filename of each storage file of distributed memory system of the present invention is storage to correspondence client,
And the filename includes the corresponding server position and its path where described each storage file, thus it is possible to effectively
Solve the storage file name to take the substantial amounts of memory space of distributor and reduce the operating efficiency of the distributor, promote
The distributor is set only to be parsed to filename, there is provided the operating efficiency of the distributor.
4th, because first file to be stored is not carried out burst by the present invention, but directly by the described first text to be stored
Part is stored as an entirety, therefore, it is possible to effectively solve existing distributed memory system reduce storage file when, it is necessary to
Multiple bursts are re-assemblied the problem of complexity and expense together, realized when carrying out small documents and storing, improve
The storage performance of distributed system of the present invention.
, but those skilled in the art once know basic creation although preferred embodiments of the present invention have been described
Property concept, then can make other change and modification to these embodiments.So, appended claims are intended to be construed to include excellent
Select embodiment and fall into having altered and changing for the scope of the invention.
Obviously, those skilled in the art can carry out various changes and modification without deviating from essence of the invention to the present invention
God and scope.So, if these modifications of the invention and modification belong to the scope of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to comprising these changes and modification.
Claims (14)
1. a kind of method of storage file, applies in a distributed memory system, it is characterised in that methods described includes：
Obtain the first level of the first file to be stored from the first client；
The M rank storage server group set in the distributed memory system is obtained, including：Based in storage server group
The quantity of server, determines the M rank storage server group；Wherein, the M is the integer not less than 1；
A first level storage server corresponding with the first level is determined from the M rank storage server group
Group, wherein, the first level storage server group includes at least one storage server group；
Obtain the weight of each storage server group of the first level storage server group, the weight and storage server
The available memory space parameter of group is related；
Based on the size of the weight, a first storage server group is determined from the first level storage server group；
Wherein, the first storage server group includes at least one server；
First file to be stored is assigned in the first storage server group；
Based on a preset rules, a first server is determined from the first storage server group；
The first file to be stored of the first level is obtained, and the first service is arrived into first file to be stored storage
In device, wherein, first file to be stored can be synchronized to the first storage server group by the first server
On each server in other servers.
2. the method for claim 1, it is characterised in that first file to be stored of the acquisition from the first client
First level, specifically include：
Obtain the related parameter information of first file to be stored；
Based on the parameter information, the first level of first file to be stored is determined.
3. the method for claim 1, it is characterised in that M set in the acquisition distributed memory system
Rank storage server group, specifically includes：
Obtain the quantity of server in each storage server group in the distributed system；
Based on the quantity of server in described each storage server group, it is determined that the rank of each storage server；
Based on the rank of each storage server, obtain the M rank set in the distributed memory system and deposit
Storage server group.
4. the method for claim 1, it is characterised in that each of the acquisition first level storage server group
The weight of storage server group, specifically includes：
Obtain the available memory space parameter of each storage server group of the first level storage server group；
The weight of each storage server group is determined based on the available memory space parameter.
5. the method for claim 1, it is characterised in that by first file to be stored storage to the described first clothes
After in business device, methods described also includes：
The filename of first file to be stored is generated, and the filename is transferred to first client, for institute
The first client is stated to be stored, wherein, the filename includes that the first server position and the first server are returned
The position of the first storage server group of category.
6. the method for claim 1, it is characterised in that after first file to be stored is obtained, the side
Method also includes：Stored first file to be stored as an entirety.
7. the method for claim 1, it is characterised in that described to be based on a preset rules, from first storage service
A first server is determined in device group, specially：
A second server is randomly choosed in the first storage server group, the second server is defined as described
First server.
8. a kind of storage system of distributed document, it is characterised in that the system includes：
M rank storage server group, wherein, each rank storage server group bag in the M rank storage server group
At least one storage server group is included, one storage server group includes at least one server, and M is whole not less than 1
Number；
Distributor, is connected with each server group in the M rank storage server group, and the first client is come from for obtaining
The first file to be stored first level, from the M rank storage server group determine one with the first level
Corresponding first level storage server group, the distributor is additionally provided with Weight Acquisition module, for obtaining the first order
The available memory space of the weight of each storage server group of other storage server group, the weight and storage server group
Parameter is related, and based on the size of the weight, the distributor determines one the from the first level storage server group
One storage server group；Wherein, the first storage server group includes at least one server, by the described first text to be stored
Part is assigned in the first storage server group, and based on a preset rules, is determined from the first storage server group
One first server；
Memory, the first file to be stored for obtaining the first level, and the first file to be stored storage is arrived
In the first server, wherein, first file to be stored can be synchronized to described first and deposited by the first server
Store up on each server of server group.
9. system as claimed in claim 8, it is characterised in that the system is additionally provided with storage file rank determination module,
It is arranged in first client, the parameter information related for obtaining first file to be stored, and based on the ginseng
Number information, determines the first level of first file to be stored.
10. system as claimed in claim 8, it is characterised in that the system also includes server rank setup module, obtains
In the distributed system in each storage server group server quantity, and based in described each storage server group take
The quantity of business device, it is determined that the rank of each storage server group, based on the rank of each storage server group, obtains
The M rank storage server group.
11. systems as claimed in claim 8, it is characterised in that the Weight Acquisition module also includes that memory space obtains son
Module, the available memory space parameter of each storage server group for obtaining the first level storage server group, institute
The weight for stating each storage server group is determined based on the available memory space parameter.
12. systems as claimed in claim 8, it is characterised in that the system also includes：
Filename generation unit, it is described for generating after by first file to be stored storage to the first server
The filename of the first file to be stored, and the filename is transferred to first client is stored, wherein, the text
Part name includes the first of the first server position of first file to be stored storage and first server ownership
The position of storage server group.
13. systems as claimed in claim 8, it is characterised in that the system also includes：Storage file arranges unit, is obtaining
After taking first file to be stored, for being stored first file to be stored as an entirety.
14. systems as claimed in claim 8, it is characterised in that the distributor is additionally provided with selecting module, for described
A second server is randomly choosed in first storage server group, the second server is defined as the first service
Device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210170213.XA CN103457963B (en) | 2012-05-28 | 2012-05-28 | The method and distributed memory system of storage file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210170213.XA CN103457963B (en) | 2012-05-28 | 2012-05-28 | The method and distributed memory system of storage file |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103457963A CN103457963A (en) | 2013-12-18 |
| CN103457963B true CN103457963B (en) | 2017-06-27 |
Family
ID=49739915
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210170213.XA Active CN103457963B (en) | 2012-05-28 | 2012-05-28 | The method and distributed memory system of storage file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103457963B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105912612B (en) * | 2016-04-06 | 2019-04-05 | 中广天择传媒股份有限公司 | A kind of distributed file system and the data balancing location mode towards the system |
| CN111208934B (en) | 2018-11-21 | 2021-07-09 | 华为技术有限公司 | Data storage method and device |
| CN109413207A (en) * | 2018-12-11 | 2019-03-01 | 深圳市网心科技有限公司 | A kind of file uploading method, system, device and computer readable storage medium |
| CN111913909B (en) * | 2019-05-08 | 2024-02-23 | 厦门网宿有限公司 | Repartitioning method and system in distributed storage system |
| CN112527187B (en) * | 2019-12-24 | 2024-01-26 | 许昌学院 | Distributed online storage system and method for personal users |
| CN113704024B (en) * | 2021-08-27 | 2023-05-09 | 腾讯科技（上海）有限公司 | Data processing method, device, related equipment and storage medium |
| CN113986116A (en) * | 2021-09-07 | 2022-01-28 | 广东珠江智联信息科技股份有限公司 | Distributed storage system and data management method based on distributed storage system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1828556A (en) * | 2005-03-05 | 2006-09-06 | 腾讯科技（深圳）有限公司 | Method, system and apparatus for data memory |
| CN1972280A (en) * | 2005-11-25 | 2007-05-30 | 华为技术有限公司 | Communication system and method with classification service function |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1317711A1 (en) * | 2000-08-11 | 2003-06-11 | 3Ware, Inc. | Architecture for providing block-level storage access over a computer network |
| CN101201801B (en) * | 2006-12-11 | 2010-12-29 | 南京理工大学 | Classification storage management method for VOD system |
| CN101686123B (en) * | 2008-09-24 | 2012-01-25 | 中国移动通信集团公司 | Method and system for managing key, method and device for generating and authenticating key |
-
2012
- 2012-05-28 CN CN201210170213.XA patent/CN103457963B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1828556A (en) * | 2005-03-05 | 2006-09-06 | 腾讯科技（深圳）有限公司 | Method, system and apparatus for data memory |
| CN1972280A (en) * | 2005-11-25 | 2007-05-30 | 华为技术有限公司 | Communication system and method with classification service function |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103457963A (en) | 2013-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103457963B (en) | The method and distributed memory system of storage file | |
| CN105511801B (en) | The method and apparatus of data storage | |
| EP1649387B1 (en) | Distributed database system | |
| CN110169008B (en) | Distributed data redundancy storage method based on consistent hash algorithm | |
| CN102855294B (en) | Intelligent hash data layout method, cluster storage system and method thereof | |
| US8880502B2 (en) | Searching a range in a set of values in a network with distributed storage entities | |
| US9225593B2 (en) | Methods of structuring data, pre-compiled exception list engines and network appliances | |
| CN105138678B (en) | A kind of reading, distribution method and the reading of map tile data, dissemination system | |
| CN102880705B (en) | Database key generating apparatus and database key generation method | |
| EP1498831A2 (en) | Data allocation in a distributed storage system | |
| CN107426258A (en) | A kind of upload of image file and method for down loading and device | |
| Dai et al. | A new replica placement policy for hadoop distributed file system | |
| CN100578469C (en) | Storage and polling method and storage controller and polling system | |
| CN105357322B (en) | A kind of virtual machine distribution method based on Topology partition | |
| Park | A generalization of multiple choice balls-into-bins | |
| CN102004789A (en) | Application method of uniform/universal resource locator (URL) filter system | |
| CN108073641A (en) | The method and apparatus for inquiring about tables of data | |
| CN102378407A (en) | Object name resolution system and method in internet of things | |
| CN111309260B (en) | Data storage node selection method | |
| CN103546522B (en) | Determine the method and distributed memory system of storage server | |
| CN105871726A (en) | Mode matching method for dynamically adding tree node and unit based on common prefix | |
| CN113205424B (en) | Method and apparatus for deploying and invoking contracts in blockchain | |
| Mense et al. | Spread: An adaptive scheme for redundant and fair storage in dynamic heterogeneous storage systems | |
| CN106960052A (en) | A kind of collage-credit data acquisition method and system | |
| CN1581188A (en) | Storage managment method and system for large capacity electronic mail |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |