CN106527960A - Management method for multi-memory disk loads, device, document system and memory network system - Google Patents
Management method for multi-memory disk loads, device, document system and memory network system Download PDFInfo
- Publication number
- CN106527960A CN106527960A CN201510582124.XA CN201510582124A CN106527960A CN 106527960 A CN106527960 A CN 106527960A CN 201510582124 A CN201510582124 A CN 201510582124A CN 106527960 A CN106527960 A CN 106527960A
- Authority
- CN
- China
- Prior art keywords
- storage dish
- storage
- file
- factor
- dish
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a management method for multi-memory disk loads, a device, a document system and a memory network system. The method comprises the steps that a memory disk list containing each memory disk recognition mark is acquired at first; after receiving of a document access request, document full-path information in the document access request is extracted; and according to the document full-path information and the recognition mark of each memory disk, a hash algorithm is used to select one memory disk and take the memory disk as a target memory disk to be accessed by the document access request. According to the invention, the hash algorithm is used to achieve a multi-disk load balance mechanism; massive documents can be distributed on multiple disks very evenly; metadata does not need to exist; the system structure becomes very simple and efficient; only low requirements are proposed for hardware (mainly the memory); no metadata exists; single-point failures caused by metadata damage can be avoided; and safety of system storage can be enhanced.
Description
Technical field
The present invention relates to the communications field, and in particular to a kind of many storage dish load management methods, device, file
System and logical volume transfer method and storage network system.
Background technology
With hardware designs and the lifting of manufacturing process, present server product can often extend more
Storage dish (mechanical type hard disk or solid state hard disc are below collectively referred to as " polydisc "), how efficient utilization polydisc
Formed one can towards " polydisc load balance, high concurrent, high-throughput " storage service system,
People have carried out many designs and have realized.Currently, the angle from " polydisc load balance " is said, most of
Traditional mode is to provide one piece of meta-data region (i.e. metadata), in meta-data region completes multivolume file
Balanced access, will position uniform mapping of the file on polydisc in meta-data region, each file path is looked into
Look for, after meta-data region finds file physical location, carry out actual access
Operation.In order to safeguard the metadata, need to be additionally provided metadata controller, metadata controller is in storage
Substantial amounts of cpu resource (problem brought is that cpu performance is lifted, and increases cost) can be expended during system busy, together
When, with sharply increasing for quantity of documents, a large amount of valuable physical memories can be expended in meta-data region, and (that what is brought asks
Topic is internal memory dilatation, increased cost again), even using most simplifying efficient data structure, meta-data region
Memory cost it is still very important.On the other hand, if meta-data region is damaged, or metadata controller
Collapse, then mean " paralysis " of system.
It can be seen that, it is existing to there is big expense, high cost and unit by way of polydisc load balance is realized in meta-data region
Data field failure causes the problem of systemic breakdown.
The content of the invention
The main technical problem to be solved in the present invention is to provide a kind of many storage dish load management method and apparatus,
Solution is existing to have big expense, high cost and metadata by way of polydisc load balance is realized in meta-data region
Area's failure causes the problem of systemic breakdown.
For solving above-mentioned technical problem, the present invention provides a kind of many storage dish load management methods, including：
Obtain storage dish list, identification marking of the storage dish list comprising each storage dish；
File access request is received, the file full path footpath information in the file access request is obtained；
According to file full path footpath information and the identification marking of each storage dish, using hashing algorithm from institute
A target storage dish accessed as the file access request is selected in stating storage dish.
In an embodiment of the present invention, the identification mark according to file full path footpath information and each storage dish
Know, a target accessed as the file access request is selected from the storage dish using hashing algorithm
Storage dish includes：
To the identification marking of each storage dish by hashing algorithm process obtain the storage medium of each storage dish because
Son；
The file full path footpath factor is obtained by hashing algorithm process to file full path footpath information；
The storage medium factor of the file full path footpath factor and each storage dish is carried out integration process to obtain
Integration factor corresponding with each storage dish；
One is selected from the storage dish as the file according to the corresponding integration factor of each storage dish
The target storage dish that access request is accessed.
In an embodiment of the present invention, according to the corresponding integration factor of each storage dish from the storage
A target storage dish accessed as the file access request is selected to include in disk：
Each storage dish correspondence is obtained by hashing algorithm process to the corresponding integration factor of each storage dish
Selective factor B；
The maximum corresponding storage dish of selective factor B of selected value is used as the target storage dish.
In an embodiment of the present invention, by the storage of the file full path footpath factor and each storage dish
Mediating factors carry out integration to be included：By the storage medium of the file full path footpath factor and each storage dish because
Son carries out XOR process respectively, obtains integration factor corresponding with each storage dish.
In an embodiment of the present invention, also include：The working condition of each storage dish is monitored,
Abnormal storage dish be will appear from according to monitoring result to be changed.
In an embodiment of the present invention, the identification marking is the physical location identification marking of each storage dish.
In an embodiment of the present invention, the physical location identification marking includes storage dish place framework
The slot number of frame number and storage dish place slot.
In order to solve the above problems, present invention also offers a kind of many storage dish load management devices, including：
Polydisc position management module, for obtaining storage dish list, the storage dish list includes each storage dish
Identification marking；
Request receiving module, for receiving the file access request comprising file full path footpath information；
Polydisc load-store management module, for according to the file full path footpath information in the file access request
With the identification marking of each storage dish, one is selected from the storage dish as described using hashing algorithm
The target storage dish that file access request is accessed.
In an embodiment of the present invention, the polydisc load-store management module include calculating sub module,
Integrate submodule and select submodule；
The calculating sub module obtains each for processing by hashing algorithm to the identification marking of each storage dish
The storage medium factor of storage dish；And for being processed by hashing algorithm to file full path footpath information
To the file full path footpath factor；
It is described integrate submodule for by the storage medium of the file full path footpath factor and each storage dish because
Son carries out integration and processes obtaining integration factor corresponding with each storage dish；
The selection submodule is for selecting from the storage dish according to the corresponding integration factor of each storage dish
Select a target storage dish accessed as the file access request.
In an embodiment of the present invention, the selection submodule is according to the corresponding integration of each storage dish
The factor selects a target storage dish accessed as the file access request to include from the storage dish：
Each storage dish correspondence is obtained by hashing algorithm process to the corresponding integration factor of each storage dish
Selective factor B；
The maximum corresponding storage dish of selective factor B of selected value is used as the target storage dish.
In an embodiment of the present invention, also including state monitoring module, the work to each storage dish
State is monitored.
In an embodiment of the present invention, the identification marking is the physical location identification marking of each storage dish.
In order to solve the above problems, present invention also offers a kind of distributed file system, including file access
Client, file access interface, multiple storage dishes and many storage dish load management devices as above；
The file access client is by the file access interface to many storage dish load management devices
Send file access request；
Many storage dish load management devices receive the file access request, and from the plurality of storage dish
It is middle to select a target storage dish accessed as the file access request.
In order to solve the above problems, present invention also offers a kind of distributed storage network system, including file
Access client, file access interface, multiple memory nodes and many storage dish load managements as above
Device；The memory node includes multiple storage dishes；
The file access client is by the file access interface to many storage dish load management devices
Send file access request；
Many storage dish load management devices receive the file access request, please according to the file access
Ask one is selected from the plurality of memory node as target storage node, and from the target storage node
Multiple storage dishes in select one as the file access request access target storage dish.
The invention has the beneficial effects as follows：
Many storage dish load management methods, device, file system and logical volume transfer method and storage network system that the present invention is provided,
The storage dish list comprising each storage dish recognition marks is first obtained, after then receiving file access request, is carried
The file full path footpath information in this document access request is taken, and then according to file full path footpath information and each storage dish
Identification marking, using hashing algorithm select from storage dish one as the file access request access
Target storage dish.Namely the present invention realizes polydisc load balancing using hashing algorithm, can be by magnanimity text
Part is spread evenly across on polydisc, and does not need the presence of metadata, system structure to become very simple
Efficiently, the requirement to hardware (mainly internal memory) is relatively low, no metadata, is also not in that metadata is damaged
Caused Single Point of Faliure, can lift system storage safety.
In addition, the present invention can be also monitored to the state of each storage dish, bad storage dish is changed,
Ensure the normal storage of file；And in terms of resilient expansion, only need to increase storage dish, whole system is in capacity
Can all get a promotion with handling capacity.
Description of the drawings
Fig. 1 is many storage dish load management method flow schematic diagrams that the embodiment of the present invention one is provided；
Fig. 2 is the schematic diagram of the employing hashing algorithm selection target storage dish process that the embodiment of the present invention one is provided；
Fig. 3 is the schematic diagram according to integration factor selection target storage dish process that the embodiment of the present invention one is provided；
Fig. 4 is many storage dish load management device structural representations one that the embodiment of the present invention two is provided；
Fig. 5 is many storage dish load management device structural representations two that the embodiment of the present invention two is provided；
Fig. 6 is many storage dish load management device structural representations three that the embodiment of the present invention two is provided；
Fig. 7 is the distributed file system structural representation that the embodiment of the present invention three is provided；
Fig. 8 is the storage dish and mount point mapping relations that the embodiment of the present invention three is provided；
Fig. 9 is many storage dish load management method flow schematic diagrams that the embodiment of the present invention three is provided；
Figure 10 is the distributed storage network system structure diagram that the embodiment of the present invention four is provided.
Specific embodiment
Accompanying drawing is combined below by specific embodiment to be described in further detail the present invention.
Embodiment one：
The present embodiment is realized being managed the load of many storage dishes using hashing algorithm, and system architecture is very simple
Single, it is not necessary to extra meta-data region, as long as having server and storage dish, (i.e. storage dish, including mechanical type
Hard disk and/or solid state hard disc), it is possible to carry out the file access business of magnanimity, be highly convenient for deployment and implement.
Its access performance is high, is operated by original metadata retrieval and is evolved to hash calculating operation, no matter hundred million grades, also
It is the file of 1,000,000,000 number of stages, is all once quick three column operations, you can obtains the physical bit of file storage
Put；Also the state of each storage dish can be monitored, bad storage dish is changed, it is ensured that file is just
Often store；The characteristic of easy extension is also equipped with simultaneously, as long as increasing storage dish, (mechanical type hard disk or solid-state are hard
Disk), power system capacity and handling capacity all linearly can be lifted.The present invention is done further in detail with specific example below
Describe in detail bright：
Many storage dish load management methods that the present embodiment is provided, are said by taking file storing process as an example below
It is bright, it is shown in Figure 1, including：
Step 101：Storage dish list is obtained, the identification marking comprising each storage dish in the storage dish list；
Step 102：File access request is received, this document access request includes file full path footpath information；
Step 103：According to file full path footpath information and the identification marking of each storage dish, using hash
Algorithm selects a target storage dish accessed as the file access request from the storage dish；
Step 104：Corresponding file access operation is carried out in the target storage dish.File in the present embodiment
Access request can be file storage request, it is also possible to file read request；When asking for file storage, then
The write operation of corresponding document is carried out in the target storage dish；For file read request when, then deposit in the target
The read operation of corresponding document is carried out on storage disk.
In above-mentioned steps 103, according to file full path footpath information and the identification marking of each storage dish, using hash
Algorithm selects a target storage dish as file access request access shown in Figure 2 from storage dish,
Including：
Step 201：The identification marking of each storage dish is situated between by the storage that hashing algorithm process obtains each storage dish
Prime factor；Specifically identification marking can be mapped to by positive integer by hashing algorithm herein, be not precluded from certainly reflecting
Penetrate into other forms；As long as distribution character can be divided equally using hashing algorithm；
Step 202：The file full path footpath factor is obtained by hashing algorithm process to file full path footpath information；Herein
Specifically also identification marking can be mapped to by positive integer by hashing algorithm, be not precluded from certainly being mapped to other shapes
Formula；As long as distribution character can be divided equally using hashing algorithm；Specifically hash in corresponding the present embodiment
Algorithm also flexibly can be selected, as long as above-mentioned purpose can be realized；
Step 203：The storage medium factor of the file full path footpath factor for obtaining with each storage dish is carried out at integration
Reason obtains integration factor corresponding with each storage dish；Namely how many storage dish is with regard to how many integration factor；
Step 204：From each storage dish, one is selected to visit as file according to the corresponding integration factor of each storage dish
Ask the target storage dish that request is accessed.
Hashing algorithm in above-mentioned steps 201 and step 202 can adopt identical algorithm.
Integration in above-mentioned steps 203 is processed concretely by the file full path footpath factor for obtaining and each storage dish
The storage medium factor carry out XOR process respectively, obtain integration factor corresponding with each storage dish.
The detailed process of above-mentioned steps 204 is shown in Figure 3, including：
Step 301：Each storage dish correspondence is obtained by hashing algorithm process to the corresponding integration factor of each storage dish
Selective factor B；Algorithm employed in the step is identical with step 201 and step 202；
Step 302：The maximum corresponding storage dish of selective factor B of selected value is used as target storage dish.
Identification marking in the present embodiment in storage dish list is the physical location identification marking of each storage dish；And
Storage device specifically may include storage server and/or disk cluster (JBOD：Just a Bunch Of Disks),
Storage server, disk cluster all include polylith storage dish, and storage dish can be solid state hard disc, or
Mechanical hard disk.The present embodiment is additionally provided with the application program for carrying out file access on a storage device, namely
File access client.
In the present embodiment, storage server and JBOD can be numbered, such as storage server numbering is frame
Frame number 0, first disk cluster numbering are frame number 1, and second disk cluster numbering is frame number 2, the like,
Nth disk cluster numbering is frame number N；
Further, distributed file system finger daemon is to each the storage dish (mechanical type in storage server
Hard disk or solid state hard disc) slot be numbered, the slot of each storage dish in disk cluster is compiled
Number；
Each storage dish in the present embodiment in storage device just sets a unification and unique physical location is compiled
Number, i.e., " frame number+slot number ", the referred to as physical location identification marking of storage dish；Storage service is obtained during startup
All storage dishes on device and disk cluster, the frame number sequence being located by storage dish first, then by inserting in framework
Groove number sorts, and forms one group of one-dimensional storage dish physical location identification marking list, i.e. storage dish list：
Frame number 0_ slot number 0
Frame number 0_ slot number 1
Frame number 0_ slot number 2
Frame number 0_ slot N'
Frame number 1_ slot number 0
Frame number 1_ slot number 1
Frame number 1_ slot number 2
Frame number 1_ slot N "
Frame number 2_ slot number 0
Frame number 2_ slot number 1
Frame number 2_ slot number 2
Frame number 2_ slot N " '
……
Frame number N_ slot number 0
Frame number N_ slot number 1
Frame number N_ slot number 2
Frame number N_ slot N " "
Then the physical location identification marking of each storage dish is calculated by (i.e. HASH) is hashed, and then will
Storage dish physical location identification marking is mapped to one group of discrete and equally distributed positive integer, and referred to as " storage is situated between
Prime factor ".The physical location identification marking of each storage dish is adopted in the present embodiment, no matter physical location inserts assorted
The storage medium of sample, " the storage medium factor " calculated according to physical location character string group is all one
Sample, that is to say, that " the storage medium factor " in the present embodiment is only related to each physical location, and with deposit
Storage disk is unrelated, can further lift reliability.In the present embodiment, can also number plus storage dish, to each
The storage dish numbering moment carries out unique number, such as disk0001, disk0002 for each storage dish,
Disk0003 ... ..., disk000N.
Physical location identification marking now is then by frame number+_ slot number+storage dish numbering.
After obtaining each storage dish corresponding " the storage medium factor ", can further by storage dish corresponding drive
(i.e. Linux or the corresponding block device files of other Unix-like, such as/dev/sda) and storage dish physics
Station location marker carries out correspondence carry, for example：
/dev/sda /mnt/mydisks/01_00
/dev/sdb /mnt/mydisks/01_01
/dev/sdc /mnt/mydisks/01_02
/dev/sdd /mnt/mydisks/01_03
/dev/sde /mnt/mydisks/01_04
/dev/sdf /mnt/mydisks/01_05
/dev/sdg /mnt/mydisks/01_06
/dev/sdh /mnt/mydisks/01_07
/dev/sdi /mnt/mydisks/01_08
/dev/sdj /mnt/mydisks/01_09
/dev/sdk /mnt/mydisks/01_10
/dev/sdj /mnt/mydisks/01_11
File full path footpath information in the present embodiment may include file type information+some storage catalogue path+texts
Part name；Can be by file full path footpath information MAP into a positive integer using hashing algorithm.
Chosen after target storage dish stored using the balance hashing algorithm of the present embodiment, when user needs
When reading this document, the maximum storage dish of " selective factor B " value is found still according to same method, and should
Target storage dish when storage dish must be storage.
In the present embodiment, in above process, the working condition of each storage dish can be monitored, according to prison
Survey result and will appear from abnormal storage dish rejecting, then changed.When rejecting, can be by the storage dish
File equilibrium be transferred in other storage dishes, can also be completely transferred to change after new storage dish on.
It is current that in storage industry, solid state hard disc (SSD) increasingly tends to main flow, the present embodiment can by SSD with
Traditional mechanical type hard disk is independently organized into groups, i.e. SSD forms one group of solid state hard disc and stores sublist, and this is solid
State hard-disc storage sublist includes the identification marking of each solid state hard disc, such as ssd_0001, ssd_0002 ...
ssd_000N；
Traditional mechanical type hard disk forms one group of mechanical hard disk and stores sublist, such as
disk_0001,disk_0002…disk_000N。
When status monitoring is carried out, two groups of sublist of real-time monitoring can be distinguished.
Now during load management is carried out, can be according to the behavior of user's request, such as, user is want visit
Frequently file (i.e. " heat " file) is asked, is stored in the corresponding solid state hard disc storage sublist of SSD, then
The identification marking that each solid state hard disc in sublist is stored only for solid state hard disc carries out hash calculating, will access
Frequently file (i.e. " heat " file) is mapped in solid state hard disc storage sublist.
If user is want little file (i.e. " cold " file) is accessed, traditional mechanical type hard disk is stored in
In corresponding mechanical hard disk storage sublist, then each machinery of sublist is stored only for traditional mechanical hard disk
The identification marking of hard disk carries out hash calculating, and the file (i.e. " cold " file) for accessing little is mapped to machine
In tool hard-disc storage sublist.The satisfaction of Consumer's Experience can be further lifted so.
Embodiment two：
A kind of many storage dish load management devices are present embodiments provided, it is shown in Figure 4, including：
Polydisc position management module 1, for obtaining storage dish list, knowledge of the storage dish list comprising each storage dish
Do not identify；
Request receiving module 2, for receiving the file access request comprising file full path footpath information；
Polydisc load-store management module 3, for according to the file full path footpath information in file access request and each
The identification marking of storage dish, selects one to access as file access request from storage dish using hashing algorithm
Target storage dish.
Polydisc load-store management module 3 in the present embodiment includes calculating sub module 31, integrates submodule 32
With selection submodule 33；
Calculating sub module 31 obtains each storage dish for processing by hashing algorithm to the identification marking of each storage dish
The storage medium factor；And for obtaining file full path to file full path footpath information by hashing algorithm process
The footpath factor；Herein calculating sub module 31 specifically can by hashing algorithm by identification marking and file full path footpath because
Son is mapped to positive integer, is not precluded from certainly being mapped to other forms；As long as can dividing equally using hashing algorithm
Distribution character.
Integrate submodule 32 to be integrated for the storage medium factor by the file full path footpath factor with each storage dish
Process obtains integration factor corresponding with each storage dish；
Submodule 33 is selected for a conduct being selected from storage dish according to the corresponding integration factor of each storage dish
The target storage dish that the file access request is accessed, detailed process include：
To the corresponding integration factor of each storage dish by hashing algorithm process obtain the corresponding selection of each storage dish because
Son；
The maximum corresponding storage dish of selective factor B of selected value is used as the target storage dish.
In the present embodiment, the calculating of three hashing algorithms can specifically adopt identical algorithm.
Identification marking in the present embodiment in storage dish list is the physical location identification marking of each storage dish；And
Storage device specifically may include storage server and/or disk cluster (JBOD：Just a Bunch Of Disks),
Storage server, disk cluster all include polylith storage dish, and storage dish can be solid state hard disc, or
Mechanical hard disk.The present embodiment is additionally provided with the application program for carrying out file access on a storage device, namely
File access client.
In the present embodiment, storage server and JBOD can be numbered, such as storage server numbering is frame
Frame number 0, first disk cluster numbering are frame number 1, and second disk cluster numbering is frame number 2, the like,
Nth disk cluster numbering is frame number N；
Further, distributed file system finger daemon is to each the storage dish (mechanical type in storage server
Hard disk or solid state hard disc) slot be numbered, the slot of each storage dish in disk cluster is compiled
Number；
Each storage dish in the present embodiment in storage device just sets a unification and unique physical location is compiled
Number, i.e., " frame number+slot number ", the referred to as physical location identification marking of storage dish；Storage service is obtained during startup
All storage dishes on device and disk cluster, the frame number sequence being located by storage dish first, then by inserting in framework
Groove number sorts, and forms one group of one-dimensional storage dish physical location identification marking list, i.e. storage dish list.So
The physical location identification marking of each storage dish is calculated by calculating sub module 31 by (i.e. HASH) is hashed afterwards,
And then storage dish physical location identification marking is mapped to into one group of discrete and equally distributed positive integer, referred to as " deposit
The storage media factor ".The physical location identification marking of each storage dish is adopted in the present embodiment, no matter physical location is inserted
Which type of storage medium entered, " the storage medium factor " calculated according to physical location character string group is all
It is the same, that is to say, that " the storage medium factor " in the present embodiment is only related to each physical location, and
It is unrelated with storage dish, can further lift reliability.In the present embodiment, can also plus storage dish number,
Unique number is carried out for each storage dish to each storage dish numbering moment, physical location identification marking now then by
Frame number+_ slot number+storage dish numbering.
After obtaining each storage dish corresponding " the storage medium factor ", can further by storage dish corresponding drive
(i.e. Linux or the corresponding block device files of other Unix-like, such as/dev/sda) and storage dish physics
Station location marker carries out correspondence carry.
File full path footpath information in the present embodiment may include file type information+some storage catalogue path+texts
Part name；Calculating sub module 31 can be by file full path footpath information MAP into a positive integer using hashing algorithm.
Chosen after target storage dish stored using the balance hashing algorithm of the present embodiment, when user needs
When reading this document, the maximum storage dish of " selective factor B " value is found still according to same method, and should
Target storage dish when storage dish must be storage.
In the present embodiment, in above process, the working condition of each storage dish can be monitored, according to prison
Survey result and will appear from abnormal storage dish rejecting, then changed.When rejecting, can be by the storage dish
File equilibrium be transferred in other storage dishes, can also be completely transferred to change after new storage dish on.
Shown in Figure 5, many storage dish load management devices in the present embodiment can also further include shape
State monitoring modular 4, for being monitored to the working condition of each storage dish.And then be able to will be gone out according to monitoring result
Now abnormal storage dish is rejected, and is then changed.When rejecting, can be balanced by the file in the storage dish
It is transferred in other storage dishes, in the new storage dish after being also completely transferred to change.
Shown in Figure 6, many storage dish load management devices in the present embodiment also include Classification Management mould
Block 5, for SSD is independently organized into groups with traditional mechanical type hard disk, i.e. SSD forms one group of solid state hard disc
Storage sublist, the solid state hard disc storage sublist include the identification marking of each solid state hard disc, such as
ssd_0001,ssd_0002…ssd_000N；
Traditional mechanical type hard disk forms one group of mechanical hard disk and stores sublist, such as
disk_0001,disk_0002…disk_000N。
When status monitoring is carried out, two groups of sublist of real-time monitoring can be distinguished.
Now during load management is carried out, can be according to the behavior of user's request, such as, user is want visit
Frequently file (i.e. " heat " file) is asked, is stored in the corresponding solid state hard disc storage sublist of SSD, then
The identification marking that each solid state hard disc in sublist is stored only for solid state hard disc carries out hash calculating, will access
Frequently file (i.e. " heat " file) is mapped in solid state hard disc storage sublist.
If user is want little file (i.e. " cold " file) is accessed, traditional mechanical type hard disk is stored in
In corresponding mechanical hard disk storage sublist, then each machinery of sublist is stored only for traditional mechanical hard disk
The identification marking of hard disk carries out hash calculating, and the file (i.e. " cold " file) for accessing little is mapped to machine
In tool hard-disc storage sublist.The satisfaction of Consumer's Experience can be further lifted so.
Hashing algorithm employed in the present embodiment can support that hot-swappable mode carries out real-time update.
Embodiment three：
A kind of distributed file system is present embodiments provided, shown in Figure 7, which includes file access
Many storage dishes shown in client 71, file access interface 72, multiple storage dishes 73 and embodiment two are born
Carry managing device 74；File access client 71 can be realized by various user programs, file access interface 72
Then can be realized using general-purpose interface dynamic link library.
As shown in figure 8, illustrating the multiple storage dishes 73 in " distributed file system " in the present embodiment
With mount point mapping relations, it is related to storage server and some JBOD, in storage server, there are some storages
Disk, also has some storage dishes on JBOD, storage server uses SAS ((Serial Attached with JBOD
SCSI) i.e. Serial Attached SCSI (SAS)) cable is connected.Each storage dish has a unique physical location identification
Mark, i.e., identified using " frame number-slot number ", on an operating system, using the physical location mark of storage dish
Know as mount directory.Fig. 8 shows that all storage dishes and the mapping one by one of the mount point in operating system are closed
System；Meanwhile, each storage dish has uniquely " the storage medium factor ", and which is calculated process and refers to reality
Apply shown in example two.
File access client 71 is sent to many storage dish load management devices 74 by file access interface 72
File access request；Many storage dish load management devices 74 receive file access request, and from multiple storage dishes
It is middle to select a target storage dish accessed as file access request.Specifically shown with a file storage below
Example is illustrated, shown in Figure 9, including：
Step 901：File access client 71 calls file access interface 72 to initiate file access request,
And " comspec of file " is provided；
Step 902：Many storage dish load management devices 74 by " comspec of file " be just mapped as one
Integer, referred to as " the file full path footpath factor "；
Step 903：Many storage dish load management devices 74 provide available storage dish matter physical location and list,
And obtain " the storage medium factor " of each storage dish；
Step 904：Many storage dish load management devices 74 will be each " storage medium factor " and " literary
The part complete trails factor " merges into " integration factor ", and (how many " the storage medium factor " is individual " whole with regard to how many
Close the factor ")；
Step 905：Many storage dish load management devices 74 calculate each " whole factor ", obtain multiple
" selective factor B " is (how many " storage medium factor " with regard to how many " integration factor ", so as to how many
Individual " selective factor B ")；
Step 906：Many storage dish load management devices 74 select " selective factor B " of maximum, most at last
File Mapping is in the maximum storage dish of " selective factor B " value；
Step 907：Many storage dish load management devices 74 complete the read-write of file in the storage dish for choosing
Operation.
Example IV：
A kind of distributed storage network system is present embodiments provided, it is shown in Figure 10, visit including file
Ask that client 01, file access interface 02, multiple memory nodes 03 and as shown in embodiment two deposit more
Storage disk load management device 04；Memory node 03 includes multiple storage dishes；Also will be in embodiment three it is multiple
Used as a memory node, multiple memory nodes are bonded logical volume transfer method and storage network system to storage dish.The present embodiment
In can to store network in each memory node be numbered, such as shape such as node1, node2 ... ..., nodeN；
The numbering and way to manage of the multiple storage dishes in each memory node is using the side in the various embodiments described above
Formula.Specific control process is as follows：
File access client 01 is sent to many storage dish load management devices 04 by file access interface 02
File access request；
Many storage dish load management devices 04 receive the file access request, according to file access request from institute
One is selected in stating multiple memory nodes as target storage node nodeX, and its selection mode may also be employed above-mentioned
The mode of selection target storage dish in each embodiment, other modes may also be employed carries out selection determination；Then
And a mesh accessed as file access request is selected from multiple storage dishes of target storage node nodeX
Mark storage dish.
In the present embodiment, complete to select storage in storage network using many storage dish load management devices 04
The mechanism of node, further, completes the selection operation of polydisc inside memory node.The present embodiment supports bullet
Property extension, a large-scale storage network can be built by extension storage node, whole storage network is deposited
Storage load is shared on each memory node by equilibrium, and inside each memory node, storage load is again equal
Weighing apparatus is shared on each disk of polydisc.
The present invention and the contrast of prior art, at least possess advantages below：
(1) system architecture is very simple, it is not necessary to extra metadata controller, as long as having server and depositing
Storage media (mechanical type hard disk or solid state hard disc), it is possible to carry out the file access business of magnanimity, very just
In deployment and enforcement.
(2) performance is high, is operated by original metadata retrieval and is evolved to hash calculating operation, no matter hundred million grades,
Or the file of 1,000,000,000 number of stages, is all once quick three-stage operational, you can obtain the physics of file storage
Position
(3) easily extend, as long as increasing storage medium (mechanical type hard disk or solid state hard disc), power system capacity and
Handling capacity all linearly can be lifted.
Above content is with reference to specific embodiment further description made for the present invention, it is impossible to recognized
Being embodied as of the fixed present invention is confined to these explanations.For the ordinary skill of the technical field of the invention
For personnel, without departing from the inventive concept of the premise, some simple deduction or replace can also be made,
Protection scope of the present invention should be all considered as belonging to.
Claims (14)
1. a kind of many storage dish load management methods, it is characterised in that include：
Obtain storage dish list, identification marking of the storage dish list comprising each storage dish；
File access request is received, the file full path footpath information in the file access request is obtained；
According to file full path footpath information and the identification marking of each storage dish, using hashing algorithm from institute
A target storage dish accessed as the file access request is selected in stating storage dish.
2. many storage dish load management methods as claimed in claim 1, it is characterised in that according to institute
The identification marking of file full path footpath information and each storage dish is stated, is selected from the storage dish using hashing algorithm
One target storage dish accessed as the file access request includes：
To the identification marking of each storage dish by hashing algorithm process obtain the storage medium of each storage dish because
Son；
The file full path footpath factor is obtained by hashing algorithm process to file full path footpath information；
The storage medium factor of the file full path footpath factor and each storage dish is carried out integration process to obtain
Integration factor corresponding with each storage dish；
One is selected from the storage dish as the file according to the corresponding integration factor of each storage dish
The target storage dish that access request is accessed.
3. many storage dish load management methods as claimed in claim 2, it is characterised in that according to institute
Stating the corresponding integration factor of each storage dish selects one to visit as the file access request from the storage dish
The target storage dish asked includes：
Each storage dish correspondence is obtained by hashing algorithm process to the corresponding integration factor of each storage dish
Selective factor B；
The maximum corresponding storage dish of selective factor B of selected value is used as the target storage dish.
4. many storage dish load management methods as claimed in claim 3, it is characterised in that will be described
The file full path footpath factor carries out integration with the storage medium factor of each storage dish to be included：Will be the file complete
The path factor carries out XOR process respectively with the storage medium factor of each storage dish, obtains respectively depositing with described
The corresponding integration factor of storage disk.
5. many storage dish load management methods as described in any one of claim 1-4, it is characterised in that
Also include：The working condition of each storage dish is monitored, abnormal depositing will appear from according to monitoring result
Storage disk is changed.
6. many storage dish load management methods as described in any one of claim 1-4, it is characterised in that
The identification marking is the physical location identification marking of each storage dish.
7. many storage dish load management methods as claimed in claim 6, it is characterised in that the thing
Reason location recognition mark includes the frame number of storage dish place framework and the slot number of storage dish place slot.
8. a kind of many storage dish load management devices, it is characterised in that include：
Polydisc position management module, for obtaining storage dish list, the storage dish list includes each storage dish
Identification marking；
Request receiving module, for receiving the file access request comprising file full path footpath information；
Polydisc load-store management module, for according to the file full path footpath information in the file access request
With the identification marking of each storage dish, one is selected from the storage dish as described using hashing algorithm
The target storage dish that file access request is accessed.
9. many storage dish load management devices as claimed in claim 8, it is characterised in that described many
Disk load-store management module includes calculating sub module, integrates submodule and select submodule；
The calculating sub module obtains each for processing by hashing algorithm to the identification marking of each storage dish
The storage medium factor of storage dish；And for being processed by hashing algorithm to file full path footpath information
To the file full path footpath factor；
It is described integrate submodule for by the storage medium of the file full path footpath factor and each storage dish because
Son carries out integration and processes obtaining integration factor corresponding with each storage dish；
The selection submodule is for selecting from the storage dish according to the corresponding integration factor of each storage dish
Select a target storage dish accessed as the file access request.
10. many storage dish load management devices as claimed in claim 8, it is characterised in that the choosing
Select submodule one is selected from the storage dish as described according to the corresponding integration factor of each storage dish
The target storage dish that file access request is accessed includes：
Each storage dish correspondence is obtained by hashing algorithm process to the corresponding integration factor of each storage dish
Selective factor B；
The maximum corresponding storage dish of selective factor B of selected value is used as the target storage dish.
The 11. many storage dish load management devices as described in any one of claim 8-11, it is characterised in that
Also include state monitoring module, the working condition of each storage dish is monitored.
The 12. many storage dish load management devices as described in any one of claim 8-11, it is characterised in that
The identification marking is the physical location identification marking of each storage dish.
13. a kind of distributed file systems, it is characterised in that visit including file access client, file
Ask interface, multiple storage dishes and many storage dish load management devices as described in any one of claim 8-12；
The file access client is by the file access interface to many storage dish load management devices
Send file access request；
Many storage dish load management devices receive the file access request, and from the plurality of storage dish
It is middle to select a target storage dish accessed as the file access request.
14. a kind of distributed storage network systems, it is characterised in that including file access client, text
Part access interface, multiple memory nodes and many storage dish load pipes as described in any one of claim 8-12
Reason device；The memory node includes multiple storage dishes；
The file access client is by the file access interface to many storage dish load management devices
Send file access request；
Many storage dish load management devices receive the file access request, please according to the file access
Ask one is selected from the plurality of memory node as target storage node, and from the target storage node
Multiple storage dishes in select one as the file access request access target storage dish.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510582124.XA CN106527960B (en) | 2015-09-14 | 2015-09-14 | Multi-storage-disk load management method and device, file system and storage network system |
| PCT/CN2016/098071 WO2017045545A1 (en) | 2015-09-14 | 2016-09-05 | Method and apparatus for managing loads of multiple storage disks, file system, and storage network system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510582124.XA CN106527960B (en) | 2015-09-14 | 2015-09-14 | Multi-storage-disk load management method and device, file system and storage network system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106527960A true CN106527960A (en) | 2017-03-22 |
| CN106527960B CN106527960B (en) | 2021-04-02 |
Family
ID=58288162
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510582124.XA Active CN106527960B (en) | 2015-09-14 | 2015-09-14 | Multi-storage-disk load management method and device, file system and storage network system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106527960B (en) |
| WO (1) | WO2017045545A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111488127A (en) * | 2020-04-16 | 2020-08-04 | 苏州浪潮智能科技有限公司 | Data parallel storage method and device based on disk cluster and data reading method |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112988065B (en) * | 2021-02-08 | 2023-11-17 | 北京星网锐捷网络技术有限公司 | Data migration method, device, equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1641610A (en) * | 2004-01-08 | 2005-07-20 | 英业达股份有限公司 | Hard disk replacement control and management method for network storage system |
| US20090043774A1 (en) * | 2007-08-11 | 2009-02-12 | Gosukonda Naga Sudhakar | Techniques for retaining security restrictions with file versioning |
| CN104660643A (en) * | 2013-11-25 | 2015-05-27 | 南京中兴新软件有限责任公司 | Request response method and device and distributed file system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9043334B2 (en) * | 2012-12-26 | 2015-05-26 | Industrial Technology Research Institute | Method and system for accessing files on a storage system |
| CN104375781B (en) * | 2013-08-16 | 2019-07-23 | 深圳市腾讯计算机系统有限公司 | Data access method and device |
| CN104123359B (en) * | 2014-07-17 | 2017-03-22 | 江苏省邮电规划设计院有限责任公司 | Resource management method of distributed object storage system |
-
2015
- 2015-09-14 CN CN201510582124.XA patent/CN106527960B/en active Active
-
2016
- 2016-09-05 WO PCT/CN2016/098071 patent/WO2017045545A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1641610A (en) * | 2004-01-08 | 2005-07-20 | 英业达股份有限公司 | Hard disk replacement control and management method for network storage system |
| US20090043774A1 (en) * | 2007-08-11 | 2009-02-12 | Gosukonda Naga Sudhakar | Techniques for retaining security restrictions with file versioning |
| CN104660643A (en) * | 2013-11-25 | 2015-05-27 | 南京中兴新软件有限责任公司 | Request response method and device and distributed file system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111488127A (en) * | 2020-04-16 | 2020-08-04 | 苏州浪潮智能科技有限公司 | Data parallel storage method and device based on disk cluster and data reading method |
| CN111488127B (en) * | 2020-04-16 | 2023-01-10 | 苏州浪潮智能科技有限公司 | Data parallel storage method and device based on disk cluster and data reading method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106527960B (en) | 2021-04-02 |
| WO2017045545A1 (en) | 2017-03-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10761758B2 (en) | Data aware deduplication object storage (DADOS) | |
| US9965539B2 (en) | Computer implemented method for dynamic sharding | |
| US8285757B2 (en) | File system for a storage device, methods of allocating storage, searching data and optimising performance of a storage device file system | |
| US8312242B2 (en) | Tracking memory space in a storage system | |
| US9424274B2 (en) | Management of intermediate data spills during the shuffle phase of a map-reduce job | |
| US8694554B2 (en) | Cluster storage using subsegmenting for efficient storage | |
| US9529545B1 (en) | Managing data deduplication in storage systems based on storage space characteristics | |
| US8874627B2 (en) | Enumerating metadata in file system directories | |
| CN103793534B (en) | Distributed file system and balanced metadata storage and the implementation method for accessing load | |
| CN105683898A (en) | Set-associative hash table organization for efficient storage and retrieval of data in a storage system | |
| US10552056B2 (en) | Data storage system tiering accounting for limited write endurance | |
| CN103229173A (en) | Metadata management method and system | |
| CN110603518B (en) | Composite aggregation architecture | |
| WO2014063073A1 (en) | Migrating deduplicated data | |
| CN103067525A (en) | Cloud storage data backup method based on characteristic codes | |
| US11755557B2 (en) | Flat object storage namespace in an object storage system | |
| US20140074834A1 (en) | Storage Block Metadata Tagger | |
| US10481820B1 (en) | Managing data in storage systems | |
| US7792966B2 (en) | Zone control weights | |
| CN106527960A (en) | Management method for multi-memory disk loads, device, document system and memory network system | |
| CN101997919A (en) | Storage resource management method and device | |
| US20140075149A1 (en) | Storage Mechanism with Variable Block Size | |
| US20150177984A1 (en) | Management system and management method | |
| US8776049B2 (en) | Address aligned resource set allocation in a memory space | |
| Lee et al. | Optimizing Read Operations of Hadoop Distributed File System on Heterogeneous Storages. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |