CN108287660A - Date storage method and equipment - Google Patents
Date storage method and equipment Download PDFInfo
- Publication number
- CN108287660A CN108287660A CN201710012670.9A CN201710012670A CN108287660A CN 108287660 A CN108287660 A CN 108287660A CN 201710012670 A CN201710012670 A CN 201710012670A CN 108287660 A CN108287660 A CN 108287660A
- Authority
- CN
- China
- Prior art keywords
- storage device
- order
- group
- memory node
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
This application discloses a kind of date storage method and equipment.The date storage method includes：Data to be stored is divided into N number of object, wherein N is positive integer；N number of object is assigned to M to put in order in group, wherein M is the positive integer less than N；For M put in order any one in group put in order group, it is determined based on memory map assignments and puts in order group corresponding at least three object storage device with this, wherein, comprising the mapping relations put in order between group and object storage device in memory map assignments, and this is put in order to each object storage for including in group based on pseudo-random data Distribution Algorithm and is put in order in group corresponding corresponding object storage device to this.The date storage method and equipment of the embodiment of the present application, can improve the data storage efficiency in Ceph distributed data-storage systems, effectively realize high-speed read-write of the data in Ceph distributed data-storage systems.
Description
Technical field
This application involves computer realm more particularly to a kind of date storage method and equipment.
Background technology
With the arrival in big data epoch, traditional centralized data-storage system can no longer meet large-scale data and deposit
The demand of storage.In order to meet the needs of mass data storage, it is ensured that distribution occur in the reliability of data storage and safety
Formula data-storage system.Ceph is a kind of distributed data-storage system increased income, and can be disperseed to be stored in by more by data
A memory node, that is, in more storage servers, realize the distributed storage of data, improve data-storage system reliability,
Availability and access efficiency.
In practical applications, Ceph distributed data-storage systems are mapped by three-level, realize the distributed storage of data.
First, it would be desirable to which the file data (File) of storage is divided into the consistent object data of several size of data (Object), real
The mapping of existing File to Object；Then, any one Object is assigned to one by hash algorithm and puts in order group
In (Placement Group, PG), the mapping of Object to PG is realized；Finally, pass through pseudo-random data Distribution Algorithm
The Object that (Controlled Replication Under Scalable Hashing, CRUSH) will include in any one PG
It stores in object storage device different in object storage cluster (Object Storage Device, OSD), realizes that PG is arrived
The mapping of OSD.
But since Ceph distributed data-storage systems need to carry out Hash operation, realization in data storage procedure
The mapping storage of data, therefore the storage efficiency of data is relatively low, cannot be satisfied the demand of high-speed read-write.
Invention content
In consideration of it, a kind of date storage method of the embodiment of the present application offer and equipment, are deposited with improving Ceph distributed datas
The data storage efficiency of storage system.
According to a kind of date storage method of the embodiment of the present application, it is applied to Ceph distributed data-storage systems, and
Including：Data to be stored is divided into N number of object, wherein N is positive integer；N number of object is assigned to M to put in order in group,
In, M is the positive integer less than N；For M put in order any one in group put in order group, determined based on memory map assignments and returned with this
Set corresponding at least three object storage device of group, wherein include to put in order between group and object storage device in memory map assignments
Mapping relations, and be based on pseudo-random data Distribution Algorithm, this is put in order include in group each object storage return to this
It sets in the corresponding corresponding object storage device of group.
According to a kind of data storage device of the embodiment of the present application, it is applied to Ceph distributed data-storage systems, and
Including：Division unit is configured for data to be stored being divided into N number of object, wherein N is positive integer；Allocation unit,
It is configured for N number of object being assigned to M and put in order in group, wherein M is the positive integer less than N；Storage unit is configured to use
In for M put in order any one in group put in order group, determine to put in order with this based on memory map assignments and organize corresponding at least three
Object storage device, wherein comprising the mapping relations put in order between group and object storage device in memory map assignments, and be based on
This is put in order each object storage that includes in group and organizes a corresponding corresponding object and deposit to being put in order with this by pseudo-random data Distribution Algorithm
It stores up in equipment.
According to the date storage method and system of the embodiment of the present application, data to be stored is being divided into N number of object and by N
A object is assigned to after M puts in order in group, is directly found by predetermined memory map assignments and is put in order with any one
Corresponding at least three object storage device of group, and then any one is put in order in group by pseudo-random data Distribution Algorithm and includes
Object storage put in order in the corresponding corresponding object storage device of group to this, Ceph distributed data-storage systems can be improved
In data storage efficiency, effectively realize high-speed read-write of the data in Ceph distributed data-storage systems.
Description of the drawings
Attached drawing described herein is used for providing further understanding of the present application, constitutes part of this application, this Shen
Illustrative embodiments and their description please do not constitute the improper restriction to the application for explaining the application.In the accompanying drawings：
Fig. 1 is the flow diagram according to a kind of date storage method of the embodiment of the present application；
Fig. 2 is the process schematic started according to the system of the Ceph distributed data-storage systems of the embodiment of the present application；
Fig. 3 is to be illustrated according to the process of the setting memory node of the Ceph distributed data-storage systems of the embodiment of the present application
Figure；
Fig. 4 is the process schematic according to the data storage of the Ceph distributed data-storage systems of the embodiment of the present application；
Fig. 5 is the fault restoration according to the object storage device of the Ceph distributed data-storage systems of the embodiment of the present application
Process schematic；
Fig. 6 is the mistake according to the fault restoration of the memory node of the Ceph distributed data-storage systems of the embodiment of the present application
Journey schematic diagram；
Fig. 7 is the structural schematic diagram according to the data storage device of the embodiment of the present application.
Specific implementation mode
In practical applications, Ceph distributed data-storage systems are mainly deposited including client, meta data server, object
Accumulation and four parts monitor (Monitor, hereinafter referred to as Ceph Mon), wherein：Client indicates current data users
The memory node at place；Meta data server is used for caching with the information of synchronization data attribute (for example, the storage position of data
It sets, historical data, record file etc.)；Object storage cluster includes multiple memory nodes for data storage, and monitor is used for
Monitoring function is executed to entire Ceph distributed data-storage systems.
During Ceph distributed data-storage systems store data, a rope is distributed for file to be stored (File)
Draw node number (Inode Number, INO), which is the unique identifier of the File；When the data volume of File to be stored is larger
When, it needs the File being divided into a series of Object of unified size and stores.Here, the last one Object's is big
The small Object that can be different from front.
In Ceph large-scale storage clusters, the number of Object is more, and the data volume that each Object includes is smaller, such as
Fruit stores Object by traversing the read-write mode of addressing, can seriously affect data storage rate.Meanwhile it if will
Object is mapped on OSD by the hash algorithm of certain fixation mapping and is stored, and when this OSD is damaged, Object can not be automatic
It migrates to other free time OSD, to cause loss of data.Therefore, usually the larger Object of data volume is assigned to several
In a PG.
It can determine the object identity code of any one Object by INO and object number (Object Number, ONO)
(Object Indentification, OID).For any one Object, done using static Hashing HASH function pairs OID
HASH determines the cryptographic Hash of the Object, and carrying out modulo operation with the cryptographic Hash and the number of PG determines the corresponding PG of the Object
PG identification codes (PGID), and then realize the mapping of Object to the PG.
PG is the concept container and a logical concept of an Object, in Ceph distributed data-storage systems
It is virtual presence, for carrying out tissue and position mapping to the storage of Object.One PG is responsible for several Object of tissue, but
It is an Object to be mapped in a PG, is i.e. is " one-to-many " mapping between PG and Object.Rationally it is arranged PG's
Quantity can ensure the uniformity of data distribution.
Ceph distributed data-storage systems determine that any one PG is corresponded to by pseudo-random data Distribution Algorithm (CRUSH)
OSD, and then the Object storages in the PG are realized into the mapping of PG to OSD in the corresponding OSD in the PG.On one OSD
A large amount of PG can be carried, is i.e. is " multi-to-multi " mapping between PG and OSD.By CRUSH algorithms, it was both avoided that memory node single-point
Loss of data when failure, and it is avoided that relying on metadata is stored, and data storage efficiency is effectively increased.
But since Ceph distributed data-storage systems need to carry out Hash operation and Kazakhstan in data storage procedure
Modulo operation between uncommon value and PG numbers, causes data storage efficiency relatively low, cannot be satisfied the demand of high-speed read-write.
In order to realize the purpose of the application, a kind of date storage method of the embodiment of the present application offer and equipment, will wait depositing
After storage data are divided into N number of Object and N number of Object is assigned in M PG, pass through predetermined memory map assignments
At least three OSDs corresponding with any one PG are directly searched, and then each by include in any one PG by CRUSH algorithms
In a Object storages to corresponding OSD corresponding to the PG, so as to improve the number in Ceph distributed data-storage systems
According to storage efficiency, high-speed read-write of the data in Ceph distributed data-storage systems is effectively realized.
Technical scheme is clearly and completely retouched with reference to the application specific embodiment and corresponding attached drawing
It states.Obviously, the described embodiments are only a part but not all of the embodiments of the present application.Based in the application
Embodiment, every other embodiment obtained by those of ordinary skill in the art without making creative efforts,
It shall fall in the protection scope of this application.
Below in conjunction with attached drawing, the technical solution that each embodiment of the application provides is described in detail.
Embodiment 1
Fig. 1 is a kind of flow diagram of date storage method provided by the embodiments of the present application.The date storage method can
To be applied in Ceph distributed data-storage systems, and may include steps of.
Step 11：Data to be stored is divided into N number of object Object, wherein N is positive integer.
In a step 11, in order to realize that object stores, data to be stored is divided into N number of Object.Here, each
Object has the object identity code different from other Object.The data volume of N number of Object can be identical, can not also phase
Together, it is not specifically limited here.
Step 12：N number of Object is assigned to M according to object size to put in order in group PG, wherein M is just whole less than N
Number.
In step 12, the N number of Object divided in step 11 is assigned to according to object size in M PG, it is real
Now the grouping of Object is stored.It should be noted that any one PG has puts in order group identification code different from other PG.For
Realize being uniformly distributed for data, N number of Object is evenly distributed to according to object size in M PG.For example, when 500
Include 5 Object when Object is assigned in 100 PG, in each PG.
Step 13：At least three object storage device OSD corresponding with any one PG is determined by memory map assignments,
In, the mapping relations between PG and OSD are included in memory map assignments.
In step 13, according to pre-stored memory map assignments in Ceph distributed data-storage systems, Ceph Mon
Determine at least three OSDs corresponding with any one PG.In Ceph distributed data-storage systems, an Object will be preserved
At least three copies, i.e. an Object will be stored at least three OSD.Since an Object is only mapped to a PG
In, therefore any one PG needs to establish mapping relations between at least three OSD, just can ensure that an Object can be stored
Into at least three OSD.
Step 14：For any one PG, each Object for including in PG storages are arrived and this by CRUSH algorithms
In the corresponding corresponding OSD of PG.
At step 14, for any one PG, due to having determined that corresponding with the PG at least three in step 13
A OSD, therefore each Object for including in the PG can be stored by CRUSH algorithms and arrive corresponding OSD corresponding to the PG
In, realize the distributed storage of data to be stored.
In the alternative embodiment of the application, memory map assignments can be created in the following manner.Specifically, storage is created
The processing of mapping table includes：
First, the cryptographic Hash of each OSD is read from memory.Specifically, it is stored in Ceph Mon readings memory each
The cryptographic Hash of OSD.
Secondly, the mapping relations between any one PG and at least three OSD are established.Specifically, CephMon establishes arbitrary
Mapping relations between one PG and at least three OSD.It should be noted that the OSD for establishing mapping relations with PG is idle
OSD can realize the OSD of data storage function.
Finally, the mapping relations between PG and OSD are stored in memory map assignments.
By reading the cryptographic Hash of each OSD stored in memory, the mapping between each PG and at least three OSD is established
Relationship avoids the relatively low problem of storage efficiency caused by calculating cryptographic Hash.
In the alternative embodiment of the application, the cryptographic Hash of each OSD can be determined in the following manner.Specifically, really
The processing of the cryptographic Hash of fixed each OSD includes：
First, the facility information of the memory node of preset number is called from system folder, wherein any one storage
Include at least three OSD in node.It should be noted that the facility information of memory node includes but not limited to and memory node pair
The facility informations such as IP address, the machine name answered.
Secondly, according to the facility information of any one memory node, the Hash of each OSD in the memory node is calculated
Value.
Finally, in memory by the storage of the cryptographic Hash of each OSD in the memory node of preset number.
Fig. 2 is the process schematic started according to the system of the Ceph distributed data-storage systems of the embodiment of the present application.
As shown in Fig. 2, when Ceph distributed data-storage systems start, 3 stored in Ceph Mon calling system files deposit
The facility information for storing up node, the Kazakhstan of each OSD in the memory node is calculated according to the facility information of any one memory node
Uncommon value, and in memory by the storage of the cryptographic Hash of these OSD.
In the alternative embodiment of the application, the facility information of memory node can be determined in the following manner：
First, the facility information of the memory node of preset number is set in node scan script；
Then, by parsing node scan script, the facility information of the memory node of preset number is stored in system text
In part folder.
Fig. 3 is to be illustrated according to the process of the setting memory node of the Ceph distributed data-storage systems of the embodiment of the present application
Figure.As shown in figure 3, Ceph Mon set 3 memory nodes in the node scan script of Ceph distributed data-storage systems
Facility information, and then by parse node scan script the facility information of 3 memory nodes is stored in system folder
In.
Fig. 4 is the process schematic according to the data storage of the Ceph distributed data-storage systems of the embodiment of the present application.
As shown in figure 4, Ceph Mon pass through after three OSD (OSD1, OSD2 and OSD3) corresponding with some PG are determined
CRUSH algorithms store each Object for including in the PG into corresponding OSD corresponding to the PG.
In the alternative embodiment of the application, further include according to the date storage method of the embodiment of the present application：When being stored with
When the OSD of Object breaks down, the update cryptographic Hash of each OSD in the memory node where the OSD is calculated；It is deposited according to this
The update cryptographic Hash for storing up each OSD in node, determines the idle OSD in the memory node；It will be stored in the OSD to break down
Object storages in idle OSD.
Fig. 5 is to be illustrated according to the process of the OSD fault restorations of the Ceph distributed data-storage systems of the embodiment of the present application
Figure.As shown in figure 5, when event occurs for the OSD2 for being stored with Object in the object storage cluster of Ceph distributed data-storage systems
When barrier, the memory node startup separator reparation where OSD2, recalculate each OSD in the memory node cryptographic Hash (that is,
Calculate the update cryptographic Hash of these OSD), and according to the update cryptographic Hash of each OSD in the memory node, determine the storage
Idle OSDx in node, and then will be in the Object that stored in the OSD2 that broken down storages to idle OSDx.
In the alternative embodiment of the application, further include according to the date storage method of the embodiment of the present application：When being stored with
The memory node of group is put in order when breaking down, increases idle memory node in node scan script；It will be sent out by CRUSH algorithms
What is stored in the memory node of raw failure puts in order in group storage to idle memory node.
Fig. 6 is the process according to the memory node fault restoration of the Ceph distributed data-storage systems of the embodiment of the present application
Schematic diagram.As shown in fig. 6, when any one memory node in the object storage cluster of Ceph distributed data-storage systems
When Node3 breaks down, Ceph Mon update node scan script, increase idle memory node in node scan script
The facility information of Node4, and then the PG stored in failed storage node Node3 was stored to the free time by CRUSH algorithms and is stored
In node Node4.
In Ceph distributed data-storage systems, data to be stored is being divided into N number of Object and by N number of Object
After being assigned in M PG according to object size, by predetermined memory map assignments can directly find with it is any one
Corresponding at least three OSD of a PG, so can by CRUSH algorithms by any one PG storages to corresponding with the PG each
In OSD, to improve the data storage efficiency in Ceph distributed data-storage systems, effectively realize data in Ceph distributions
High-speed read-write in data-storage system.
Embodiment 2
Fig. 7 is the structural schematic diagram according to a kind of data storage device of the embodiment of the present application.As shown in fig. 7, according to this
Apply for that the data storage device 70 of embodiment includes division unit 701, allocation unit 702 and storage unit 703, wherein：It divides
Unit 701 is configured for data to be stored being divided into N number of object Object, wherein N is positive integer；Allocation unit 702
It is configured for N number of Object being assigned to M according to object size and put in order in group PG, wherein M is the positive integer less than N；It deposits
Storage unit 703 be configured for for M put in order any one in group put in order group, determined based on memory map assignments and put in order with this
The corresponding at least three object storage devices OSD of group PG, wherein include the mapping relations between PG and OSD in memory map assignments,
And each object for including in the PG is stored to corresponding corresponding to the PG based on pseudo-random data distribution CRUSH algorithms
In OSD.
In the alternative embodiment of the application, data storage device 70 further includes：Reading unit 704 and map unit 705,
Wherein：Reading unit 705 is configured for reading the cryptographic Hash of each OSD from memory；Map unit 706 is configured for pair
In M put in order any one in group put in order group, establish the mapping relations between the PG and at least three OSD, and will be established
Mapping relations be stored in memory map assignments.
In the alternative embodiment of the application, data storage device 70 further includes call unit 706 and computing unit 707,
Wherein：Call unit 706 is configured for calling the facility information of the memory node of preset number from system folder,
In, include at least three OSD in any one memory node；Computing unit 707 is configured for the storage for preset number
Any one memory node in node calculates each OSD's in the memory node according to the facility information of the memory node
Cryptographic Hash, and in memory by the storage of the cryptographic Hash of each OSD.
In the alternative embodiment of the application, data storage device 70 further includes setup unit 708, wherein：Setup unit
709 are configured for setting the facility information of the memory node of preset number in node scan script, and by parsing node
The facility information of the memory node of preset number is stored in system folder by scan script.
In the alternative embodiment of the application, computing unit 707 is additionally configured to that event occurs as the OSD for being stored with object
When barrier, the update cryptographic Hash of each OSD in the memory node where the OSD is calculated；Storage unit 703 is additionally configured to root
It according to the update cryptographic Hash of each OSD in the memory node, determines the idle OSD in the memory node, and will break down
In the object storage to idle OSD stored in OSD.
In the alternative embodiment of the application, setup unit 709, which is additionally configured to work as to be stored with, puts in order any one of group
When a memory node breaks down, increase idle memory node in node scan script；Storage unit 703 is additionally configured to
The PG stored in the memory node to break down is stored into idle memory node by CRUSH algorithms.
Data to be stored is being divided into N number of Object and will be N number of according to the data storage device of the embodiment of the present application
After Object is assigned to according to object size in M PG, by predetermined memory map assignments can directly find with
Corresponding at least three OSD of any one PG, and then can be stored any one PG to corresponding with the PG by CRUSH algorithms
Each OSD in, to improve the data storage efficiency in Ceph distributed data-storage systems, effectively realize data in Ceph
High-speed read-write in distributed data-storage system.
It should be understood by those skilled in the art that, the embodiment of the present invention can be provided as method, system or computer program
Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the present invention
Apply the form of example.Moreover, the present invention can be used in one or more wherein include computer usable program code computer
The computer program production implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
The form of product.
It will be understood by those skilled in the art that embodiments herein can be provided as method, system or computer program product.
Therefore, complete hardware embodiment, complete software embodiment or embodiment combining software and hardware aspects can be used in the application
Form.It is deposited moreover, the application can be used to can be used in the computer that one or more wherein includes computer usable program code
The shape for the computer program product implemented on storage media (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
Formula.
Above is only an example of the present application, it is not intended to limit this application.For those skilled in the art
For, the application can have various modifications and variations.It is all within spirit herein and principle made by any modification, equivalent
Replace, improve etc., it should be included within the scope of claims hereof.
Claims (12)
1. a kind of date storage method, which is characterized in that the date storage method is applied to Ceph distributed data-storage systems,
And including：
Data to be stored is divided into N number of object, wherein N is positive integer；
N number of object is assigned to M to put in order in group, wherein M is the positive integer less than N；
For the M put in order any one in group put in order group,
It is determined based on memory map assignments and puts in order group corresponding at least three object storage device with described, wherein the storage is reflected
Comprising the mapping relations put in order between group and object storage device in firing table, and
Based on pseudo-random data Distribution Algorithm, by it is described put in order each object storage that includes in group to it is described put in order it is group corresponding
Corresponding object storage device in.
2. date storage method as described in claim 1, which is characterized in that further include：
The cryptographic Hash of each object storage device is read from memory；
For the M put in order any one in group put in order group,
The mapping relations of group put in order between group identification code and the cryptographic Hash of at least three object storage devices are put in order described in foundation,
The mapping relations are stored in the memory map assignments.
3. date storage method as claimed in claim 2, which is characterized in that further include：
The facility information of the memory node of preset number is transferred from system folder, wherein the storage section of the preset number
Any one memory node in point includes at least three object storage devices；
For any one memory node in the memory node of the preset number, the equipment letter based on the memory node
Breath, calculates the cryptographic Hash of each object storage device in the memory node；
The cryptographic Hash of each object storage device in the memory node of the preset number is stored in the memory.
4. date storage method as claimed in claim 3, which is characterized in that further include：
The facility information of the memory node of the preset number is set in node scan script；
By parsing the node scan script, the facility information of the memory node of the preset number is stored in the system
In file.
5. date storage method as claimed in claim 4, which is characterized in that further include：
When any one object storage device for being stored with object breaks down, the storage where the object storage device is calculated
The update cryptographic Hash of each object storage device in node；
According to the update cryptographic Hash of each object storage device in the memory node, the free time in the memory node is determined
Object storage device；And
It will be in the object that stored in the object storage device that broken down storage to the idle object storage device.
6. date storage method as claimed in claim 3, which is characterized in that further include：
When be stored with put in order group any one memory node break down when, in the node scan script increase the free time deposit
Store up node；And
Based on the pseudo-random data Distribution Algorithm, the sky is arrived into group storage of putting in order stored in the memory node to break down
In not busy memory node.
7. a kind of data storage device, which is characterized in that the data storage device is applied to Ceph distributed data-storage systems,
And including：
Division unit is configured for data to be stored being divided into N number of object, wherein N is positive integer；
Allocation unit is configured for N number of object being assigned to M and put in order in group, wherein M is the positive integer less than N；
Storage unit, be configured for for the M put in order any one in group put in order group,
It is determined based on memory map assignments and puts in order group corresponding at least three object storage device with described, wherein the storage is reflected
Comprising the mapping relations put in order between group and object storage device in firing table, and
Based on pseudo-random data Distribution Algorithm, by it is described put in order each object storage that includes in group to it is described put in order it is group corresponding
Corresponding object storage device in.
8. data storage device as claimed in claim 7, which is characterized in that further include：
Reading unit is configured for reading the cryptographic Hash of each object storage device from memory；
Map unit, be configured for for the M put in order any one in group put in order group,
The mapping relations of group put in order between group identification code and the cryptographic Hash of at least three object storage devices are put in order described in foundation,
The mapping relations are stored in the memory map assignments.
9. data storage device as claimed in claim 8, which is characterized in that further include：
Call unit is configured for calling the facility information of the memory node of preset number from system folder, wherein institute
It includes at least three object storage devices to state any one memory node in the memory node of preset number；
Computing unit, any one memory node being configured in the memory node for the preset number are based on institute
The facility information for stating memory node calculates the cryptographic Hash of each object storage device in the memory node, and will be described pre-
If the cryptographic Hash of each object storage device in the memory node of number is stored in the memory.
10. the equipment of data storage as claimed in claim 9, which is characterized in that further include：
Setup unit is configured for setting the facility information of the memory node of the preset number in node scan script,
And the facility information of the memory node of the preset number is stored in the system text by parsing the node scan script
In part folder.
11. data storage device as claimed in claim 10, which is characterized in that
The computing unit is also configured to when the object storage device for being stored with object breaks down, and calculates object storage
The update cryptographic Hash of each object storage device in memory node where equipment；
The storage unit is additionally configured to the update cryptographic Hash according to each object storage device in the memory node,
Determine the idle object storage device in the memory node, and the object that will be stored in the object storage device to break down
It stores in the idle object storage device.
12. data storage device as claimed in claim 10, which is characterized in that
The setup unit be additionally configured to when be stored with put in order group any one memory node break down when, described
Increase idle memory node in node scan script；
The storage unit is also configured to will be in the memory node that broken down based on the pseudo random number data distribution algorithms
Storage is put in order in group storage to the idle memory node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710012670.9A CN108287660B (en) | 2017-01-09 | 2017-01-09 | Data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710012670.9A CN108287660B (en) | 2017-01-09 | 2017-01-09 | Data storage method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108287660A true CN108287660A (en) | 2018-07-17 |
| CN108287660B CN108287660B (en) | 2021-07-09 |
Family
ID=62819128
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710012670.9A Active CN108287660B (en) | 2017-01-09 | 2017-01-09 | Data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108287660B (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109464564A (en) * | 2018-10-24 | 2019-03-15 | 岭南师范学院 | A kind of rhizoma cyperi phenolic acid compound supercritical extraction method |
| CN109669822A (en) * | 2018-11-28 | 2019-04-23 | 平安科技（深圳）有限公司 | The creation method and computer readable storage medium of electronic device, spare memory pool |
| CN109933284A (en) * | 2019-02-26 | 2019-06-25 | 启迪云计算有限公司 | A kind of data distribution algorithms of distributed block storage system |
| CN110908606A (en) * | 2019-11-15 | 2020-03-24 | 浪潮电子信息产业股份有限公司 | Data reconstruction method of distributed file system |
| CN110955733A (en) * | 2020-01-02 | 2020-04-03 | 北京同有飞骥科技股份有限公司 | Data equalization method and system for distributed system |
| CN111026720A (en) * | 2019-12-20 | 2020-04-17 | 深信服科技股份有限公司 | File processing method, system and related equipment |
| CN111125011A (en) * | 2019-12-20 | 2020-05-08 | 深信服科技股份有限公司 | File processing method, system and related equipment |
| CN111258508A (en) * | 2020-02-16 | 2020-06-09 | 西安奥卡云数据科技有限公司 | Metadata management method in distributed object storage |
| CN111324298A (en) * | 2018-12-17 | 2020-06-23 | 西部数据技术公司 | Data storage system and method for autonomic tuning of data storage system performance, capacity and/or operational requirements |
| CN112486413A (en) * | 2020-11-27 | 2021-03-12 | 杭州朗和科技有限公司 | Data reading method, device, medium and computing equipment |
| CN112596973A (en) * | 2020-11-17 | 2021-04-02 | 新华三大数据技术有限公司 | Data object storage method and device and storage medium |
| WO2021077745A1 (en) * | 2019-10-25 | 2021-04-29 | 浪潮电子信息产业股份有限公司 | Data reading and writing method of distributed storage system |
| CN113778341A (en) * | 2021-09-17 | 2021-12-10 | 北京航天泰坦科技股份有限公司 | Distributed storage method and device for remote sensing data and remote sensing data reading method |
| CN114253481A (en) * | 2021-12-23 | 2022-03-29 | 深圳市名竹科技有限公司 | Data storage method and device, computer equipment and storage medium |
| CN114253482A (en) * | 2021-12-23 | 2022-03-29 | 深圳市名竹科技有限公司 | Data storage method and device, computer equipment and storage medium |
| CN117609195A (en) * | 2024-01-24 | 2024-02-27 | 济南浪潮数据技术有限公司 | Object management method, device and equipment of distributed storage system and medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103905540A (en) * | 2014-03-25 | 2014-07-02 | 浪潮电子信息产业股份有限公司 | Object storage data distribution mechanism based on two-sage Hash |
| US20150248252A1 (en) * | 2008-08-15 | 2015-09-03 | International Business Machines Corporation | Mapping of logical volumes to host clusters |
| CN105450734A (en) * | 2015-11-09 | 2016-03-30 | 上海爱数信息技术股份有限公司 | Distributed storage CEPH data distribution optimization method |
-
2017
- 2017-01-09 CN CN201710012670.9A patent/CN108287660B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150248252A1 (en) * | 2008-08-15 | 2015-09-03 | International Business Machines Corporation | Mapping of logical volumes to host clusters |
| CN103905540A (en) * | 2014-03-25 | 2014-07-02 | 浪潮电子信息产业股份有限公司 | Object storage data distribution mechanism based on two-sage Hash |
| CN105450734A (en) * | 2015-11-09 | 2016-03-30 | 上海爱数信息技术股份有限公司 | Distributed storage CEPH data distribution optimization method |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109464564A (en) * | 2018-10-24 | 2019-03-15 | 岭南师范学院 | A kind of rhizoma cyperi phenolic acid compound supercritical extraction method |
| CN109669822A (en) * | 2018-11-28 | 2019-04-23 | 平安科技（深圳）有限公司 | The creation method and computer readable storage medium of electronic device, spare memory pool |
| CN109669822B (en) * | 2018-11-28 | 2023-06-06 | 平安科技（深圳）有限公司 | Electronic device, method for creating backup storage pool, and computer-readable storage medium |
| CN111324298B (en) * | 2018-12-17 | 2023-10-24 | 西部数据技术公司 | Data storage system and method for autonomously adjusting data storage system performance, capacity, and/or operational requirements |
| CN111324298A (en) * | 2018-12-17 | 2020-06-23 | 西部数据技术公司 | Data storage system and method for autonomic tuning of data storage system performance, capacity and/or operational requirements |
| CN109933284A (en) * | 2019-02-26 | 2019-06-25 | 启迪云计算有限公司 | A kind of data distribution algorithms of distributed block storage system |
| WO2021077745A1 (en) * | 2019-10-25 | 2021-04-29 | 浪潮电子信息产业股份有限公司 | Data reading and writing method of distributed storage system |
| CN110908606A (en) * | 2019-11-15 | 2020-03-24 | 浪潮电子信息产业股份有限公司 | Data reconstruction method of distributed file system |
| CN110908606B (en) * | 2019-11-15 | 2021-06-29 | 浪潮电子信息产业股份有限公司 | Data reconstruction method of distributed file system |
| CN111125011A (en) * | 2019-12-20 | 2020-05-08 | 深信服科技股份有限公司 | File processing method, system and related equipment |
| CN111125011B (en) * | 2019-12-20 | 2024-02-23 | 深信服科技股份有限公司 | File processing method, system and related equipment |
| CN111026720B (en) * | 2019-12-20 | 2023-05-12 | 深信服科技股份有限公司 | File processing method, system and related equipment |
| CN111026720A (en) * | 2019-12-20 | 2020-04-17 | 深信服科技股份有限公司 | File processing method, system and related equipment |
| CN110955733A (en) * | 2020-01-02 | 2020-04-03 | 北京同有飞骥科技股份有限公司 | Data equalization method and system for distributed system |
| CN111258508A (en) * | 2020-02-16 | 2020-06-09 | 西安奥卡云数据科技有限公司 | Metadata management method in distributed object storage |
| CN112596973A (en) * | 2020-11-17 | 2021-04-02 | 新华三大数据技术有限公司 | Data object storage method and device and storage medium |
| CN112486413B (en) * | 2020-11-27 | 2022-08-05 | 杭州朗和科技有限公司 | Data reading method, device, medium and computing equipment |
| CN112486413A (en) * | 2020-11-27 | 2021-03-12 | 杭州朗和科技有限公司 | Data reading method, device, medium and computing equipment |
| CN113778341A (en) * | 2021-09-17 | 2021-12-10 | 北京航天泰坦科技股份有限公司 | Distributed storage method and device for remote sensing data and remote sensing data reading method |
| CN114253482A (en) * | 2021-12-23 | 2022-03-29 | 深圳市名竹科技有限公司 | Data storage method and device, computer equipment and storage medium |
| CN114253481A (en) * | 2021-12-23 | 2022-03-29 | 深圳市名竹科技有限公司 | Data storage method and device, computer equipment and storage medium |
| CN117609195A (en) * | 2024-01-24 | 2024-02-27 | 济南浪潮数据技术有限公司 | Object management method, device and equipment of distributed storage system and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108287660B (en) | 2021-07-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108287660A (en) | Date storage method and equipment | |
| US10776396B2 (en) | Computer implemented method for dynamic sharding | |
| US9811546B1 (en) | Storing data and metadata in respective virtual shards on sharded storage systems | |
| CN102411637B (en) | Metadata management method of distributed file system | |
| US20120197958A1 (en) | Parallel Serialization of Request Processing | |
| CN106250226B (en) | Method for scheduling task and system based on consistency hash algorithm | |
| CN108874803B (en) | Data storage method, device and storage medium | |
| CN104050102B (en) | Object storage method and device in a kind of telecommunication system | |
| EP2419845A2 (en) | Policy-based storage structure distribution | |
| CN101997823A (en) | Distributed file system and data access method thereof | |
| CN111723073B (en) | Data storage processing method, device, processing system and storage medium | |
| CN104407879A (en) | A power grid timing sequence large data parallel loading method | |
| CN108108476A (en) | The method of work of highly reliable distributed information log system | |
| CN104750757B (en) | A kind of date storage method and equipment based on HBase | |
| CN109756573B (en) | File system based on block chain | |
| CN108491163B (en) | Big data processing method and device and storage medium | |
| CN113655969B (en) | Data balanced storage method based on streaming distributed storage system | |
| CN106790422A (en) | A kind of data buffer storage cluster and data retrieval method for WEB application | |
| CN107315745B (en) | Private letter storage method and system | |
| US10984059B2 (en) | Data retrieval method and system | |
| CN102999571A (en) | Realizing method for multiple nodes of single computer in cluster | |
| CN110427364A (en) | A kind of data processing method, device, electronic equipment and storage medium | |
| CN110134338A (en) | A kind of distributed memory system and its data redundancy protection method and relevant device | |
| CN107018170A (en) | A kind of method of data synchronization and equipment | |
| KR101428649B1 (en) | Encryption system for mass private information based on map reduce and operating method for the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |