CN105511801B - The method and apparatus of data storage - Google Patents
The method and apparatus of data storage Download PDFInfo
- Publication number
- CN105511801B CN105511801B CN201510810711.XA CN201510810711A CN105511801B CN 105511801 B CN105511801 B CN 105511801B CN 201510810711 A CN201510810711 A CN 201510810711A CN 105511801 B CN105511801 B CN 105511801B
- Authority
- CN
- China
- Prior art keywords
- data
- rack
- back end
- memory space
- data copy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
- G06F3/0667—Virtualisation aspects at data level, e.g. file, record or object virtualisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The invention discloses a kind of method and apparatus of data storage.Wherein, this method includes：Obtain data to be stored；The corresponding rack of data is determined in memory space by Hash operation；The corresponding data copy of data is obtained, and the corresponding storage region of the back end of data copy on the rack is determined by Hash operation；Wherein, memory space is annular memory space, and back end corresponds to multiple virtual data nodes, and the responsible memory space of multiple virtual data nodes is made of multiple storage regions divided equally；Data copy is stored in storage region.The technical issues of present invention can be rapidly completed data and store while guaranteeing data balancing distribution, lifting system process performance.
Description
Technical field
The present invention relates to field of data storage, in particular to a kind of method and apparatus of data storage.
Background technique
Distributed type assemblies storage is a kind of key technology of big data storage management, wherein HDFS (Hadoop
Distributed File System, Hadoop distributed file system) because its high transmission rates and high fault tolerance become solve it is big
The effective ways of data efficient storage application.
Existing one kind, which is achieved in that, is described as integer programming problem for optimal Buffer sizing problem, but existing
HDFS data Placement Strategy while selecting data reliability to be promoted, do not take into account data balancing problem and system and execute
Performance causes data storage slowly and system process performance is low.
For above-mentioned problem, currently no effective solution has been proposed.
Summary of the invention
The embodiment of the invention provides a kind of method and apparatus of data storage, can guarantee the same of data balancing distribution
When, data storage, lifting system process performance is rapidly completed.
According to an aspect of an embodiment of the present invention, a kind of method of data storage is provided, including：It obtains to be stored
Data；The corresponding rack of the data is determined in memory space by Hash operation；Obtain the corresponding data pair of the data
This, and the corresponding storage region of back end of the data copy in the rack is determined by Hash operation；Wherein, institute
Memory space is stated as annular memory space, the back end corresponds to multiple virtual data nodes, the multiple virtual data section
The responsible memory space of point is made of multiple storage regions divided equally；The data copy is stored in the storage region.
Optionally, before data to be stored being taken in reed, the method also includes：The memory space is respectively operated
Obtain multiple storage regions divided equally.
Optionally, before the corresponding data copy of the acquisition data, the method also includes：Determine the rack
It is whether abnormal；The corresponding data copy of the data that obtains includes：When determining that the rack is normal, the data are obtained
Corresponding data copy；When determining the rack exception, next machine is searched in the memory space according to clock-wise order
Frame.
Optionally, it is described the data copy is stored in the storage region before, the method also includes：?
Determine that the data copy after the corresponding storage region of back end in the rack, determines the machine by Hash operation
Whether the back end on frame is abnormal；When determining the back end exception, according to clock-wise order in the memory space
The interior next rack of search；When determining that the back end is normal, determine whether the corresponding storage region of the back end fills
Foot；The described data copy is stored in the storage region include：It, will be described when determining the storage region abundance
Data copy is stored in the storage region；When determining that the storage region is inadequate, according to clock-wise order described
Next back end in memory space in the search rack.
Optionally, after the data copy is stored in the storage region, the method also includes：It determines complete
Whether portion's data copy stores completion；When determining the not stored completion of total data copy, deposited according to clock-wise order described
Store up the next rack of search in space.
According to another aspect of an embodiment of the present invention, a kind of device of data storage is additionally provided, including：Acquiring unit,
For obtaining data to be stored；Processing unit, for determining that the data are corresponding in memory space by Hash operation
Rack, and the corresponding data copy of the data is obtained, and determine the data copy in the rack by Hash operation
The corresponding storage region of back end；Wherein, the memory space is annular memory space, and the back end corresponds to multiple
Virtual data node, the responsible memory space of the multiple virtual data node are made of multiple storage regions divided equally；Storage
Unit, for the data copy to be stored in the storage region.
Optionally, the processing unit is also used to before obtaining data to be stored, and the memory space is divided equally
Operation obtains multiple storage regions divided equally.
Optionally, described device further includes：First judging unit, for obtaining the corresponding data pair of the data described
Before this, determine whether the rack is abnormal：The processing unit, for determining that the rack is normal in first judging unit
When, obtain the corresponding data copy of the data；When first judging unit determines the rack exception, according to clockwise
Sequence searches for next rack in the memory space.
Optionally, described device further includes：Second judgment unit, for it is described the data copy is stored in it is described
Before in storage region, the corresponding storage of back end of the data copy in the rack is being determined by Hash operation
After region, determine whether the back end in the rack is abnormal；The processing unit, in the second judgment unit
When determining the back end exception, next rack is searched in the memory space according to clock-wise order；Second judgement is single
Member is also used to when determining that the back end is normal, determines whether the corresponding storage region of the back end is sufficient；It is described
Storage unit, for when the second judgment unit determines the storage region abundance, the data copy to be stored in institute
It states in storage region；The processing unit, for when the second judgment unit determines that the storage region is inadequate, according to
Clock-wise order searches for next back end in the rack in the memory space.
Optionally, described device further includes：The data copy is being stored in the storage region by third judging unit
After interior, determine whether total data copy stores completion；The processing unit, for determining that total data copy is not stored
After the completion, next rack is searched in the memory space according to clock-wise order.
In embodiments of the present invention, data to be stored are obtained；Data pair are determined in memory space by Hash operation
The rack answered；The corresponding data copy of data is obtained, and the interior data section of data copy on the rack is determined by Hash operation
The corresponding storage region of point；Wherein, memory space is annular memory space, and back end corresponds to multiple dummy nodes, Duo Gexu
The responsible memory space of quasi- node is made of multiple storage regions divided equally；Data copy is stored in storage region, in this way,
The present invention passes through hash algorithm and introduces virtual data node and the completion data storage of equal part storage region, so as to guarantee
While data balancing is distributed, data storage is rapidly completed, the technical issues of lifting system process performance.
Detailed description of the invention
The drawings described herein are used to provide a further understanding of the present invention, constitutes part of this application, this hair
Bright illustrative embodiments and their description are used to explain the present invention, and are not constituted improper limitations of the present invention.In the accompanying drawings：
Fig. 1 is a kind of flow diagram of optional date storage method according to an embodiment of the present invention；
Fig. 2 is a kind of schematic diagram of optional rack migration according to an embodiment of the present invention；
Fig. 3 is a kind of optional rack backup distribution schematic diagram according to an embodiment of the present invention；
Fig. 4 is a kind of optional data connection inquiry runing time comparison diagram according to an embodiment of the present invention；
Fig. 5 is that one kind according to an embodiment of the present invention is optionally actually backed up distribution situation and runing time comparison diagram；
Fig. 6 is backup distribution situation figure after a kind of optional back end adjustment according to an embodiment of the present invention；
Fig. 7 is the flow diagram of another optional date storage method according to an embodiment of the present invention；
Fig. 8 is a kind of flow diagram of optional date storage method according to an embodiment of the present invention；
Fig. 9 is a kind of structural schematic diagram of optional data storage device according to an embodiment of the present invention；
Figure 10 is the structural schematic diagram of another optional data storage device according to an embodiment of the present invention；
Figure 11 is the structural schematic diagram of the third optional data storage device according to an embodiment of the present invention.
Specific embodiment
In order to enable those skilled in the art to better understand the solution of the present invention, below in conjunction in the embodiment of the present invention
Attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is only
The embodiment of a part of the invention, instead of all the embodiments.Based on the embodiments of the present invention, ordinary skill people
The model that the present invention protects all should belong in member's every other embodiment obtained without making creative work
It encloses.
It should be noted that description and claims of this specification and term " first " in above-mentioned attached drawing, "
Two " etc. be to be used to distinguish similar objects, without being used to describe a particular order or precedence order.It should be understood that using in this way
Data be interchangeable under appropriate circumstances, so as to the embodiment of the present invention described herein can in addition to illustrating herein or
Sequence other than those of description is implemented.In addition, term " includes " and " having " and their any deformation, it is intended that cover
Cover it is non-exclusive include, for example, the process, method, system, product or equipment for containing a series of steps or units are not necessarily limited to
Step or unit those of is clearly listed, but may include be not clearly listed or for these process, methods, product
Or other step or units that equipment is intrinsic.
Fig. 1 is a kind of method of data storage according to an embodiment of the present invention, as shown in Figure 1, the embodiment of the present invention is held
Row main body is the device of data storage, and this method comprises the following steps：
Step S101, data to be stored are obtained.
Step S102, the corresponding rack of the data is determined in memory space by Hash operation.
Step S103, the corresponding data copy of the data is obtained, and determines the data copy in the machine by Hash operation
The corresponding storage region of back end on frame.
Wherein, which is annular memory space, which corresponds to multiple virtual data nodes, multiple void
The responsible memory space of quasi- back end is made of multiple storage regions divided equally.
Further, before obtaining data to be stored, which is respectively operated to obtain multiple divide equally
Storage region.
Wherein, annular memory space is respectively operated, all dummy nodes in memory space all include more at this time
A to divide equally storage region, i.e. the responsible memory space of each dummy node is made of multiple regions of dividing equally.Work as back end
After being mapped to memory space by Hash operation, first equal subregion of back end present position node thus is found in addressing
Mapping position, continue to search next equal subregion clockwise using the method for linear hashing if the conflict of position
Domain, until the memory space that each back end is responsible for waits subregions to constitute by multiple.
Optionally, before the corresponding data copy of the acquisition data, determine whether the rack is abnormal；Determining the rack
When normal, the corresponding data copy of the data is obtained；When determining the rack exception, according to clock-wise order in the memory space
The interior next rack of search.
Optionally, before the data copy is stored in the storage region by this, which is being determined by Hash operation
According to copy after the corresponding storage region of back end in the rack, determine whether the back end in the rack is abnormal；?
When determining the back end exception, next rack is searched in the memory space according to clock-wise order；Determining the data section
When point is normal, determine whether the corresponding storage region of the back end is sufficient；When determining the storage region abundance, by the data
Copy is stored in the storage region；When determining that the storage region is inadequate, according to clock-wise order in the memory space
Search for next back end in the rack.
Step S104, the data copy is stored in the storage region.
In this step, data can complete grouping storage according to ready-portioned memory space, when having machine to add in cluster
When adding or deleting, by while keeping monotonicity, reducing number by the consistency hash algorithm rule migrated clockwise
According to the amount of migration and server pressure.
Further, after the data copy is stored in the storage region, determine whether total data copy is deposited
Storage is completed；When determining the not stored completion of total data copy, next machine is searched in the memory space according to clock-wise order
Frame.
It should be noted that the embodiment of the present invention passes through Hash operation Function Mapping storage to n for multiple data copies
A rack sets clockwise and chooses two different data nodes storages two in successively each rack in k rack
Copy chooses suitable back end in the sequentially connected rack of each of n-k rack and is used to store remaining copy.
The back end of rack is found in cyclic annular memory space by Hash operation mapping mode twice, detailed process is as shown in Figure 1.
For the addition and deletion of back end, the embodiment of the present invention can use consistency according to migration rules clockwise
Hash operation maps to obtain back end new on cyclic annular memory space, sends out in equal subregions of the data in single rack at this time
Raw mapping variation, storing data can be able to maintain original storage location or generate small range migration；Storing data backup is according to one
The Hash operation mapping of cause property is continuously stored in the rack that annulus adjoins, when cluster adds rack or encounters failure
When exiting rack, the data of rack adjacent thereto require to be migrated.When rack enters cyclic annular memory space, change rack
The migration situations of the several racks relevant to Backup Data in front and back is as shown in Fig. 2, as seen from Figure 2, at such algorithm
Reason distributed type assemblies data storage problem can inherit consistency hash algorithm while keeping monotonicity, can be to avoid a large amount of
Data Migration and the advantages of increase server stress.
Verifying explanation is carried out below for the above method；
HDFS storage cluster is built in a local network, and operating system uses Ubuntu14.04, installs Hadoop cloud platform meter
It calculates, Intel to strong 4 core CPU, 8G memory, 1TBSATA hard disk, Gigabit Ethernet is configured to the company of storage cluster back end
It connects, simulates 6 racks (each rack has 6,8,12,9,8,10 back end respectively).
The data of acquisition are shown in Table 1, the performance of back end be it is identical, back end is needed according to actual storage cluster
Situation is configured, and uses mac address of nic to combine with LAN IP as unique identification position, mapping function uses consistent
Property Hash Chord algorithm, equal part storage size is set as 64MB, and upload data scale is 50GB, stores about 1200
Storage region.
| File | Data copy number | File size | Occupied space | Record strip number |
| User information | 3 | 1MB | 3MB | 2130 |
| It gives out information | 3 | 16GB | 45GB | 512k |
| Environmental data | 3 | 640MB | 1900MB | 9680 |
Table 1
It is tested using the above-mentioned multi-frame connecting test parallel performance variation tendency that is provided with, verifying multi-frame parallel connection
With backup partition density relationship.Here it is considered that each back end equiblibrium mass distribution and individual performance difference is little, theoretically each machine
Back end is assigned having equal opportunities for data backup according to consistency hash algorithm in frame.From figure 3, it can be seen that according to number
According to load balancing theory actual conditions in rack back end backup should be close with theoretical case, can guarantee between rack
The balance of data stores.Fig. 4 show runing time comparative situation, uses consistency Hash optimization algorithm and standard hadoop
The end Connection inquiring Processing Algorithm that about subtracts under platform is attached inquiry to data, compares it is found that mapping tasks are in local completion
Data connection spends the time to be obviously shortened than transmitting from mapping end to the data for about subtracting end, and the starting for reducing data transmission is opened
Pin.
Experiment is uploaded using data, and data set is uploaded to HDFS, verifying optimization storage strategy and number from local file system
The number between back end is understood by considering the backup number distribution condition in rack back end according to the relationship of uploading speed
According to equalization performance.Theoretically, it is believed that back end is distributed in homogeneous area, and performance difference is little, then back end meeting
There is equal opportunity to be assigned the backup that rack obtains.As shown in Figure 5 as can be seen that with data scale growth, data pass
Defeated final call duration time can kept stable (51.5s or so and floating amplitude is little), each back end in rack
The probability for obtaining backup is able to maintain to float up and down in theoretical value (10%), shows that the back end in rack is being assigned backup
When can proof load it is balanced.
Consistency hash algorithm monotonicity and load balancing are verified according to Data Migration and backup number ratio variation tendency
Characteristic carries out back end addition and delete operation to cluster, obtains for rack by mapping actual value adjusted shown in Fig. 6
Take backup instances, it can be seen that newly added back end acquisition backup probability is higher, and the reason of such case occur is data
In the probability for completing that layout data node is needed again to obtain backup during mapping, but oscillation amplitude is smaller in zone of reasonableness
Interior (1.8%), the overall load not having an impact in rack are balanced；Such case occur is since it is desired that spending the other time
To select back end.Fig. 6, which gives, carries out data using consistency hash algorithm and the Hadoop platform strategy of random distribution
The runing time of operation compares, it can be seen that as data scale growth runing time increasing degree is smaller, data-handling efficiency
It is higher, illustrate that this algorithm for design can satisfy the processing of fairly large data.
Fig. 7 is a kind of method of data storage provided in an embodiment of the present invention, as shown in fig. 7, the embodiment of the present invention is held
Row main body is the device of data storage, and this method includes：
Step S701, data to be stored are obtained.
Step S702, the corresponding rack of the data is determined in memory space by Hash operation.
Wherein, which is annular memory space, which corresponds to multiple virtual data nodes, multiple void
The responsible memory space of quasi- back end is made of multiple storage regions divided equally.
Step S703, determine whether the rack is abnormal；
When determining that the rack is normal, step S704 to step S706 is executed；
When determining the rack exception, step S710 is executed.
Step S704, the corresponding data copy of the data is obtained.
Step S705, the corresponding memory block of back end of the data copy in the rack is determined by Hash operation
Domain.
Step S706, determine whether the back end in the rack is abnormal.
When determining that the back end in the rack is normal, step S707 is executed；
When determining the back end exception in the rack, step S710 is executed.
Step S707, determine whether the corresponding storage region of the back end is sufficient.
When determining the corresponding storage region abundance of the back end, step S708 is executed.
When determining that the corresponding storage region of the back end is inadequate, step S711 is executed.
Step S708, the data copy is stored in the storage region.
Step S700, determine whether total data copy stores completion.
When determining that the storage of total data copy is completed, then data storage terminates.
When determining the not stored completion of total data copy, step S710 is executed.
Step S710, next rack is searched in the memory space according to clock-wise order and continues to execute step S703.
Step S711, next back end for being searched in the rack in the memory space according to clock-wise order simultaneously continues
Execute step S707.
In this way, the present invention passes through hash algorithm and introduces virtual data node and the completion data storage of equal part storage region,
So as to data storage be rapidly completed, the technical issues of lifting system process performance while guaranteeing data balancing distribution.
It should be noted that for simple description, therefore, it is stated as a series of dynamic for above method embodiment
It combines, but those skilled in the art should understand that, the present invention is not limited by the sequence of acts described, because of foundation
The present invention, some steps may be performed in other sequences or simultaneously.Secondly, those skilled in the art should also know that, it says
Embodiment described in bright book belongs to preferred embodiment, and related actions and modules not necessarily present invention institute is necessary
's.
Fig. 8 is a kind of device of data storage provided in an embodiment of the present invention, as shown in figure 8, the device includes；
Acquiring unit 801, for obtaining data to be stored.
Processing unit 802, for determining the corresponding rack of the data in memory space by Hash operation, and obtaining should
The corresponding data copy of data, and the corresponding storage of back end of the data copy in the rack is determined by Hash operation
Region
Wherein, which is annular memory space, which corresponds to multiple virtual data nodes, multiple void
The responsible memory space of quasi- back end is made of multiple storage regions divided equally.
Storage unit 803, for the data copy to be stored in the storage region.
Optionally, the processing unit 802, is also used to before obtaining data to be stored, which is divided equally
Operation obtains multiple storage regions divided equally.
Optionally, as shown in figure 9, the device further includes：
First judging unit 804, for determining whether the rack is different before the corresponding data copy of the acquisition data
Often；
The processing unit 801, for it is corresponding to obtain the data when first judging unit 804 determines that the rack is normal
Data copy；When first judging unit 804 determines the rack exception, searched in the memory space according to clock-wise order
The lower rack of rope.
Optionally, as shown in Figure 10, which further includes：
Second judgment unit 805, for passing through Hash before the data copy is stored in the storage region by this
Operation determines that the data copy after the corresponding storage region of back end in the rack, determines the data section in the rack
Whether point is abnormal；
The processing unit 802 is used for when the second judgment unit 805 determines the back end exception, according to clockwise
Sequence searches for next rack in the memory space；
Second judgment unit 805 is also used to when determining that the back end is normal, determines the corresponding storage of the back end
Whether region is sufficient；
The storage unit 803 is used for when the second judgment unit 805 determines the storage region abundance, by the data pair
Originally it is stored in the storage region；
The processing unit 802 is used for when the second judgment unit 805 determines that the storage region is inadequate, according to up time
Needle sequence searches for next back end in the rack in the memory space.
Optionally, as shown in figure 11, which further includes：
Third judging unit 806 determines total data copy after the data copy is stored in the storage region
Whether storage is completed；
The processing unit 802 is deposited at this according to clock-wise order for after the completion of determining that total data copy is not stored
Store up the next rack of search in space.
Using data provided in an embodiment of the present invention store device, pass through hash algorithm and introduce virtual data node with
Equal part storage region completes data storage, so as to data storage be rapidly completed, mentions while guaranteeing data balancing distribution
The technical issues of rising system process performance.
It should be noted that affiliated those skilled in the art can be understood that, for convenience and simplicity of description,
The specific work process and description of the device of the data storage of foregoing description, can be with reference to corresponding in preceding method embodiment
Journey, details are not described herein.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
In the above embodiment of the invention, it all emphasizes particularly on different fields to the description of each embodiment, does not have in some embodiment
The part of detailed description, reference can be made to the related descriptions of other embodiments.
In several embodiments provided herein, it should be understood that disclosed technology contents can pass through others
Mode is realized.Wherein, the apparatus embodiments described above are merely exemplary, such as the division of the unit, Ke Yiwei
A kind of logical function partition, there may be another division manner in actual implementation, for example, multiple units or components can combine or
Person is desirably integrated into another system, or some features can be ignored or not executed.Another point, shown or discussed is mutual
Between coupling, direct-coupling or communication connection can be through some interfaces, the INDIRECT COUPLING or communication link of unit or module
It connects, can be electrical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit
The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple
On unit.It can some or all of the units may be selected to achieve the purpose of the solution of this embodiment according to the actual needs.
It, can also be in addition, the functional units in various embodiments of the present invention may be integrated into one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of software functional units.
If the integrated unit is realized in the form of SFU software functional unit and sells or use as independent product
When, it can store in a computer readable storage medium.Based on this understanding, technical solution of the present invention is substantially
The all or part of the part that contributes to existing technology or the technical solution can be in the form of software products in other words
It embodies, which is stored in a storage medium, including some instructions are used so that a computer
Equipment (can for personal computer, server or network equipment etc.) execute each embodiment the method for the present invention whole or
Part steps.And storage medium above-mentioned includes：USB flash disk, read-only memory (ROM, Read-Only Memory), arbitrary access are deposited
Reservoir (RAM, Random Access Memory), mobile hard disk, magnetic or disk etc. be various to can store program code
Medium.
The above is only a preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, various improvements and modifications may be made without departing from the principle of the present invention, these improvements and modifications are also answered
It is considered as protection scope of the present invention.
Claims (8)
1. a kind of method of data storage, which is characterized in that including：
Obtain data to be stored；
The corresponding rack of the data is determined in memory space by Hash operation；
The corresponding data copy of the data is obtained, and number of the data copy in the rack is determined by Hash operation
According to the corresponding storage region of node；Wherein, the memory space is annular memory space, and the back end corresponds to multiple virtual
Back end, the responsible memory space of the multiple virtual data node are made of multiple storage regions divided equally；
The data copy is stored in the storage region；Before the corresponding data copy of the acquisition data, institute
The method of stating further includes：Determine whether the rack is abnormal；The corresponding data copy of the data that obtains includes：Determining
State rack it is normal when, obtain the corresponding data copy of the data；When determining the rack exception, exist according to clock-wise order
The next rack of search in the memory space.
2. the method according to claim 1, wherein before obtaining data to be stored, the method also includes：
The memory space is respectively operated to obtain multiple storage regions divided equally.
3. the method according to claim 1, wherein the data copy is stored in the memory block described
Before in domain, the method also includes：
Determined by Hash operation the data copy after the corresponding storage region of back end in the rack, determine
Whether the back end in the rack is abnormal；
When determining the back end exception, next rack is searched in the memory space according to clock-wise order；
When determining that the back end is normal, determine whether the corresponding storage region of the back end is sufficient；
The described data copy is stored in the storage region include：When determining the storage region abundance, by institute
Data copy is stated to be stored in the storage region；
When determining that the storage region is inadequate, searched in the rack in the memory space according to clock-wise order
Next back end.
4. the method according to claim 1, wherein being stored in the storage region by the data copy
Later, the method also includes：
Determine whether total data copy stores completion；
When determining the not stored completion of total data copy, next machine is searched in the memory space according to clock-wise order
Frame.
5. a kind of device of data storage, which is characterized in that including：
Acquiring unit, for obtaining data to be stored；
Processing unit for determining the corresponding rack of the data in memory space by Hash operation, and obtains the number
According to corresponding data copy, and determines that back end of the data copy in the rack is corresponding by Hash operation and deposit
Storage area domain；Wherein, the memory space is annular memory space, and the back end corresponds to multiple virtual data nodes, described
The responsible memory space of multiple virtual data nodes is made of multiple storage regions divided equally；
Storage unit, for the data copy to be stored in the storage region；
Described device further includes：First judging unit, for determining institute before the corresponding data copy of the acquisition data
Whether abnormal state rack；The processing unit, for when first judging unit determines that the rack is normal, described in acquisition
The corresponding data copy of data；When first judging unit determines the rack exception, according to clock-wise order described
The next rack of search in memory space.
6. device according to claim 5, which is characterized in that the processing unit is also used to obtaining number to be stored
According to preceding, the memory space is respectively operated to obtain multiple storage regions divided equally.
7. device according to claim 5, which is characterized in that described device further includes：
Second judgment unit, for it is described the data copy is stored in the storage region before, passing through Hash
Operation determines that the data copy after the corresponding storage region of back end in the rack, determines in the rack
Whether back end is abnormal；
The processing unit, for when the second judgment unit determines the back end exception, according to clock-wise order
Next rack is searched in the memory space；
Second judgment unit is also used to when determining that the back end is normal, determines the corresponding memory block of the back end
Whether domain is sufficient；
The storage unit, for when the second judgment unit determines the storage region abundance, by the data copy
It is stored in the storage region；
The processing unit, for when the second judgment unit determines that the storage region is inadequate, according to suitable clockwise
Sequence searches for next back end in the rack in the memory space.
8. device according to claim 5, which is characterized in that described device further includes：
Third judging unit determines that total data copy is after the data copy is stored in the storage region
No storage is completed；
The processing unit, for after the completion of determining that total data copy is not stored, according to clock-wise order in the storage
Next rack is searched in space.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510810711.XA CN105511801B (en) | 2015-11-12 | 2015-11-12 | The method and apparatus of data storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510810711.XA CN105511801B (en) | 2015-11-12 | 2015-11-12 | The method and apparatus of data storage |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105511801A CN105511801A (en) | 2016-04-20 |
| CN105511801B true CN105511801B (en) | 2018-11-16 |
Family
ID=55719821
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510810711.XA Active CN105511801B (en) | 2015-11-12 | 2015-11-12 | The method and apparatus of data storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105511801B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105915626B (en) * | 2016-05-27 | 2019-02-26 | 南京邮电大学 | A kind of data copy initial placement method towards cloud storage |
| CN106503139A (en) * | 2016-10-20 | 2017-03-15 | 上海携程商务有限公司 | Dynamic data access method and system |
| CN107171971A (en) * | 2017-05-11 | 2017-09-15 | 北京云端智度科技有限公司 | Network traffic load method in a balanced way is realized using uniformity Hash strategy |
| CN109388335B (en) * | 2017-08-10 | 2021-10-29 | 中国移动通信集团宁夏有限公司 | Data storage method and system |
| CN110099112B (en) * | 2019-04-28 | 2022-03-29 | 平安科技（深圳）有限公司 | Data storage method, device, medium and terminal equipment based on point-to-point network |
| CN110275873A (en) * | 2019-06-28 | 2019-09-24 | 重庆紫光华山智安科技有限公司 | File memory method, device, storage management apparatus and storage medium |
| CN111563070A (en) * | 2020-05-15 | 2020-08-21 | 苏州浪潮智能科技有限公司 | Method and device for storing result of Hash algorithm |
| CN114328563B (en) * | 2021-12-31 | 2023-07-14 | 北京深演智能科技股份有限公司 | Data updating control method and device |
| CN115297131B (en) * | 2022-08-01 | 2023-05-26 | 东北大学 | Sensitive data distributed storage method based on consistent hash |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102737127A (en) * | 2012-06-20 | 2012-10-17 | 厦门聚海源物联网络技术有限公司 | Massive data storage method |
| CN102831120A (en) * | 2011-06-15 | 2012-12-19 | 腾讯科技（深圳）有限公司 | Data processing method and system |
| CN103761167A (en) * | 2014-01-23 | 2014-04-30 | 浪潮(北京)电子信息产业有限公司 | Method and device for achieving data center backup |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9398018B2 (en) * | 2014-03-18 | 2016-07-19 | nTrust Technology Solutions Corp. | Virtual currency system |
-
2015
- 2015-11-12 CN CN201510810711.XA patent/CN105511801B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102831120A (en) * | 2011-06-15 | 2012-12-19 | 腾讯科技（深圳）有限公司 | Data processing method and system |
| CN102737127A (en) * | 2012-06-20 | 2012-10-17 | 厦门聚海源物联网络技术有限公司 | Massive data storage method |
| CN103761167A (en) * | 2014-01-23 | 2014-04-30 | 浪潮(北京)电子信息产业有限公司 | Method and device for achieving data center backup |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105511801A (en) | 2016-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105511801B (en) | The method and apparatus of data storage | |
| CN112738820B (en) | Dynamic deployment method and device of service function chain and computer equipment | |
| EP3014444B1 (en) | Computing connected components in large graphs | |
| CN110169008B (en) | Distributed data redundancy storage method based on consistent hash algorithm | |
| CN102457571B (en) | Method for uniformly distributing data in cloud storage | |
| CN108769111A (en) | A kind of server connection method, computer readable storage medium and terminal device | |
| CN106302172A (en) | Support Hash lookup and the storage of route querying, lookup method and device simultaneously | |
| CN107256132B (en) | Performance test-based heterogeneous Redis cluster storage allocation method | |
| CN110362380A (en) | A kind of multiple-objection optimization virtual machine deployment method in network-oriented target range | |
| CN108282522A (en) | Data storage access method based on dynamic routing and system | |
| CN107070645A (en) | Compare the method and system of the data of tables of data | |
| CN106651000B (en) | Black-start partitioning method based on semi-supervised spectral clustering | |
| CN103229480A (en) | Data processing method, device and client in distributed storage system | |
| CN108073641A (en) | The method and apparatus for inquiring about tables of data | |
| CN106796588B (en) | The update method and equipment of concordance list | |
| CN105138536B (en) | Mobile social networking data fragmentation method based on Directed Hypergraph | |
| CN105302838A (en) | Classification method as well as search method and device | |
| CN106911777A (en) | A kind of data processing method and server | |
| CN111880926B (en) | Load balancing method and device and computer storage medium | |
| CN108304555A (en) | Distributed maps data processing method | |
| CN103414756B (en) | A kind of task distribution method, distribution node and system | |
| CN113835823A (en) | Resource scheduling method and device, electronic equipment and computer readable storage medium | |
| CN107818113A (en) | File access method for determining position and device | |
| CN108809680A (en) | A kind of method and apparatus of equipment management | |
| CN107547622B (en) | Resource adjusting method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220316 Address after: 130022 21 / F, Yuanchuang international, Weixing Road, Changchun City, Jilin Province Patentee after: Jilin Kasite Technology Co.,Ltd. Address before: 130022 No. 7186 Satellite Road, Changchun, Jilin, Chaoyang District Patentee before: CHANGCHUN University OF SCIENCE AND TECHNOLOGY |