CN110287197A - A kind of date storage method, moving method and device - Google Patents

A kind of date storage method, moving method and device Download PDF

Info

Publication number
CN110287197A
CN110287197A CN201910582925.4A CN201910582925A CN110287197A CN 110287197 A CN110287197 A CN 110287197A CN 201910582925 A CN201910582925 A CN 201910582925A CN 110287197 A CN110287197 A CN 110287197A
Authority
CN
China
Prior art keywords
virtual
fragment
key
server
destination
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910582925.4A
Other languages
Chinese (zh)
Other versions
CN110287197B (en
Inventor
郝忠秀
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Weimeng Chuangke Network Technology China Co Ltd
Original Assignee
Weimeng Chuangke Network Technology China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Weimeng Chuangke Network Technology China Co Ltd filed Critical Weimeng Chuangke Network Technology China Co Ltd
Priority to CN201910582925.4A priority Critical patent/CN110287197B/en
Publication of CN110287197A publication Critical patent/CN110287197A/en
Application granted granted Critical
Publication of CN110287197B publication Critical patent/CN110287197B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/214Database migration support
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2228Indexing structures
    • G06F16/2255Hash tables
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/256Integrating or interfacing systems involving database management systems in federated or virtual databases
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

This application discloses a kind of date storage method, moving method and devices.The date storage method includes: the virtual partition mark of virtual partition corresponding to calculation key according to the quantity of virtual partition in the cryptographic Hash of the key of data to be stored and preset first virtual level;Destination virtual fragment is determined according to the quantity of fragment virtual in virtual partition mark and preset second virtual level;The mapping table that virtual fragment and server are inquired by the virtual segmental identification of destination virtual fragment determines destination server, and wherein the corresponding relationship of virtual fragment and server is able to carry out adjustment in mapping table;The key-value pair of key and key should be stored in destination server.Pass through the date storage method, even if the quantity of server changes, it only needs to adjust the corresponding relationship of partial virtual fragment and server in mapping table, and data corresponding to the virtual fragment by the part are migrated, can reduce Data Migration amount compared with the existing technology.

Description

A kind of date storage method, moving method and device
Technical field
This application involves Internet technical field more particularly to a kind of date storage methods, moving method and device.
Background technique
Usually data are stored in the server of cluster in a manner of key-value pair (key-value) at present.It is inciting somebody to action When data are stored, the quantity of server determines data institute in the cryptographic Hash and cluster that the prior art passes through key (key) The server of storage.However in practical applications, since the quantity of server in cluster may be sent out due to dilatation etc. Changing, this data storage method of the prior art frequently can lead to a large amount of number when the quantity of server changes It is migrated from the server stored originally according to needs.
Summary of the invention
The embodiment of the present invention provides a kind of date storage method, moving method and device, can occur in number of servers When variation, the amount of migration of data is reduced.
In a first aspect, the embodiment of the invention provides a kind of date storage methods, this method comprises:
According to the quantity of virtual partition in the cryptographic Hash of the key of data to be stored and preset first virtual level, institute is calculated The virtual partition mark of virtual partition corresponding to key is stated, includes multiple virtual partitions in first virtual level;
Destination virtual is determined according to the quantity of fragment virtual in virtual partition mark and preset second virtual level Fragment includes multiple virtual fragments in second virtual level;
The mapping table of virtual fragment and server is inquired by the virtual segmental identification of the destination virtual fragment, really Set the goal server, wherein the corresponding relationship of virtual fragment and server is able to carry out adjustment in the mapping table;
The key-value pair of the key and the key should be stored in the destination server.
Second aspect, the embodiment of the invention provides a kind of data migration methods, this method comprises:
Determine virtual fragment to be migrated;
The mapping table of virtual fragment and server is inquired by the virtual segmental identification of virtual fragment to be migrated, really Fixed server to be moved into, wherein the corresponding relationship of virtual fragment and server is able to carry out adjustment in the mapping table;
By the corresponding data of virtual fragment to be migrated from former server migration into server to be moved into.
The third aspect, the embodiment of the invention provides a kind of data storage device, which includes: that virtual partition mark is true Order member, destination virtual fragment determination unit, destination server determination unit and storage unit, in which:
Virtual partition identifies determination unit, according in the cryptographic Hash of the key of data to be stored and preset first virtual level The quantity of virtual partition calculates the virtual partition mark of virtual partition corresponding to the key, includes in first virtual level Multiple virtual partitions;
Destination virtual fragment determination unit is virtually divided according in virtual partition mark and preset second virtual level The quantity of piece determines destination virtual fragment, includes multiple virtual fragments in second virtual level;
Destination server determination unit inquires virtual fragment and clothes by the virtual segmental identification of the destination virtual fragment The mapping table of business device, determines destination server, closes wherein virtual fragment is corresponding with server in the mapping table System is able to carry out adjustment;
The key-value pair of the key and the key should be stored in the destination server by storage unit.
Fourth aspect, the embodiment of the invention provides a kind of data migration device, described device includes: virtual fragment second Determination unit, the second determination unit of server and migration units, in which:
The second determination unit of virtual fragment, determines virtual fragment to be migrated;
The second determination unit of server inquires virtual fragment and clothes by the virtual segmental identification of virtual fragment to be migrated Be engaged in the mapping table of device, determine server to be moved into, wherein in the mapping table virtual fragment and server pair It should be related to and be able to carry out adjustment;
Migration units, by the corresponding data of virtual fragment to be migrated from former server migration to server to be moved into In.
Using date storage method provided by the embodiment of the present application, according to the cryptographic Hash of the key of data to be stored and in advance If the first virtual level in virtual partition quantity, calculate the virtual partition mark of virtual partition corresponding to the key, then root Destination virtual fragment is determined according to the quantity of virtual fragment in virtual partition mark and preset second virtual level, is then passed through The virtual segmental identification of the destination virtual fragment inquires the mapping table of virtual fragment and server, so that it is determined that destination service Device, and the key assignments of the key and the key is stored in the destination server, the wherein virtual fragment in mapping table and clothes The corresponding relationship of business device is able to carry out adjustment, by the date storage method, even if the quantity of server changes, it is only necessary to Adjust the corresponding relationship of partial virtual fragment and server in mapping table, and number corresponding to the virtual fragment by the part According to being migrated, Data Migration amount can reduce compared with the existing technology, to solve the problems of the prior art.
Detailed description of the invention
The drawings described herein are used to provide a further understanding of the present application, constitutes part of this application, this Shen Illustrative embodiments and their description please are not constituted an undue limitation on the present application for explaining the application.In the accompanying drawings:
Fig. 1 is a kind of flow diagram of date storage method provided by the embodiments of the present application;
Fig. 2 is a kind of flow diagram of data migration method provided by the embodiments of the present application;
Fig. 3 is the scenario simulation schematic diagram of data migration method provided by the embodiments of the present application;
Fig. 4 is a kind of structural schematic diagram of data storage device provided by the embodiments of the present application;
Fig. 5 is a kind of structural schematic diagram of data migration device provided by the embodiments of the present application.
Specific embodiment
To keep the purposes, technical schemes and advantages of the application clearer, below in conjunction with the application specific embodiment and Technical scheme is clearly and completely described in corresponding attached drawing.Obviously, described embodiment is only the application one Section Example, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art are not doing Every other embodiment obtained under the premise of creative work out, shall fall in the protection scope of this application.
Below in conjunction with attached drawing, the technical scheme provided by various embodiments of the present application will be described in detail.
Embodiment 1
Usually data can be stored in the following way at present, calculate the cryptographic Hash of the key of the data first, so The quantity of server in the cryptographic Hash and cluster is subjected to modulo operation afterwards, is taken later according in the result of modulo operation and cluster The corresponding relationship of business device, which is stored in corresponding server.But this data storage method, work as server When quantity increases or decreases, the result of number of servers modulo operation would generally change in the cryptographic Hash and cluster of key, lead A large amount of data needs are caused to be migrated from the server stored originally.
Based on this, embodiments herein 1 provides a kind of date storage method, is counted by the date storage method According to storage can also reduce the amount of migration of data compared with the existing technology even if number of servers increases or decreases, to solve Certainly the problems of the prior art.
The application constructs the first virtual level and the second virtual level in advance, wherein the first virtual level includes multiple virtual partitions (partition), the second virtual level includes multiple virtual fragments (shard), and each virtual partition respectively corresponds one Virtual partition identifies (partition_ID), and the quantity (partition_number) of virtual partition can root in the first virtual level According to being set, for example it is set as some fixed value, each virtual fragment also respectively corresponds a virtual fragment mark Know (shard_ID).
Include one or more keys (key) in tables of data, the key assignments (value) corresponding to key can be belonged in key It is stored in tables of data, includes one or more tables of data (table) in database, each tables of data also has one respectively For the data table name (table_name) of the unique identification tables of data, each database has one to be used for unique identification The wide area information server title (db_name).
The purpose of first virtual level be in order to can the more accurate quantity in tables of data rank control data table, thus The different dispersion of business demand data and extent of polymerization are coped with, the quantity of virtual partition can reflect cluster in the first virtual level The reading performance of middle server.Second virtual level is the lower layer in the first virtual level, for the virtual fragment that server tier is done, Therefore the virtual fragment in the second virtual level can be used for the migration that cluster conveniently realizes data during dilatation or capacity reducing.
As shown in Figure 1, the date storage method of the application includes the following steps:
Step S11: it according to the quantity of virtual partition in the cryptographic Hash of the key of data to be stored and the first virtual level, calculates The virtual partition of virtual partition corresponding to the key identifies.
In practical applications, data to be stored can be the characteristic of user etc., the sample data of model, model data Deng these data to be stored include key (key) and corresponding key assignments (value), these data can be stored in tables of data In.
Cryptographic Hash is by carrying out one group of numerical value that cryptographic calculation obtains to file content.It can be counted by hash function The cryptographic Hash for calculating key, is not defined the concrete type of hash function here, can be in conjunction with the process performance and not of hardware Used hash function is determined with the complexity of hash function, for example when the process performance of hardware is higher, can be used The higher hash function of complexity carrys out the cryptographic Hash of calculation key, when the process performance of hardware is lower, then can use complexity Lower hash function carrys out the cryptographic Hash of calculation key.
According to the quantity of virtual partition in the cryptographic Hash of key and the first virtual level, virtual partition corresponding to the key is calculated Virtual partition mark mode can there are many, can be with for example, after through cryptographic Hash of the hash function to calculate key The cryptographic Hash of key and the quantity of virtual partition are subjected to modulo operation, to calculate virtual point of virtual partition corresponding to key Area's mark, can also be by carrying out complementation for the quantity of the cryptographic Hash of key and virtual partition, to calculate corresponding to key Virtual partition virtual partition mark.
For example virtual partition mark can be calculated using following calculation formula:
Partition_ID=hash (key) %partition_number
In formula, partition_ID is identified for calculated virtual partition, hash (key) is key cryptographic Hash, Partition_number is the quantity of virtual partition in the first virtual level, " % " is modulo operator, which passes through key The modulo operation of cryptographic Hash and the quantity of virtual partition identifies to calculate virtual partition.
Step S12: target is determined according to the quantity of fragment virtual in virtual partition mark and preset second virtual level Virtual fragment.
In the second virtual level, the quantity of virtual fragment can be set according to actual needs, such as according to virtual point The quantity of virtual fragment can also be set as certain to determine the quantity of virtual fragment by the ratio of piece quantity and number of servers One fixed value.
The mode of destination virtual fragment is determined according to the quantity of fragment virtual in virtual partition mark and the second virtual level Can also there are many, such as can first calculate virtual partition mark cryptographic Hash, then by virtual partition identify cryptographic Hash Modulo operation or complementation are carried out with the quantity of fragment virtual in the second virtual level, calculates the virtual of destination virtual fragment Segmental identification, so that it is determined that destination virtual fragment out, this mode is carried out by virtual partition mark and the quantity of virtual fragment Remainder or modulus realize, more fast.
Furthermore it is also possible to by the title (table_name) and data of virtual partition mark the belonged to tables of data of associative key The title (db_name) in library calculates virtual partition mark, Hash whole after table_name and db_name triplicity Then the quantity of fragment virtual in the cryptographic Hash and the second virtual level is carried out modulo operation or complementation, calculated by value The virtual segmental identification of destination virtual fragment, so that it is determined that destination virtual fragment out, this mode is in the process for calculating cryptographic Hash In, db_name, table_name and virtual partition mark are combined, it can be by adjusting the side such as sequence of positions of this three Formula calculated cryptographic Hash to adjust, therefore safety is higher, any one can be selected to determine mesh according to actual needs Mark the mode of virtual fragment.
For example virtual partition mark can be calculated using following calculation formula:
Shard_ID=hash (db_name, table_name, partition_ID) %shard_number
In formula, shard_ID is by the virtual segmental identification of calculated destination virtual fragment, shard_number The quantity of virtual fragment in second virtual level, " % " are modulo operator, hash (db_name, table_name, Partition_ID the data table name and virtual partition that) database-name, the key belonged to by key is belonged to identify three In conjunction with whole cryptographic Hash, which passes through the quantity of virtual fragment in whole cryptographic Hash and the second virtual level after triplicity Modulo operation is carried out, to calculate the virtual segmental identification of destination virtual fragment.
In particular, in the first virtual level the quantity of virtual partition be set as some fixed values (for example, 100 or other Value), and when the quantity of virtual fragment is also set to some fixed value (for example, 1000 or other values) in the second virtual level, By step S11 and step S12 it is found that destination virtual fragment corresponding to key will uniquely determine.
Step S13: the corresponding relationship of virtual fragment and server is inquired by the virtual segmental identification of destination virtual fragment Table determines destination server.
Wherein, the corresponding relationship of virtual fragment and server is able to carry out adjustment in the mapping table.Also, due to In the mapping table, the corresponding relationship of virtual fragment and server is able to carry out adjustment, therefore when the quantity of server occurs Variation, it is only necessary to adjust the corresponding relationship of the mapping table virtual fragment and server, then will carry out relationship adjustment Data corresponding to virtual fragment are migrated, and do not need to redefine number according to the key of data and number of servers According to the new demand servicing device to be stored, to reduce the amount of migration of data.
It usually can be by modification configuration file to the corresponding relationship of virtual fragment and server in the mapping table It is adjusted, the corresponding relationship can also be adjusted by other means.
Therefore, in practical applications can by periodically (such as every 10 minutes 1 time or other times period), or Person in real time detects configuration file, when detecting that configuration file is modified, can pass through destination virtual fragment Mapping table after virtual segmental identification query modification, and mesh corresponding to destination virtual fragment is judged according to query result Whether mark server is adjusted, and in case of adjustment, illustrates that the quantity of server may have occurred variation, or due to Certain server breaks down, and can determine fresh target server adjusted later, then takes the target before adjustment The corresponding Data Migration of destination virtual fragment is into fresh target server adjusted in business device.
Such as when server progress dilatation, it is only necessary to, will be certain virtual in mapping table by modifying configuration file Fragment adjust it is to being carried out with the server newly increased corresponding, and by Data Migration corresponding to these virtual fragments to newly increasing In server, and other data are not needed to migrate.
When server carries out capacity reducing, the service of capacity reducing can also will be needed in mapping table by modifying configuration file Virtual fragment in device, adjust it is corresponding to being carried out with other servers, and by the virtual fragment institute in the server for needing capacity reducing The corresponding corresponding server of Data Migration, does not need to migrate other data yet.
It, can be by right when therefore the quantity of server being caused to need to be adjusted due to dilatation or capacity reducing etc. Destination server corresponding to the destination virtual fragment in relation table is answered to be adjusted, it, can be with after corresponding relationship adjustment Mapping table is inquired by the virtual segmental identification of destination virtual fragment, so that it is determined that fresh target server adjusted, so Afterwards by Data Migration corresponding to destination virtual fragment in the destination server before adjustment into fresh target server adjusted.
Furthermore it is possible to by the mapping table in the memory of management system long-term O&M, in this way number of servers need When being adjusted, the corresponding relationship of virtual fragment and server in mapping table can be adjusted, in time convenient for data Migration.
Step S14: the key-value pair of key and the key should be stored in destination server.
When the key assignments of the key and the key is stored in destination server, phase in destination server can be stored in In the tables of data answered, which belongs to the database in destination server.
Using date storage method provided by the embodiment of the present application, according to the cryptographic Hash of the key of data to be stored and in advance If the first virtual level in virtual partition quantity, calculate the virtual partition mark of virtual partition corresponding to the key, then root Destination virtual fragment is determined according to the quantity of virtual fragment in virtual partition mark and preset second virtual level, is then passed through The virtual segmental identification of the destination virtual fragment inquires the mapping table of virtual fragment and server, so that it is determined that target takes out Be engaged in device, and the key-value pair of the key and the key should be stored in destination server, wherein the virtual fragment in mapping table with The corresponding relationship of server is able to carry out adjustment.By the date storage method, even if the quantity of server changes, only need The corresponding relationship of partial virtual fragment and server in mapping table is adjusted, and corresponding to the virtual fragment by the part Data are migrated, and can reduce Data Migration amount compared with the existing technology, to solve the problems of the prior art.
For example, existing 100 servers are used to store the value of 10000 key, the prior art is respectively by this 10000 The corresponding cryptographic Hash of key carries out modulus operation with 100, and determines the server to be stored according to the result of modulus operation, but Be when dilatation increases to 101 servers, this corresponding cryptographic Hash of 10000 key with 101 modulus operation result It would generally change, be migrated so as to cause mass data.
And by method provided herein, when carrying out the storage of data, pass through the cryptographic Hash of the key of data first And first virtual partition in virtual level quantity, calculate the virtual partition mark of virtual partition corresponding to the key, then Destination virtual fragment is determined according to the quantity of fragment virtual in virtual partition mark and the second virtual level, then passes through target void The virtual segmental identification of quasi- fragment inquires the mapping table of virtual fragment and server, destination server is determined, to carry out The corresponding storage of corresponding key and value.Accordingly even when number of servers increases to 101 by 100, it can be by adjusting correspondence The corresponding relationship of virtual fragment and server in relation table, will need virtual fragment corresponding to the key of data migrated adjust to It is carried out with newly-increased server corresponding, and by these Data Migrations into newly-increased server, does not need to count other According to being migrated, therefore it can reduce the amount of migration of data.
Need to obtain the same user it should be noted that being frequently encountered in practical applications (such as in algorithm engineering) Multiple characteristic informations the case where, need to obtain the multi-group data of the user at this time, the multi-group data can be stored in In same server, so that efficiency increases when obtaining the multi-group data.It can be by following mode by same user Multi-group data be stored in same server:
Major key (primary key) and column are determined from the multiple groups data to be stored of same user respectively corresponding key Key (column key), wherein the key assignments of major key is used to identify the user, for example can be user name, identification card number, phone number Unique identification etc. of code, login account name or system distribution, can using the key except primary key as column key, For example can be age, gender etc..
After determining primary key and column key, the cryptographic Hash of primary key can be first calculated, so Afterwards according to the quantity of virtual partition in the cryptographic Hash and preset first virtual level, void corresponding to primary key is calculated The virtual partition mark of quasi- subregion, for example pass through the modulus operation of the quantity of virtual partition in the cryptographic Hash and the first virtual level, To calculate the virtual partition mark of virtual partition.After the virtual partition mark for calculating the virtual partition, Ke Yigen Destination virtual fragment is determined according to above-mentioned steps S12, and destination server is determined according to step S13, then by the user's The key assignments of primary key and primary key in multiple groups data to be stored and the column key and column The key assignments of key is stored in the destination server, so that the subsequent multiple groups that can obtain the user from the destination server Data increase the efficiency of acquisition.
Due to the dilatation of server or capacity reducing etc., number of servers is caused to change, partial data need into When row migration, Data Migration can be carried out by data migration method as shown in Figure 2, this method comprises the following steps:
Step S21: virtual fragment to be migrated is determined.
Virtual fragment to be migrated can be determined in several ways, such as when certain data are migrated, Virtual fragment corresponding to these data can be determined as virtual fragment to be migrated, can also be according to corresponding to virtual fragment The data volume size of data determines virtual fragment to be migrated, for example, in server expansion, in order to reduce Data Migration Amount, can be determined as virtual fragment for the lesser virtual fragment of data volume, or in server capacity reducing, by the service to capacity reducing Virtual fragment in device is determined as virtual fragment to be migrated.
Step S22: it is corresponding with server that virtual fragment is inquired by the virtual segmental identification of virtual fragment to be migrated Relation table determines server to be moved into.
Wherein the corresponding relationship of virtual fragment and server is able to carry out adjustment in the mapping table.For example, can lead to Modification configuration file is crossed to be adjusted to the corresponding relationship of fragment virtual in the mapping table and server.
Step S23: by the corresponding data of virtual fragment to be migrated from former server migration into server to be moved into.
The concrete mode of Data Migration can be, first by the corresponding data of virtual fragment to be migrated from former server replicates Into server to be moved into, after duplication is completed, data corresponding in former server are deleted.
Using the data migration method, during server expansion or capacity reducing, it is only necessary to determine go out to be migrated virtual Fragment, and mapping table is inquired by the virtual segmental identification of the virtual fragment to be migrated, so that it is determined that going out to be moved into Server, and will corresponding Data Migration to should in server be moved into, can determine according to actual needs to The virtual fragment of migration, and adjusts the corresponding relationship of virtual fragment and server in mapping table, can then be directed to Data corresponding to the virtual fragment of migration are migrated, and other data, compared with the existing technology, data can't be related to The amount of migration is less.
It is illustrated in figure 3 the corresponding data of virtual fragment B and moves to server 3 (node3) from server 1 (node1) Process schematic.It include initially server 1 and server 2 (node2) two-server in cluster, wherein servicing in the Fig. 3 Include data corresponding to virtual fragment A and B in device 1, include data corresponding to virtual fragment C in server 2, cluster passes through It include server 1, server 2 and server 3 after dilatation, using virtual fragment B as virtual fragment to be migrated, and by its institute Data corresponding to virtual fragment B are first replicated that (data are same into server 3 by corresponding Data Migration from server 1 at this time Step) into device 3 to be serviced, it is completed in duplication and then deletes data corresponding in server 1.
Embodiment 2
Based on inventive concept identical with the date storage method in the embodiment of the present application 1, the embodiment of the present application 2 is provided A kind of data storage device can be used in solving the problems of the prior art, in addition, for the explanation in the embodiment 2, if any Do not know place, it can be with the related content in reference implementation example 1.As shown in figure 4, the data storage device 30 includes virtual partition Determination unit 301, destination virtual fragment determination unit 302, destination server determination unit 303 and storage unit 304 are identified, Wherein:
Virtual partition identifies determination unit 301, virtually according to the cryptographic Hash of the key of data to be stored and preset first The quantity of virtual partition in layer, the virtual partition for calculating virtual partition corresponding to the key identifies, in first virtual level Including multiple virtual partitions, the data to be stored includes key and key assignments;
Destination virtual fragment determination unit 302, according to empty in virtual partition mark and preset second virtual level The quantity of quasi- fragment determines destination virtual fragment, includes multiple virtual fragments in second virtual level;
Destination server determination unit 303 inquires virtual fragment by the virtual segmental identification of the destination virtual fragment With the mapping table of server, determine destination server, wherein in the mapping table virtual fragment and server pair It should be related to and be able to carry out adjustment;
The key-value pair of the key and the key should be stored in the destination server by storage unit 304.
The data storage device 30 as provided in the embodiment of the present application 2, using with the data in the embodiment of the present application 1 The identical inventive concept of storage method, therefore the problems of the prior art are also able to solve, this is just repeated no more here.Separately Outside, in practical applications, by the way that the data storage device 30 is combined specific hardware device, acquired others technical effect Apply to distributed type assemblies In, due to can reduce Data Migration amount during cluster dilatation or capacity reducing, to improve the business valence of the distributed type assemblies Value.
In practical applications, which can also include key determination unit, key determination unit, from same use The multiple groups data to be stored at family determines major key in corresponding key respectively, wherein the key assignments of the major key is for identifying the use Family;Then, virtual partition identifies determination unit, according to virtual partition in the cryptographic Hash of the major key and preset first virtual level Quantity, calculate virtual partition corresponding to the major key virtual partition mark;And storage unit, by the user's Key except major key and key assignments and the major key and key assignments in each group data to be stored respectively correspond and are stored in the target Server.
The data storage device 30 can also include the second detection migration units, detect that configuration file is repaired for working as When changing, pass through the mapping table after the virtual segmental identification query modification of the destination virtual fragment;Sentenced according to query result Whether the destination server corresponding to the destination virtual fragment that breaks adjusts;If so, by the destination server before adjustment Described in the corresponding Data Migration of destination virtual fragment into fresh target server adjusted.
The data storage device 30 can also include mapping table adjustment unit, for by modifying configuration file to institute The corresponding relationship of virtual fragment and server in mapping table is stated to be adjusted.
Virtual partition identifies determination unit 301, can will be virtual in the cryptographic Hash of the key and preset first virtual level The quantity of subregion carries out modulo operation or complementation, and modulo operation or the operation result of complementation is right as the key institute The virtual partition for the virtual partition answered identifies.
Destination virtual fragment determination unit 302, the cryptographic Hash that the virtual partition is identified and preset second virtual level In the quantity of virtual fragment carry out modulo operation or complementation, using modulo operation or the operation result of complementation as described in The virtual segmental identification of destination virtual fragment determines the destination virtual fragment according to the virtual segmental identification;Or, will be described It is empty in the cryptographic Hash of the title of virtual partition mark, the title of the belonged to tables of data of the key and database and the second virtual level The quantity of quasi- fragment carries out modulo operation or complementation, using modulo operation or the operation result of complementation as the target The virtual segmental identification of virtual fragment, determines the destination virtual fragment according to the virtual segmental identification.
Embodiment 3
Based on inventive concept identical with the data migration method in the embodiment of the present application 1, the embodiment of the present application 3 is provided A kind of data migration device can be used in solving the problems of the prior art, in addition, for the explanation in the embodiment 3, if any Do not know place, it can also be with the related content in reference implementation example 1.As shown in figure 5, the data migration device 40 includes: virtual The second determination unit of fragment 401, the second determination unit of server 402 and migration units 403, in which:
The second determination unit of virtual fragment 401, determines virtual fragment to be migrated;
The second determination unit of server 402 inquires virtual fragment by the virtual segmental identification of virtual fragment to be migrated With the mapping table of server, server to be moved into is determined, wherein virtual fragment and server in the mapping table Corresponding relationship be able to carry out adjustment;
Migration units 403, by the corresponding data of virtual fragment to be migrated from former server migration to service to be moved into In device.
The data migration device 40 as provided in the embodiment 3 uses and the data migration method phase in embodiment 1 Same inventive concept, therefore also can solve the problems of the prior art, it just repeats no more here.
It in practical applications, can be by modification configuration file come pair to fragment and server virtual in mapping table It should be related to and be adjusted.
It should be understood by those skilled in the art that, the embodiment of the present invention can provide as method, system or computer program Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the present invention Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the present invention, which can be used in one or more, The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces The form of product.
The present invention be referring to according to the method for the embodiment of the present invention, the process of equipment (system) and computer program product Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates, Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one The step of function of being specified in a box or multiple boxes.
In a typical configuration, calculating equipment includes one or more processors (CPU), input/output interface, net Network interface and memory.
Memory may include the non-volatile memory in computer-readable medium, random access memory (RAM) and/or The forms such as Nonvolatile memory, such as read-only memory (ROM) or flash memory (flash RAM).Memory is computer-readable medium Example.
Computer-readable medium includes permanent and non-permanent, removable and non-removable media can be by any method Or technology come realize information store.Information can be computer readable instructions, data structure, the module of program or other data. The example of the storage medium of computer includes, but are not limited to phase change memory (PRAM), static random access memory (SRAM), moves State random access memory (DRAM), other kinds of random access memory (RAM), read-only memory (ROM), electric erasable Programmable read only memory (EEPROM), flash memory or other memory techniques, read-only disc read only memory (CD-ROM) (CD-ROM), Digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape magnetic disk storage or other magnetic storage devices Or any other non-transmission medium, can be used for storage can be accessed by a computing device information.As defined in this article, it calculates Machine readable medium does not include temporary computer readable media (transitory media), such as the data-signal and carrier wave of modulation.
It should also be noted that, the terms "include", "comprise" or its any other variant are intended to nonexcludability It include so that the process, method, commodity or the equipment that include a series of elements not only include those elements, but also to wrap Include other elements that are not explicitly listed, or further include for this process, method, commodity or equipment intrinsic want Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including described want There is also other identical elements in the process, method of element, commodity or equipment.
It will be understood by those skilled in the art that embodiments herein can provide as method, system or computer program product. Therefore, complete hardware embodiment, complete software embodiment or embodiment combining software and hardware aspects can be used in the application Form.It is deposited moreover, the application can be used to can be used in the computer that one or more wherein includes computer usable program code The shape for the computer program product implemented on storage media (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) Formula.
The above description is only an example of the present application, is not intended to limit this application.For those skilled in the art For, various changes and changes are possible in this application.All any modifications made within the spirit and principles of the present application are equal Replacement, improvement etc., should be included within the scope of the claims of this application.

Claims (10)

1. a kind of date storage method characterized by comprising
According to the quantity of virtual partition in the cryptographic Hash of the key of data to be stored and preset first virtual level, the key is calculated The virtual partition of corresponding virtual partition identifies, and includes multiple virtual partitions in first virtual level;
Destination virtual fragment is determined according to the quantity of fragment virtual in virtual partition mark and preset second virtual level, It include multiple virtual fragments in second virtual level;
The mapping table that virtual fragment and server are inquired by the virtual segmental identification of the destination virtual fragment, determines mesh Server is marked, wherein the corresponding relationship of virtual fragment and server is able to carry out adjustment in the mapping table;
The key of the data to be stored and key-value pair should be stored in the destination server.
2. method as described in claim 1, which is characterized in that the method also includes:
Major key is determined from the multiple groups data to be stored of same user respectively corresponding key, wherein the key assignments of the major key is used In the mark user;Then,
The quantity of virtual partition in the cryptographic Hash of the key according to data to be stored and preset first virtual level calculates institute The virtual partition mark for stating virtual partition corresponding to key, specifically includes: according to the cryptographic Hash of the major key and preset the The quantity of virtual partition in one virtual level calculates the virtual partition mark of virtual partition corresponding to the major key;And
It is described that the key of the data to be stored and key-value pair should be stored in the destination server, it specifically includes: will be described Major key and key assignments in each group data to be stored of user and the key except the major key and key assignments respectively correspond and are stored in institute State destination server.
3. method as described in claim 1, which is characterized in that the method also includes:
When detecting that configuration file is modified, after the virtual segmental identification query modification of the destination virtual fragment Mapping table;
According to query result judge the destination virtual fragment corresponding to destination server whether adjust;
If so, by the corresponding Data Migration of destination virtual fragment described in the destination server before adjustment to new mesh adjusted It marks in server.
4. method as described in claim 1, which is characterized in that according to the cryptographic Hash of the key of data to be stored and preset first The quantity of virtual partition in virtual level calculates the virtual partition mark of virtual partition corresponding to the key, specifically includes:
The quantity of virtual partition in the cryptographic Hash of the key and preset first virtual level is subjected to modulo operation or complementation, It is identified using modulo operation or the operation result of complementation as the virtual partition of virtual partition corresponding to the key.
5. method as described in claim 1, which is characterized in that according to virtual partition mark and preset second virtual level In the quantity of virtual fragment determine destination virtual fragment, specifically include:
The quantity of virtual fragment carries out modulo operation in the cryptographic Hash that the virtual partition is identified and preset second virtual level Or complementation, using modulo operation or the operation result of complementation as the virtual segmental identification of the destination virtual fragment, The destination virtual fragment is determined according to the virtual segmental identification;Or,
By the cryptographic Hash and second of the title of virtual partition mark, the title of the belonged to tables of data of the key and database The quantity of virtual fragment carries out modulo operation or complementation in virtual level, and modulo operation or the operation result of complementation are made For the virtual segmental identification of the destination virtual fragment, the destination virtual fragment is determined according to the virtual segmental identification.
6. a kind of data migration method characterized by comprising
Determine virtual fragment to be migrated;
Inquire the mapping table of virtual fragment and server by the virtual segmental identification of virtual fragment to be migrated, determine to The server moved into, wherein the corresponding relationship of virtual fragment and server is able to carry out adjustment in the mapping table;
By the corresponding data of virtual fragment to be migrated from former server migration into server to be moved into.
7. method as claimed in claim 6, which is characterized in that by modification configuration file to virtually dividing in the mapping table The corresponding relationship of piece and server is adjusted.
8. a kind of data storage device, which is characterized in that described device includes: virtual partition mark determination unit, destination virtual Fragment determination unit, destination server determination unit and storage unit, in which:
Virtual partition identifies determination unit, according to virtual in the cryptographic Hash of the key of data to be stored and preset first virtual level The quantity of subregion calculates the virtual partition mark of virtual partition corresponding to the key, includes multiple in first virtual level Virtual partition;
Destination virtual fragment determination unit, according to fragment virtual in virtual partition mark and preset second virtual level Quantity determines destination virtual fragment, includes multiple virtual fragments in second virtual level;
Destination server determination unit inquires virtual fragment and server by the virtual segmental identification of the destination virtual fragment Mapping table, determine destination server, wherein in the mapping table virtual fragment and server corresponding relationship energy Enough it is adjusted;
The key of the data to be stored and key-value pair should be stored in the destination server by storage unit.
9. device as claimed in claim 8, which is characterized in that described device further include: key determination unit, from the more of same user Group data to be stored determines major key in corresponding key respectively, wherein the key assignments of the major key is for identifying the user;Then,
Virtual partition identifies determination unit, according to virtual partition in the cryptographic Hash of the major key and preset first virtual level Quantity calculates the virtual partition mark of virtual partition corresponding to the major key;And
Storage unit, by each group data to be stored of the user major key and key assignments,
And key except the major key and key assignments respectively correspond and are stored in the destination server.
10. a kind of data migration device, which is characterized in that described device includes: virtual the second determination unit of fragment, server Two determination units and migration units, in which:
The second determination unit of virtual fragment, determines virtual fragment to be migrated;
The second determination unit of server inquires virtual fragment and server by the virtual segmental identification of virtual fragment to be migrated Mapping table, server to be moved into is determined, wherein virtual fragment pass corresponding with server in the mapping table System is able to carry out adjustment;
Migration units, by the corresponding data of virtual fragment to be migrated from former server migration into server to be moved into.
CN201910582925.4A 2019-06-28 2019-06-28 Data storage method, migration method and device Active CN110287197B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910582925.4A CN110287197B (en) 2019-06-28 2019-06-28 Data storage method, migration method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910582925.4A CN110287197B (en) 2019-06-28 2019-06-28 Data storage method, migration method and device

Publications (2)

Publication Number Publication Date
CN110287197A true CN110287197A (en) 2019-09-27
CN110287197B CN110287197B (en) 2022-02-08

Family

ID=68021453

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910582925.4A Active CN110287197B (en) 2019-06-28 2019-06-28 Data storage method, migration method and device

Country Status (1)

Country Link
CN (1) CN110287197B (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110968265A (en) * 2019-11-05 2020-04-07 北京字节跳动网络技术有限公司 Fragmentation expansion method and device and electronic equipment
CN111125059A (en) * 2019-12-17 2020-05-08 南京甄视智能科技有限公司 Data migration method and device, storage medium and server
CN111176830A (en) * 2019-12-03 2020-05-19 支付宝实验室(新加坡)有限公司 Information flow distribution method and device and server system
CN111708759A (en) * 2020-06-12 2020-09-25 北京思特奇信息技术股份有限公司 Optimization method and system for big table data migration
CN111708763A (en) * 2020-06-18 2020-09-25 北京金山云网络技术有限公司 Data migration method and device of fragment cluster and fragment cluster system
CN112039993A (en) * 2020-09-02 2020-12-04 中国银行股份有限公司 Long connection address processing method and device
CN113238993A (en) * 2021-05-14 2021-08-10 中国人民银行数字货币研究所 Data processing method and device
CN113507390A (en) * 2021-07-08 2021-10-15 中国工商银行股份有限公司 Distributed system cluster online fragmentation and capacity expansion method and device
CN114490826A (en) * 2022-01-29 2022-05-13 杭州宇信数字科技有限公司 Request response method, data storage method, device, server and storage medium
CN115113798A (en) * 2021-03-17 2022-09-27 中国移动通信集团山东有限公司 Data migration method, system and equipment applied to distributed storage

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103078927A (en) * 2012-12-28 2013-05-01 合一网络技术(北京)有限公司 Key-value data distributed caching system and method thereof
CN103166992A (en) * 2011-12-14 2013-06-19 中国电信股份有限公司 Method and system of fast file updating applied to cloud storage
CN103473334A (en) * 2013-09-18 2013-12-25 浙江中控技术股份有限公司 Data storage method, inquiry method and system
CN104008012A (en) * 2014-05-30 2014-08-27 长沙麓云信息科技有限公司 High-performance MapReduce realization mechanism based on dynamic migration of virtual machine
CN104077239A (en) * 2014-06-24 2014-10-01 华为技术有限公司 IP hard disk, and memory system and data operation method thereof
US20160034551A1 (en) * 2014-07-31 2016-02-04 International Business Machines Corporation Determination of data partition
CN106375404A (en) * 2016-08-30 2017-02-01 华为技术有限公司 Data storage control method and apparatus, data storage method and apparatus, and data acquisition method and apparatus
CN106471472A (en) * 2014-06-23 2017-03-01 甲骨文国际公司 System and method for the zoned migration in multi-tenant application server environment
CN106909556A (en) * 2015-12-23 2017-06-30 中国电信股份有限公司 The storage equalization methods and device of main memory cluster
US20170300490A1 (en) * 2016-04-15 2017-10-19 Apple Inc. Distributed offline indexing
CN108334551A (en) * 2017-12-29 2018-07-27 谷米科技有限公司 Date storage method and system, data query method and system
CN109815231A (en) * 2018-12-26 2019-05-28 亚信科技(中国)有限公司 Data fragmentation method, device, computer equipment and storage medium

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103166992A (en) * 2011-12-14 2013-06-19 中国电信股份有限公司 Method and system of fast file updating applied to cloud storage
CN103078927A (en) * 2012-12-28 2013-05-01 合一网络技术(北京)有限公司 Key-value data distributed caching system and method thereof
CN103473334A (en) * 2013-09-18 2013-12-25 浙江中控技术股份有限公司 Data storage method, inquiry method and system
CN104008012A (en) * 2014-05-30 2014-08-27 长沙麓云信息科技有限公司 High-performance MapReduce realization mechanism based on dynamic migration of virtual machine
CN106471472A (en) * 2014-06-23 2017-03-01 甲骨文国际公司 System and method for the zoned migration in multi-tenant application server environment
CN104077239A (en) * 2014-06-24 2014-10-01 华为技术有限公司 IP hard disk, and memory system and data operation method thereof
US20160034551A1 (en) * 2014-07-31 2016-02-04 International Business Machines Corporation Determination of data partition
CN106909556A (en) * 2015-12-23 2017-06-30 中国电信股份有限公司 The storage equalization methods and device of main memory cluster
US20170300490A1 (en) * 2016-04-15 2017-10-19 Apple Inc. Distributed offline indexing
CN106375404A (en) * 2016-08-30 2017-02-01 华为技术有限公司 Data storage control method and apparatus, data storage method and apparatus, and data acquisition method and apparatus
CN108334551A (en) * 2017-12-29 2018-07-27 谷米科技有限公司 Date storage method and system, data query method and system
CN109815231A (en) * 2018-12-26 2019-05-28 亚信科技(中国)有限公司 Data fragmentation method, device, computer equipment and storage medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
XIULEI QIN 等: "Optimizing data migration for cloud-based key-value stores", 《ACM》 *
李志远 等: "数据中心中服务器操作系统热迁移", 《电信快报》 *

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110968265A (en) * 2019-11-05 2020-04-07 北京字节跳动网络技术有限公司 Fragmentation expansion method and device and electronic equipment
CN110968265B (en) * 2019-11-05 2023-08-08 北京字节跳动网络技术有限公司 Method and device for expanding capacity by fragments and electronic equipment
CN111176830A (en) * 2019-12-03 2020-05-19 支付宝实验室(新加坡)有限公司 Information flow distribution method and device and server system
CN111125059A (en) * 2019-12-17 2020-05-08 南京甄视智能科技有限公司 Data migration method and device, storage medium and server
CN111125059B (en) * 2019-12-17 2022-08-12 南京甄视智能科技有限公司 Data migration method and device, storage medium and server
CN111708759A (en) * 2020-06-12 2020-09-25 北京思特奇信息技术股份有限公司 Optimization method and system for big table data migration
CN111708763A (en) * 2020-06-18 2020-09-25 北京金山云网络技术有限公司 Data migration method and device of fragment cluster and fragment cluster system
CN111708763B (en) * 2020-06-18 2023-12-01 北京金山云网络技术有限公司 Data migration method and device of sliced cluster and sliced cluster system
CN112039993A (en) * 2020-09-02 2020-12-04 中国银行股份有限公司 Long connection address processing method and device
CN112039993B (en) * 2020-09-02 2022-10-18 中国银行股份有限公司 Long connection address processing method and device
CN115113798A (en) * 2021-03-17 2022-09-27 中国移动通信集团山东有限公司 Data migration method, system and equipment applied to distributed storage
CN115113798B (en) * 2021-03-17 2024-03-19 中国移动通信集团山东有限公司 Data migration method, system and equipment applied to distributed storage
CN113238993A (en) * 2021-05-14 2021-08-10 中国人民银行数字货币研究所 Data processing method and device
CN113238993B (en) * 2021-05-14 2023-12-05 中国人民银行数字货币研究所 Data processing method and device
CN113507390A (en) * 2021-07-08 2021-10-15 中国工商银行股份有限公司 Distributed system cluster online fragmentation and capacity expansion method and device
CN114490826A (en) * 2022-01-29 2022-05-13 杭州宇信数字科技有限公司 Request response method, data storage method, device, server and storage medium
CN114490826B (en) * 2022-01-29 2023-01-13 杭州宇信数字科技有限公司 Request response method, data storage method, device, server and storage medium

Also Published As

Publication number Publication date
CN110287197B (en) 2022-02-08

Similar Documents

Publication Publication Date Title
CN110287197A (en) A kind of date storage method, moving method and device
CN107807796B (en) Data layering method, terminal and system based on super-fusion storage system
CN107229518B (en) Distributed cluster training method and device
CN107122126B (en) Data migration method, device and system
CN107645403A (en) Terminal rule engine apparatus, terminal rule operation method
CN106407207B (en) Real-time newly-added data updating method and device
CN103136243A (en) File system duplicate removal method and device based on cloud storage
CN107133228A (en) A kind of method and device of fast resampling
CN104899079B (en) A kind of method and device of data processing
CN113177050B (en) Data equalization method, device, query system and storage medium
CN108595268A (en) A kind of data distributing method, device and computer readable storage medium based on MapReduce
US20190158367A1 (en) Selection of cloud service providers to host applications
CN115129782A (en) Partition level connection method and device for distributed database
CN109510852A (en) The method and device of gray scale publication
CN110427364A (en) A kind of data processing method, device, electronic equipment and storage medium
Boukhelef et al. Optimizing the cost of DBaaS object placement in hybrid storage systems
CN109871181A (en) A kind of Object Access method and device
US10489074B1 (en) Access rate prediction in a hybrid storage device
CN107391039B (en) Data object storage method and device
CN112817540B (en) Data block redistribution method and device based on disk array expansion
US11164086B2 (en) Real time ensemble scoring optimization
Mai et al. Dynamic Data Migration in Hybrid Main Memories for In‐Memory Big Data Storage
CN110659296B (en) Storage method, device, equipment and computer readable medium
US20210165677A1 (en) Load balancing vm selection and movement
CN104239520A (en) Historical-information-based HDFS (hadoop distributed file system) data block placement strategy

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant