CN107465729B - Virtual node copy adjusting system and method based on access time ranking - Google Patents
Virtual node copy adjusting system and method based on access time ranking Download PDFInfo
- Publication number
- CN107465729B CN107465729B CN201710609966.9A CN201710609966A CN107465729B CN 107465729 B CN107465729 B CN 107465729B CN 201710609966 A CN201710609966 A CN 201710609966A CN 107465729 B CN107465729 B CN 107465729B
- Authority
- CN
- China
- Prior art keywords
- node
- virtual
- physical
- nodes
- copy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1029—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers using data related to the state of servers by a load balancer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1031—Controlling of the operation of servers by a load balancer, e.g. adding or removing servers that serve requests
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The invention relates to a virtual node copy adjusting system and method based on access time ranking. Relates to the field of computer information storage. The invention provides a copy adjustment method design based on virtual node access ranking, when a new node is added or an old node is deleted, the mapping relation between the virtual node and the physical node is dynamically adjusted according to the access frequency ranking of the virtual node, the load balance of a storage system is realized, and the storage performance is improved.
Description
Technical Field
The invention relates to the technical field of computer information storage, in particular to a virtual node copy adjusting system and method based on access frequency ranking in object storage.
Background
With the development of technologies such as the internet, the internet of things and the like, especially the wide application of a web2.0 data-intensive network technology, unstructured data is exponentially increased. It is expected that by 2020, the amount of data will reach 40ZB, 80% of which is unstructured data, most of which is inactive data. The storage of extended capacity blocks and files with PB level capability cannot meet the requirement of data storage, and a new storage system rack-object storage comes up. The object storage is one of three services of cloud storage, and two requirements of storage filing and cloud service are met. In the field of aerospace, the method can be used for storing unstructured data such as satellite cloud pictures and aircraft operation orbit data, and provides storage service of elastic expansion for a cloud platform. In order to ensure the security of data, a copy technology is introduced. In the aspect of copy adjustment, when physical nodes are added or deleted, due to the randomness of virtual node copy adjustment, the load of the system is unbalanced.
Disclosure of Invention
Technical problem to be solved
The technical problem to be solved by the invention is as follows: how to realize the mapping relationship between the virtual nodes and the physical nodes when a new node is added into the storage system or an old node is deleted from the storage system, the load balance of the storage system is realized, and the storage performance is improved.
(II) technical scheme
In order to solve the above technical problem, the present invention provides a system for adjusting a virtual node copy based on access number ranking, comprising:
the virtual node access time sequencing module is used for determining the access times of the virtual nodes according to the access records of the users to the virtual nodes and ranking the virtual nodes to which each physical node in the storage system belongs according to the access times; each physical node corresponds to one device in the storage system, and each virtual node to which the physical node belongs is a copy, or a virtual node copy;
a virtual node copy migration list calculation module, configured to, when a physical node joins the storage system, obtain a weight of each physical node, calculate a virtual node number of a unit weight of the physical node, and add a virtual node copy to be adjusted to a migration list according to the unit weight of each physical node; when a physical node needs to be deleted or quit from the storage system, removing the corresponding equipment from the equipment list, and calculating the number of copies of the virtual nodes which can be received by each remaining physical node according to the number of the virtual nodes of the unit weight; the unit weight is equal to the device weight divided by the device capacity;
the virtual node copy adjusting module is used for adjusting the mapping relation between the virtual node and the physical node; when a physical node joins the storage system, directly modifying a mapping table from the virtual node to the physical node according to the migration list; when physical nodes need to be deleted or quit from the storage system, the number of virtual node copies which can be received by each physical node is calculated, the access quantity of unit weight is calculated according to the total access quantity of the physical nodes, the access quantity of the unit weight is sequenced according to the access quantity of the unit weight, and the virtual node copies with the corresponding number are sequentially obtained from the virtual node list of the deleted physical nodes and distributed to the corresponding physical nodes.
Preferably, the virtual node copy access sorting module comprises a virtual node copy access number recording submodule, a physical node owned virtual node copy extracting submodule and a virtual node copy access number sorting submodule;
the virtual node copy access recording submodule is used for recording the access times of the virtual node copies when a user requests an object, the object read by the user is/account/container/object, a Ring class get _ part () method is called to obtain a virtual node part where the object is located, the Ring class get _ node () method returns a physical node where the virtual node is located, and a shared dictionary, namely, visit _ Number _ Dic, is created for all devices to record the access times of each virtual node copy;
the physical node has a virtual node copy extraction submodule for traversing a two-dimensional array reapplica 2part2dev in a first-row and last-row mode, wherein the two-dimensional array stores the mapping relation from the virtual node to the physical node and extracts the virtual node copy to which each physical node belongs; then traversing a one-dimensional array partition _ single _ array by using an enumeration function, wherein the partition _ single _ array stores a virtual node copy serial Number to obtain an id of a physical node where a virtual node copy is located, judging whether the virtual node copy belongs to one of the physical nodes by comparing the ids of the physical nodes, and then storing the extracted virtual node copy in a shared dictionary view _ Number _ Dic mode, wherein the value is the access times of the virtual node copy;
the virtual node copy access sequencing submodule is used for ranking the access times of the virtual node copies owned by the physical nodes according to the access times of the virtual node copies, storing the virtual node copies owned by the physical nodes of each device and the access times into the dictionary dev _ access _ partition _ dic, obtaining a stored virtual node copy access time list by using the dictionary dev _ access _ partition _ dic, sequencing the virtual node copy access time list, and ranking the virtual nodes according to the sequenced access time list.
Preferably, when a physical node is added, the virtual node copy migration list calculation module includes: the system comprises a node adding submodule, a unit weight virtual node metering operator module and a virtual node copy migration list calculating submodule; the node joining submodule is used for joining the physical node to the storage system, and controlling the joining of the node by executing a command, namely: judging whether the command input is wrong or not, using an izip and islice tool to put forward relevant information of the equipment from a command line, then realizing the addition of a physical node, setting equipment id and equipment weight when adding, and finally adding the equipment into an equipment list; the unit weight virtual node metering operator module is used for adding and summing the weights of all the devices, solving the number of virtual nodes owned by the physical nodes of the unit weight according to the total number of the virtual nodes, traversing the device list, acquiring the weight of each physical node, and calculating the number of the virtual nodes of the unit weight; the virtual node copy migration list calculation submodule is used for adding the virtual node copy to be adjusted into the migration list according to the unit weight of each physical node;
when a physical node is deleted, the virtual node copy migration list calculation module comprises: the system comprises a node deleting submodule, a unit weight calculating submodule and a virtual node copy migration list calculating submodule; the node deletion submodule is used for finding the physical equipment from the equipment list according to the id of the equipment node, setting the weight of the physical node to be 0 and removing the equipment from the equipment list; the unit weight virtual node metering operator module is used for adding and summing the weights of all the devices, solving the number of virtual nodes owned by the physical nodes of the unit weight according to the total number of the virtual nodes, traversing the device list, acquiring the weight of each physical node, and calculating the number of the virtual nodes of the unit weight; the virtual node copy migration list calculation submodule is used for calculating the number of virtual node copies to be received by each physical node, traversing the equipment list in sequence, calculating the number of virtual node copies which can be received by each physical node according to the number of virtual nodes of unit weight of the physical equipment, and storing the virtual node copies in a list form.
Preferably, when a physical node is added, the virtual node copy adjusting module comprises a virtual node copy adjusting submodule, the virtual node copy to be adjusted is already added into the virtual node migration list, the migration list is traversed by using a map function, the adjustable virtual node copy is determined, the virtual node copy to be adjusted is sequentially searched in a preceding and following mode, and the mapping relation is modified according to the weight of the added physical node; is provided with a weight wn+1Node D ofn+1Adding the node D into a storage system, wherein n is the number of devices in the storage system when the node D is not added, and calculating the node D according to the weightn+1The number of virtual nodes to be allocated is
Combining existing physical nodes D in a storage system1,D2,D3…DnAnd weight w thereof1,w2,w3…wnCalculating the need to reassign N to each physical node1,N2,N3…NnA virtual node copy to Dn+1，
Let physical node D1Is named as P11,P12,P13…P1mIn turn will be
Total N1A virtual node copy is allocated to Dn+1To physical node D2,D3…DnTreating with the same method;
when physical nodes are deleted, the virtual node copy adjusting module comprises a unit weight access quantity sequencing submodule and a virtual node copy adjusting submodule, the total access times of the physical nodes are obtained by adding the access times of all the virtual nodes, then the total access times are divided by the weight of the physical nodes, the access quantity of the unit weight can be obtained, the physical nodes are ranked according to the access times of the unit weight by using bubble sequencing algorithm bubbleSort () due to the limited number of the physical nodes, when access adjustment is carried out, the virtual node copy of the physical node with small access quantity is adjusted to the physical node with large access quantity of the unit weight, the virtual node copy of the physical node with large access quantity is adjusted to the physical node with small access quantity of the unit weight, and when the weight w exists in the storage system, the virtual node copy of the physical node with large access quantity is adjusted to the physicaliPhysical node D ofiSetting the physical node D when exiting the storage systemiThe number of owned virtual nodes is NiCalculating the number of virtual nodes distributed to each storage node according to the weight of each remaining physical node
According to N1,N2,N3…NnCalculating the access times of virtual nodes of unit weight of the rest nodes in the storage system, and accessing the virtual nodes according to the unit weightThe times are ranked from large to small, and the physical node sequence is V1,V2,Vi…VnN ≠ i, i ∈ [1, n), sequentially from the virtual node list of deleted physical nodes according to the unit weight of the remaining physical nodes
And obtaining the corresponding number of virtual node copies, thereby completing the adjustment.
The invention also provides a method for adjusting the virtual node copies by using the system based on the ranking of the access times, which comprises the following steps:
determining the access times of the virtual nodes according to the access records of the users to the virtual nodes by using the virtual node access time sequencing module, and ranking the virtual nodes to which each physical node in the storage system belongs according to the access times; each physical node corresponds to one device in the storage system, and each virtual node to which the physical node belongs is a copy, or a virtual node copy;
acquiring the weight of each physical node by using the virtual node copy migration list calculation module when a physical node is added into the storage system, calculating the number of virtual nodes of unit weight of the physical node, and adding the virtual node copy to be adjusted into a migration list according to the unit weight of each physical node; when a physical node needs to be deleted or quit from the storage system, removing the corresponding equipment from the equipment list, and calculating the number of copies of the virtual nodes which can be received by each remaining physical node according to the number of the virtual nodes of the unit weight; the unit weight is equal to the device weight divided by the device capacity;
utilizing the virtual node copy adjusting module to adjust the mapping relation between the virtual node and the physical node; when a physical node joins the storage system, directly modifying a mapping table from the virtual node to the physical node according to the migration list; when physical nodes need to be deleted or quit from the storage system, the number of virtual node copies which can be received by each physical node is calculated, the access quantity of unit weight is calculated according to the total access quantity of the physical nodes, the access quantity of the unit weight is sequenced according to the access quantity of the unit weight, and the virtual node copies with the corresponding number are sequentially obtained from the virtual node list of the deleted physical nodes and distributed to the corresponding physical nodes.
(III) advantageous effects
The invention provides a copy adjustment method design based on virtual node access ranking, when a new node is added or an old node is deleted, the mapping relation between the virtual node and the physical node is dynamically adjusted according to the access frequency ranking of the virtual node, the load balance of a storage system is realized, and the storage performance is improved.
Drawings
FIG. 1 is a block diagram of a hardware architecture of a method for adjusting policy for virtual node replicas in object storage according to the present invention;
wherein:
1 proxy node 2 communication link between proxy nodes
3-proxy cluster 4-proxy cluster and storage cluster communication link
5 communication links between storage nodes 6 storage nodes
7 storage cluster
FIG. 2 is a diagram of the logical architecture of the invention;
wherein:
21 virtual node copy adjusting module 22 virtual node access ordering calculating module
23 virtual node migration list calculation module 24 virtual nodes
25 Ring
FIG. 3 is a schematic diagram of a physical node joining a storage system;
FIG. 4 is a schematic diagram of a physical node exiting a storage system.
Detailed Description
In order to make the objects, contents, and advantages of the present invention clearer, the following detailed description of the embodiments of the present invention will be made in conjunction with the accompanying drawings and examples.
The hardware platform required by the invention is an object storage system. As shown in fig. 1. The object storage system comprises a proxy cluster 3, a storage cluster 7, communication links 4 of the proxy cluster and the storage cluster. The agent cluster 3 comprises a plurality of agent nodes 1 and communication links 2 among the agent nodes; the storage cluster 7 comprises a number of storage nodes 6 and communication links 5 between the storage nodes.
The system of the invention is composed of a virtual node access sequencing calculation module 22, a virtual node migration list calculation module 23 and a virtual node copy adjustment module 21. Ring 25 represents a data structure that stores a mapping of virtual nodes to physical nodes.
1) Virtual node access sequencing calculation module
The virtual node copy access ordering module 22 is configured to rank virtual nodes owned by one physical node according to access times, and may be divided into a virtual node copy access time recording sub-module, a physical node owned virtual node copy extraction sub-module, and a virtual node copy access time ordering sub-module.
The virtual node copy access recording submodule is used for recording the access times of the virtual node copy when a user requests an object. The object read by the user is/account/container/object, and the get _ part () method of the Ring class is called to obtain the virtual node part where the object is located. The invention rewrites the method of the Ring type get _ node () to return a physical node where a virtual node is located, the return value is in the form of a triple, the first element in the triple is a physical node of a virtual node copy, the second element is a virtual node, and the third element is a mark of the virtual node copy. A shared dictionary, visit _ Number _ Dic, is created for all devices to record the Number of accesses of each virtual node copy, where key is a virtual node copy in tuple form, and value is the Number of accesses of the virtual node copy, account.
The physical node has a virtual node copy extraction submodule for traversing the two-dimensional array, namely the mapping replay 2part2dev from the virtual node to the physical node, in a preceding and following way, and extracting a virtual node copy owned by each physical node, wherein the virtual node copy is specifically realized as for part, part _ single _ array entity (replay 2part2 dev); and traversing a one-dimensional array partition _ single _ array (storing virtual node copy serial numbers) by using an enumeration function to obtain the id of a physical node where the virtual node copy is located, judging whether the virtual node copy belongs to a certain physical node by comparing the id of the physical node, and storing the extracted virtual node copy in a shared dictionary view _ Number _ Dic form, wherein the key is (part, replication) and the value is the access times of the virtual node copy.
And the virtual node copy access ordering submodule is used for ranking the access times of the virtual node copies owned by the physical node according to the access times of the virtual node copies. And storing the virtual node copy owned by the physical node of each device and the access times into a dictionary dev _ access _ partition _ dic. And obtaining a list of the access times of the saved virtual node copies by using a value () method of the dictionary dev _ access _ partition _ dic. As the physical node has more virtual node copies and the list of the access times of the virtual node copies is larger, the quick sorting algorithm quick () is selected to sort the list of the access times of the virtual node copies, the virtual nodes are ranked according to the sorted list of the access times, and the ranked virtual nodes are stored in the dev _ partition _ rank _ list.
2) Virtual node copy migration list calculation module
The virtual node copy migration list calculation module 23 is configured to calculate the number of the reallocated virtual node copies and add them to the virtual node migration list when a physical node is added or deleted. The method is divided into two cases of node addition and node deletion.
When a physical node is added, the virtual node copy migration list calculation module 23 is divided into: the system comprises a node adding submodule, a unit weight virtual node metering operator module and a virtual node copy migration list calculating submodule. The node adding submodule adds the physical node to the storage cluster 7 by calling the add () method of the swift-ring-builder class. The method comprises the steps of controlling the addition of a node by executing commands of swift-Ring-builder < builder _ file > add [ r < region > ] z < zone > < ip > < port >/< device _ name > _ meta > < weight >, judging whether the command input is wrong, extracting relevant information of equipment from a command line by utilizing an izip tool and an islice tool, such as values of region, zone, ip, port, device _ name, weight and meta, and calling the swift-common-Ring class and the Ring class add _ dev method to realize the addition of the physical node. When executing the dev _ add method, the device id and the device weight are set, and finally the device is added to the device list. And the unit weight virtual node meter operator module is used for adding and summing the weights of all the devices, and calculating the number of virtual nodes owned by the physical nodes of the unit weight according to the total number of the virtual nodes. The device list is traversed to obtain the weights of the physical nodes, and the number of virtual nodes of unit weight (equal to the device weight divided by the device capacity) is calculated. And the virtual node copy migration list calculation submodule adds the virtual node copy to be adjusted into the migration list according to the unit weight of each physical node.
When physical nodes are deleted, the virtual node copy migration list calculation module is divided into: the system comprises a node deleting submodule, a unit weight calculating submodule and a virtual node copy migration list calculating submodule. And the node deletion submodule finds the physical equipment from the equipment list according to the id of the equipment node by calling a remove _ dev method in the Ring class, sets the weight of the physical node to be 0, and removes the equipment from the equipment list by using a list remove method. The function of the unit weight calculation submodule is the same as that of the unit weight calculation submodule when the node is added, and the design and the implementation of the unit weight virtual node meter submodule can be referred to when the node is added.
When physical nodes are deleted, the virtual node copy migration list calculation submodule calculates the number of virtual node copies to be received by each physical node. By rewriting the Ring-like distribution function get _ part _ allowed (self, actstr), using the sequential traversal device list, calculating the number of virtual node copies that can be received by each physical node according to the number of virtual nodes of the unit weight of the physical device, and implementing the following steps:
weight _ of _ one _ part × dev [ wlan ] len (dev _ partiton _ rank _ list) is stored in a list, and the element type is a binary (dev, part _ found).
3) Virtual node copy adjusting module
The virtual node copy adjusting module is used for adjusting the mapping relation from the virtual node copy to the physical node, and the method comprises two conditions of node adding and node deleting. The system comprises a unit weight access amount calculation module and a virtual node copy adjustment submodule.
When the node is added, only the virtual node copy adjusting submodule is included. The virtual node copies needing to be adjusted are added into the virtual node migration list, the adjusted virtual node copies are determined by rewriting the Ring type's reassurance _ parts (actionstr, moved _ list), traversing the migration list moved _ list by using a map function (elements in the moved _ list exist in a binary form), searching the virtual node copies needing to be adjusted in sequence by using an enum function according to a preceding and following mode, and modifying the mapping relation according to the weight of the added physical node. Is provided with a weight wn+1Node D ofn+1Adding the virtual nodes into the storage cluster 7, and calculating the number of the virtual nodes needing to be distributed according to the weight
Combining storage nodes D1,D2,D3…DnAnd weight w thereof1,w2,w3…wnCalculating that each physical node needs to be redistributed
A virtual node copy to Dn+1Let physical node D1Is named as P11,P12,P13…P1mIn turn will be
Total N1A virtual node copy is allocated to Dn+1To physical node D2,D3…DnThe same procedure was used. The process is shown in figure 3.
When the node is deleted, the virtual node copy adjustment strategy design comprises two modules, a unit weight access quantity ordering submodule and a virtual nodeAnd a copy adjustment submodule. The virtual node copies owned by the physical node are ranked according to the access times, the access times of all the virtual nodes are added to obtain the total access times of the physical node, and then the total access times are divided by the weight of the physical node, so that the access amount of the unit weight, access _ num _ of _ one _ weight, can be obtained. Due to the fact that the number of the physical nodes is limited, ranking is conducted on the physical nodes according to the access times of the unit weight through a bubble sorting algorithm bubbleSort (). When access adjustment is performed, the virtual node copy with a small access amount is adjusted to the node with a large unit weight access amount, and the virtual node with a large access amount is adjusted to the node with a small unit weight access amount. When there is a weight w in the storage cluster 7iPhysical node D ofiWhen exiting the storage cluster 7, the number of virtual nodes owned by the node is set to NiThe number of virtual nodes distributed by each storage node 6 is calculated according to the weight of each other node
Calculating the number of virtual node accesses per unit weight of the remaining nodes in the storage cluster 7
Ranking from large to small according to the access times of the virtual nodes with unit weight, wherein the physical node sequence is V1,V2,Vi…VnN ≠ i, i ∈ [1, n ] according to the unit weight of undeleted (remaining) physical nodes, in turn from the virtual node list of the physical nodes that have been deleted
The corresponding number of virtual node copies is obtained, and the adjustment is completed, and the adjustment process is as shown in fig. 4.
The invention also provides a method for adjusting the virtual node copies by using the system based on the ranking of the access times, which comprises the following steps:
determining the access times of the virtual nodes according to the access records of the users to the virtual nodes by using the virtual node access time sequencing module, and ranking the virtual nodes to which each physical node in the storage system belongs according to the access times; each physical node corresponds to one device in the storage system, and each virtual node to which the physical node belongs is a copy, or a virtual node copy;
acquiring the weight of each physical node by using the virtual node copy migration list calculation module when a physical node is added into the storage system, calculating the number of virtual nodes of unit weight of the physical node, and adding the virtual node copy to be adjusted into a migration list according to the unit weight of each physical node; when a physical node needs to be deleted or quit from the storage system, removing the corresponding equipment from the equipment list, and calculating the number of copies of the virtual nodes which can be received by each remaining physical node according to the number of the virtual nodes of the unit weight; the unit weight is equal to the device weight divided by the device capacity;
utilizing the virtual node copy adjusting module to adjust the mapping relation between the virtual node and the physical node; when a physical node joins the storage system, directly modifying a mapping table from the virtual node to the physical node according to the migration list; when physical nodes need to be deleted or quit from the storage system, the number of virtual node copies which can be received by each physical node is calculated, the access quantity of unit weight is calculated according to the total access quantity of the physical nodes, the access quantity of the unit weight is sequenced according to the access quantity of the unit weight, and the virtual node copies with the corresponding number are sequentially obtained from the virtual node list of the deleted physical nodes and distributed to the corresponding physical nodes. .
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (2)
1. A system for adjusting virtual node copy ranking based on access times, comprising:
the virtual node access time sequencing module is used for determining the access times of the virtual nodes according to the access records of the users to the virtual nodes and ranking the virtual nodes to which each physical node in the storage system belongs according to the access times; each physical node corresponds to one device in the storage system, and each virtual node to which the physical node belongs is a copy called a virtual node copy;
a virtual node copy migration list calculation module, configured to, when a physical node joins the storage system, obtain a weight of each physical node, calculate a virtual node number of a unit weight of the physical node, and add a virtual node copy to be adjusted to a migration list according to the unit weight of each physical node; when a physical node needs to be deleted or quit from the storage system, the corresponding equipment is removed from the equipment list, and the number of the virtual node copies which can be received by each remaining physical node is calculated according to the number of the virtual nodes of the unit weight; the unit weight is equal to the device weight divided by the device capacity;
the virtual node copy adjusting module is used for adjusting the mapping relation between the virtual node and the physical node; when a physical node joins the storage system, directly modifying a mapping table from the virtual node to the physical node according to the migration list; when a physical node needs to be deleted or quit from the storage system, the method is used for calculating the number of virtual node copies which can be received by each physical node, calculating the access amount of unit weight according to the total access amount of the physical node, sequencing according to the access amount of the unit weight, and sequentially acquiring the corresponding number of virtual node copies from the virtual node list of the deleted physical node to distribute to the corresponding physical node;
when a physical node is added, the virtual node copy adjusting module comprises a virtual node copy adjusting submodule, a virtual node copy to be adjusted is added into a virtual node migration list, a map function is utilized to traverse the migration list, the adjustable virtual node copy is determined, the virtual node copy to be adjusted is sequentially searched according to a front-to-back mode, and the mapping relation is modified according to the weight of the added physical node; is provided with a weight wn+1Section (A) ofPoint Dn+1Adding the node D into a storage system, wherein n is the number of devices in the storage system when the node D is not added, and calculating the node D according to the weightn+1The number of virtual nodes to be allocated is
Combining existing physical nodes D in a storage system1,D2,D3…DnAnd weight w thereof1,w2,w3…wnCalculating the need to reassign N to each physical node1,N2,N3…NnA virtual node copy to Dn+1，
Let physical node D1Is named as P11,P12,P13…P1mIn turn will be
Total N1A virtual node copy is allocated to Dn+1To physical node D2,D3…DnTreating with the same method;
when physical nodes are deleted, the virtual node copy adjusting module comprises a unit weight access quantity sequencing submodule and a virtual node copy adjusting submodule, the total access times of the physical nodes are obtained by adding the access times of all the virtual nodes, then the total access times are divided by the weight of the physical nodes, the access quantity of the unit weight can be obtained, the physical nodes are ranked according to the access times of the unit weight by using bubble sequencing algorithm bubbleSort () due to the limited number of the physical nodes, when access adjustment is carried out, the virtual node copy of the physical node with small access quantity is adjusted to the physical node with large access quantity of the unit weight, the virtual node copy of the physical node with large access quantity is adjusted to the physical node with small access quantity of the unit weight, and when the physical node is deleted in the storage system, the virtual node copy of the physical node with large access quantityHas a weight wiPhysical node D ofiSetting the physical node D when exiting the storage systemiThe number of owned virtual nodes is NiCalculating the number of virtual nodes distributed by each storage node according to the weight of each remaining physical node to be N1,N2,N3…Nn，n≠i,
According to N1,N2,N3…NnCalculating the access times of the virtual nodes of the unit weight of the rest nodes in the storage system, ranking from large to small according to the access times of the virtual nodes of the unit weight, and setting the sequence of the physical nodes as V1,V2,Vi…Vn，n ≠ i, i ∈ [1, n), sequentially from the virtual node list of deleted physical nodes according to the unit weight of the remaining physical nodes
And obtaining the corresponding number of virtual node copies, thereby completing the adjustment.
2. A method for adjusting virtual node copy ranking based on access times by using the system of claim 1, comprising:
determining the access times of the virtual nodes according to the access records of the users to the virtual nodes by using the virtual node access time sequencing module, and ranking the virtual nodes to which each physical node in the storage system belongs according to the access times; each physical node corresponds to one device in the storage system, and each virtual node to which the physical node belongs is a copy called a virtual node copy;
acquiring the weight of each physical node by using the virtual node copy migration list calculation module when a physical node is added into the storage system, calculating the number of virtual nodes of unit weight of the physical node, and adding the virtual node copy to be adjusted into a migration list according to the unit weight of each physical node; when a physical node needs to be deleted or quit from the storage system, removing the corresponding equipment from the equipment list, and calculating the number of copies of the virtual nodes which can be received by each remaining physical node according to the number of the virtual nodes of the unit weight; the unit weight is equal to the device weight divided by the device capacity;
utilizing the virtual node copy adjusting module to adjust the mapping relation between the virtual node and the physical node; when a physical node joins the storage system, directly modifying a mapping table from the virtual node to the physical node according to the migration list; when physical nodes need to be deleted or quit from the storage system, calculating the number of virtual node copies which can be received by each physical node, calculating the access amount of unit weight according to the total access amount of the physical nodes, sequencing according to the access amount of the unit weight, and sequentially acquiring the corresponding number of virtual node copies from the virtual node list of the deleted physical nodes to be distributed to the corresponding physical nodes;
when a physical node is added, the virtual node copy adjusting module comprises a virtual node copy adjusting submodule, a virtual node copy to be adjusted is added into a virtual node migration list, a map function is utilized to traverse the migration list, the adjustable virtual node copy is determined, the virtual node copy to be adjusted is sequentially searched according to a front-to-back mode, and the mapping relation is modified according to the weight of the added physical node; is provided with a weight wn+1Node D ofn+1Adding the node D into a storage system, wherein n is the number of devices in the storage system when the node D is not added, and calculating the node D according to the weightn+1The number of virtual nodes to be allocated is
Combining existing physical nodes D in a storage system1,D2,D3…DnAnd weight w thereof1,w2,w3…wnCalculating the need to reassign N to each physical node1,N2,N3…NnA virtual node copy to Dn+1，
Let physical node D1Is named as P11,P12,P13…P1mIn turn will be
Total N1A virtual node copy is allocated to Dn+1To physical node D2,D3…DnTreating with the same method;
when physical nodes are deleted, the virtual node copy adjusting module comprises a unit weight access quantity sequencing submodule and a virtual node copy adjusting submodule, the total access times of the physical nodes are obtained by adding the access times of all the virtual nodes, then the total access times are divided by the weight of the physical nodes, the access quantity of the unit weight can be obtained, the physical nodes are ranked according to the access times of the unit weight by using bubble sequencing algorithm bubbleSort () due to the limited number of the physical nodes, when access adjustment is carried out, the virtual node copy of the physical node with small access quantity is adjusted to the physical node with large access quantity of the unit weight, the virtual node copy of the physical node with large access quantity is adjusted to the physical node with small access quantity of the unit weight, and when the weight w exists in the storage system, the virtual node copy of the physical node with large access quantity is adjusted to the physicaliPhysical node D ofiSetting the physical node D when exiting the storage systemiThe number of owned virtual nodes is NiCalculating the number of virtual nodes distributed by each storage node according to the weight of each remaining physical node to be N1,N2,N3…Nn，n≠i,
According to N1,N2,N3…NnCalculating the access times of the virtual nodes with unit weight of the rest nodes in the storage system, ranking from large to small according to the access times of the virtual nodes with unit weight,the physical node sequence is V1,V2,Vi…Vn，n ≠ i, i ∈ [1, n), sequentially from the virtual node list of deleted physical nodes according to the unit weight of the remaining physical nodes
And obtaining the corresponding number of virtual node copies, thereby completing the adjustment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710609966.9A CN107465729B (en) | 2017-07-25 | 2017-07-25 | Virtual node copy adjusting system and method based on access time ranking |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710609966.9A CN107465729B (en) | 2017-07-25 | 2017-07-25 | Virtual node copy adjusting system and method based on access time ranking |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107465729A CN107465729A (en) | 2017-12-12 |
| CN107465729B true CN107465729B (en) | 2020-09-22 |
Family
ID=60546940
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710609966.9A Active CN107465729B (en) | 2017-07-25 | 2017-07-25 | Virtual node copy adjusting system and method based on access time ranking |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107465729B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108228106A (en) * | 2017-12-30 | 2018-06-29 | 广东技术师范学院 | A kind of self-adaptation control method of cost driving copy |
| CN110086840B (en) * | 2018-01-26 | 2022-03-11 | 浙江宇视科技有限公司 | Image data storage method, device and computer readable storage medium |
| CN110244901B (en) * | 2018-03-07 | 2021-03-26 | 杭州海康威视系统技术有限公司 | Task allocation method and device and distributed storage system |
| CN109586951B (en) * | 2018-11-07 | 2020-04-17 | 北京达佳互联信息技术有限公司 | Method, device, electronic equipment and readable storage medium for implementing fault transfer |
| CN111338843B (en) * | 2018-12-19 | 2023-08-15 | 中国移动通信集团云南有限公司 | Data backup method and device for production system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100476742C (en) * | 2007-02-09 | 2009-04-08 | 华中科技大学 | Load balancing method based on object storage device |
| EP2532137B1 (en) * | 2010-02-05 | 2015-08-12 | Telefonaktiebolaget L M Ericsson (PUBL) | Method and node entity for enhancing content delivery network |
| CN102457571B (en) * | 2011-09-15 | 2014-11-05 | 中标软件有限公司 | Method for uniformly distributing data in cloud storage |
| CN103139302B (en) * | 2013-02-07 | 2015-07-08 | 浙江大学 | Real-time copy scheduling method considering load balancing |
| CN105657064B (en) * | 2016-03-24 | 2019-03-12 | 东南大学 | Swift load-balancing method based on dummy node storage optimization |
-
2017
- 2017-07-25 CN CN201710609966.9A patent/CN107465729B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN107465729A (en) | 2017-12-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107465729B (en) | Virtual node copy adjusting system and method based on access time ranking | |
| EP3314477B1 (en) | Systems and methods for parallelizing hash-based operators in smp databases | |
| EP2480974B1 (en) | Distributed content storage and retrieval | |
| CN106294352B (en) | A kind of document handling method, device and file system | |
| US20160350302A1 (en) | Dynamically splitting a range of a node in a distributed hash table | |
| US9800575B1 (en) | Assigning storage responsibility in a distributed data storage system with replication | |
| Xu et al. | Drop: Facilitating distributed metadata management in eb-scale storage systems | |
| US11126641B2 (en) | Optimized data distribution system | |
| EP4075295A1 (en) | System and method for structuring and accessing tenant data in a hierarchical | |
| US20220237506A1 (en) | Using container and model information to select containers for executing models | |
| CN107203623B (en) | Load balancing and adjusting method of web crawler system | |
| CN116150160B (en) | Adjustment method and device for database cluster processing nodes and storage medium | |
| CN107818113A (en) | File access method for determining position and device | |
| JP2015513333A (en) | Database workload balancing through migration | |
| Irandoost et al. | Learning automata-based algorithms for MapReduce data skewness handling | |
| KR102054068B1 (en) | Partitioning method and partitioning device for real-time distributed storage of graph stream | |
| ELomari et al. | New data placement strategy in the HADOOP framework | |
| Shabeera et al. | A novel approach for improving data locality of mapreduce applications in cloud environment through intelligent data placement | |
| Singh | An intelligent reallocation of load for cluster cloud environment | |
| Al Nuaimi et al. | ssCloud: A Smart Storage for Distributed DaaS on the Cloud | |
| US8015207B2 (en) | Method and apparatus for unstructured data mining and distributed processing | |
| CN104537081A (en) | File management system and method | |
| Noraziah et al. | Managing MyGRANTS Fragmented Database Using Binary Vote Assignment Grid Quorum with Association Rule (BVAGQ-AR) Replication Model | |
| Ghandour et al. | User-based Load Balancer in HBase. | |
| Xu et al. | C 2: adaptive load balancing for metadata server cluster in cloud-scale storage systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |