CN105260376B - Method, apparatus and system for clustered node reducing and expansion - Google Patents
Method, apparatus and system for clustered node reducing and expansion Download PDFInfo
- Publication number
- CN105260376B CN105260376B CN201510505399.3A CN201510505399A CN105260376B CN 105260376 B CN105260376 B CN 105260376B CN 201510505399 A CN201510505399 A CN 201510505399A CN 105260376 B CN105260376 B CN 105260376B
- Authority
- CN
- China
- Prior art keywords
- node
- cluster
- thread
- client
- node cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of method, apparatus and system for clustered node reducing and expansion.This method includes from the redis database snapshots for receiving multiple first node parallel generations；Receive the write operation that client carries out after the transmission of redis database snapshots；Appended document corresponding with multiple second nodes is generated, and write operation is separately recorded in appended document；Redis database snapshots are simultaneously recombined into database file corresponding with multiple second nodes by analysis redis database snapshots；Send corresponding database file and appended document respectively to multiple second nodes；And the clustered node topology of second node cluster is sent to client by the configuration management system of instruction configuration client, so that client can be written and read operation according to the clustered node topology of second node cluster.This method, equipment and system solve the problems, such as that Data Migration existing in the prior art takes and is difficult to carry out online.
Description
Technical field
The present invention relates to node cluster fields, and in particular to is used for the method, apparatus and system of clustered node reducing and expansion.
Background technology
Redis (long-range dictionary services, Remote Dictionary Service) is a kind of Key/Value of full memory
(key-value pair) buffer/store system.Due to by server memory capacity limit where individual node, cache in large-scale data/
It stores under scene, generally forms cluster by multiple nodes, each node is only responsible for storage section data, referred to as sharding.Visitor
Family end will be sent to specific node according to the mapping relations of Key (key) and node to the request of specific Key.
The mapping relations of Key and node, currently used mode are realized by client routing algorithm.By node by suitable
Sequence is numbered, using Key as parameter, by Hash function one numerical value of calculating, then according to the quantity of node come modulus (or
Person's consistency hash algorithm) to navigate to a node.It is route by then passing through algorithm, algorithm itself also can guarantee road
By time complexity be (1) O, and do not need to anywhere to store Key to node mapping relations, therefore it is main
Using in the storage scene for having a large amount of, ever-increasing Key/Value.
But there is also drawbacks for client routing algorithm.Destination node quantity and number once change, key and section
The mapping relations of point will change, and be written to the key/value numbers of node A before client by old mapping relations
According to due to being mapped to node B by new mapping relations, may going on node B to read and can not read.Therefore, by
When data volume increases and needs to increase node, it is necessary to traverse all key and (command interface of redis typically directly be called to hold
Row traversal), the migrating data between node, to reflect newest mapping relations.It is this to increase whole collection by increasing node
The mode of group's capacity is referred to as horizontal extension.Data Migration caused by horizontal extension would generally take very much and be difficult online into
Row.
Data Migration caused by changing in order to avoid node.Pre-sharding schemes can be taken, that is, pre-estimate number
According to amount and its growth rate, suitable enough number of nodes is set.Moreover, in order to avoid occupying excessive physics clothes
Business device, usual initial stage can dispose multiple nodes on same server, with the growth of data volume, by moving to newly node
On increased physical server so that each node possesses more memory sources, so that the memory size of whole cluster
Expand.Node is migrated to other servers, the leader follower replication realization that redis itself is provided is taken to can be realized.Due to being more
Change the mapping relations of node and physical machine, number of nodes and number do not change, need not migrating data among the nodes.
The also referred to as vertical extension of this not changed mode of capacity, number of nodes by increasing single node.
Vertical extension needs advance layout data capacity and growth.The suitable enough nodes of assessment are not easy to.Section
Setting up for point quantity needs to consider memory consumption and data growth, as data more than expected increase, even if an object
One node of server disposition is managed, can not continue to extend due to being limited by physical server memory, unavoidably need level
It extends to solve the capacity expansion of whole cluster.
And Data Migration caused by the horizontal extension spoken of above needs to be traversed for all key, according to new mapping relations,
By on Data Migration to new node, this whole process would generally take very much.And it also is difficult to carry out online.
There is also offline horizontal extension schemes, i.e.,：First stop the read-write of all clients before extension, migration is completed
Afterwards, then client is allowed to be written and read according to new mapping relations.This offline horizontal extension avoids migrator and visitor
Family end program be carried out at the same time read-write and caused by data collision and inconsistency, but need longer to forbid access time.So
And business demand does not usually allow to forbid reading and writing for a long time, it is therefore desirable to a side that online horizontal extension can be carried out
Case.
Invention content
To solve the above problems, an embodiment of the present invention provides a kind of method, apparatus and system of clustered node reducing and expansion.
According to an aspect of the invention, there is provided a kind of method for clustered node reducing and expansion, including：From first node
Multiple first nodes of cluster receive the redis database snapshots of the multiple first node parallel generation；From the multiple
One node receives the write operation that client carries out after the transmission of the redis database snapshots；It generates and second node collection
The corresponding appended document of multiple second nodes of group, and according to key to the mapping relations of the second node cluster, by institute
Write operation is stated to be separately recorded in the appended document；According to the key to the mapping relations of the second node cluster, analysis
The redis database snapshots and by the redis database snapshots be recombined into described in the second node cluster
In the corresponding database file of multiple second nodes；Send corresponding database respectively to the multiple second node
File and appended document；And instruction configures the configuration management system of the client by the cluster section of the second node cluster
Point topology be sent to the client so that the client can according to the clustered node topology of the second node cluster come
It is written and read operation.
According to the second aspect of the invention, a kind of equipment for clustered node reducing and expansion is provided, including：Receiving unit,
Redis databases for receiving the multiple first node parallel generation from multiple first nodes of first node cluster are fast
According to, and from the multiple first node receive client carried out after the transmission of the redis database snapshots write behaviour
Make；Appended document generation unit, for generating appended document corresponding with multiple second nodes of second node cluster, and
According to key to the mapping relations of the second node cluster, the write operation that the receiving unit receives is recorded respectively
In the appended document；Recombination unit, for according to the key to the mapping relations of the second node cluster, analysis
The redis database snapshots are simultaneously recombined into and institute by redis database snapshots that the receiving unit receives
In the corresponding database file of the multiple second node for stating second node cluster；Transmission unit is used for described more
A second node sends corresponding database file and appended document respectively, and is used to indicate the configuration client
The clustered node topology of the second node cluster is sent to the client by configuration management system, so that the client
It can be written and read operation according to the clustered node topology of the second node cluster.
According to the third aspect of the invention we, a kind of system for clustered node reducing and expansion is provided, including：Client；Packet
Include the first node cluster of multiple first nodes；Second node cluster including multiple second nodes；It is above-mentioned to be used for clustered node
The equipment of reducing and expansion；And the configuration management system for configuring the client, the configuration management system are saved described second
The clustered node topology of point cluster is sent to the client, so that the client can be according to the second node cluster
Clustered node topology operates to be written and read.
By using above-mentioned method, apparatus and system, it is time-consuming and difficult to solve Data Migration existing in the prior art
The problem of to carry out online.
Description of the drawings
It is by the detailed description carried out below in conjunction with the accompanying drawings to invention, the features described above and advantage that make the present invention is brighter
It is aobvious, wherein：
Fig. 1 is the schematic diagram for showing clustered node reducing and expansion system according to an embodiment of the invention；
Fig. 2 is the flow chart for showing clustered node reducing and expansion method according to an embodiment of the invention；And
Fig. 3 is the schematic block diagram for showing clustered node reducing and expansion equipment according to an embodiment of the invention.
Specific implementation mode
In the following, the preferred embodiment of the present invention is described in detail with reference to the accompanying drawings.In the accompanying drawings, although being shown in different attached drawings
In, but identical reference numeral is for indicating identical or similar component.For clarity and conciseness, including herein known
The detailed description of function and structure will be omitted, to avoid keeping subject of the present invention unclear.
In common technical solution, the bulk migration dependence of redis node datas is redis leader follower replication functions.
From node initializing when, request host node by the data snapshot at that moment, be packaged into rdb (binary format of redis db)
It is sent to from node, self EMS memory is initialized from node load rdb.Host node has recorded from after that snapshot time simultaneously
All write operations, and occur to arrive from node successively by write operation time sequencing.From snapshot data of the node in load on the basis of, according to
The secondary all write operations for executing host node and subsequently sending, to reach the data final consistency of master and slave node.
And the scheme that the embodiment of the present invention is provided provides one and is known as redis bridges under the support of redis leader follower replications
Middleware, by original n nodes cluster A, pass through replicate and reproduction process in press the new mapping relations of key and node, will
Cluster B of the Data Migration to m node.In transition process, cluster A continues the read-write requests of service client, has waited until B clusters
Through complete copy data and the follow-up increment write operation for being happened at cluster A is pulled up to, the duplication delay of B clusters is non-
Normal hour just allows B cluster services in the read-write requests of client.Specifically, in some embodiments, allowed according to business demand
Time window, the new number of nodes of B clusters and number (clustered node topology) are sent to client by configuring system, together
When forbid client to write.In the time window, client still can carry out a read request according to cluster A node topologies.Work as A
When reaching fully synchronized with B company-datas, configuration notifications client is written and read according to B clustered node topologys asks
It asks.So far, Data Migration and number of nodes change are completed.Old cluster A node resources, that is, releasable.
Redis bridges in the present invention are the middlewares for realizing redis replication protocols.After redis bridge process initiations, as
All nodes on cluster A are connected to from the role of node, request replicates.Meanwhile it also allows B clusters as the role of host node
All nodes connect and respond the duplicate requests of the latter.Due to the leader follower replication of redis be with node (process) be unit, because
It can be by realizing the slave section of above-mentioned multinode to be connected as the replication protocol of granularity realization redis in this embodiment of the present invention
Point and host node.
The system schematic of the design principle for aforementioned present invention is briefly described below.Fig. 1 is shown according to the present invention
The schematic diagram of the clustered node reducing and expansion system of embodiment.
As shown in Figure 1, the system includes application client (abbreviation client), the configuration pipe for configuring client
Reason system including two nodes cluster A (being referred to herein as first node cluster), the packet that will be expanded to from cluster A
The cluster B (being referred to herein as second node cluster) and redis bridges for including three nodes (are referred to herein as used for
The equipment of clustered node reducing and expansion).
It should be noted that although it includes two nodes in cluster A to be shown in Fig. 1, include three nodes, sheet in cluster B
In the case that inventive embodiments can also be applied to there are other various several destination nodes.For example, in Fig. 1, cluster A interior joints
Number be less than cluster B interior joints number, to carry out cluster dilatation.However in some embodiments, the number of cluster A interior joints
Mesh can be more than the number of cluster B interior joints, to realize cluster capacity reducing.Therefore, although being with cluster dilatation in the following description
The case where example is described, and the method for the embodiment of the present invention can also be used for cluster capacity reducing on an equal basis.
Below according to System describe shown in FIG. 1 clustered node reducing and expansion method according to the ... of the embodiment of the present invention.Fig. 2 shows
The flow chart of clustered node reducing and expansion method according to the ... of the embodiment of the present invention.
It, can be multiple from the reception of multiple first nodes of first node cluster by redis bridges in the step 210 of Fig. 2
Redis databases (rdb) snapshot of first node parallel generation.
Before this step, the node topology of cluster A and B cluster can be configured in redis bridges.
In addition, the node that redis bridges can be also directed in cluster A and cluster B establishes thread and connection respectively.
Specifically, 3 nodes of B clusters initiate synchronous (sync) life to be connected to redis bridges from role in master-slave relationship
It enables.Redis bridges are 3 connections c1, c2, c3 correspondence establishment 3 threads b1, b2, b3 of 3 nodes, and create 3 synchronous teams
Arrange (SynchronousQueue) q1, q2, q3.Wherein, b1, b2, b3 consume q1, q2, q3 respectively.When there is no data,
3 connections c1, c2, c3 enter blocked state.
In addition, redis bridges can also create 1 thread t1, it is responsible for carrying out rdb conversions.T1 may include a count synchronization device
(CountdownLatch)L1.The initial value of count synchronization device L1 is identical as the number of cluster A interior joints.Shown in Fig. 1
In example initial count can value be 2.T1 is blocked in count synchronization device L1 when running, and is continued to execute when L1 becomes 0.
2 nodes that redis bridges can be also directed in cluster A establish 2 threads a1, a2 respectively.2 threads include to meter
The reference of number synchronizer L1.Redis bridges establish the connection to 2 nodes of cluster A respectively, initiate sync orders；
2 nodal parallels of cluster A generate rdb snapshots and issue redis bridges at this time.
In the step 220 of Fig. 2, redis bridges from first node receive client redis database snapshots transmission it
The write operation carried out afterwards.
Specifically, thread a1 (or a2) receives rdb, and is stored in local file system.For thread a1 and a2 (that is,
For first node A1 and A2) filename be respectively a1.rdb, a2.rdb.According to rdb agreements, ending is FF+8 bytes
Verification and.After rdb is received, count synchronization device L1 is successively decreased 1 by a1 (or a2).
Next, in the step 230 of Fig. 2, it is right respectively with multiple second nodes of second node cluster that redis bridges are generated
The appended document answered, and write operation is separately recorded in appended document to the mapping relations of second node cluster according to key.
In some specific examples of the present invention, this can be completed by thread a1 and a2.After receiving rdb, thread
A1, a2 continue to the write operation from cluster A, and according to the mapping relations of newly-established key to B clustered nodes, will write behaviour
Work is stored in corresponding b1.aof, b2.aof, b3.aof file in a manner of adding.
Then, in the step 240 of Fig. 2, redis analyzes redis according to key to the mapping relations of second node cluster
Redis database snapshots are simultaneously recombined into corresponding respectively with multiple second nodes in second node cluster by database snapshot
Database file in.
In some specific examples of the present invention, when L1 is reduced to 0, the rdb snapshots of all nodes in cluster A are indicated
(number is 2 in this example) is all received by redis bridges.Thread t1 is continued to run at this time, is read and is parsed two rdb texts
Part (for example, traversal key/value), according to the mapping relations of new key to B clustered nodes, 3 are written to by key/value
New rdb files b1.rdb, b2.rdb, b3.rdb.Time carried out with the command interface for directly invoking redis in the prior art
It goes through and compares, workable traversal is called without using tcp (transmission control protocol) protocol stacks herein, but direct logarithm
According to being operated, speed is faster, and more efficient.
After rdb snapshots reconfigure, in the step 250 of Fig. 2, redis bridges are sent respectively to second node and it
Corresponding database file and appended document.
For example, in some examples of the present invention, after thread t1 writes 3 rdb files, sent respectively toward q1, q2, q3
File completes signal, including corresponding rdb filenames.
After thread b1, b2, b3 receive the signal from q1, q2, q3, corresponding rdb files are write with sendfile mechanism
Enter each self-sustaining connection c1, c2, c3 with B clustered nodes.
Thread b1 reads b1.aof since file header, and all write operations are sent to the company with B clustered nodes according to this
C1 is met, B cluster corresponding nodes execute write operation accordingly.Until thread b1 reads the end-of-file of b1.aof, record current
offsetl。
Thread b2, b3 with described in aforesaid operations read corresponding b2.aof, b3.aof, send it to corresponding B collection
Group node, and record last offset2, offset3.
So far, in the step 260 of Fig. 2, by configuration management system by the topological (node of the clustered node of second node cluster
Number and numbering etc.) it is sent to client, so that client can be carried out according to the clustered node topology of second node cluster
Read-write operation.
Specifically, the clustered node (second node) in step 250 into second node cluster is sent corresponding respectively
Database file and appended document after, redis bridges can pass through the communication link (not shown in figure 1) with configuration management system
It notifies configuration management system, is sent with indicating that configuration management system carries out above-mentioned clustered node topology.
Clustered node reducing and expansion method according to the ... of the embodiment of the present invention can terminate at step 260.However in some cases,
During sending corresponding database file and appended document to cluster B, client may perform some write operations, and according to
The embodiment of the present invention is stored in after the labels of the offset in corresponding .aof files.Therefore, other realities according to the present invention
Example is applied, some improvement to such case are provided.
Specifically, the new number of nodes of B clusters and number (clustered node topology) are sent to visitor in configuration management system
When the end of family, client can be forbidden to write simultaneously.
Then thread b1 reads b1.aof since offsetl, and until end-of-file, all write operations are sent to according to this
C1 is connected, B cluster corresponding nodes execute according to this.Thread b2, b3 are similarly executed.
After the operation of thread b1, b2 and b3 all terminate, configuration management system can notify client according to B clustered nodes
Topology operates to be written and read, that is, restarts the write operation of client.
Since the write operation recorded later in offset labels in each .aof files will not be too many, above-mentioned additional behaviour
Work will not bring delay.This minimize the inconsistent time window of data, it ensure that data final consistency is write with business and stop
Influence balance.
According to the above method, one can be utilized under the premise of not changing redis storage organizations and increasing new function
Realize that the middleware of the replication protocol of redis reaches the increase of back end by the fractionation and recombination of data in reproduction process
(horizontal dilatation) or reduce (horizontal capacity reducing).
Although it should be noted that in order to narration convenience and be easy to reader understand in fig. 2 use different attached drawing marks
Remember to identify each step, but the size of reference numeral is not offered as executing the exact sequence of correlation step.For example, generating additional
The step 230 of file and record write operation can be in analysis redis database snapshots and the step for reconfiguring redis database snapshots
It executes after rapid 240, or simultaneously executes.Further for example, for each node in first node cluster and second node cluster
The sequence for establishing connection and thread can be by different from sequencing as described herein progress etc..The present invention is not by these sequences
It is limited.
In other embodiments of the present invention, clustered node reducing and expansion equipment, such as the redis in Fig. 1 are additionally provided
Bridge.Fig. 3 shows the brief configuration block diagram of the clustered node reducing and expansion equipment.
As shown in figure 3, the equipment includes：Receiving unit 310, for being received from multiple first nodes of first node cluster
The redis database snapshots of multiple first node parallel generation, and client is received in redis from multiple first node
The write operation carried out after the transmission of database snapshot；Appended document generation unit 320, for generating and second node cluster
The corresponding appended document of multiple second nodes, and according to key to the mapping relations of the second node cluster, by receiving unit
310 write operations received are separately recorded in appended document；Recombination unit 330, for according to key to second node
The mapping relations of cluster, the redis database snapshots that receive of analysis receiving unit 310 and by redis database snapshots again
It is combined in database file corresponding with multiple second node of second node cluster；And transmission unit 340, it uses
In sending corresponding database file and appended document respectively to multiple second node, and it is used to indicate configuration client
The clustered node topology of second node cluster is sent to client by the configuration management system at end, so that client being capable of basis
The clustered node topology of second node cluster operates to be written and read.
Equipment in Fig. 3 can also forbid unit 350 including write operation, in the configuration pipe configured to client
When the clustered node topology of second node cluster is sent to client by reason system, by configuration management system forbid client into
Row write operates.At this point, transmission unit 340 is also sent in appended document to multiple second nodes to multiple second node respectively
The write operation of the client recorded after corresponding database file and appended document is sent respectively.
Equipment in Fig. 3 can also include：Connection establishment unit 360, for multiple first segments with first node cluster
Point respectively establishes the first connection, and respectively establishes second with multiple second nodes of second node cluster and connect；And thread is built
Vertical unit 370 is established for respectively establishing first thread for multiple first nodes of first node cluster for rdb conversions
Thread, and respectively establish the second thread for multiple second nodes of second node cluster.
In some instances, as described above, the thread for rdb conversions may include count synchronization device.Count synchronization device
Initial value is equal to the number of first node cluster interior joint, and each first thread is received from the node in first node cluster
Redis database snapshots, count synchronization device successively decrease one.When count synchronization device is equal to zero, the thread for rdb conversions can
Redis database snapshots are simultaneously recombined into multiple second section with second node cluster by analysis redis database snapshots
In the corresponding database file of point.
The equipment of Fig. 3 can also establish unit 380 including queue, equal number of same with multiple second nodes for establishing
Walk queue.In some instances, after the thread completion converted for rdb reconfigures, it is notified via isochronous queue
To the second thread.At this point, transmission unit 340 is also right respectively with it to the transmission of the multiple second node using second thread
The database file answered.
In some instances, as described above, first thread, the second thread and for rdb conversion thread when unused
Into blocked state.
It should be noted that the clustered node reducing and expansion equipment described in figure 3 is merely to make those skilled in the art more
The figure that the present invention is expressly understood and makees, wherein some are omitted to understanding the unnecessary modules/components of the present invention, it is of the invention
Protection domain should not be limited by the detail of these attached drawings.For example, may include more module/groups in actual equipment
Part, such as display, Operation and Maintenance interface, input/output interface.For another example said units 320,330,350-380 etc. can
It is realized by one or more of actual computer processor/module.The module can be software, hardware, firmware or ability
Other modes known to field technique personnel.The present invention does not limit these.
Above description is only used for realizing embodiments of the present invention, and it should be appreciated by those skilled in the art do not taking off
Any modification or partial replacement from the scope of the present invention, the range that should belong to the claim of the present invention to limit, because
This, the scope of protection of the invention shall be subject to the scope of protection specified in the patent claim.
Claims (13)
1. a kind of method for clustered node reducing and expansion, including：
The redis databases that the multiple first node parallel generation is received from multiple first nodes of first node cluster are fast
According to；
The write operation that client carries out after the transmission of the redis database snapshots is received from the multiple first node；
Appended document corresponding with multiple second nodes of second node cluster is generated, and according to key to second section
The mapping relations of point cluster, the write operation is separately recorded in the appended document；
According to the key to the mapping relations of the second node cluster, the redis database snapshots are analyzed and will be described
Redis database snapshots are recombined into database corresponding with the multiple second node of second node cluster
In file；
Send corresponding database file and appended document respectively to the multiple second node；And
Indicate that the clustered node topology of the second node cluster is sent to institute by the configuration management system for configuring the client
Client is stated, so that the client can be written and read behaviour according to the clustered node topology of the second node cluster
Make.
2. according to the method described in claim 1, the clustered node topology of the second node cluster is sent to the visitor
When the end of family, further include：
The client is forbidden to carry out write operation by the configuration management system configured to the client；And
To the multiple second node send respectively in the appended document sent respectively to the multiple second node and its
The write operation of the client recorded after corresponding database file and appended document.
3. according to the method described in claim 1, further including：
Second is respectively established with the multiple second node of the second node cluster to connect；
The multiple second node for the second node cluster respectively establishes the second thread；
Establish the thread for rdb conversions；
The multiple first node for the first node cluster respectively establishes first thread；And
First is respectively established with the multiple first node of the first node cluster to connect.
4. according to the method described in claim 3, wherein, the thread for rdb conversions includes count synchronization device, the meter
The initial value of number synchronizer is equal to the number of the first node cluster interior joint, and the method further includes：
Each first thread receives the redis database snapshots from the node in the first node cluster, described
Count synchronization device successively decreases one；And
When the count synchronization device is equal to zero, redis database snapshots described in the Thread Analysis for rdb conversions simultaneously will
The redis database snapshots are recombined into number corresponding with the multiple second node of second node cluster
According in library file.
5. according to the method described in claim 4, further including：
It establishes and the equal number of isochronous queue of the multiple second node；
After being reconfigured described in being completed in the thread for rdb conversions, it is notified via the isochronous queue to institute
State the second thread；And
Second thread sends database file corresponding with its to the multiple second node.
6. method according to any one of claim 4 or 5, wherein the first thread, second thread and described
Thread for rdb conversions enters blocked state when unused.
7. a kind of equipment for clustered node reducing and expansion, including：
Receiving unit, for receiving the multiple first node parallel generation from multiple first nodes of first node cluster
Redis database snapshots, and from the multiple first node receive client the redis database snapshots transmission it
The write operation carried out afterwards；
Appended document generation unit, for generating appended document corresponding with multiple second nodes of second node cluster,
And according to key to the mapping relations of the second node cluster, the write operation that the receiving unit receives is remembered respectively
Record is in the appended document；
Recombination unit, for according to the key to the mapping relations of the second node cluster, analyzing the receiving unit and connecing
The redis database snapshots are simultaneously recombined into and the second node cluster by the redis database snapshots that receive
The corresponding database file of the multiple second node in；And
Transmission unit, for sending corresponding database file and appended document respectively to the multiple second node, with
And it is used to indicate and configures the configuration management system of the client and be sent to the clustered node topology of the second node cluster
The client, so that the client can be written and read behaviour according to the clustered node topology of the second node cluster
Make.
8. equipment according to claim 7, further includes：
Write operation forbids unit, in the configuration management system configured to the client by the second node cluster
Clustered node topology when being sent to the client, forbid the client to be grasped into row write by the configuration management system
Make；
Wherein, the transmission unit is also sent in the appended document to the multiple second node to the multiple respectively
Two nodes send the write operation of the client recorded after corresponding database file and appended document respectively.
9. equipment according to claim 7, further includes：
Connection establishment unit is connect for respectively establishing first with the multiple first node of the first node cluster, and
Second is respectively established with the multiple second node of the second node cluster to connect；And
Thread establishes unit, for respectively establishing first thread for the multiple first node of the first node cluster, builds
The thread converted for rdb is found, and the second thread is respectively established for the multiple second node of the second node cluster.
10. equipment according to claim 9, wherein the thread for rdb conversions includes count synchronization device, described
The initial value of count synchronization device is equal to the number of the first node cluster interior joint, and each first thread is from described first
Node in node cluster receives the redis database snapshots, and the count synchronization device successively decreases one, when the count synchronization
When device is equal to zero, redis database snapshots described in the Thread Analysis for rdb conversions are simultaneously fast by the redis databases
According to being recombined into database file corresponding with the multiple second node of second node cluster.
11. equipment according to claim 10, further includes：
Unit is established in queue, is used for foundation and the equal number of isochronous queue of the multiple second node,
Wherein, after being reconfigured described in being completed in the thread for rdb conversions, it is notified via the isochronous queue
To second thread；And the transmission unit is also sent and its point using second thread to the multiple second node
Not corresponding database file.
12. the equipment according to any one of claim 9 to 11, wherein the first thread, second thread and institute
It states the thread converted for rdb and enters blocked state when unused.
13. a kind of system for clustered node reducing and expansion, including：
Client；
First node cluster including multiple first nodes；
Second node cluster including multiple second nodes；
Equipment according to any one of claim 7 to 11；And
Configuration management system for configuring the client, the configuration management system is by the cluster of the second node cluster
Node topology is sent to the client, so that the client can be opened up according to the clustered node of the second node cluster
It flutters to be written and read operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510505399.3A CN105260376B (en) | 2015-08-17 | 2015-08-17 | Method, apparatus and system for clustered node reducing and expansion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510505399.3A CN105260376B (en) | 2015-08-17 | 2015-08-17 | Method, apparatus and system for clustered node reducing and expansion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105260376A CN105260376A (en) | 2016-01-20 |
| CN105260376B true CN105260376B (en) | 2018-08-14 |
Family
ID=55100069
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510505399.3A Active CN105260376B (en) | 2015-08-17 | 2015-08-17 | Method, apparatus and system for clustered node reducing and expansion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105260376B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105808643A (en) * | 2016-02-25 | 2016-07-27 | 浪潮通信信息系统有限公司 | Method for refreshing Redis memory database |
| CN107919977B (en) * | 2016-10-11 | 2021-09-03 | 阿里巴巴集团控股有限公司 | Online capacity expansion and online capacity reduction method and device based on Paxos protocol |
| CN108073414B (en) * | 2016-11-15 | 2020-07-28 | 北京亿阳信通科技有限公司 | Implementation method for merging multithreading concurrent requests and submitting and distributing results in batches based on Jedis |
| CN106843755B (en) * | 2017-01-04 | 2019-10-11 | 北京百度网讯科技有限公司 | Data balancing method and device for server cluster |
| CN107330035B (en) * | 2017-06-26 | 2020-08-07 | 宁波图灵奇点智能科技有限公司 | Operation log synchronization method in database, mobile terminal and computer readable storage medium |
| CN107463612B (en) * | 2017-06-29 | 2019-10-01 | 北京北信源软件股份有限公司 | High concurrent REDIS database operation method and system |
| CN109407964A (en) * | 2017-08-18 | 2019-03-01 | 阿里巴巴集团控股有限公司 | A kind of moving method of data, device and equipment |
| CN107741963A (en) * | 2017-09-30 | 2018-02-27 | 北京奇虎科技有限公司 | Across company-data merging method and device, electronic equipment, computer-readable storage medium |
| CN109818767B (en) * | 2017-11-21 | 2021-08-03 | 网宿科技股份有限公司 | Method and device for adjusting Redis cluster capacity and storage medium |
| CN107943589A (en) * | 2017-11-29 | 2018-04-20 | 苏宁云商集团股份有限公司 | The management method and device of a kind of data buffer storage |
| CN110333944B (en) * | 2019-04-19 | 2021-07-27 | 中国联合网络通信集团有限公司 | Ticket data service processing method and equipment |
| CN111309805B (en) * | 2019-12-13 | 2023-10-20 | 华为技术有限公司 | Data reading and writing method and device for database |
| CN114945910A (en) * | 2020-02-24 | 2022-08-26 | 深圳市欢太科技有限公司 | Redis cluster capacity expansion method and device, electronic equipment and storage medium |
| CN112632189A (en) * | 2020-12-26 | 2021-04-09 | 中国农业银行股份有限公司 | Data synchronization system and method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102148850A (en) * | 2010-08-09 | 2011-08-10 | 华为软件技术有限公司 | Cluster system and service processing method thereof |
| CN102521297A (en) * | 2011-11-30 | 2012-06-27 | 北京人大金仓信息技术股份有限公司 | Method for achieving system dynamic expansion in shared-nothing database cluster |
| CN103034664A (en) * | 2011-10-10 | 2013-04-10 | 上海盛霄云计算技术有限公司 | Method, system and device for controlling data migration of database |
| CN103078927A (en) * | 2012-12-28 | 2013-05-01 | 合一网络技术(北京)有限公司 | Key-value data distributed caching system and method thereof |
| CN103634375A (en) * | 2013-11-07 | 2014-03-12 | 华为技术有限公司 | Method, device and equipment for cluster node expansion |
| CN104407807A (en) * | 2014-10-24 | 2015-03-11 | 华中科技大学 | Storage and expansion method aiming at RS coding storage cluster |
| CN104484469A (en) * | 2014-12-31 | 2015-04-01 | 天津南大通用数据技术股份有限公司 | Expansion method for supporting non-halt of multi-hash map database cluster system |
-
2015
- 2015-08-17 CN CN201510505399.3A patent/CN105260376B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102148850A (en) * | 2010-08-09 | 2011-08-10 | 华为软件技术有限公司 | Cluster system and service processing method thereof |
| CN103034664A (en) * | 2011-10-10 | 2013-04-10 | 上海盛霄云计算技术有限公司 | Method, system and device for controlling data migration of database |
| CN102521297A (en) * | 2011-11-30 | 2012-06-27 | 北京人大金仓信息技术股份有限公司 | Method for achieving system dynamic expansion in shared-nothing database cluster |
| CN103078927A (en) * | 2012-12-28 | 2013-05-01 | 合一网络技术(北京)有限公司 | Key-value data distributed caching system and method thereof |
| CN103634375A (en) * | 2013-11-07 | 2014-03-12 | 华为技术有限公司 | Method, device and equipment for cluster node expansion |
| CN104407807A (en) * | 2014-10-24 | 2015-03-11 | 华中科技大学 | Storage and expansion method aiming at RS coding storage cluster |
| CN104484469A (en) * | 2014-12-31 | 2015-04-01 | 天津南大通用数据技术股份有限公司 | Expansion method for supporting non-halt of multi-hash map database cluster system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105260376A (en) | 2016-01-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105260376B (en) | Method, apparatus and system for clustered node reducing and expansion | |
| Ferguson et al. | Orion: Google's {Software-Defined} Networking Control Plane | |
| CN110209726B (en) | Distributed database cluster system, data synchronization method and storage medium | |
| US20190354628A1 (en) | Asynchronous replication of synchronously replicated data | |
| US11789638B2 (en) | Continuing replication during storage system transportation | |
| CN102265277B (en) | Operation method and device for data memory system | |
| US11042503B1 (en) | Continuous data protection and restoration | |
| DE102021113808A1 (en) | Handling replications between different networks | |
| CN115668172A (en) | Managing host mapping of replication endpoints | |
| EP1712998B1 (en) | Remote copy system and remote copy method | |
| US20220188196A1 (en) | Continuous data protection | |
| US7143112B2 (en) | Method and apparatus for data integration | |
| US10353872B2 (en) | Method and apparatus for conversion of virtual machine formats utilizing deduplication metadata | |
| WO2018133578A1 (en) | Method and device for migrating virtual machine, computer storage medium, and electronic apparatus | |
| CN109840166A (en) | Across the cluster object storage async backup methods, devices and systems of one kind | |
| WO2022021856A1 (en) | Method and apparatus for online migration of multi-disk virtual machine into different storage pools | |
| US11860892B2 (en) | Offline index builds for database tables | |
| JP2007241486A (en) | Memory system | |
| US20180267735A1 (en) | Pre-forking replicas for efficient scaling of a distributed data storage system | |
| CN104468274A (en) | Cluster monitor and management method and system | |
| JP2004164611A (en) | Management of attribute data | |
| CN109639773A (en) | A kind of the distributed data cluster control system and its method of dynamic construction | |
| US10949102B2 (en) | Leveraging snapshot for time series pattern analysis prediction | |
| WO2016174739A1 (en) | Multicomputer system, management computer, and data linkage management method | |
| WO2016117007A1 (en) | Database system and database management method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |