CN103593347A - Load balancing method and distributed database system - Google Patents
Load balancing method and distributed database system Download PDFInfo
- Publication number
- CN103593347A CN103593347A CN201210288434.7A CN201210288434A CN103593347A CN 103593347 A CN103593347 A CN 103593347A CN 201210288434 A CN201210288434 A CN 201210288434A CN 103593347 A CN103593347 A CN 103593347A
- Authority
- CN
- China
- Prior art keywords
- migration
- node
- file
- migrated
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 230000005012 migration Effects 0.000 claims abstract description 521
- 238000013508 migration Methods 0.000 claims abstract description 521
- 239000000284 extract Substances 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 description 6
- 238000011084 recovery Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 5
- 230000006835 compression Effects 0.000 description 3
- 238000007906 compression Methods 0.000 description 3
- 230000006837 decompression Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a load balancing method. The load balancing method comprises the steps: server nodes add currently-stored data to migration unit files in real time and establish migration unit file statistical tables according to the migration unit files; a master control server node periodically obtains the migration unit file statistical tables of the server nodes; the master control server node selects emigration nodes, emigration files of the emigration nodes and migration nodes corresponding to the emigration files as migration decisions according to the migration unit file statistical tables and migration strategies; the master control server node adjusts loads of the emigration nodes and the migration nodes according to the migration decisions when confirming the migration execution. The invention further discloses a distributed database system. By adopting the load balancing method, the load of the distributed database system can be balanced according to the load conditions of the server nodes.
Description
Technical Field
The invention relates to a database management technology in the field of computers, in particular to a load balancing method and a distributed database system.
Background
The distributed database can linearly improve the overall performance and the storage capacity by dynamically increasing the computing nodes and the storage nodes under the condition of not interrupting the service, and the safety of data can be ensured by a multi-copy technology among the nodes of the system. The distributed database consists of a plurality of server nodes, and data are distributed on the plurality of server nodes; the Central Processing Unit (CPU) and the hard disk in each server node synchronously process data. The method for storing data in the distributed database comprises the following steps: and the client selects a server node through a consistency algorithm and then sends the data to the server node for storage.
However, the server node is selected by the client through a consistency algorithm, such as a Hash (Hash) algorithm, but the client cannot determine the situation of the stored data in each server when selecting the server using the consistency algorithm, which may cause a problem of unbalanced load of the distributed database system, and further affect the storage service of the system. For example, the data stored in the server node a is close to 100% of the disk capacity of the server, while the data stored in the server node B occupies less than 10% of the disk capacity, but the client selects the server node a to store the data according to the consistency algorithm, so that the server node a may not store the data.
Therefore, in the prior art, the load condition of each server node in the current system is not considered when data is stored, so that the problem of load imbalance of the distributed database occurs.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a method for balancing load and a distributed database system, which can balance the load of a distributed database according to the load condition of each server node.
In order to achieve the purpose, the technical scheme of the invention is realized as follows:
the invention provides a method for balancing load, which comprises the following steps:
the server node adds the currently stored data to the migration unit file in real time, and establishes a migration unit file statistical table according to the migration unit file;
the method comprises the steps that a master control server node periodically obtains a migration unit file statistical table of each server node;
the master control server node selects a migration node, a migration file of the migration node and a migration node corresponding to the migration file as migration decisions according to the migration unit file statistical table and the migration strategy;
and when the master control server node determines the migration, adjusting the loads of the migrated node and the migrated node according to the migration decision.
In the foregoing solution, the migration unit file statistics table includes: the identification of the server node, the total capacity of the disk and the used capacity of the disk, the file information of all migration units stored in the server node, the disk capacity occupied by each migration unit file and the access frequency of each migration unit file.
In the foregoing solution, the migration policy includes: one or more of the following policies:
selecting a migration unit file statistical table with the used capacity of the disk exceeding a threshold value, and taking a server node corresponding to the selected migration unit file statistical table as a migration node; selecting the migration unit file in the migration node as a migration file from the selected migration unit file statistical table; selecting migration nodes corresponding to all migration files from server nodes corresponding to the residual migration unit file statistical table;
calculating the difference between the server node with the maximum disk used capacity and the server node with the minimum disk used capacity, when the difference is larger than a preset value, taking the server node with the maximum disk used capacity as a migrated node, selecting a migrated unit file as a migrated file from a corresponding migrated unit file statistical table, and selecting a migrated node for each migrated file from the server nodes corresponding to the remaining migrated unit file statistical table;
and selecting a server node corresponding to the migration unit file statistical table with the used capacity of the disk lower than the threshold value as a migration node according to the migration unit file statistical table, selecting a migration node from the server nodes corresponding to the residual migration unit file statistical table, and selecting a migration unit file as a migration file.
In the foregoing solution, before the master server node periodically obtains the migration unit file statistics table of each server node, the method further includes: and the master control server node detects the states of all the server nodes in real time, if the states of the server nodes are ready to quit, the server nodes are taken as migration nodes, all migration unit files in the migration nodes are taken as migration files, and the migration nodes are selected for all the migration files from the server nodes corresponding to the residual migration unit file statistical table.
In the foregoing solution, the adjusting the loads of the migrated node and the migrated node according to the migration decision includes: the master control server respectively sends a notification of sending the migrated files to the migrated nodes to the corresponding migrated nodes according to the selected migrated nodes, the migrated files of the migrated nodes and the migrated nodes corresponding to the migrated files;
the emigration node extracts the emigration file and sends the emigration file to the emigration node according to the notice of sending the emigration file;
when the master control server node receives the notification that the migration node returns the success of migration, all migration nodes corresponding to the migration files of the migration node are notified to process the received migration files;
and when the master control server node receives the notification of the completion of the migration returned by the migration node, updating the self migration unit file statistical table.
The invention also provides a distributed database system, comprising: a master server node and a server node; wherein,
the server node is used for adding the currently stored data to the migration unit file in real time, establishing a migration unit file statistical table according to the migration unit file, providing the migration unit file statistical table for the main control server node, and receiving the load adjustment of the main control server node as an immigration node or an emigration node;
the main control server node is used for periodically acquiring a migration unit file statistical table of each server node; selecting a migration node, a migration file of the migration node and a migration node corresponding to the migration file as migration decisions according to the migration unit file statistical table and the migration strategy; and when the migration is determined, adjusting the loads of the migration node and the migration node according to the migration decision.
In the foregoing scheme, the server node is specifically configured to establish a migration unit file statistics table by using the identifier of the server node, the total capacity of the disk, the used capacity of the disk, information of all migration unit files stored in the server node, the disk capacity occupied by each migration unit file, and the access frequency of each migration unit file.
In the foregoing solution, the master server node is specifically configured to store a migration policy, where the migration policy includes one or more of the following policies:
selecting a migration unit file statistical table with the used capacity of the disk exceeding a threshold value, and taking a server node corresponding to the selected migration unit file statistical table as a migration node; selecting the migration unit file in the migration node as a migration file from the selected migration unit file statistical table; selecting migration nodes corresponding to all migration files from server nodes corresponding to the residual migration unit file statistical table;
calculating the difference between the server node with the maximum disk used capacity and the server node with the minimum disk used capacity, when the difference is larger than a preset value, taking the server node with the maximum disk used capacity as a migrated node, selecting a migrated unit file as a migrated file from a corresponding migrated unit file statistical table, and selecting a migrated node for each migrated file from the server nodes corresponding to the remaining migrated unit file statistical table;
according to the migration unit file statistical table, selecting a server node corresponding to the migration unit file statistical table with the used capacity of the disk lower than the threshold value as a migration node, selecting a migration node from server nodes corresponding to the residual migration unit file statistical table, and selecting a migration unit file as a migration file
In the above scheme, the master control server node is further configured to detect the states of all server nodes in real time, and if the state of a server node is ready to exit, the server node is used as an outbound node, all migration unit files in the outbound node are used as outbound files, and an inbound node is selected for each outbound file from the server nodes corresponding to the remaining migration unit file statistics table;
correspondingly, the server node is also used for providing the state of the server node for the main control server node to be ready for quitting or normal operation.
In the above scheme, the master server node is specifically configured to send, to the corresponding migration node, a notification to send the migrated file to the server node serving as the migration node, according to the selected migration node, the migrated file of the migration node, and the migration node corresponding to the migrated file; when receiving a notification that the migration is successful returned by the server node serving as the migration node, notifying all migration nodes corresponding to the migration file of the migration node to process the received migration file; updating a migration unit file statistical table of the server node after receiving a notification of completion of migration returned by the server node as a migration node;
correspondingly, the server node is specifically configured to, when serving as a migration node, extract the migration file and send the extracted migration file to the migration node according to the notification that the master server node sends the migration file, and return a notification of successful migration to the master server node; and when the node is used as the migration node, after the migration is completed, a notification of completing the migration is returned to the master control server node.
According to the load balancing method and the distributed database system, the master control server node can select the emigration node, the emigration file of the emigration node and the emigration node corresponding to the emigration file as migration decision periodically according to the migration unit file statistical table and the migration strategy established by all the server nodes; and adjusting the loads of the migrated node and the migrated node according to the migration decision. Therefore, the load of each server node can be adjusted according to the condition of the stored data of each server node in the distributed database system, and further the load balance of the distributed database is realized.
Drawings
FIG. 1 is a flow chart of a method for balancing loads according to the present invention;
fig. 2 is a schematic diagram of the composition structure of the distributed database system of the present invention.
Detailed Description
The basic idea of the invention is: each server node adds the currently stored data to a migration unit file in real time, and establishes a migration unit file statistical table according to the migration unit file; the method comprises the steps that a master control server node periodically obtains a migration unit file statistical table of each server node; the master control server node selects a migration node, a migration file of the migration node and a migration node corresponding to the migration file as migration decisions according to the migration unit file statistical table and the migration strategy; and when the master control server node determines the migration, adjusting the loads of the migrated node and the migrated node according to the migration decision.
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
The method for balancing load, as shown in fig. 1, includes the following steps:
step 101: each server node adds the currently stored data to the migration unit file in real time, and establishes a migration unit file statistical table according to the migration unit file.
Here, the adding of the currently stored data to the migration unit file is: the server node detects whether data sent by the client side is received or not in real time, if so, the server node detects whether the current migration unit file exceeds a maximum recording threshold value or not, if so, the server node stops updating the current migration unit file, newly establishes a migration unit file as the current migration unit file, and stores the data into the current migration unit file; and if not, directly storing the data into the current migration unit file by the server node. The maximum recording threshold is a numerical value preset according to an actual situation, and may be the maximum amount of data recorded in the migration unit file, or the maximum disk capacity occupied by the data stored in the migration unit file.
Step 102: the master control server node periodically acquires a migration unit file statistical table of each server node.
Here, the migration unit file statistic table includes: the identification of the server node, the total capacity of the disk and the used capacity of the disk, the file information of all migration units stored in the server node, the disk capacity occupied by each migration unit file and the access frequency of each migration unit file;
the identifier of the server node can be the address of the server node in the system;
the migration unit file information includes: the number of the migration unit files and the file names can be composed of the identification of the server node and/or the establishment sequence of the migration unit files; the periodicity is a time length set according to actual conditions, and for example, the periodicity can be set to be one day; the obtaining may report the migration unit file statistical table to the master control server periodically for each server node, or may extract the migration unit file statistical information from each server node managed by the master control server periodically for the master control server.
Step 103: and the master control server node selects the emigration node, the emigration file of the emigration node and the emigration node corresponding to the emigration file as migration decision according to all the migration unit file statistical tables and the migration strategies.
Here, the migration policy is one or more of the following policies preset in the master server according to actual conditions:
selecting a migration unit file statistical table with the used capacity of a disk exceeding a threshold value, and taking a server node corresponding to the selected migration unit file statistical table as a migration node; selecting the migration unit file in the migration node as a migration file from the selected migration unit file statistical table; selecting migration nodes corresponding to all migration files from server nodes corresponding to the residual migration unit file statistical table;
in the selected migration unit file statistical table, the migration unit file in the selected migration node as the migration file may set a selection rule according to an actual situation, for example, the selection rule may be set as: calculating a difference value between the used capacity of the disk and a threshold value, and taking the sum of the capacities of the disks occupied by the migration unit files and one or more migration unit files consistent with the difference value as migration files; the threshold value may be set according to actual conditions, for example, may be set to 90% of the total capacity of the disk;
the selecting of the migration node corresponding to each migration file from the server nodes corresponding to the remaining migration unit file statistics table may be: and calculating the average used disk capacity according to the used disk capacities of all the current server nodes, selecting a server node with the used disk capacity lower than the average used disk capacity from the server nodes corresponding to the file statistical table of the residual migration unit as a migration node, calculating the difference between the used disk capacity of the migration node and the average used disk capacity, and selecting one or more migration files with the sum of the occupied disk capacities being the same as the difference from all the migration files and corresponding to the migration node.
Calculating a difference value between the server node with the largest disk used capacity and the server node with the smallest disk used capacity, when the difference value is larger than a preset value, taking the server node with the largest disk used capacity as a migrated node, selecting one or more migrated unit files from a corresponding migrated unit file statistical table as migrated files, and selecting migrated nodes for each migrated file from the server nodes corresponding to the remaining migrated unit file statistical table;
the preset value may be a ratio of a proportional value to a total disk capacity corresponding to the server node with the largest disk capacity.
Selecting a server node corresponding to a migration unit file statistical table with the used capacity of the disk lower than a threshold value as a migration node according to the migration unit file statistical table, selecting a migration node from server nodes corresponding to the remaining migration unit file statistical tables, and selecting one or more migration unit files as migration files;
wherein, selecting a migration node from the server nodes corresponding to the remaining migration unit file statistical table, and selecting one or more migration unit files as migration files may be set according to an actual situation, for example, the following may be set: and calculating the average used disk capacity according to the used disk capacities of all the current server nodes, selecting the server nodes with the used disk capacities higher than the average used disk capacity from the rest server nodes as migration nodes, and selecting one or more migration files according to the difference between the used disk capacity of the migration nodes and the average used disk capacity.
Step 104: and when the master control server node determines the migration, adjusting the loads of the migrated node and the migrated node according to the migration decision.
Here, the migration may be determined manually by executing the migration command, or the master server node is configured to first detect whether a migration node and a migration node have been selected, if yes, determine the migration, and if not, end the processing flow;
the adjusting the load of the migrated node and the migrated node according to the migration decision comprises:
step a: the main control server locks all migration unit file statistical tables stored by the main control server, and notifies all selected migration nodes and migration nodes to lock the migration unit file statistical tables of the main control server; and the master control server respectively sends a notice of sending the migrated files to the migrated nodes to the corresponding migrated nodes according to the selected migrated nodes, the migrated files of the migrated nodes and the migrated nodes corresponding to the migrated files.
Wherein the locking is to stop any operation on the migration unit file statistical table.
Step b: and the emigration node extracts the emigration file and sends the emigration file to the emigration node according to the notification of sending the emigration file, and updates the migration unit file statistical table of the emigration node.
The sending mode may be transferred by using a File Transfer Protocol (FTP); the step of sending the extracted migrated file to the migration node may be to extract and compress the migrated file, and send the compressed migrated file to the migration node; the compression is prior art and is not described herein.
Further, after the extracted migrated file is sent to the migration node, the master control server node also monitors whether the compressed file fails or not and whether network transmission is abnormal or not in real time, if the compressed file fails or the network transmission is abnormal, whether recovery is possible is judged, if the recovery is possible, whether the current repeated execution frequency exceeds the maximum value is judged, if yes, the processing flow is ended, and if not, the extracted migrated file is sent to the migration node, and the repeated execution frequency is increased by one; if the recovery is impossible, the migration operation fails, a system abnormal log is recorded, and the processing flow is ended; and c, if the compressed file is successful or the network transmission is normal, executing the step c.
Step c: when the master control server node receives the notification that the migration of any migration node is successful, all migration nodes corresponding to the migration files of the migration node are notified, and the received migration files are processed.
Before the migrating node returns the notification of successful migration, the method further includes: after receiving the notification of the data migration start, the migration node notifies the identification number of the session instance executing the data migration to the master control server node; after receiving the message containing the session identification number, the master control server periodically sends a heartbeat message to the emigration node; and after receiving the heartbeat message, the emigration node detects whether the session with the appointed session identification number is running, if so, the emigration node continues responding the heartbeat message to the main control server node, and if the heartbeat is not detected after the preset time length is exceeded, the data migration operation is judged to be failed, and the main control node is informed of the emigration failure.
The session identification number may be a current migration operation sequence number; the sending and responding of the heartbeat message are both methods adopted when managing system devices in the prior art, and are not described herein.
The step of the master control server node receiving the notification that the migration is successful returned by any one of the migration nodes comprises the following steps: the master control server node detects whether a notification of successful migration returned by the migration node is received or not in real time, and when the notification of successful migration returned by the migration node is received, all migration nodes corresponding to the migration files of the migration node are notified to process the received migration files; and when receiving a notification of migration failure returned by the migration node, ending the migration processing corresponding to the migration node.
The processing of the received migrated file is as follows: and the migration node saves the migration file sent by the migration node, updates the self migration unit file statistical table and returns a notification of completing migration to the master control server node. Wherein the saving the migrated file may include: decompressing the received compressed migrated file and storing the decompressed migrated file; the decompression is a method corresponding to the compression method, and is a prior art, and is not described herein again.
Step d: and when the master control server node receives a notification that the migration is completed returned by any migration node, updating the self migration unit file statistical table.
And the updating is to change the record corresponding to the migrated file.
In addition, before the step 101 is executed, the method may further include: and the master control server node detects the states of all the server nodes in real time, if the state of any one server node is ready to quit, the server node is taken as an emigration node, all migration unit files in the emigration node are taken as emigration files, and emigration nodes are selected for all the migration files from the server nodes corresponding to the residual migration unit file statistical tables.
The selecting of the migration node for each migration file from the server nodes corresponding to the remaining migration unit file statistics table may be: and calculating the average used disk capacity according to the used disk capacities of all the current residual server nodes, selecting the server nodes with the used disk capacity lower than the average used disk capacity as migration nodes, and selecting the migration nodes corresponding to one or more migration files one by one according to the difference between the used disk capacity of the migration nodes and the average used disk capacity.
As shown in fig. 2, the present invention provides a distributed database system, comprising: a master server node 21 and a server node 22; wherein,
the server node 22 is used for adding currently stored data to the migration unit file in real time, establishing a migration unit file statistical table according to the migration unit file, providing the migration unit file statistical table for the main control server node 21, and receiving load adjustment of the main control server node 21 as an in-transit node or an out-transit node;
the main control server node 21 is configured to periodically obtain a migration unit file statistical table of each server node 22; selecting a migration node, a migration file of the migration node and a migration node corresponding to the migration file as migration decisions according to the migration unit file statistical table and the migration strategy; and when the migration is determined, adjusting the loads of the migration node and the migration node according to the migration decision.
The server node 22 is specifically configured to detect whether data sent by the client is received in real time, detect whether the current migration unit file exceeds a maximum recording threshold if the data is received, stop updating the current migration unit file if the data exceeds the maximum recording threshold, create a new migration unit file as the current migration unit file, and store the data in the current migration unit file; and if not, directly storing the data into the current migration unit file.
The server node 22 is specifically configured to store a migration unit file statistical table, and includes: the identification of the server node, the total capacity and the used capacity of the disk, the file information of all migration units stored in the server node, the disk capacity occupied by each migration unit file and the access frequency of each migration unit file.
The master server node 21 is specifically configured to store a migration policy; the migration policy is one or more of the following policies preset according to actual conditions:
selecting a migration unit file statistical table with the used capacity of a disk exceeding a threshold value, and taking a server node corresponding to the selected migration unit file statistical table as a migration node; selecting the migration unit file in the migration node as a migration file from the selected migration unit file statistical table; selecting migration nodes corresponding to all migration files from server nodes corresponding to the residual migration unit file statistical table;
in the selected migration unit file statistical table, the migration unit file in the selected migration node as the migration file may set a selection rule according to an actual situation, for example, the selection rule may be set as: calculating a difference value between the used capacity of the disk and a threshold value, and taking the sum of the capacities of the disks occupied by the migration unit files and one or more migration unit files consistent with the difference value as migration files; the threshold value may be set according to actual conditions, for example, may be set to 90% of the total capacity of the disk;
the selecting of the migration node corresponding to each migration file from the server nodes corresponding to the remaining migration unit file statistics table may be: and calculating the average used disk capacity according to the used disk capacities of all the current server nodes, selecting a server node with the used disk capacity lower than the average used disk capacity from the server nodes corresponding to the file statistical table of the residual migration unit as a migration node, calculating the difference between the used disk capacity of the migration node and the average used disk capacity, and selecting one or more migration files with the sum of the occupied disk capacities being the same as the difference from all the migration files and corresponding to the migration node.
Calculating a difference value between the server node with the largest disk used capacity and the server node with the smallest disk used capacity, when the difference value is larger than a preset value, taking the server node with the largest disk used capacity as a migrated node, selecting one or more migrated unit files from a corresponding migrated unit file statistical table as migrated files, and selecting migrated nodes for each migrated file from the server nodes corresponding to the remaining migrated unit file statistical table;
the preset value may be a ratio of a proportional value to a total disk capacity corresponding to the server node with the largest disk capacity.
Selecting a server node corresponding to a migration unit file statistical table with the used capacity of the disk lower than a threshold value as a migration node according to the migration unit file statistical table, selecting a migration node from server nodes corresponding to the remaining migration unit file statistical tables, and selecting one or more migration unit files as migration files;
wherein, selecting a migration node from the server nodes corresponding to the remaining migration unit file statistical table, and selecting one or more migration unit files as migration files may be set according to an actual situation, for example, the following may be set: and calculating the average used disk capacity according to the used disk capacities of all the current server nodes, selecting the server nodes with the used disk capacities higher than the average used disk capacity from the rest server nodes as migration nodes, and selecting one or more migration files according to the difference between the used disk capacity of the migration nodes and the average used disk capacity.
The main control server node 21 is specifically configured to execute a migration command manually to determine that migration is to be started, or to first detect whether a migration node and a migration node have been selected, determine migration if yes, and end the processing flow if no.
The master control server node 21 is specifically configured to lock all migration unit file statistics tables currently stored by itself, and notify all selected migration nodes and migration nodes to lock their own migration unit file statistics tables, and the master control server respectively sends a notification of sending an migration file to a migration node to the corresponding migration node according to the selected migration node, the migration file of the migration node, and the migration node corresponding to the migration file.
The server node 22 is specifically configured to, when serving as a migration node, extract the migration file and send the migration file to the migration node according to the notification of sending the migration file, and update the migration unit file statistical table of the server node.
The server node 22 is specifically configured to extract and compress the migrated file, and send the compressed migrated file to the migration node.
The main control server node 21 is specifically configured to monitor in real time whether the compressed file of the server node 22 serving as the migration node fails and whether network transmission is abnormal, and if the compressed file fails or the network transmission is abnormal, determine whether recovery is possible, if recovery is possible, determine whether the current repeated execution frequency exceeds a maximum value, if yes, end the processing flow, otherwise, extract the migration file and send the file to the migration node, and add one to the repeated execution frequency; if the recovery is impossible, the migration operation fails, a system abnormal log is recorded, and the processing flow is ended.
The server node 22 is further configured to, when serving as an emigration node, notify the main control module of an identification number of a session instance for executing data migration after receiving a notification of starting data migration, and detect whether a session with a session identification number corresponding to a heartbeat message is running after receiving the heartbeat message sent by the main control server node 21, if so, continue to respond to the main control server node 21 with a message, and if the heartbeat message is not detected after exceeding a preset time period, determine that a data migration operation is failed, and notify the main control server node 21 of an emigration failure; correspondingly, the main control server node 21 is further configured to periodically send a heartbeat message to the server node 22 serving as the migrating node after receiving the message that includes the session identification number and is sent by the server node 22.
The main control server node 21 is specifically configured to detect whether a notification that the migration node returns a successful migration is received or not in real time, notify all migration nodes corresponding to the migration file of the migration node to process the received migration file when the notification that the migration node returns a successful migration is received, and end the migration processing corresponding to the migration node when the notification that the migration node returns a failed migration is received.
The server node 22 is specifically configured to, when serving as an migration node, store a migration file sent by the migration node, update a migration unit file statistics table of itself, and return a notification of completing migration to the master server node 21.
The server node 22 is specifically configured to, when serving as an migration node, decompress and store the received compressed migration file; the decompression is a method corresponding to the compression method, and is a prior art, and is not described herein again.
The main control server node 21 is further configured to detect the state of the server node 22 in real time, and if the state of the server node 22 is ready to exit, use the server node 22 as an outbound node, use all migration unit files in the outbound node as outbound files, and select an inbound node for each outbound file from the server nodes 22 corresponding to the remaining migration unit file statistics table; correspondingly, the server node 22 is further configured to provide the master server node 21 with its own status as ready to exit or normal operation.
The main control server node 21 is specifically configured to calculate an average used disk capacity according to the used disk capacities of all the remaining server nodes 22 at present, select a server node 22 having a used disk capacity lower than the average used disk capacity as an migration node, and select migration nodes corresponding to one or more migration files one by one according to a difference between the used disk capacity of the migration node and the average used disk capacity.
The master server node 21 and the server node 22 in the system have all functions that the node devices in the distributed database specified in the prior art should have, except the functions provided by the present invention, and are not described herein again.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention.
Claims (10)
1. A method for balancing loads, the method comprising:
the server node adds the currently stored data to the migration unit file in real time, and establishes a migration unit file statistical table according to the migration unit file;
the method comprises the steps that a master control server node periodically obtains a migration unit file statistical table of each server node;
the master control server node selects a migration node, a migration file of the migration node and a migration node corresponding to the migration file as migration decisions according to the migration unit file statistical table and the migration strategy;
and when the master control server node determines the migration, adjusting the loads of the migrated node and the migrated node according to the migration decision.
2. The method of claim 1, wherein the migration unit file statistics table comprises: the identification of the server node, the total capacity of the disk and the used capacity of the disk, the file information of all migration units stored in the server node, the disk capacity occupied by each migration unit file and the access frequency of each migration unit file.
3. The method of claim 1, wherein the migration policy comprises: one or more of the following policies:
selecting a migration unit file statistical table with the used capacity of the disk exceeding a threshold value, and taking a server node corresponding to the selected migration unit file statistical table as a migration node; selecting the migration unit file in the migration node as a migration file from the selected migration unit file statistical table; selecting migration nodes corresponding to all migration files from server nodes corresponding to the residual migration unit file statistical table;
calculating the difference between the server node with the maximum disk used capacity and the server node with the minimum disk used capacity, when the difference is larger than a preset value, taking the server node with the maximum disk used capacity as a migrated node, selecting a migrated unit file as a migrated file from a corresponding migrated unit file statistical table, and selecting a migrated node for each migrated file from the server nodes corresponding to the remaining migrated unit file statistical table;
and selecting a server node corresponding to the migration unit file statistical table with the used capacity of the disk lower than the threshold value as a migration node according to the migration unit file statistical table, selecting a migration node from the server nodes corresponding to the residual migration unit file statistical table, and selecting a migration unit file as a migration file.
4. The method of claim 1, wherein before the master server node periodically obtains the migration unit file statistics table of each server node, the method further comprises: and the master control server node detects the states of all the server nodes in real time, if the states of the server nodes are ready to quit, the server nodes are taken as migration nodes, all migration unit files in the migration nodes are taken as migration files, and the migration nodes are selected for all the migration files from the server nodes corresponding to the residual migration unit file statistical table.
5. The method of claim 1, wherein adjusting the load of the migrating node and the migrating node according to the migration decision comprises: the master control server respectively sends a notification of sending the migrated files to the migrated nodes to the corresponding migrated nodes according to the selected migrated nodes, the migrated files of the migrated nodes and the migrated nodes corresponding to the migrated files;
the emigration node extracts the emigration file and sends the emigration file to the emigration node according to the notice of sending the emigration file;
when the master control server node receives the notification that the migration node returns the success of migration, all migration nodes corresponding to the migration files of the migration node are notified to process the received migration files;
and when the master control server node receives the notification of the completion of the migration returned by the migration node, updating the self migration unit file statistical table.
6. A distributed database system, the system comprising: a master server node and a server node; wherein,
the server node is used for adding the currently stored data to the migration unit file in real time, establishing a migration unit file statistical table according to the migration unit file, providing the migration unit file statistical table for the main control server node, and receiving the load adjustment of the main control server node as an immigration node or an emigration node;
the main control server node is used for periodically acquiring a migration unit file statistical table of each server node; selecting a migration node, a migration file of the migration node and a migration node corresponding to the migration file as migration decisions according to the migration unit file statistical table and the migration strategy; and when the migration is determined, adjusting the loads of the migration node and the migration node according to the migration decision.
7. The system according to claim 6, wherein the server node is specifically configured to establish a migration unit file statistical table by using the identifier of the server node, the total capacity of the disk, and the used capacity of the disk, information of all migration unit files stored in the server node, the disk capacity occupied by each migration unit file, and the access frequency of each migration unit file.
8. The system according to claim 6, wherein the master server node is specifically configured to store a migration policy, where the migration policy includes one or more of the following policies:
selecting a migration unit file statistical table with the used capacity of the disk exceeding a threshold value, and taking a server node corresponding to the selected migration unit file statistical table as a migration node; selecting the migration unit file in the migration node as a migration file from the selected migration unit file statistical table; selecting migration nodes corresponding to all migration files from server nodes corresponding to the residual migration unit file statistical table;
calculating the difference between the server node with the maximum disk used capacity and the server node with the minimum disk used capacity, when the difference is larger than a preset value, taking the server node with the maximum disk used capacity as a migrated node, selecting a migrated unit file as a migrated file from a corresponding migrated unit file statistical table, and selecting a migrated node for each migrated file from the server nodes corresponding to the remaining migrated unit file statistical table;
and selecting a server node corresponding to the migration unit file statistical table with the used capacity of the disk lower than the threshold value as a migration node according to the migration unit file statistical table, selecting a migration node from the server nodes corresponding to the residual migration unit file statistical table, and selecting a migration unit file as a migration file.
9. The system of claim 8,
the master control server node is also used for detecting the states of all the server nodes in real time, if the states of the server nodes are ready to quit, the server node is taken as a migration node, all migration unit files in the migration node are taken as migration files, and migration nodes are selected for all the migration files from the server nodes corresponding to the residual migration unit file statistical table;
correspondingly, the server node is also used for providing the state of the server node for the main control server node to be ready for quitting or normal operation.
10. The system of claim 6,
the master control server node is specifically configured to send a notification of sending the migrated file to the server node serving as the migrated node to the corresponding migrated node according to the selected migrated node, the migrated file of the migrated node, and the migrated node corresponding to the migrated file; when receiving a notification that the migration is successful returned by the server node serving as the migration node, notifying all migration nodes corresponding to the migration file of the migration node to process the received migration file; updating a migration unit file statistical table of the server node after receiving a notification of completion of migration returned by the server node as a migration node;
correspondingly, the server node is specifically configured to, when serving as a migration node, extract the migration file and send the extracted migration file to the migration node according to the notification that the master server node sends the migration file, and return a notification of successful migration to the master server node; and when the node is used as the migration node, after the migration is completed, a notification of completing the migration is returned to the master control server node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210288434.7A CN103593347B (en) | 2012-08-14 | 2012-08-14 | The method and distributed data base system of a kind of equally loaded |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210288434.7A CN103593347B (en) | 2012-08-14 | 2012-08-14 | The method and distributed data base system of a kind of equally loaded |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103593347A true CN103593347A (en) | 2014-02-19 |
| CN103593347B CN103593347B (en) | 2017-06-13 |
Family
ID=50083496
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210288434.7A Active CN103593347B (en) | 2012-08-14 | 2012-08-14 | The method and distributed data base system of a kind of equally loaded |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103593347B (en) |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105049508A (en) * | 2015-07-21 | 2015-11-11 | 齐鲁工业大学 | Cloud data migration method |
| CN105227645A (en) * | 2015-09-15 | 2016-01-06 | 齐鲁工业大学 | A kind of cloud data migration method |
| CN105245381A (en) * | 2015-10-22 | 2016-01-13 | 上海斐讯数据通信技术有限公司 | Cloud server downtime monitoring migration system and method |
| CN105488079A (en) * | 2014-10-11 | 2016-04-13 | 苏州精易会信息技术有限公司 | Method for processing multi-tenant data |
| WO2016165441A1 (en) * | 2015-09-06 | 2016-10-20 | 中兴通讯股份有限公司 | Migration policy adjustment method, capacity-change suggestion method and device |
| WO2018014613A1 (en) * | 2016-07-19 | 2018-01-25 | Zhejiang Dahua Technology Co., Ltd. | Systems and methods for migrating data |
| CN108710925A (en) * | 2018-05-15 | 2018-10-26 | 南京博内特信息科技有限公司 | A method of the clothes commodity shelf system based on Internet of Things |
| CN108781189A (en) * | 2016-09-26 | 2018-11-09 | 华为技术有限公司 | A kind of method and relevant device of load balancing |
| WO2019061132A1 (en) * | 2017-09-28 | 2019-04-04 | 深圳清华大学研究院 | Hybrid file system architecture, file storage, dynamic migration, and application thereof |
| CN110377430A (en) * | 2019-07-24 | 2019-10-25 | 中南民族大学 | Data migration method, equipment, storage medium and device |
| CN111143324A (en) * | 2019-12-20 | 2020-05-12 | 浪潮软件股份有限公司 | Kudu database data balancing system based on size and implementation method |
| EP3703342A1 (en) * | 2019-03-01 | 2020-09-02 | ABB Schweiz AG | Dynamic load balancing in network centric process control systems |
| CN113268203A (en) * | 2021-05-18 | 2021-08-17 | 天津中科曙光存储科技有限公司 | Capacity balancing method and device of storage system, computer equipment and storage medium |
| CN113448941A (en) * | 2020-03-27 | 2021-09-28 | 北京金山云网络技术有限公司 | Database migration method and device and electronic equipment |
| CN113821176A (en) * | 2021-09-29 | 2021-12-21 | 重庆紫光华山智安科技有限公司 | Data migration processing method and device and storage medium |
| CN115016727A (en) * | 2022-04-27 | 2022-09-06 | 阿里云计算有限公司 | Data dump method and device |
| CN116389442A (en) * | 2023-02-10 | 2023-07-04 | 上海弘积信息科技有限公司 | Web page request system and method based on load balancing |
| CN116991580A (en) * | 2023-07-27 | 2023-11-03 | 上海沄熹科技有限公司 | Distributed database system load balancing method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101582850B (en) * | 2009-06-19 | 2012-05-02 | 优万科技（北京）有限公司 | Method and system for realizing load balance |
| CN102457428B (en) * | 2010-10-27 | 2015-10-21 | 中兴通讯股份有限公司 | The implementation of load balancing of distributed hashtable network and device |
-
2012
- 2012-08-14 CN CN201210288434.7A patent/CN103593347B/en active Active
Cited By (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105488079A (en) * | 2014-10-11 | 2016-04-13 | 苏州精易会信息技术有限公司 | Method for processing multi-tenant data |
| CN105049508A (en) * | 2015-07-21 | 2015-11-11 | 齐鲁工业大学 | Cloud data migration method |
| WO2016165441A1 (en) * | 2015-09-06 | 2016-10-20 | 中兴通讯股份有限公司 | Migration policy adjustment method, capacity-change suggestion method and device |
| CN105227645A (en) * | 2015-09-15 | 2016-01-06 | 齐鲁工业大学 | A kind of cloud data migration method |
| CN105245381A (en) * | 2015-10-22 | 2016-01-13 | 上海斐讯数据通信技术有限公司 | Cloud server downtime monitoring migration system and method |
| CN105245381B (en) * | 2015-10-22 | 2019-08-16 | 上海斐讯数据通信技术有限公司 | Cloud Server delay machine monitors migratory system and method |
| WO2018014613A1 (en) * | 2016-07-19 | 2018-01-25 | Zhejiang Dahua Technology Co., Ltd. | Systems and methods for migrating data |
| US11620067B2 (en) | 2016-07-19 | 2023-04-04 | Zhejiang Dahua Technology Co., Ltd. | Systems and methods for migrating data |
| US10922004B2 (en) | 2016-07-19 | 2021-02-16 | Zhejiang Dahua Technology Co., Ltd. | Systems and methods for migrating data records |
| CN108781189B (en) * | 2016-09-26 | 2020-12-22 | 华为技术有限公司 | Load balancing method and related equipment |
| CN108781189A (en) * | 2016-09-26 | 2018-11-09 | 华为技术有限公司 | A kind of method and relevant device of load balancing |
| WO2019061132A1 (en) * | 2017-09-28 | 2019-04-04 | 深圳清华大学研究院 | Hybrid file system architecture, file storage, dynamic migration, and application thereof |
| CN111095233A (en) * | 2017-09-28 | 2020-05-01 | 深圳清华大学研究院 | Hybrid file system architecture, file storage, dynamic migration and applications thereof |
| CN111095233B (en) * | 2017-09-28 | 2023-09-26 | 深圳清华大学研究院 | Hybrid file system architecture, file storage, live migration and applications thereof |
| US10810169B1 (en) | 2017-09-28 | 2020-10-20 | Research Institute Of Tsinghua University In Shenzhen | Hybrid file system architecture, file storage, dynamic migration, and application thereof |
| CN108710925A (en) * | 2018-05-15 | 2018-10-26 | 南京博内特信息科技有限公司 | A method of the clothes commodity shelf system based on Internet of Things |
| US11500690B2 (en) | 2019-03-01 | 2022-11-15 | Abb Schweiz Ag | Dynamic load balancing in network centric process control systems |
| CN111641669B (en) * | 2019-03-01 | 2024-05-17 | Abb瑞士股份有限公司 | Dynamic load balancing in a network-centric process control system |
| CN111641669A (en) * | 2019-03-01 | 2020-09-08 | Abb瑞士股份有限公司 | Dynamic load balancing in a network-centric process control system |
| EP3703342A1 (en) * | 2019-03-01 | 2020-09-02 | ABB Schweiz AG | Dynamic load balancing in network centric process control systems |
| CN110377430A (en) * | 2019-07-24 | 2019-10-25 | 中南民族大学 | Data migration method, equipment, storage medium and device |
| CN110377430B (en) * | 2019-07-24 | 2021-08-31 | 中南民族大学 | Data migration method, device, storage medium and device |
| CN111143324A (en) * | 2019-12-20 | 2020-05-12 | 浪潮软件股份有限公司 | Kudu database data balancing system based on size and implementation method |
| CN111143324B (en) * | 2019-12-20 | 2023-05-02 | 浪潮软件股份有限公司 | Kudu database data equalization system based on size and implementation method |
| CN113448941A (en) * | 2020-03-27 | 2021-09-28 | 北京金山云网络技术有限公司 | Database migration method and device and electronic equipment |
| CN113448941B (en) * | 2020-03-27 | 2024-10-11 | 北京金山云网络技术有限公司 | Database migration method and device and electronic equipment |
| CN113268203A (en) * | 2021-05-18 | 2021-08-17 | 天津中科曙光存储科技有限公司 | Capacity balancing method and device of storage system, computer equipment and storage medium |
| CN113268203B (en) * | 2021-05-18 | 2022-11-04 | 天津中科曙光存储科技有限公司 | Capacity balancing method and device of storage system, computer equipment and storage medium |
| CN113821176A (en) * | 2021-09-29 | 2021-12-21 | 重庆紫光华山智安科技有限公司 | Data migration processing method and device and storage medium |
| CN115016727A (en) * | 2022-04-27 | 2022-09-06 | 阿里云计算有限公司 | Data dump method and device |
| CN116389442A (en) * | 2023-02-10 | 2023-07-04 | 上海弘积信息科技有限公司 | Web page request system and method based on load balancing |
| CN116991580A (en) * | 2023-07-27 | 2023-11-03 | 上海沄熹科技有限公司 | Distributed database system load balancing method and device |
| CN116991580B (en) * | 2023-07-27 | 2024-09-13 | 上海沄熹科技有限公司 | Distributed database system load balancing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103593347B (en) | 2017-06-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103593347B (en) | The method and distributed data base system of a kind of equally loaded | |
| US9262287B2 (en) | Computer information system and dynamic disaster recovery method therefor | |
| CN106603319B (en) | Fault processing method, management server and logic server | |
| CN108924202B (en) | Distributed cluster data disaster tolerance method and related device | |
| CN106936623A (en) | The management method of distributed cache system and cache cluster | |
| JP5753030B2 (en) | Caching system using cache cloud structure and method for providing caching service | |
| CN112272291B (en) | Video storage method, device, management equipment and readable storage medium | |
| US20130117423A1 (en) | Methods for managing data synchronization and devices thereof | |
| CN107729205B (en) | Fault processing method and device for business system | |
| WO2016139938A1 (en) | Thin client system, connection management device, virtual machine operating device, method, and storage medium | |
| CN112612769B (en) | File processing method, device and storage medium | |
| CN108989106A (en) | A kind of data processing method of distributed type assemblies, apparatus and system | |
| EP3570169B1 (en) | Method and system for processing device failure | |
| CN113312234B (en) | Health detection optimization method and terminal | |
| CN110545197B (en) | Node state monitoring method and device | |
| CN114844912A (en) | Data link distribution method and device and distributed block storage system | |
| CN108200151B (en) | ISCSI Target load balancing method and device in distributed storage system | |
| CN103888310B (en) | Monitor processing method and system | |
| CN115145782A (en) | Server switching method, mooseFS system and storage medium | |
| WO2015196769A1 (en) | Method for processing data in iptv system, and network element device | |
| CN114039836A (en) | Fault processing method and device for Exporter collector | |
| CN110365936B (en) | Code stream obtaining method, device and system | |
| CN109462639B (en) | Port expansion equipment management method and device | |
| CN105307198A (en) | System and device for implementing change of anchor base station of terminal | |
| CN115174356B (en) | Cluster alarm reporting method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |