WO2017020742A1 - 负载均衡方法及设备 - Google Patents

负载均衡方法及设备 Download PDF

Info

Publication number
WO2017020742A1
WO2017020742A1 PCT/CN2016/091521 CN2016091521W WO2017020742A1 WO 2017020742 A1 WO2017020742 A1 WO 2017020742A1 CN 2016091521 W CN2016091521 W CN 2016091521W WO 2017020742 A1 WO2017020742 A1 WO 2017020742A1
Authority
WO
WIPO (PCT)
Prior art keywords
server
partition
load
target
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/091521
Other languages
English (en)
French (fr)
Chinese (zh)
Inventor
沈春辉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to JP2018505621A priority Critical patent/JP6886964B2/ja
Publication of WO2017020742A1 publication Critical patent/WO2017020742A1/zh
Priority to US15/890,319 priority patent/US20180167461A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/5033—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering data affinity
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system
    • G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0852—Delays
    • H04L43/087—Jitter
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1004—Server selection for load balancing
    • H04L67/101—Server selection for load balancing based on network conditions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0876—Network utilisation, e.g. volume of load or congestion level
    • H04L43/0882—Utilisation of link capacity
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1004—Server selection for load balancing
    • H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload

Definitions

  • the present application relates to the field of computers, and in particular, to a load balancing method and device.
  • a data table is sliced lexicographically, each slice is called a partition, and these partitions are distributed to servers in the cluster. How to distribute these partitions evenly or dynamically is the problem to be solved by the load balancing method in the system.
  • the quality of the load balancing method directly affects whether the data storage is uniform and the service read and write requests are uniform. In extreme cases, if the balance fails and all partitions are scheduled to a physical machine, then the service capability of the entire cluster is equivalent to the service capability of a physical machine, and the performance of the cluster becomes the same as that of a single machine.
  • the load balancing methods in the existing distributed storage systems are generally based on the equalization of the number of partitioned loads, and the goal is to make the number of partitions on each server substantially similar.
  • the basic method is as follows: first obtain the number of partition loads on all servers, calculate the average partition load on each server, and migrate the partitions on the server that exceeds the partition load to the server with too few partition loads to Implement load balancing. When some physical machines in the system are restarted, the partitions will be reallocated. In the process, since the current load balancing method only considers the number of partitioned loads, the probability that the partitions are assigned to each server is the same. Therefore, the probability of remote reading of partition data is greatly increased. Because of the need to access disk data on a remote server during remote reads, remote reads require additional network overhead compared to local reads, so read performance is poor.
  • the load balancing method when the load balancing method is applied to the distributed storage system by using the existing load balancing method, the data localization rate after the partition allocation is not high, so that the read performance of the entire system is poor.
  • An object of the present application is to provide a load balancing method and device to solve the prior art.
  • the data localization rate of the middle partition is not high and the read performance is poor.
  • the present application provides a load balancing method, including:
  • the partition is migrated to the target server corresponding to the partition.
  • the server with the highest data localization rate of each partition is determined as the target server corresponding to the partition, including:
  • the server with the highest data localization rate of the partition is determined as the partition.
  • Corresponding target server If the difference between the data localization rate of the server where the current partition is located and the data localization rate of the server with the highest data localization rate is greater than a preset value, the server with the highest data localization rate of the partition is determined as the partition. Corresponding target server.
  • the method further includes:
  • the partition corresponding to the partition is changed to the low load server, where N is a positive integer.
  • determining the high load server and the low load server according to the predicted partition load number including:
  • the server is configured to determine the server whose predicted partition load number is lower than the preset load range lower limit as the low load server.
  • N is the difference between the predicted partition load number of the high load server and the average predicted partition load number of all servers.
  • the method further includes:
  • the server where the current partition is located is a high-load server, and the partition is one of the N partitions with lower data localization rate in all partitions on the current server, the target server corresponding to the partition is changed to The low load server, wherein N is a positive integer;
  • the partition is migrated to the target server corresponding to the partition.
  • determining the high load server and the low load server according to the current partition load number including:
  • the server whose current partition load number is higher than the preset load range upper limit is determined as a high load server, and the server whose current partition load number is lower than the preset load range lower limit is determined as a low load server.
  • N is the difference between the current partition load of the high load server and the average current partition load of all servers.
  • changing the target server corresponding to the partition to the low load server includes:
  • the target server corresponding to the partition is changed to the low-load server with the highest data localization rate of the partition according to the data localization rate of the partition at the low-load server.
  • the partition is migrated to a target server corresponding to the partition, including:
  • Each partition is sequentially migrated to a target server corresponding to the partition according to a preset interval.
  • a load balancing device comprising:
  • a localization rate obtaining device configured to acquire a data localization rate of each partition on each server, where the data localization rate is a local data and a local data stored in a physical machine corresponding to the partition The ratio of the total data of the partition;
  • a target determining device configured to determine a server with the highest data localization rate of each partition as a target server corresponding to the partition
  • the partition migration device is configured to migrate the partition to a target server corresponding to the partition if the current server of the partition is a different server from the corresponding target server.
  • the target determining apparatus is configured to: if the difference between the data localization rate of the server where the current partition is located and the data localization rate of the server with the highest data localization rate is greater than a preset value, the data of the partition is used.
  • the server with the highest localization rate is determined as the target server corresponding to the partition.
  • the device further includes:
  • a load determining device configured to calculate a predicted partition load number of each server after determining a server having the highest data localization rate of each partition as a target server corresponding to the partition, and determining a high according to the predicted partition load number a load server and a low load server; wherein the predicted partition load number is a number of partitions that will exist on each server after migrating each partition to a target server corresponding to the partition;
  • a target changing device configured to: before the partition is migrated to the target server corresponding to the partition, if the target server corresponding to a certain partition is a high-load server, and the partition is all the partitions that will exist on the target server If one of the N partitions with a lower data localization rate is changed, the target server corresponding to the partition is changed to the low load server, where N is a positive integer.
  • the load determining apparatus is configured to determine, as a high load server, the server whose predicted partition load number is higher than a preset load range upper limit, and determine the server whose predicted partition load number is lower than a preset load range lower limit. For low load servers.
  • N is the difference between the predicted partition load number of the high load server and the average predicted partition load number of all servers.
  • the device further includes:
  • a load determining device configured to acquire a current partition load number of each server after migrating the partition to a target server corresponding to the partition, and determine a high load server and a low load server according to the current partition load number;
  • the target changing device is configured to: if the server where the current partition is located is a high-load server, and the partition is one of the N partitions with lower data localization rate in all partitions on the current server, the partition is Corresponding target server is changed to the low load server, where N is a positive integer;
  • the partitioning device is further configured to: after the target changing device changes the target server corresponding to the partition to the low load server, if the server where the partition is currently located and the corresponding target server are different servers, Migrate the partition to the target server corresponding to the partition.
  • the load determining apparatus is configured to determine, as a high load server, the server whose current partition load number is higher than a preset load range upper limit, and determine, by the server whose current partition load number is lower than a preset load range lower limit. For low load servers.
  • N is the difference between the current partition load of the high load server and the average current partition load of all servers.
  • the target changing means is configured to change the target server corresponding to the partition to the one according to a data localization rate of the partition at the low load server.
  • the partition migration apparatus is configured to sequentially migrate each partition to a target server corresponding to the partition according to a preset interval time.
  • the technical solution provided by the present application obtains the data localization rate of each partition on each server, and allocates each partition to the server with the highest localization rate according to the data localization rate, and processes
  • the data query request is made, since the data localization rate of the server to which each partition is currently allocated is high, most of the data can be acquired on the disk of the local server, thereby greatly reducing the probability of remote reading of the partition data.
  • To improve read performance by further adjusting the allocation of partitions by combining the number of partitioned loads, it is possible to optimize the read performance while avoiding certain servers in certain situations (such as data hotspots or system expansion). The problem of relatively concentrated partition load that can occur.
  • FIG. 1 is a flowchart of a load balancing method according to an embodiment of the present application
  • FIG. 2 is a schematic diagram of storage of a distributed data storage system based on a distributed file system according to an embodiment of the present application
  • FIG. 3 is a flowchart of a preferred load balancing method according to an embodiment of the present disclosure
  • FIG. 4 is a flowchart of a more preferred load balancing method according to an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of a load balancing device according to an embodiment of the present disclosure.
  • FIG. 6 is a schematic structural diagram of a preferred load balancing device according to an embodiment of the present disclosure.
  • FIG. 7 is a schematic structural diagram of a more preferred load balancing device according to an embodiment of the present disclosure.
  • the terminal, the device of the service network, and the trusted party each include one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
  • processors CPUs
  • input/output interfaces network interfaces
  • memory volatile and non-volatile memory
  • the memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory.
  • RAM random access memory
  • ROM read only memory
  • Memory is an example of a computer readable medium.
  • Computer readable media includes both permanent and non-persistent, removable and non-removable media.
  • Information storage can be implemented by any method or technology.
  • the information can be computer readable instructions, data structures, modules of programs, or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electricity Erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, A tape storage or other magnetic storage device or any other non-transportable medium that can be used to store information that can be accessed by a computing device.
  • computer readable media does not include non-transitory computer readable media, such as modulated data signals and carrier waves.
  • FIG. 1 shows a load balancing method provided by an embodiment of the present application, which includes the following steps:
  • Step S101 Acquire a data localization rate of each partition on each server, where the data localization rate is local data of the partition stored on a physical machine corresponding to a certain server and total data of the partition.
  • Step S102 determining, by the server with the highest data localization rate of each partition, the target server corresponding to the partition;
  • Step S103 If the server where the partition is currently located is a different server from the corresponding target server, migrate the partition to the target server corresponding to the partition.
  • the existing load balancing method is generally based on the equalization of the partition load number, only the factor of the partition load is taken into consideration, so that the probability that the partition is allocated to each server when the partition is reallocated is the same, so it is easy to cause the partition.
  • the data localization rate of this partition in the server is low.
  • SSD Solid State Drive
  • the Region is a logical table (Table). According to a data unit that is divided according to a preset rule, there is no intersection between the partitions, and all the partitions constitute a complete logical table.
  • a partition will contain multiple files, and a file will consist of one or more blocks, which are the basic unit of physical storage.
  • each data block has multiple copies that are allocated to multiple servers in a distributed file system for redundant storage.
  • 2 shows a storage schematic diagram of a distributed data storage system based on a distributed file system.
  • the distributed data storage system includes three servers: server 1, server 2, and server 3, respectively, and each server is allocated a plurality of servers.
  • the partitions, such as the partition of server 1, are partition A and partition B.
  • a plurality of files are further included.
  • the partition A includes the file 1 and the file 2.
  • the file 1 further includes a data block 11 and a data block 12, and the file 2 includes the data block 21 and the data block 22.
  • the server 1', the server 2', and the server 3' are the same physical machine as the server 1, the server 2, and the server 3, respectively.
  • Each data block has two copies, deployed in a server with a distributed file system, wherein two copies of the data block 11 are respectively deployed on the server 1' and the server 3', and two copies of the data block 12 are respectively deployed on The server 2' and the server 3', two copies of the data block 21 are respectively deployed to the server 1' and the server 3', and two copies of the data block 22 are respectively deployed to the server 2' and the server 3'.
  • a Server1 , A Server2 , and A Server3 respectively represent the data localization rate of the partition A in the servers 1 to 3, and Block11, Block12, Block21, and Block22 respectively represent the data block 11, the data block 12, the data block 21, and the data block 22.
  • Size, File1, File2 represent the size of file 1 and file 2, respectively.
  • multiple copies of a file's data block are peer-to-peer, that is, multiple parts of the storage medium are the same.
  • each copy is stored in an HDD (Hard Disk Drive) or SSD.
  • the data blocks on each physical machine storage medium are calculated.
  • multiple copies of the data block of the file are not equal, such as two copies of the aforementioned data block 11, one stored in the HDD and the other stored in the SSD
  • the data localization calculation only the data blocks stored in the SSD are calculated, for example, two copies of the data block 11 are stored in the server 1' in the data localization calculation.
  • the HDD it is stored in the SSD in the server 3'.
  • the execution body of the load balancing method may be a central server in a distributed data storage system.
  • the central server includes, but is not limited to, implementations such as a network host, a single network server, a plurality of network server sets, or a cloud computing based computer collection.
  • the cloud is composed of a large number of host or network servers based on Cloud Computing, which is a kind of distributed computing, a virtual computer composed of a group of loosely coupled computers.
  • the central server can periodically collect the data localization rate of the partition on each server by means of a heartbeat report.
  • the server with the highest data localization rate of each partition can be determined, and the server is the preferred server of the corresponding partition, and will be migrated as a partition.
  • Target server Taking the scenario shown in FIG. 2 as an example, it is assumed that the data block 11, the data block 12, the data block 21, and the data block 22 have the same size.
  • the data localization rate distribution is:
  • a partition migration plan is generated, and the server 3 is determined as the target server of the partition A. Since the server currently deployed by the partition A is the server 1, and the target server is not the same server, the partition migration plan is executed, and the partition A is migrated to the target server corresponding to the partition, that is, the server 3. If the server where the partition is currently located is the same server as its target server, it means that the currently deployed server of the partition already has the highest data localization rate because no partition migration is required. After the migration is completed, the data localization rate of the partition A in the server 3 can reach 100%, that is, for any data query request, only the local disk of the physical machine where the server 3 is located is locally read, and the data can be obtained. Data is needed, so read performance is greatly improved.
  • the number of servers, partitions, files, and data blocks involved will be significantly larger than the number shown in Figure 2.
  • the number of data blocks of the remaining partitions stored on the physical machine corresponding to the server will be relatively higher.
  • the data localization rate of the remaining partitions on the server is relatively low, so after the partition migration according to the data localization rate, the number of partition loads on each server will be relatively balanced, so that the load of each server is relatively close. .
  • the server with the highest data localization rate of each partition is determined as the target server corresponding to the partition, including: if the data localization rate of the server where the partition is currently located and the data locality of the server with the highest data localization rate are local If the difference between the conversion rates is greater than the preset value, the server with the highest localization rate of the partition is confirmed. It is defined as the target server corresponding to the partition.
  • the preset value may be set according to an actual application scenario, for example, it may be set to 10%, that is, if the difference between the data localization rate of the current server and the highest data localization rate that can be achieved is greater than 10%, The server that uses this highest data localization rate is targeted.
  • partition A assume that the data localization rate of partition A in the currently deployed server 1 is 70%, and the data localization rates thereof at server 2 and server 3 are 30% and 75%, respectively.
  • the server with the highest data localization rate is server 3, but the difference between the data localization rate of the server and the current server is only 5%, and the overall read performance is not improved after the migration. Therefore, the migration may not be performed, and the target server of the partition A may be set to the server where it is currently located (ie, the server 1) during the actual processing. If the data localization rate of the partition in the server 3 reaches 90%, the data localization rate can be increased by 20% after the migration. At this time, the read performance is improved, so the server 3 is used as the target server of the partition A. .
  • the data localization rate of each partition in the adjusted distributed data storage system can be maximized, and in general, the load between the servers can be relatively balanced.
  • the data on some server nodes may be concentrated, causing many partitions to be loaded on individual servers, while other servers load fewer partitions, resulting in no partition load. balanced.
  • the embodiment of the present application further provides a preferred load balancing method.
  • the processing flow of the method is as shown in FIG. 3, and includes the following steps:
  • Step S101 Acquire a data localization rate of each partition on each server, where the data localization rate is local data of the partition stored on a physical machine corresponding to a certain server and total data of the partition.
  • Step S102 determining, by the server with the highest data localization rate of each partition, the target server corresponding to the partition;
  • Step S103 if the current server of the partition and its corresponding target server are different servers, the partition is migrated to the target server corresponding to the partition;
  • Step S104 obtaining the current partition load number of each server, and according to the current partition The number of loads determines the high load server and the low load server;
  • Step S105 if the server where the current partition is located is a high-load server, and the partition is one of the N partitions with lower data localization rate among all the partitions on the current server, the target corresponding to the partition is The server is changed to the low load server, where N is a positive integer;
  • Step S106 If the server where the partition is currently located and its corresponding target server are different servers, the partition is migrated to the target server corresponding to the partition.
  • the partition load of all the servers is combined with the number of partitions to adjust the partition load imbalance caused by the data localization rate only, and the data is guaranteed. Under the premise of reading performance, the server's partition load number is more balanced.
  • the low-load server and the high-load server refer to servers that have a deployed partition number lower than and higher than the average partition load number, respectively.
  • a preset load range may be set according to the average partition load number, and whether a server belongs to a low load server or a high load server according to the upper limit and the lower limit of the preset load range may be used. Therefore, the step S104 may be: obtaining the current partition load number of each server, and determining, as the high load server, the server whose current partition load number is higher than the preset load range upper limit, and lowering the current partition load number.
  • the server with the preset load range lower limit is determined to be a low load server.
  • the upper limit of the preset load range can be set as: the average partition load number ⁇ (1 + coefficient), and the preset load range upper limit is set as: the average partition load number ⁇ (1 + coefficient), the coefficient can be specific
  • the server is determined to be a low load server.
  • the current partition load of the server 3 is 57, it is determined as a high load server, and the current partition load of the server 1 is 40, which is determined to be a low load server, and the data localization rate in the server 3 can be the lowest.
  • the partitions are migrated to Server 1 to make the partition load more balanced.
  • the number of migrations can be determined according to actual needs, and may be one or more.
  • the partition load of the server 3 cannot be reduced to the preset load range, Still able to make the load tend to be balanced.
  • the number of partitions migrated from a high load server to a low load server can be determined based on the average partition load count.
  • N is the difference between the current partition load of the high load server and the average current partition load of all servers.
  • the number of partitions that need to be migrated is seven, that is, if a partition is one of the seven partitions with lower data localization rate in all partitions on the current server, the target corresponding to the partition The server will be changed to server 1. For the remaining six partitions with lower data localization rates, the target server will also be changed to a low-load server.
  • the target server corresponding to the partition may be changed to a low-load server by means of random allocation.
  • the target server corresponding to the partition is changed to a low load server with the highest data localization rate of the partition.
  • the server 1 is a high-load server
  • the server 3, the server 4, and the server 6 are both low-load servers
  • the partition B is the partition with the lowest data localization rate in the server 1, which is 52%
  • the partition B is in the server 3 and the server. 4.
  • the data localization rates on server 6 are 40%, 33%, and 17%, respectively.
  • Partition B will still be migrated from a high-load server to a low-load server to ensure a balanced number of partitioned loads.
  • the optimal server can still be selected according to the data localization rate. For example, for partition B, its optimal low load server is server 3.
  • the following may occur: if the server where the partition B is currently located is the server 6 at step S101, the data localization rate is 17%; at step S102, the target server of the partition B is determined. For server 1, the data localization rate is 52%; in step S103, partition B is migrated to the currently set target server to have an optimal data localization rate.
  • steps S104 to S106 based on the consideration of the number of partitioned loads, it is necessary to change the target server of the partition B to the server 3 and perform migration. In this process, partition B is migrated twice, and from the final result, partition B is migrated from server 6 to server 3. In theory, only one migration is required. Therefore, the above scheme is carried out in S103. The move may be invalid.
  • the embodiment of the present application further provides a more preferred load balancing method.
  • the processing flow of the method is as shown in FIG. 4, and includes the following steps:
  • Step S101 Acquire a data localization rate of each partition on each server, where the data localization rate is local data of the partition stored on a physical machine corresponding to a certain server and total data of the partition.
  • Step S102 determining, by the server with the highest data localization rate of each partition, the target server corresponding to the partition;
  • Step S107 calculating a predicted partition load number of each server, and determining a high load server and a low load server according to the predicted partition load number; wherein the predicted partition load number is if each partition is migrated to the partition corresponding to The number of partitions that will exist on each server after the target server;
  • Step S108 if the target server corresponding to a certain partition is a high-load server, and the partition is one of N partitions with lower data localization rate among all the partitions that will exist on the target server, the The target server corresponding to the partition is changed to the low load server, where N is a positive integer;
  • Step S103 If the server where the partition is currently located is a different server from the corresponding target server, migrate the partition to the target server corresponding to the partition.
  • the solution predicts that each server will exist after the corresponding partition is migrated according to the determined target server.
  • the number of partitioned loads By predicting the number of partitioned loads and balancing the number of partitioned partitions, the target server of the partition is changed, and the target server determined at this time is unified to perform partition migration. Since the computational cost of analog computing is much less than the actual migration, it can avoid invalid migration, save processing costs, and improve load balancing efficiency while paying less computational cost.
  • the predicted partition load number used to determine the high load server and the low load server is a calculated value based on the first determined target server, and is not directly obtained by each server. The actual value taken.
  • the manner of determining the high load server and the low load server according to the predicted partition load number and when there are multiple low load servers, how to select one of them as the target server is similar to the load balancing method shown in FIG. 3 described above. For the sake of brevity, it will not be discussed here.
  • step S107 determining a high load server and a low load server according to the predicted partition load number, including: determining, by the server whose predicted partition load number is higher than a preset load range upper limit, as a high load server, The server that predicts that the partition load number is lower than the lower limit of the preset load range is determined to be a low load server.
  • N is the difference between the predicted partition load number of the high load server and the average predicted partition load number of all servers. And changing the target server corresponding to the partition to the low load server, specifically: when the low load server is multiple, according to the data localization rate of the partition at the low load server, The target server corresponding to the partition is changed to the low load server with the highest data localization rate of the partition.
  • the partition when the partitioning is performed, the partition is migrated to the target server corresponding to the partition, which includes: sequentially according to a preset interval time. Each partition is migrated to the target server for that partition. Because the related settings of the distributed data storage system will change during the partition migration process, if a large number of migrations occur in a short period of time, the internal settings of the system may change too fast, causing the system to jitter. To avoid this situation, you can set a certain interval (for example, 100ms) when each partition is migrated to prevent jitter caused by partition migration.
  • a certain interval for example, 100ms
  • an embodiment of the present application further provides a load balancing device, which is configured as shown in FIG. 5, and includes a localization rate obtaining device 510, a target determining device 520, and a partition migrating device. 530.
  • the localization rate obtaining apparatus 510 is configured to obtain a data localization rate of each partition on each server, where the data localization rate is locally stored on a physical machine corresponding to a certain server.
  • the ratio of the data to the total data of the partition is configured to determine the server with the highest data localization rate of each partition as the target server corresponding to the partition; and the partition migration device 530 is configured to use the partition current If the server and its corresponding target server are different servers, then the The partition is migrated to the target server corresponding to the partition.
  • the existing load balancing devices are generally based on the equalization of the number of partitioned loads, only the factor of the number of partitioned loads is taken into account, so that the probability that the partitions are allocated to the respective servers when the partitions are reallocated is the same, so that the partitions are easily caused.
  • the data localization rate of this partition in the server is low. For most data query requests, it is often necessary to obtain the disk by remotely accessing other physical machines, which will greatly reduce the read performance of the system.
  • the QPS capability that can be provided is close to 30,000 times; if it is remotely read, the 100MB/S capability provided by the Gigabit network card Calculate that a random read access to at least one 16KB block can provide up to 6,000 QPS capabilities. Regardless of QPS throughput, remote reads will have at least 0.5ms of overhead over local reads in terms of response latency.
  • each partition by obtaining the data localization rate of the partition on each server, and assigning each partition to the server with the highest localization rate according to the data localization rate, when processing the data query request, since each partition is currently allocated
  • the data localization rate is high in the server, and most of the data can be obtained in the disk of the local server, so the probability of remote reading of the partition data can be greatly reduced, and the reading performance is improved.
  • a partition is a logical table according to a data unit that is divided according to a preset rule, and there is no intersection between the partitions, and all partitions constitute one. Complete logical table.
  • a partition will contain multiple files, and a file will consist of one or more data blocks, which are the basic unit of physical storage.
  • each data block has multiple copies that are allocated to multiple servers in a distributed file system for redundant storage.
  • 2 shows a storage schematic diagram of a distributed data storage system based on a distributed file system.
  • the distributed data storage system includes three servers: server 1, server 2, and server 3, respectively, and each server is allocated a plurality of servers.
  • the partitions such as the partition of server 1, are partition A and partition B.
  • a plurality of files are further included.
  • the partition A includes the file 1 and the file 2.
  • the file 1 further includes a data block 11 and a data block 12, and the file 2 includes the data block 21 and the data block 22.
  • the server 1', the server 2', and the server 3' are the same physical machine as the server 1, the server 2, and the server 3, respectively.
  • Each number There are two copies of the block, deployed in the server with the distributed file system, wherein two copies of the data block 11 are respectively deployed on the server 1' and the server 3', and two copies of the data block 12 are respectively deployed on the server 2 'And server 3', two copies of the data block 21 are respectively deployed to the server 1' and the server 3', and two copies of the data block 22 are respectively deployed to the server 2' and the server 3'.
  • the data localization rate distribution of partition A that is, the data localization rate of partition A on each server is:
  • a Server1 , A Server2 , and A Server3 respectively represent the data localization rate of the partition A in the servers 1 to 3, and Block11, Block12, Block21, and Block22 respectively represent the data block 11, the data block 12, the data block 21, and the data block 22.
  • Size, File1, File2 represent the size of file 1 and file 2, respectively.
  • multiple copies of a file's data block are peer-to-peer, that is, multiple parts of the storage medium are the same, for example, each copy is stored in the HDD or SSD.
  • each physical Data blocks on the machine's storage media are counted.
  • multiple copies of the data block of the file are not equal, such as two copies of the aforementioned data block 11, one stored in the HDD and the other stored in the SSD
  • the data localization calculation only the data blocks stored in the SSD are calculated, for example, two copies of the data block 11 are stored in the server 1' in the data localization calculation.
  • the HDD it is stored in the SSD in the server 3'.
  • the device 5 may be a central server in a distributed data storage system.
  • the central server includes, but is not limited to, implementations such as a network host, a single network server, a plurality of network server sets, or a cloud computing based computer collection.
  • the cloud is composed of a large number of host or network servers based on cloud computing, wherein cloud computing is a kind of distributed computing, a virtual computer composed of a group of loosely coupled computers.
  • the central server can periodically collect the data localization rate of the partition on each server by means of a heartbeat report.
  • the localization rate obtaining means 510 can determine the server with the highest data localization rate of each partition according to the acquired data localization rate of each partition on each server, and the server is the preferred server of the corresponding partition, and will serve as The target server for partition migration. Taking the scenario shown in FIG. 2 as an example, it is assumed that the data block 11, the data block 12, the data block 21, and the data block 22 have the same size. At this time, the data localization rate distribution of the partition A is:
  • a partition migration plan is generated, and the server 3 is determined as the target server of the partition A. Since the server currently deployed by the partition A is the server 1, and the target server is not the same server, the partition migration plan is executed, and the partition A is migrated to the target server corresponding to the partition, that is, the server 3. If the server where the partition is currently located is the same server as its target server, it means that the currently deployed server of the partition already has the highest data localization rate because no partition migration is required. After the migration is completed, the data localization rate of the partition A in the server 3 can reach 100%, that is, for any data query request, only the local disk of the physical machine where the server 3 is located is locally read, and the data can be obtained. Data is needed, so read performance is greatly improved.
  • the number of servers, partitions, files, and data blocks involved will be significantly larger than the number shown in Figure 2.
  • the number of remaining partitions is limited due to storage space limitation.
  • the number of blocks stored on the physical machine corresponding to the server will be relatively small, so that the data localization rate of the remaining partitions on the server is relatively low, so after partition migration according to the data localization rate, on each server
  • the number of partitioned loads will also be balanced, making the load of each server relatively close.
  • the target determining apparatus 520 is configured to: if the difference between the data localization rate of the server where the current partition is located and the data localization rate of the server with the highest data localization rate is greater than a preset value, the data of the partition is used. The server with the highest localization rate is determined as the target server corresponding to the partition.
  • the preset value may be set according to an actual application scenario, for example, it may be set to 10%, that is, if the difference between the data localization rate of the current server and the highest data localization rate that can be achieved is greater than 10%, The server that uses this highest data localization rate is targeted.
  • partition A assume that the data localization rate of partition A in the currently deployed server 1 is 70%, and the data localization rates thereof at server 2 and server 3 are 30% and 75%, respectively.
  • the server with the highest data localization rate is server 3, but the difference between the data localization rate of the server and the current server is only 5%, and the overall read performance is not improved after the migration. Therefore, the migration may not be performed, and the target server of the partition A may be set to the server where it is currently located (ie, the server 1) during the actual processing. If the data localization rate of the partition in the server 3 reaches 90%, the data localization rate can be increased by 20% after the migration. At this time, the read performance is improved, so the server 3 is used as the target server of the partition A. .
  • the data localization rate of each partition in the adjusted distributed data storage system can be maximized, and in general, the load between the servers can be relatively balanced.
  • the data on some server nodes may be concentrated, causing many partitions to be loaded on individual servers, while other servers load fewer partitions, resulting in no partition load. balanced.
  • the embodiment of the present application further provides a preferred load balancing device, and the device 5 The structure is as shown in FIG. 6.
  • the load determination means 540 and the target change means 550 are also included.
  • the load determining apparatus 540 is configured to acquire a current partition load number of each server after migrating the partition to a target server corresponding to the partition, and determine a high load server and a low load server according to the current partition load number.
  • the target changing device 550 is configured to: if the server where the current partition is currently located is a high-load server, and the partition is one of the N partitions with lower data localization rate in all partitions on the current server, the The target server corresponding to the partition is changed to the low load server, where N is a positive integer.
  • the partition migration device 530 is further configured to change, in addition to the target server determined by the target determining device, the target server corresponding to the partition to the low load, in addition to migrating the partition according to the target server determined by the target determining device. After the server, if the current server of the partition is a different server from the corresponding target server, the partition is migrated to the target server corresponding to the partition.
  • the content of the localization rate obtaining device 510 and the object determining device 520 are the same as or substantially the same as those of the corresponding device in the embodiment of FIG. 5 for the sake of brevity, and therefore will not be further described herein. It is hereby incorporated by reference.
  • the partition load of all the servers is combined with the number of partitions to adjust the partition load imbalance caused by the data localization rate only, and the data is guaranteed. Under the premise of reading performance, the server's partition load number is more balanced.
  • the low-load server and the high-load server refer to servers that have a deployed partition number lower than and higher than the average partition load number, respectively.
  • a preset load range may be set according to the average partition load number, and whether a server belongs to a low load server or a high load server according to the upper limit and the lower limit of the preset load range may be used. Therefore, when determining the high load server and the low load server, the load determining apparatus 540 determines that the server whose current partition load number is higher than the preset load range upper limit is determined as a high load server, and the current partition load number is low.
  • the server at the lower limit of the preset load range is determined to be a low load server.
  • the current partition load of the server 3 is 57, it is determined as a high load server, and the current partition load of the server 1 is 40, which is determined to be a low load server, and the data localization rate in the server 3 can be the lowest.
  • the partitions are migrated to Server 1 to make the partition load more balanced.
  • the number of migrations can be determined according to actual needs, and may be one or more.
  • the partition load of the server 3 cannot be reduced to the preset load range, Still able to make the load tend to be balanced.
  • the number of partitions migrated from a high load server to a low load server can be determined based on the average partition load count.
  • N used in the target changing means 550 is the difference between the current partition load number of the high load server and the average current partition load number of all the servers.
  • the number of partitions that need to be migrated is seven, that is, if a partition is one of the seven partitions with lower data localization rate in all partitions on the current server, the target corresponding to the partition The server will be changed to server 1. For the remaining six partitions with lower data localization rates, the target server will also be changed to a low-load server.
  • the target changing device 550 may change the target server corresponding to the partition to a certain low load server by means of random allocation. Further, the target changing means 550 also changes the target server corresponding to the partition to a low load server having the highest data localization rate of the partition according to the data localization rate of the partition at the low load server.
  • the server 1 is a high-load server
  • the server 3, the server 4, and the server 6 are both low-load servers
  • the partition B is the partition with the lowest data localization rate in the server 1, which is 52%
  • the partition B is in the server 3 and the server. 4.
  • the data localization rates on server 6 are 40%, 33%, and 17%, respectively.
  • Partition B will still be migrated from a high-load server to a low-load server to ensure a balanced number of partitioned loads.
  • the server is the server 3.
  • the following may occur: if the localization rate obtaining means 510 obtains the partition data localization rate, the server where the partition B is currently located is the server 6, and the data localization rate is 17%; the target is determined.
  • the device 520 determines that the target server of the partition B is the server 1 according to the localization rate obtaining device 510, and the data localization rate is 52%; at this time, the partition migration device 530 determines the target according to the target determining device 520.
  • the server migrates partition B to the currently set target server to have the optimal data localization rate.
  • the load determining device 540, the target changing device 550, and the partition migrating device 530 may change the target server of the partition B to the server 3 based on the number of partition loads in the subsequent processing, and perform migration.
  • partition B is migrated twice, and from the final result, partition B is migrated from server 6 to server 3.
  • only one migration is required. Therefore, the first migration by the partition migration device 530 for a certain partition (i.e., the migration by the target server determined by the target determination device 520) in the above scheme may be invalid.
  • the embodiment of the present application further provides a more preferred load balancing device.
  • the structure of the device 5 is as shown in FIG. 7, except for the localization rate obtaining device 510 shown in FIG. 5.
  • a load determining device 540' and a target changing device 550' are also included.
  • the load determining means 540' is configured to calculate the predicted partition load number of each server after determining the server with the highest data localization rate of each partition as the target server corresponding to the partition, and determine the number of predicted partition loads according to the predicted partition load number.
  • the predicted partition load number is a number of partitions that will exist on each server after migrating each partition to a target server corresponding to the partition; the target changing device 550 ′ Before the partition is migrated to the target server corresponding to the partition, if the target server corresponding to a certain partition is a high load server, and the partition is a data localization rate of all partitions that will exist on the target server One of the lower N partitions changes the target server corresponding to the partition to the low load server, where N is a positive integer.
  • the localization rate acquisition device 510, the target determination device 520, and the partition migration device 530 are respectively corresponding to the corresponding devices in the embodiment of FIG. The same or substantially the same, for the sake of brevity, will not be repeated here, and is hereby incorporated by reference.
  • the solution predicts that each server will exist after the corresponding partition is migrated according to the determined target server.
  • the number of partitioned loads By predicting the number of partitioned loads and balancing the number of partitioned partitions, the target server of the partition is changed, and the target server determined at this time is unified to perform partition migration. Since the computational cost of analog computing is much less than the actual migration, it can avoid invalid migration, save processing costs, and improve load balancing efficiency while paying less computational cost.
  • the predicted partition load number used in the load determining apparatus 540' for determining the high load server and the low load server is a calculated value obtained from the first determined target server, and is not an actual value directly obtained by each server. .
  • the manner in which the load determining means 540' determines the high load server and the low load server according to the predicted partition load number and when there are a plurality of the low load servers, how the target changing means 550' selects one of them as the target server The load determining device 540 and the target changing device 550 in the load balancing device shown in FIG. 6 described above are similarly adopted.
  • the load determining apparatus 540' is configured to determine that the server whose predicted partition load number is higher than the preset load range upper limit is a high load server, and the predicted partition load number is lower than the preset load range lower limit. Determined to be a low load server.
  • the N used by the target changing means 550' is the difference between the predicted partition load number of the high load server and the average predicted partition load number of all servers.
  • the target changing device 550' is configured to change the target server corresponding to the partition to the partition according to the data localization rate of the partition at the low load server. A low-load server with the highest data localization rate.
  • the partition migration device 530 when performing multiple partition migration, sequentially migrates each partition to a target corresponding to the partition according to a preset interval time. server. Because the related settings of the distributed data storage system will change during the partition migration process, if a large number of migrations are performed in a short period of time, it may be The internal settings of the system change too fast, causing the system to jitter. To avoid this situation, you can set a certain interval (for example, 100ms) when each partition is migrated to prevent jitter caused by partition migration.
  • a certain interval for example, 100ms
  • the technical solution provided by the present application obtains the data localization rate of each partition on each server, and allocates each partition to the server with the highest localization rate according to the data localization rate, and processes the data query.
  • the data localization rate of the server to which each partition is currently assigned is high, most of the data can be acquired on the disk of the local server, so the probability of remote reading of the partition data can be greatly reduced, and the probability is improved.
  • Read performance since the data localization rate of the server to which each partition is currently assigned is high, most of the data can be acquired on the disk of the local server, so the probability of remote reading of the partition data can be greatly reduced, and the probability is improved.
  • Read performance by further adjusting the allocation of partitions by combining the number of partitioned loads, it is possible to optimize the read performance while avoiding the problem that certain servers may have relatively concentrated partition loads in certain situations (such as data hotspots or system expansion).
  • the present application can be implemented in software and/or a combination of software and hardware, for example, using an application specific integrated circuit (ASIC), a general purpose computer, or any other similar hardware device.
  • the software program of the present application can be executed by a processor to implement the steps or functions described above.
  • the software programs (including related data structures) of the present application can be stored in a computer readable recording medium such as a RAM memory, a magnetic or optical drive or a floppy disk and the like.
  • some of the steps or functions of the present application may be implemented in hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.
  • a portion of the present application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide a method and/or technical solution in accordance with the present application.
  • the program instructions for invoking the method of the present application may be stored in a fixed or removable recording medium, and/or transmitted by a data stream in a broadcast or other signal bearing medium, and/or stored in a The working memory of the computer device in which the program instructions are run.
  • an embodiment in accordance with the present application includes a device including a memory for storing computer program instructions and a processor for executing program instructions, wherein when the computer program instructions are executed by the processor, triggering
  • the apparatus operates based on the aforementioned methods and/or technical solutions in accordance with various embodiments of the present application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Environmental & Geological Engineering (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Computer And Data Communications (AREA)
PCT/CN2016/091521 2015-08-06 2016-07-25 负载均衡方法及设备 Ceased WO2017020742A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2018505621A JP6886964B2 (ja) 2015-08-06 2016-07-25 負荷平衡方法及び装置
US15/890,319 US20180167461A1 (en) 2015-08-06 2018-02-06 Method and apparatus for load balancing

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510477498.5 2015-08-06
CN201510477498.5A CN106445677A (zh) 2015-08-06 2015-08-06 负载均衡方法及设备

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/890,319 Continuation US20180167461A1 (en) 2015-08-06 2018-02-06 Method and apparatus for load balancing

Publications (1)

Publication Number Publication Date
WO2017020742A1 true WO2017020742A1 (zh) 2017-02-09

Family

ID=57943807

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/091521 Ceased WO2017020742A1 (zh) 2015-08-06 2016-07-25 负载均衡方法及设备

Country Status (4)

Country Link
US (1) US20180167461A1 (https=)
JP (1) JP6886964B2 (https=)
CN (1) CN106445677A (https=)
WO (1) WO2017020742A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10976963B2 (en) 2019-04-15 2021-04-13 International Business Machines Corporation Probabilistically selecting storage units based on latency or throughput in a dispersed storage network
CN116069594A (zh) * 2023-03-07 2023-05-05 武汉工程大学 一种负载均衡预测方法、装置、系统以及存储介质

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111124657B (zh) * 2018-10-31 2024-03-08 北京金山云网络技术有限公司 资源管理方法、装置、电子设备及存储介质
US11036698B2 (en) * 2018-12-06 2021-06-15 International Business Machines Corporation Non-relational database coprocessor for reading raw data files copied from relational databases
CN110007866B (zh) * 2019-04-11 2020-03-31 苏州浪潮智能科技有限公司 一种存储单元性能优化方法、装置、存储设备及存储介质
US11165860B2 (en) 2019-11-01 2021-11-02 Uber Technologies, Inc. Dynamically computing load balancer subset size in a distributed computing system
CN111666159B (zh) * 2020-06-28 2021-08-27 腾讯科技（深圳）有限公司 负载均衡控制方法和装置、存储介质及电子设备
US11489776B2 (en) * 2020-07-28 2022-11-01 Arista Networks, Inc. Multicore offloading of network processing
US11025710B1 (en) * 2020-10-26 2021-06-01 Verizon Digital Media Services Inc. Systems and methods for dynamic load balancing based on server utilization and content popularity
CN113608870B (zh) * 2021-07-28 2024-07-19 北京金山云网络技术有限公司 消息队列的负载均衡方法及装置、电子设备及存储介质
US11971791B2 (en) 2022-08-12 2024-04-30 Capital One Services, Llc Automated regional failover
CN116303357A (zh) * 2023-03-17 2023-06-23 苏州浪潮智能科技有限公司 一种HBase数据方法、装置、电子设备及可读介质
CN120469786B (zh) * 2025-07-15 2025-11-07 北京科杰科技有限公司 大数据的任务调度方法及系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103226467A (zh) * 2013-05-23 2013-07-31 中国人民解放军国防科学技术大学 数据并行处理方法、系统及负载均衡调度器
CN103268252A (zh) * 2013-05-12 2013-08-28 南京载玄信息科技有限公司 基于分布式存储的虚拟化平台系统及其实现方法
CN103716381A (zh) * 2013-12-12 2014-04-09 华为技术有限公司 一种分布式系统的控制方法，及管理节点
CN104008012A (zh) * 2014-05-30 2014-08-27 长沙麓云信息科技有限公司 一种基于虚拟机动态迁移的高性能MapReduce实现机制

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPWO2005001700A1 (ja) * 2003-06-27 2006-08-10 富士通株式会社 ストレージ容量管理方法、そのサーバ、記録媒体
JP5105894B2 (ja) * 2006-03-14 2012-12-26 キヤノン株式会社 文書検索システム、文書検索装置及びその方法とプログラム、記憶媒体
CN101610287B (zh) * 2009-06-16 2012-03-14 浙江大学 一种应用于分布式海量存储系统的负载均衡方法
US9740762B2 (en) * 2011-04-01 2017-08-22 Mongodb, Inc. System and method for optimizing data migration in a partitioned database
US8595267B2 (en) * 2011-06-27 2013-11-26 Amazon Technologies, Inc. System and method for implementing a scalable data storage service
CN104102523A (zh) * 2013-04-03 2014-10-15 华为技术有限公司 迁移虚拟机的方法和资源调度平台
US9934323B2 (en) * 2013-10-01 2018-04-03 Facebook, Inc. Systems and methods for dynamic mapping for locality and balance

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103268252A (zh) * 2013-05-12 2013-08-28 南京载玄信息科技有限公司 基于分布式存储的虚拟化平台系统及其实现方法
CN103226467A (zh) * 2013-05-23 2013-07-31 中国人民解放军国防科学技术大学 数据并行处理方法、系统及负载均衡调度器
CN103716381A (zh) * 2013-12-12 2014-04-09 华为技术有限公司 一种分布式系统的控制方法，及管理节点
CN104008012A (zh) * 2014-05-30 2014-08-27 长沙麓云信息科技有限公司 一种基于虚拟机动态迁移的高性能MapReduce实现机制

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10976963B2 (en) 2019-04-15 2021-04-13 International Business Machines Corporation Probabilistically selecting storage units based on latency or throughput in a dispersed storage network
US11010096B2 (en) 2019-04-15 2021-05-18 International Business Machines Corporation Probabilistically selecting storage units based on latency or throughput in a dispersed storage network
CN116069594A (zh) * 2023-03-07 2023-05-05 武汉工程大学 一种负载均衡预测方法、装置、系统以及存储介质

Also Published As

Publication number Publication date
JP2018525743A (ja) 2018-09-06
US20180167461A1 (en) 2018-06-14
CN106445677A (zh) 2017-02-22
JP6886964B2 (ja) 2021-06-16

Similar Documents

Publication Publication Date Title
WO2017020742A1 (zh) 负载均衡方法及设备
US10356150B1 (en) Automated repartitioning of streaming data
US10489343B2 (en) Cluster file system comprising data mover modules having associated quota manager for managing back-end user quotas
JP7467593B2 (ja) リソース割振り方法、記憶デバイス、および記憶システム
CN103473365B (zh) 一种基于hdfs的文件存储方法、装置及分布式文件系统
US20130332608A1 (en) Load balancing for distributed key-value store
US10394782B2 (en) Chord distributed hash table-based map-reduce system and method
WO2017028696A1 (zh) 分布式存储系统的负载监控方法及设备
US10616134B1 (en) Prioritizing resource hosts for resource placement
CN106534308B (zh) 一种分布式存储系统中解决数据块访问热点的方法及装置
CN115421924B (zh) 一种内存分配方法、装置及设备
CN104702691B (zh) 分布式负载均衡方法和装置
US10007673B1 (en) Cluster file system comprising data mover module arranged between front-end and back-end file systems
WO2021057108A1 (zh) 一种读数据方法、写数据方法及服务器
US10761726B2 (en) Resource fairness control in distributed storage systems using congestion data
US9465549B1 (en) Dynamic allocation of a high-speed memory pool between a cluster file system and a burst buffer appliance
WO2019196595A1 (zh) 管理应用程序的方法与装置
CN120085810B (zh) 数据存储方法、电子设备、存储介质和程序产品
WO2016065198A1 (en) High performance hadoop with new generation instances
US9996573B1 (en) Distributed system capacity dial-up
CN114253456A (zh) 一种缓存负载均衡方法和装置
US10721181B1 (en) Network locality-based throttling for automated resource migration
Huang et al. Resource provisioning with QoS in cloud storage
WO2016044980A1 (zh) 线程迁移方法、装置和系统
US12530122B2 (en) Methods and systems for data processing

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16832230

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2018505621

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16832230

Country of ref document: EP

Kind code of ref document: A1