WO2022217688A1 - B2b 的云端分销平台系统的数据处理方法 - Google Patents
B2b 的云端分销平台系统的数据处理方法 Download PDFInfo
- Publication number
- WO2022217688A1 WO2022217688A1 PCT/CN2021/094008 CN2021094008W WO2022217688A1 WO 2022217688 A1 WO2022217688 A1 WO 2022217688A1 CN 2021094008 W CN2021094008 W CN 2021094008W WO 2022217688 A1 WO2022217688 A1 WO 2022217688A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- cluster
- server nodes
- server
- resource utilization
- migrated
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 17
- 238000000034 method Methods 0.000 claims abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 15
- 238000013508 migration Methods 0.000 claims description 36
- 230000005012 migration Effects 0.000 claims description 36
- 230000008569 process Effects 0.000 claims description 10
- 230000003068 static effect Effects 0.000 claims description 8
- 238000012935 Averaging Methods 0.000 claims description 3
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000001174 ascending effect Effects 0.000 description 2
- 239000007795 chemical reaction product Substances 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011056 performance test Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1029—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers using data related to the state of servers by a load balancer
Definitions
- the invention relates to the technical field of data processing, in particular to a data processing method of a B2B cloud distribution platform system.
- the tourism industry has obvious characteristics of low and peak seasons.
- the processing reserves of servers need to be adjusted according to the needs of low and peak seasons. Otherwise, the off-season equipment will not be fully utilized and resources will be wasted; the peak season equipment cannot meet the corresponding needs, which will affect the user experience.
- the present invention provides a data processing method of a B2B cloud distribution platform system, comprising the following steps:
- Step S1 Divide all server nodes into a main cluster, a sub-cluster and an idle cluster pool.
- the main cluster, the sub-cluster and the idle cluster pool all include multiple server nodes, wherein the server nodes in the main cluster are used to process core services,
- the server nodes in the secondary cluster are used to process basic data services with low real-time requirements and large processing capacity, and the idle cluster pool is used to accommodate idle server nodes;
- Step S2 The distribution platform system periodically obtains the average resource utilization of multiple server nodes of the main cluster and the sub-cluster in a polling manner;
- Step S3 if the average resource utilization of the server nodes in the main cluster is greater than or equal to the minimum threshold, migrate the server nodes in the idle cluster pool to the main cluster until the average resource utilization of the server nodes in the main cluster is less than the minimum threshold, Then skip to step S4; if the average resource utilization of the server nodes in the main cluster is less than the minimum threshold, skip directly to step S4;
- Step S4 If the average resource utilization of the server nodes in the secondary cluster is less than the minimum threshold, the server nodes with high resource utilization in the secondary cluster are migrated out to the idle cluster pool in descending order until the servers in the secondary cluster are moved out.
- the average resource utilization of the nodes is greater than or equal to the minimum threshold; if the average resource utilization of server nodes in the secondary cluster is between the minimum and maximum thresholds, the migration will not be performed; if the average resource utilization of server nodes in the secondary cluster is greater than or equal to If it is equal to the highest threshold, the server nodes in the idle cluster pool will be migrated to the secondary cluster until the average resource utilization of server nodes in the secondary cluster is less than the highest threshold.
- step S3 if the average resource utilization rate of the server nodes in the main cluster is greater than or equal to the minimum threshold, after the server nodes in the idle cluster pool are migrated to the main cluster, the services in the original server nodes of the main cluster are transferred to the main cluster.
- the directory is migrated to the in-migrated server node. During migration, the service directory in the server node with the highest resource utilization is migrated preferentially according to the order of resource utilization from large to small.
- step S3 when migrating the service catalog, it is judged whether the size of the service catalog to be migrated is greater than the catalog threshold, and if it is greater than the catalog threshold, the migration catalog is disassembled into several sub-service catalogs less than or equal to the catalog threshold, And complete the migration of sub-service directories in turn.
- step S4 if the average resource utilization rate of the server nodes in the secondary cluster is less than the minimum threshold, after the server nodes in the secondary cluster are migrated out, the service directory on the migrated server nodes is migrated into the secondary cluster In the remaining server nodes, and according to the order from small to large, the server nodes with the smallest resource utilization are preferentially migrated;
- the service directory in the original server nodes of the secondary cluster will be transferred to the migrated server nodes.
- the service directory in the server node with the highest resource utilization is first migrated according to the descending order of resource utilization.
- step S4 when migrating the service catalog, it is judged whether the size of the service catalog to be migrated is greater than the catalog threshold, and if it is greater than the catalog threshold, the migration catalog is disassembled into several sub-service catalogs less than or equal to the catalog threshold, And complete the migration of sub-service directories in turn.
- the average resource utilization is obtained by taking the average of the resource utilizations of all server nodes in the main cluster or the sub-cluster, and the resource utilization of each server node is determined by the CPU utilization, memory utilization, and network sending and receiving rate.
- the rate is obtained by taking the weighted average.
- step S3 if the average resource utilization rate of the server nodes in the main cluster is greater than or equal to the minimum threshold, after the server nodes in the idle cluster pool are migrated to the main cluster, the following steps are used to realize each server in the main cluster.
- Step S31 Obtaining the static load parameters and dynamic load parameters of each server node, and weighting and averaging them to obtain comprehensive load parameters;
- Step S32 Take the average of the comprehensive load parameters of all the server nodes to obtain the average comprehensive load parameter of the main cluster, the server node whose comprehensive load parameter is lower than the average comprehensive load parameter is the target server node, and the server whose comprehensive load parameter is higher than the average comprehensive load parameter is the target server node.
- the node is the source server node;
- Step S33 arranging the source server nodes in descending order of the comprehensive load parameters, and arranging the target server nodes in descending order of the comprehensive load parameters, so that the source server nodes correspond to the target server nodes in corresponding order;
- Step S34 In descending order, preferentially migrate the service directory in the source server node with the largest comprehensive load parameter to the target server node with the smallest comprehensive load parameter until all server nodes achieve load balance.
- step S34 when migrating the service catalog, it is judged whether the size of the service catalog to be migrated is greater than the catalog threshold, and if it is greater than the catalog threshold, the migration catalog is disassembled into several sub-service catalogs less than or equal to the catalog threshold, And complete the migration of sub-service directories in turn.
- step S4 if the average resource utilization rate of the server nodes in the secondary cluster is greater than or equal to the highest threshold, after the server nodes in the idle cluster pool are migrated to the secondary cluster; If the average utilization rate is less than the minimum threshold, after the server nodes in the secondary cluster are migrated to the idle cluster pool and the service directory on the migrated server nodes is migrated back to the secondary cluster, the load of each server node in the secondary cluster is realized by the following steps balanced:
- Step S41 acquiring the static load parameters and dynamic load parameters of each server node, and weighting them to obtain comprehensive load parameters;
- Step S42 Take the average of the comprehensive load parameters of all server nodes to obtain the average comprehensive load parameter of the sub-cluster, the server node whose comprehensive load parameter is lower than the average comprehensive load parameter is the target server node, and the server whose comprehensive load parameter is higher than the average comprehensive load parameter is the target server node.
- the node is the source server node;
- Step S43 arranging the source server nodes according to the comprehensive load parameters in descending order, and arranging the target server nodes according to the comprehensive load parameters from small to large, so that the source server nodes and the target server nodes correspond in a corresponding order;
- Step S44 In descending order, preferentially migrate the service directory in the source server node with the largest comprehensive load parameter to the target server node with the smallest comprehensive load parameter until all server nodes achieve load balance.
- step S44 when migrating the service catalog, it is judged whether the size of the service catalog to be migrated is greater than the catalog threshold, and if it is greater than the catalog threshold, the migration catalog is disassembled into several sub-service catalogs less than or equal to the catalog threshold, And complete the migration of sub-service directories in turn.
- the data processing method of the B2B cloud distribution platform system provided by the present invention is based on the problem that the existing cloud distribution platform system has a large amount of data processing and high requirements for server resources, and proposes a method based on load balancing, which can make the server resources Get the most full use and improve the data processing efficiency of the platform system.
- the data processing method of the B2B cloud distribution platform system provided by the present invention is based on the problem that the existing cloud distribution platform system has a large amount of data processing and high requirements for server resources, and proposes a method based on load balancing, so that the server resources can be maximized. Make full use of it to improve the data processing efficiency of the platform system.
- any distribution platform system When processing business data, any distribution platform system has priorities. Some businesses involve core business and require the server to have sufficient resource space and processing efficiency to ensure that the core business is processed in a timely and effective manner, while other businesses involve basic service data. The processing capacity is large but the real-time requirements are low, and these services have relatively low requirements on the server.
- the present invention divides all server nodes under the platform system into three clusters: the main cluster, where the server nodes are used to process core services; the sub-cluster, where the server nodes are used to process basic services; Idle cluster pool, which is used to accommodate idle server nodes to meet the needs of the platform system between off-season and peak seasons: in off-season, the sub-cluster migrates some server nodes to the idle cluster pool to save server node consumption and improve the overall equipment During the peak season, the idle cluster pool will migrate some server nodes to the sub-cluster to ensure the work efficiency of the overall equipment.
- the minimum threshold and the maximum threshold corresponding to the resource utilization rate of the server node work can be preset according to the actual performance of the equipment, the work needs of the platform system, etc. Since the business handled by the main cluster is more important, the main The average resource utilization of server nodes in the cluster must meet the requirement of being less than the minimum threshold. That is, when the average resource utilization of server nodes in the main cluster is greater than or equal to the minimum threshold, the server nodes in the idle cluster pool will be transferred to the main cluster. Migrate until the average resource utilization of the server nodes in the main cluster is less than the minimum threshold, and the server nodes in the main cluster will not be migrated out under any circumstances.
- the average resource utilization of the server nodes in the secondary cluster does not need to meet the requirement of being less than the minimum threshold: if the average resource utilization of the server nodes in the secondary cluster is less than the minimum threshold, the From large to small, the server nodes with high resource utilization in the sub-cluster are moved out of the idle cluster pool in turn, until the average resource utilization of the server nodes in the sub-cluster is greater than or equal to the minimum threshold; if the average resource utilization of the server nodes in the sub-cluster If the resource utilization is between the lowest threshold and the highest threshold, no migration will be performed; if the average resource utilization of server nodes in the secondary cluster is greater than or equal to the highest threshold, the server nodes in the idle cluster pool will be migrated to the secondary cluster until The average resource utilization of server nodes in the secondary cluster is less than the maximum threshold.
- the present invention provides different migration schemes for the main cluster and the sub-cluster by setting the main cluster, the sub-cluster and the idle cluster pool, and based on the setting of the minimum threshold and the highest threshold, so that the present invention can target different business needs.
- the average resource utilization rate is obtained by averaging the resource utilization rates of all server nodes in the main cluster or the sub-cluster, and the resource utilization rate of each server node is determined by the CPU utilization rate, memory utilization rate, and network sending and receiving rate utilization Take the weighted average to obtain.
- the average resource utilization of server nodes in the primary cluster is greater than or equal to the minimum threshold, or the average resource utilization of server nodes in the secondary cluster is greater than or equal to the maximum threshold, it is necessary to transfer the server nodes in the idle cluster pool to the primary cluster or secondary cluster Migration, after migration, since there is no service directory on the migrated server node, it is necessary to migrate the files on the original server node in the primary cluster or the secondary cluster to the migrated server node, or, the primary cluster or secondary cluster from the original Among the existing server nodes, the files on the server nodes with higher resource utilization are migrated to the server nodes with smaller resource utilization to achieve load balancing of server nodes in each cluster; for the convenience of description, the following will migrate out the service.
- the server nodes of the directory are collectively referred to as source server nodes, also known as migration nodes, and the server nodes migrated into the service directory are collectively referred to as target server nodes, also known as migration nodes.
- the outgoing node or ingoing node is determined by the resource utilization ratio and the resource utilization of each server node: the server node whose resource utilization rate is greater than the average resource utilization rate is the outgoing node, and the server node whose resource utilization rate is less than the average resource utilization rate is the immigrating node. .
- the present invention When performing load balancing of load server node resources, the present invention provides the following two detailed load balancing solutions:
- the difference between the resource utilization and the average resource utilization determines the size of the migrated service catalog;
- the incoming nodes are arranged in ascending order of resource utilization, and the nodes with low resource utilization are preferentially moved in.
- the difference between the average resource utilization and the resource utilization determines the size of the migrated service catalog.
- the second node can take over all the service catalogs migrated out by the first node and there is remaining space, then the next step is to move the service catalogs from the migration node with the second largest resource utilization (referred to as the third node) into the second node.
- the service directory will be migrated to the ingress node with the next smallest resource utilization (referred to as the fourth node).
- the remaining service catalogs of the first node are migrated to the fourth node, and so on until all server nodes in each cluster complete the load balancing.
- the ingressing nodes After determining the average resource utilization, find the outgoing nodes whose resource utilization is greater than the average resource utilization, and the ingressing nodes whose resource utilization is less than the average resource utilization. It should be noted that the ingressing nodes here do not include newly added nodes.
- the server nodes migrated in from the idle resource pool, and the server nodes migrated from the idle resource pool are referred to as node zero, because there is no service directory above; On the zero node, until the resource utilization of each outgoing node reaches the level of the average resource utilization; migrate the service directory on each zero node to each ingress node in sequence, so that the ingress node and each zero node The resource utilization of the node number reaches the level of the average resource utilization.
- a detailed load balancing method is as follows: You can refer to the above scheme 1 or scheme 2, first perform load balancing on the remaining server nodes in the secondary cluster, and then move the service directory on the server nodes that have been migrated out of the secondary cluster. It is evenly distributed among the existing server nodes.
- the resource utilization corresponding to the server nodes that have been migrated out of the sub-cluster can be sequentially moved into the remaining server nodes of the sub-cluster in descending order.
- the remaining server nodes in the secondary cluster receive service catalogs in ascending order of resource utilization, that is, the migrated server node with the highest resource utilization moves the service catalog into the secondary cluster with the smallest remaining resource utilization.
- the server node, the migrated server node with the second largest resource utilization rate will migrate the service directory to the remaining server node with the next smallest resource utilization rate in the secondary cluster, and so on, until all the service directories in the migrated node are returned to the secondary cluster
- the load balancing of the entire sub-cluster is realized according to the above scheme 1 or scheme 2.
- a directory threshold is set. When migrating a service directory, it is judged whether the size of the service directory to be migrated is larger than the directory threshold. Sub-service directory, and complete the migration of sub-service directory in turn. In this way, the size of the service catalog data to be migrated each time is controlled, and the waiting processes are also controlled, which can reduce the waiting time of users.
- the resource utilization rate proposed by the present invention only takes into account the static load parameters, the acquisition and calculation of the static load parameters are relatively easy, and the processing is convenient when the server nodes between the main cluster, the sub-cluster and the idle cluster pool are mutually shared; However, after the server node migration is completed, only the static load parameters can be considered, and only preliminary cluster load balancing can be achieved. In order to ensure more stable load balancing of each cluster, the present invention introduces the concept of dynamic load parameters.
- the comprehensive load parameters obtained after the weighted average of the static load parameters and the dynamic load parameters can replace the resource utilization rate, so as to achieve the final load balance within each cluster.
- the dynamic load parameter is obtained by performing a benchmark performance test on each server node: the dynamic load parameter of the server node with the longest test time is set to 1, and the ratio of the test time of the remaining server nodes to the test time of the longest server node In this way, the change of the processing capability of the server node in the actual working process can be considered.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- General Engineering & Computer Science (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明涉及一种数据处理方法，包括：将服务器节点分为主集群、副集群以及闲置集群池；定期获取主集群以及副集群的多个服务器节点的平均资源利用率；若主集群中服务器节点的平均资源利用率大于或等于最低阈值，则将闲置集群池中的服务器节点向主集群中迁移；若副集群中服务器节点的平均资源利用率小于最低阈值，则将副集群中的服务器节点迁出至闲置集群池内；若介于最低阈值及最高阈值之间，则不迁移；若大于或等于最高阈值，则将闲置集群池中的服务器节点向副集群中迁移。本发明基于已有的云端分销平台系统数据处理量大，对服务器资源要求高的问题，提出了一种基于负载均衡的方法，能够使服务器资源得到最充分的利用，提高平台系统的数据处理效率。
Description
本发明涉及数据处理技术领域，具体涉及一种B2B的云端分销平台系统的数据处理方法。
信息化技术的普及推动了越来越多的云端分销平台系统的涌现，特别是针对旅游产业，大多旅游经营者都采取了线上线下相结合的方式。用户可在接受线下服务之前采取线上下单的方式实现交易。
然而，旅游产业的特点给云端分销平台系统带来了很多考验：
1、旅游行业具有明显的淡旺季特点，服务器的处理储备需要针对淡旺季的需求实现不同的调整，否则淡季设备得不到充分利用，浪费资源；旺季设备无法满足对应需求，影响用户使用体验。
2、用户在前端产品进行订单操作时，后端产品服务器负载过大会产生数据卡顿或数据丢失的现象。
因此，有必要提供一种数据处理方法，给予云端分销平台完整的负载方案。
为解决已有技术存在的不足，本发明提供了一种B2B的云端分销平台系统的数据处理方法，包括如下步骤：
步骤S1：将所有的服务器节点分为主集群、副集群以及闲置集群池，主集群、副集群以及闲置集群池内均包括多个服务器节点，其中，主集群中的服务器节点用于处理核心业务，副集群中的服务器节点用于处理实时性要求低且处理量大的基础数据业务，闲置集群池用于容纳闲置服务器节点；
步骤S2：分销平台系统以轮询方式定期获取主集群以及副集群的多个服务器节点的平均资源利用率；
步骤S3：若主集群中服务器节点的平均资源利用率大于或等于最低阈值，则将闲置集群池中的服务器节点向主集群中迁移，直至主集群中服务器节点的平均资源利用率小于最低阈值，然后跳至步骤S4；若主集群中服务器节点的平均资源利用率小于最低阈值，则直接跳至步骤S4；
步骤S4：若副集群中服务器节点的平均资源利用率小于最低阈值，则以从大到小的顺序将副集群中资源利用率高的服务器节点依次迁出至闲置集群池内，直至副集群中服务器节点的平均资源利用率大于或等于最低阈值；若副集群中服务器节点的平均资源利用率介于最低阈值及最高阈值之间，则不迁移；若副集群中服务器节点的平均资源利用率大于或等于最高阈值，则将闲置集群池中的服务器节点向副集群中迁移，直至副集群中服务器节点的平均资源利用率小于最高阈值。
其中，所述步骤S3中，若主集群中服务器节点的平均资源利用率大于或等于最低阈值，在将闲置集群池中的服务器节点向主集群迁移完毕后，将主集群原先服务器节点中的服务目录向迁入的服务器节点中迁移，迁移时，按照资源利用率由大到小的顺序，优先迁移资源利用率最大的服务器节点中的服务目录。
其中，所述步骤S3中，在迁移服务目录时，判断欲迁移服务目录的大小是否大于目录阈值，若大于目录阈值，则将迁移目录拆解为若干个小于或等于目录阈值的子服务目录，并依次完成子服务目录的迁移。
其中，所述步骤S4中，若副集群中服务器节点的平均资源利用率小于最低阈值，则在将副集群中服务器节点迁出完毕后，将迁出的服务器节点上的服务目录迁入副集群中剩余的服务器节点上，且按照由小到大的顺序，优先迁入资源利用率最小的服务器节点上；
若副集群中服务器节点的平均资源利用率大于或等于最高阈值，则在将闲置集群池中的服务器节点向副集群迁移完毕后，将副集群原先服务器节点中的服务目录向迁入的服务器节点中迁移，迁移时，按照资源利用率由大到小的顺序，优先迁移资源利用率最大的服务器节点中的服务目录。
其中，所述步骤S4中，在迁移服务目录时，判断欲迁移服务目录的大小是否大于目录阈值，若大于目录阈值，则将迁移目录拆解为若干个小于或等于目录阈值的子服务目录，并依次完成子服务目录的迁移。
其中，所述平均资源利用率，由主集群或副集群中所有的服务器节点的资源利用率取平均值所得，各服务器节点的资源利用率，由CPU使用率、内存使用率、网络收发速率使用率取加权平均值获得。
其中，所述步骤S3中，若主集群中服务器节点的平均资源利用率大于或等于最低阈值，在将闲置集群池中的服务器节点向主集群迁移完毕后，通过如下步骤实现主集群中各服务器节点的负载均衡：
步骤S31：获取各服务器节点的静态负载参数及动态负载参数，并对其加权平均得综合负载参数；
步骤S32：将所有服务器节点的综合负载参数取平均值得主集群的平均综合负载参数，综合负载参数低于平均综合负载参数的服务器节点为目标服务器节点，综合负载参数高于平均综合负载参数的服务器节点为源服务器节点；
步骤S33：按照综合负载参数由大到小的顺序排列源服务器节点，按照综合负载参数由小到大的顺序排列目标服务器节点，使源服务器节点与目标服务器节点以相应顺序对应；
步骤S34：按照由大到小的顺序，优先将综合负载参数最大的源服务器节点中的服务目录向综合负载参数最小的目标服务器节点中迁移，直至所有的服务器节点实现负载均衡。
其中，所述步骤S34中，在迁移服务目录时，判断欲迁移服务目录的大小是否大于目录阈值，若大于目录阈值，则将迁移目录拆解为若干个小于或等于目录阈值的子服务目录，并依次完成子服务目录的迁移。
其中，所述步骤S4中，若副集群中服务器节点的平均资源利用率大于或等于最高阈值，在将闲置集群池中的服务器节点向副集群迁移完毕后；或者，若副集群中服务器节点的平均利用率小于最低阈值，在将副集群中的服务器节点向闲置集群池迁移完毕并将迁移出的服务器节点上的服务目录迁移回副集群后，通过如下步骤实现副集群中各服务器节点的负载均衡：
步骤S41：获取各服务器节点的静态负载参数及动态负载参数，并对其加权平均得综合负载参数；
步骤S42：将所有服务器节点的综合负载参数取平均值得副集群的平均综合负载参数，综合负载参数低于平均综合负载参数的服务器节点为目标服务器节点，综合负载参数高于平均综合负载参数的服务器节点为源服务器节点；
步骤S43：按照综合负载参数由大到小的顺序排列源服务器节点，按照综合负载参数由小到大的顺序排列目标服务器节点，使源服务器节点与目标服务器节点以相应顺序对应；
步骤S44：按照由大到小的顺序，优先将综合负载参数最大的源服务器节点中的服务目录向综合负载参数最小的目标服务器节点中迁移，直至所有的服务器节点实现负载均衡。
其中，所述步骤S44中，在迁移服务目录时，判断欲迁移服务目录的大小是否大于目录阈值，若大于目录阈值，则将迁移目录拆解为若干个小于或等于目录阈值的子服务目录，并依次完成子服务目录的迁移。
本发明提供的B2B的云端分销平台系统的数据处理方法，基于已有的云端分销平台系统数据处理量大，对服务器资源要求高的问题，提出了一种基于负载均衡的方法，能够使服务器资源得到最充分的利用，提高平台系统的数据处理效率。
为了对本发明的技术方案及有益效果有更进一步的了解，下面详细说明本发明的技术方案及其产生的有益效果。
本发明提供的B2B的云端分销平台系统的数据处理方法，基于已有的云端分销平台系统数据处理量大，对服务器资源要求高的问题，提出一种基于负载均衡的方法，使服务器资源得到最充分的利用，提高平台系统的数据处理效率。
一、服务器节点分配原则
任何分销平台系统在处理业务数据时，均有轻重缓急之分，有些业务涉及核心业务，需要服务器具有充足的资源空间和处理效率，保证核心业务得到及时、有效处理，而另一些业务涉及基础服务数据的处理，处理量大但实时性要求低，这些业务对服务器的要求相对较低。因此，本发明基于此特点，将平台系统下的所有服务器节点划分为三个集群：主集群，其内的服务器节点用于处理核心业务；副集群，其内的服务器节点用于处理基础业务；闲置集群池，其用于容置闲置的服务器节点，以适应平台系统淡季及旺季交替的需求：淡季时，副集群将部分服务器节点迁移至闲置集群池，以节省服务器节点的消耗，提高整体设备的使用寿命，旺季时，闲置集群池将部分服务器节点迁移至副集群中，以保证整体设备的工作效率。
本发明在实际使用阶段，可根据设备的实际性能、平台系统的工作需要等，预先设定服务器节点工作的资源利用率所对应的最低阈值及最高阈值，由于主集群所处理业务较重要，主集群中服务器节点的平均资源利用率需满足恒定小于最低阈值的需要，也即，当主集群中服务器节点的平均资源利用率大于或等于最低阈值时，将闲置集群池中的服务器节点向主集群中迁移，直至主集群中服务器节点的平均资源利用率小于最低阈值，主集群中的服务器节点无论在何种情况下均不向外迁移。
对应的，由于副集群所处理业务相对次要，副集群中服务器节点的平均资源利用率不需要满足恒定小于最低阈值的需要：若副集群中服务器节点的平均资源利用率小于最低阈值，则以从大到小的顺序将副集群中资源利用率高的服务器节点依次迁出至闲置集群池内，直至副集群中服务器节点的平均资源利用率大于或等于最低阈值；若副集群中服务器节点的平均资源利用率介于最低阈值及最高阈值之间，则不迁移；若副集群中服务器节点的平均资源利用率大于或等于最高阈值，则将闲置集群池中的服务器节点向副集群中迁移，直至副集群中服务器节点的平均资源利用率小于最高阈值。
并且，在对整个平台系统中的服务器节点进行负载均衡时，优先满足主集群的业务需要，也即，在主集群实现了服务器节点的迁移需要后，再判断副集群的资源利用率。
综上，本发明通过设置主集群、副集群以及闲置集群池，并基于最低阈值和最高阈值的设置针对性地给予主集群和副集群不同的迁移方案，使得本发明能够针对不同的业务需要针对性地给予资源分配，且能够满足淡旺季的实际需求。
本发明中，平均资源利用率由主集群或副集群中所有的服务器节点的资源利用率取平均值所得，各服务器节点的资源利用率，由CPU使用率、内存使用率、网络收发速率使用率取加权平均值获得。
二、负载均衡方法
1、闲置集群池向主集群或副集群迁移
在主集群中服务器节点的平均资源利用率大于或等于最低阈值，或副集群中服务器节点的平均资源利用率大于或等于最高阈值时，需要将闲置集群池中的服务器节点向主集群或副集群迁移，迁移后，由于迁入的服务器节点上不存在服务目录，需要将主集群或副集群中原先的服务器节点上的文件向迁入服务器节点上迁移，亦或者，将主集群或副集群原先所存在的服务器节点中，资源利用率较大的服务器节点上的文件向资源利用率较小的服务器节点中迁移，以实现各集群内服务器节点的负载均衡；为描述方便，下文将迁出服务目录的服务器节点统称为源服务器节点，又称迁出节点，将迁入服务目录的服务器节点统称为目标服务器节点，又称迁入节点；具体实施时，可通过比较当前集群中的平均资源利用率与各服务器节点的资源利用率大小确定迁出节点或迁入节点：资源利用率大于平均资源利用率的服务器节点为迁出节点，资源利用率小于平均资源利用率的服务器节点为迁入节点。
在进行负载服务器节点资源的负载均衡时，本发明提供了如下两个详细的负载均衡方案：
方案一：
将所有迁出节点按照资源利用率由大至小的顺序排列，优先迁出资源利用率大的节点，资源利用率与平均资源利用率的差额决定了迁出的服务目录的大小；将所有迁入节点按照资源利用率由小至大的顺序排列，优先迁入资源利用率小的节点，平均资源利用率与资源利用率的差额决定了迁入的服务目录的大小。
将资源利用率最大的迁出节点（简称第1节点）中服务目录向资源利用率最小的迁入节点（简称第2节点）迁移，直至迁出节点资源利用率达到平均资源利用率标准，若第2节点能够承接第1节点所迁出的所有服务目录且有剩余空间，则下一步将资源利用率次大的迁出节点（简称第3节点）中服务目录接着迁入该第2节点，直至第2节点达到平均资源利用率，再向资源利用率次小的迁入节点（简称第4节点）迁移服务目录；若第2节点无法承接第1节点所迁出的所有服务目录，则在第2节点达到平均资源利用率后，将第1节点剩余的服务目录迁移至第4节点，以此逻辑类推，直至各集群内所有服务器节点完成负载均衡。
方案二：
确定平均资源利用率后，找出资源利用率大于平均资源利用率的迁出节点，以及资源利用率小于平均资源利用率的迁入节点，需要注意的是，这里的迁入节点不包括新增的由闲置资源池中迁入的服务器节点，由闲置资源池迁入的服务器节点简称零号节点，因为上面不存在服务目录；将各迁出节点中的服务目录，依次、轮流迁出至各零号节点上，直至各迁出节点中的资源利用率达到平均资源利用率的水平；将各零号节点上的服务目录再顺次迁入各迁入节点上，使迁入节点及各零号节点的资源利用率达到平均资源利用率的水平。
2、副集群向闲置集群池迁移
此种迁移操作下，将服务器节点迁出后，需要将该服务器节点上的服务目录迁回原副集群中。
一个详细的负载均衡办法如下：可参照上文的方案一或者方案二，首先对副集群现有剩下的服务器节点进行负载均衡，然后，将已迁出该副集群的服务器节点上的服务目录平均分配到各现有的服务器节点中。
亦或者，也可按照由大到小的顺序，将迁出副集群的服务器节点对应的资源利用率按照由大到小的顺序，顺次迁入副集群剩下的服务器节点中，在迁入时，副集群剩余服务器节点按照资源利用率由小到大的顺序接收服务目录，也即，资源利用率最大的被迁出的服务器节点将服务目录迁入副集群中剩余的资源利用率最小的服务器节点，资源利用率次大的被迁出的服务器节点将服务目录迁入副集群中剩余的资源利用率次小的服务器节点，依次类推，直至迁出节点中的服务目录全部回传至副集群中个，再按照上文方案一或方案二实现整个副集群的负载均衡。
由于服务目录在迁移过程中，用户无法对其访问，相应的，也无法实现其程序上的功能，服务目录的迁移不可避免会带来系统等待时间，为了将此时间尽可能降低，提高用户的体验，本发明中，设定一个目录阈值，在迁移服务目录时，判断欲迁移服务目录的大小是否大于目录阈值，若大于目录阈值，则将迁移目录拆解为若干个小于或等于目录阈值的子服务目录，并依次完成子服务目录的迁移。这样，每次迁移的服务目录数据大小得到了控制，处于等待中的进程也得到控制，可减少用户等待时间。
另外，本发明所提出的资源利用率，仅考虑到了静态负载参数，静态负载参数的获取和计算较容易，在实现主集群、副集群及闲置集群池之间的服务器节点互享时，处理方便；然而，在服务器节点迁移完毕后，仅仅考虑静态负载参数，仅能够实现初步的集群负载均衡，为了保证各集群更稳定的负载均衡，本发明引入了动态负载参数的概念。
也即，在上文的方案一及方案二所对定的负载方案中，可以静态负载参数及动态负载参数加权平均后所得的综合负载参数代替资源利用率，以实现各集群内部最终的负载均衡。
本发明中，动态负载参数通过对每个服务器节点进行基准性能测试得到：将测试时间最长的服务器节点的动态负载参数设为1，其余服务器节点的测试时间与最长服务器节点测试时间的比值为其相应的动态负载参数，如此，即可考虑到服务器节点在实际工作过程中的处理能力变化情况。
本发明的有益效果如下：
1、通过主集群、副集群的设置，能够针对不同的业务需要针对性得给予资源分配，可优先满足核心业务的需要，提升用户体验。
2、通过闲置集群池的设置，能够满足系统淡季及旺季需要。
3、通过将服务目录拆分为子服务目录迁移的设置，能够降低数据迁移过程中用户访问的等待时间，提升用户使用体验。
4、通过平均资源利用率及综合负载参数的结合设置，可针对集群之间负载均衡及集群内负载均衡给予不同的负载策略，兼具效率及稳定性。
虽然本发明已利用上述较佳实施例进行说明，然其并非用以限定本发明的保护范围，任何本领域技术人员在不脱离本发明的精神和范围之内，相对上述实施例进行各种变动与修改仍属本发明所保护的范围，因此本发明的保护范围以权利要求书所界定的为准。
Claims (10)
- 一种B2B的云端分销平台系统的数据处理方法，其特征在于，包括如下步骤：步骤S1：将所有的服务器节点分为主集群、副集群以及闲置集群池，主集群、副集群以及闲置集群池内均包括多个服务器节点，其中，主集群中的服务器节点用于处理核心业务，副集群中的服务器节点用于处理实时性要求低且处理量大的基础数据业务，闲置集群池用于容纳闲置服务器节点；步骤S2：分销平台系统以轮询方式定期获取主集群以及副集群的多个服务器节点的平均资源利用率；步骤S3：若主集群中服务器节点的平均资源利用率大于或等于最低阈值，则将闲置集群池中的服务器节点向主集群中迁移，直至主集群中服务器节点的平均资源利用率小于最低阈值，然后跳至步骤S4；若主集群中服务器节点的平均资源利用率小于最低阈值，则直接跳至步骤S4；步骤S4：若副集群中服务器节点的平均资源利用率小于最低阈值，则以从大到小的顺序将副集群中资源利用率高的服务器节点依次迁出至闲置集群池内，直至副集群中服务器节点的平均资源利用率大于或等于最低阈值；若副集群中服务器节点的平均资源利用率介于最低阈值及最高阈值之间，则不迁移；若副集群中服务器节点的平均资源利用率大于或等于最高阈值，则将闲置集群池中的服务器节点向副集群中迁移，直至副集群中服务器节点的平均资源利用率小于最高阈值。
- 如权利要求1所述的B2B的云端分销平台系统的数据处理方法，其特征在于，所述步骤S3中，若主集群中服务器节点的平均资源利用率大于或等于最低阈值，在将闲置集群池中的服务器节点向主集群迁移完毕后，将主集群原先服务器节点中的服务目录向迁入的服务器节点中迁移，迁移时，按照资源利用率由大到小的顺序，优先迁移资源利用率最大的服务器节点中的服务目录。
- 如权利要求1所述的B2B的云端分销平台系统的数据处理方法，其特征在于，所述步骤S3中，在迁移服务目录时，判断欲迁移服务目录的大小是否大于目录阈值，若大于目录阈值，则将迁移目录拆解为若干个小于或等于目录阈值的子服务目录，并依次完成子服务目录的迁移。
- 如权利要求1所述的B2B的云端分销平台系统的数据处理方法，其特征在于，所述步骤S4中，若副集群中服务器节点的平均资源利用率小于最低阈值，则在将副集群中服务器节点迁出完毕后，将迁出的服务器节点上的服务目录迁入副集群中剩余的服务器节点上，且按照由小到大的顺序，优先迁入资源利用率最小的服务器节点上；若副集群中服务器节点的平均资源利用率大于或等于最高阈值，则在将闲置集群池中的服务器节点向副集群迁移完毕后，将副集群原先服务器节点中的服务目录向迁入的服务器节点中迁移，迁移时，按照资源利用率由大到小的顺序，优先迁移资源利用率最大的服务器节点中的服务目录。
- 如权利要求4所述的B2B的云端分销平台系统的数据处理方法，其特征在于，所述步骤S4中，在迁移服务目录时，判断欲迁移服务目录的大小是否大于目录阈值，若大于目录阈值，则将迁移目录拆解为若干个小于或等于目录阈值的子服务目录，并依次完成子服务目录的迁移。
- 如权利要求1-5中任一项所述的B2B的云端分销平台系统的数据处理方法，其特征在于，所述平均资源利用率，由主集群或副集群中所有的服务器节点的资源利用率取平均值所得，各服务器节点的资源利用率，由CPU使用率、内存使用率、网络收发速率使用率取加权平均值获得。
- 如权利要求1所述的B2B的云端分销平台系统的数据处理方法，其特征在于，所述步骤S3中，若主集群中服务器节点的平均资源利用率大于或等于最低阈值，在将闲置集群池中的服务器节点向主集群迁移完毕后，通过如下步骤实现主集群中各服务器节点的负载均衡：步骤S31：获取各服务器节点的静态负载参数及动态负载参数，并对其加权平均得综合负载参数；步骤S32：将所有服务器节点的综合负载参数取平均值得主集群的平均综合负载参数，综合负载参数低于平均综合负载参数的服务器节点为目标服务器节点，综合负载参数高于平均综合负载参数的服务器节点为源服务器节点；步骤S33：按照综合负载参数由大到小的顺序排列源服务器节点，按照综合负载参数由小到大的顺序排列目标服务器节点，使源服务器节点与目标服务器节点以相应顺序对应；步骤S34：按照由大到小的顺序，优先将综合负载参数最大的源服务器节点中的服务目录向综合负载参数最小的目标服务器节点中迁移，直至所有的服务器节点实现负载均衡。
- 如权利要求7所述的B2B的云端分销平台系统的数据处理方法，其特征在于，所述步骤S34中，在迁移服务目录时，判断欲迁移服务目录的大小是否大于目录阈值，若大于目录阈值，则将迁移目录拆解为若干个小于或等于目录阈值的子服务目录，并依次完成子服务目录的迁移。
- 如权利要求1所述的B2B的云端分销平台系统的数据处理方法，其特征在于，所述步骤S4中，若副集群中服务器节点的平均资源利用率大于或等于最高阈值，在将闲置集群池中的服务器节点向副集群迁移完毕后；或者，若副集群中服务器节点的平均利用率小于最低阈值，在将副集群中的服务器节点向闲置集群池迁移完毕并将迁移出的服务器节点上的服务目录迁移回副集群后，通过如下步骤实现副集群中各服务器节点的负载均衡：步骤S41：获取各服务器节点的静态负载参数及动态负载参数，并对其加权平均得综合负载参数；步骤S42：将所有服务器节点的综合负载参数取平均值得副集群的平均综合负载参数，综合负载参数低于平均综合负载参数的服务器节点为目标服务器节点，综合负载参数高于平均综合负载参数的服务器节点为源服务器节点；步骤S43：按照综合负载参数由大到小的顺序排列源服务器节点，按照综合负载参数由小到大的顺序排列目标服务器节点，使源服务器节点与目标服务器节点以相应顺序对应；步骤S44：按照由大到小的顺序，优先将综合负载参数最大的源服务器节点中的服务目录向综合负载参数最小的目标服务器节点中迁移，直至所有的服务器节点实现负载均衡。
- 如权利要求9所述的B2B的云端分销平台系统的数据处理方法，其特征在于，所述步骤S44中，在迁移服务目录时，判断欲迁移服务目录的大小是否大于目录阈值，若大于目录阈值，则将迁移目录拆解为若干个小于或等于目录阈值的子服务目录，并依次完成子服务目录的迁移。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US17/297,250 US20240031429A1 (en) | 2021-04-13 | 2021-05-17 | Method for processing data in b2b cloud distribution platform system |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110395816 | 2021-04-13 | ||
| CN202110395816.9 | 2021-04-13 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2022217688A1 true WO2022217688A1 (zh) | 2022-10-20 |
Family
ID=83640044
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2021/094008 WO2022217688A1 (zh) | 2021-04-13 | 2021-05-17 | B2b 的云端分销平台系统的数据处理方法 |
Country Status (2)
| Country | Link |
|---|---|
| US (1) | US20240031429A1 (zh) |
| WO (1) | WO2022217688A1 (zh) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103856512A (zh) * | 2012-11-30 | 2014-06-11 | 华为技术有限公司 | 云计算的管理服务器、工作和闲置主机以及资源调度方法 |
| US20170031622A1 (en) * | 2015-07-31 | 2017-02-02 | Netapp, Inc. | Methods for allocating storage cluster hardware resources and devices thereof |
| CN109451056A (zh) * | 2018-12-20 | 2019-03-08 | 中国软件与技术服务股份有限公司 | 多集群间服务器动态分配方法及系统 |
-
2021
- 2021-05-17 US US17/297,250 patent/US20240031429A1/en active Pending
- 2021-05-17 WO PCT/CN2021/094008 patent/WO2022217688A1/zh active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103856512A (zh) * | 2012-11-30 | 2014-06-11 | 华为技术有限公司 | 云计算的管理服务器、工作和闲置主机以及资源调度方法 |
| US20170031622A1 (en) * | 2015-07-31 | 2017-02-02 | Netapp, Inc. | Methods for allocating storage cluster hardware resources and devices thereof |
| CN109451056A (zh) * | 2018-12-20 | 2019-03-08 | 中国软件与技术服务股份有限公司 | 多集群间服务器动态分配方法及系统 |
Also Published As
| Publication number | Publication date |
|---|---|
| US20240031429A1 (en) | 2024-01-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110134495B (zh) | 一种容器跨主机在线迁移方法、存储介质及终端设备 | |
| CN111124662B (zh) | 一种雾计算负载均衡方法及系统 | |
| CN113110933B (zh) | 一种具有Nginx负载均衡技术的系统 | |
| CN111132235B (zh) | 基于改进hrrn算法和多属性决策的移动卸载迁移算法 | |
| CN108196935B (zh) | 一种面向云计算的虚拟机节能迁移方法 | |
| CN106161120A (zh) | 动态均衡负载的分布式元数据管理方法 | |
| CN110933139A (zh) | 一种解决Web服务器高并发的系统及方法 | |
| CN103338228A (zh) | 基于双加权最小连接算法的云计算负载均衡调度算法 | |
| JP2019121334A (ja) | データストレージとダイナミックマイグレーション方法及びデータストレージとダイナミックマイグレーション装置 | |
| CN108684075B (zh) | 集中式基站架构下的处理资源分配方法 | |
| CN111061561A (zh) | 云计算管理平台的全阶段负载分担综合优化方法 | |
| CN112817752B (zh) | 一种分布式数据库动态负载均衡方法 | |
| CN105959411A (zh) | 云计算环境下基于协调的动态负载均衡分布式处理方法 | |
| CN112363827A (zh) | 一种基于延迟因子的多资源指标Kubernetes调度方法 | |
| Ma et al. | Dynamic task scheduling in cloud computing based on greedy strategy | |
| CN107220108A (zh) | 一种实现云数据中心负载均衡的方法和系统 | |
| CN102932271A (zh) | 负载均衡的实现方法和装置 | |
| Tao et al. | Load feedback-based resource scheduling and dynamic migration-based data locality for virtual hadoop clusters in openstack-based clouds | |
| CN110362379A (zh) | 基于改进蚁群算法的虚拟机调度方法 | |
| CN105975345A (zh) | 一种基于分布式内存的视频帧数据动态均衡存储管理方法 | |
| CN105607943A (zh) | 一种云环境下虚拟机动态部署机制 | |
| CN112506650A (zh) | 资源分配方法、系统、计算机设备和存储介质 | |
| CN113382074A (zh) | 一种基于动态反馈的微服务负载均衡优化方法 | |
| CN115718644A (zh) | 一种面向云数据中心的计算任务跨区迁移方法及系统 | |
| CN110597598B (zh) | 一种云环境中的虚拟机迁移的控制方法 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| WWE | Wipo information: entry into national phase |
Ref document number: 17297250 Country of ref document: US |
|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21936560 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 21936560 Country of ref document: EP Kind code of ref document: A1 |