CN115987990A

CN115987990A - Multi-cluster load balancing method and device, electronic equipment and storage medium

Info

Publication number: CN115987990A
Application number: CN202211588645.2A
Authority: CN
Inventors: 田文杰; 赵建星
Original assignee: Jingdong Technology Information Technology Co Ltd
Current assignee: Jingdong Technology Information Technology Co Ltd
Priority date: 2022-12-09
Filing date: 2022-12-09
Publication date: 2023-04-18

Abstract

The disclosure provides a multi-cluster load balancing method and device, electronic equipment and a computer readable storage medium, relates to the technical field of computers, and can be applied to big data, cloud computing and intelligent supply chains. The method comprises the following steps: selecting one from the plurality of sub-clusters to be determined as a main cluster, collecting service endpoint information of the main cluster and the sub-clusters through the main cluster, generating a cluster internal load balancing rule and a multi-cluster load balancing rule by the main cluster and the sub-clusters according to the service endpoint information, and forwarding data based on an expanded Berkely data packet filter technology; the disclosed embodiments can support multi-cluster services.

Description

Multi-cluster load balancing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a method and an apparatus for balancing loads of multiple clusters, an electronic device, and a computer-readable storage medium.

Background

The network agent component kube-proxy is responsible for providing service discovery and load balancing inside the cluster kubernets for the service, but a service container at the rear end of the kube-proxy load balancing needs to be inside the kubernets, and the service container does not support multi-cluster service.

It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.

Disclosure of Invention

The present disclosure provides a multi-cluster load balancing method, apparatus, electronic device and computer-readable storage medium, which at least to some extent overcome the problem in the related art that multi-cluster services are not supported.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.

According to one aspect of the present disclosure, a multi-cluster load balancing method is provided, which is applied to a node, and includes: selecting one from the plurality of sub-clusters to determine as a main cluster; collecting service endpoint information of the main cluster and the sub-cluster through the main cluster; the main cluster and the sub-cluster generate cluster rules according to the service endpoint information, and the cluster rules include: a cluster internal load balancing rule and a multi-cluster load balancing rule; forwarding data based on an extended Berkeley packet filter technique according to the clustering rules.

In one embodiment of the present disclosure, further comprising: setting a custom resource type to be deployed to the main cluster, and determining the main cluster as a global service resource object.

In one embodiment of the present disclosure, further comprising: allocating a virtual network address to the master cluster, the virtual network address comprising: an internal cluster address segment, a multi-cluster address segment.

In an embodiment of the present disclosure, the master cluster issues the service information in the customized resource type to a plurality of sub-clusters, and collects a back-end address of the sub-clusters, where the service information includes: the service endpoint information.

In an embodiment of the present disclosure, the collecting, by the master cluster, service endpoint information of the master cluster and the slave cluster includes: creating headless services within the master cluster and the sub-clusters; and acquiring the service endpoint information of the main cluster and the sub-cluster based on headless service.

In one embodiment of the present disclosure, further comprising: the main cluster monitors headless services of the sub-clusters and the main cluster; filtering the service endpoint information for the headless service through a tag selector.

In one embodiment of the present disclosure, further comprising: and when the cluster rule is failed to be configured, forwarding data through an external load balancer.

In one embodiment of the present disclosure, the selecting one of the plurality of sub-clusters to determine as the primary cluster includes: acquiring performance data of the sub-cluster; and selecting the main cluster according to the performance data.

According to another aspect of the present disclosure, there is also provided a multi-cluster load balancing apparatus, including:

the main cluster selecting module selects one from the plurality of sub-clusters and determines the selected sub-cluster as a main cluster;

the service endpoint collection module is used for collecting the service endpoint information of the main cluster and the sub-cluster through the main cluster;

the main cluster and the sub-cluster generate cluster rules according to the service endpoint information, and the cluster rules include: a cluster internal load balancing rule and a multi-cluster load balancing rule;

and the data forwarding module forwards data based on the extended Berkeley packet filter technology according to the clustering rule.

According to another aspect of the present disclosure, there is also provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the multi-cluster load balancing methods described above via execution of the executable instructions.

According to another aspect of the present disclosure, there is also provided a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the multi-cluster load balancing method of any one of the above.

The multi-cluster load balancing method, the multi-cluster load balancing device, the electronic equipment and the computer readable storage medium provided by the embodiments of the disclosure select one of the plurality of sub-clusters to be determined as a main cluster, collect service endpoint information of the main cluster and the sub-clusters through the main cluster, the main cluster and the sub-clusters generate a cluster internal load balancing rule and a multi-cluster load balancing rule according to the service endpoint information, forward data based on an expanded Berkely packet filter technology, and can support multi-cluster services.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty.

FIG. 1 is a flow chart illustrating a method for multi-cluster load balancing in an embodiment of the present disclosure;

FIG. 2 is a flow chart illustrating a method for collecting service endpoint information for multiple sub-clusters in an embodiment of the present disclosure;

FIG. 3 is a flow chart of a method for multi-cluster load balancing in an embodiment of the present disclosure;

FIG. 4 is a schematic diagram illustrating a control plane for multi-cluster load balancing in an embodiment of the present disclosure;

FIG. 5 is a data plane diagram illustrating multi-cluster load balancing in an embodiment of the present disclosure;

FIG. 6 is a schematic diagram illustrating a multi-cluster load balancing apparatus according to an embodiment of the present disclosure;

FIG. 7 is a schematic diagram of a multi-cluster load balancing system according to an embodiment of the present disclosure;

fig. 8 shows a block diagram of an electronic device in an embodiment of the present disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

For ease of understanding, the following first explains several terms to which the disclosure relates:

the CRD (Custom Resource Definition) is a Resource type built in the cluster, i.e. Definition of Custom Resource, and is used to describe what the user-defined Resource is, and the CR is each value of the CRD.

The essence of the cluster kubernets is a set of server clusters that can run specific programs on each node of the cluster to manage the containers in the nodes.

The service is an abstract concept, and is a resource object of a virtual network address, which is mapped out a designated port through a virtual network address, and a request sent by a client is forwarded to one of a group of nodes pod at the back end.

service endpoint is a resource object list in the kubernets cluster, stores the list of the port name and the IP of the back end of the corresponding service, and provides the actual ability to forward the request to the port.

The etcd is a distributed and highly-available consistent key value storage system written in a language, and is used for providing reliable functions of distributed key value storage, configuration sharing, service discovery and the like.

service vip (virtual Internet Protocol, virtual network address) is provided by a cluster, which is equivalent to a service gateway, and all requests are intercepted by the service vip and then forwarded.

ebpf (extended berkeley packet filter) aims to provide a method for filtering packets and to avoid useless packet replication behavior from kernel space to user space.

The headers services are logical abstractions and access portals of the cluster to a back-end set of pods that provide the same service.

The DNS (domain name system) stores a mapping relation of IP addresses of domain names in a distributed database.

An API (Application Programming Interface) is a predefined function that is intended to provide applications and developers the ability to access a set of routines based on certain software or hardware, without accessing source code or understanding the details of the internal workings.

nodecort is a port accessible by external systems.

kube-proxy is a simple network proxy and load balancer.

The present exemplary embodiment will be described in detail below with reference to the drawings and examples.

First, the present disclosure provides a multi-cluster load balancing method, which can be executed by any electronic device with computing processing capability.

Fig. 1 shows a flowchart of a multi-cluster load balancing method in an embodiment of the present disclosure, and as shown in fig. 1, the multi-cluster load balancing method provided in the embodiment of the present disclosure is applied to a node, and includes the following steps:

s102, selecting one from a plurality of sub-clusters and determining the selected sub-cluster as a main cluster;

in one embodiment, the user-defined resource type CRD is set and deployed to the main cluster, the main cluster is determined to be a global service resource object, automation of resource management is achieved, and service endpoint information can be efficiently collected.

The essence of the cluster kubernets is a set of server clusters that can run specific programs on each node of the cluster to manage the containers in the nodes. The CRD (Custom Resource Definition) is a Resource type built in a cluster, namely the Definition of a Custom Resource, and is used for describing what a user-defined Resource is, and a main cluster is determined as a global service Resource object through the CRD.

In one embodiment, performance data for a plurality of sub-clusters is obtained; selecting a main cluster according to the performance data; the main cluster needs to collect the service endpoint information of the main cluster and the sub-clusters, monitors the headless services of the main cluster and the sub-clusters, obtains the service endpoint information of the headless services through the filtering of the tag selector, and the like, selects the sub-clusters with the optimal performance to determine as the main cluster, and can guarantee the stability of the multi-cluster load balancing method.

S104, collecting service endpoint information of the main cluster and the sub-clusters through the main cluster;

the service is an abstract concept, is a resource object of a virtual network address, maps out a designated port through a virtual network address, and forwards a request sent by a client to one of a group of nodes, a back end, the cluster will combine into a service end point according to the IP address information associated to the node, the service end point is a resource object list in the cluster kubernets, stores the list of the node name and the IP corresponding to the back end of the service, provides the actual capability of forwarding the request to the node, stores the actual capability in the etcd, and records the access address of a service corresponding to a service container.

The etcd is a distributed and highly-available consistent key value storage system written in a language and used for providing reliable functions of distributed key value storage, configuration sharing, service discovery and the like, and the etcd can be used for storing key data and realizing distributed scheduling.

In one embodiment, each cluster deploys a first cluster component, the first cluster component is a network agent component, and the virtual network address can be allocated to the main cluster through the first cluster component, headless services are created in the main cluster and the sub-clusters, service endpoint information of the main cluster and the sub-clusters is obtained, the service information is issued to the back-end addresses of the sub-clusters collected by the sub-clusters, and the service endpoint information of the headless services is obtained through filtering.

S106, the main cluster and the sub-cluster generate a cluster rule according to the service endpoint information, and the cluster rule comprises: a cluster internal load balancing rule and a multi-cluster load balancing rule;

the cluster internal load balancing rule is a cluster internal pod load balancing rule, and the multi-cluster load balancing rule is a load balancing rule among a plurality of clusters.

In one embodiment, service endpoint information of the sub-cluster is monitored, and cluster internal load balancing rules are configured.

In one embodiment, service endpoint information of a main cluster is monitored, and load balancing rules of multiple clusters are configured.

In one embodiment, the second cluster component is a network proxy component, and the first cluster component and the second cluster component are a network proxy and a load balancer, which are basic components for realizing service registration and discovery by a cluster; the second cluster component is deployed on the main cluster and the node of each sub-cluster, monitors the service endpoint information of the sub-clusters and the service endpoint information of the main cluster respectively, configures cluster rules correspondingly, and balances loads based on the load balancing rules in the clusters and the load balancing rules of multiple clusters, so that the purposes of optimizing resource use, maximizing throughput rate, minimizing response time and avoiding overload can be achieved.

And S108, forwarding the data based on the extended Berkeley packet filter technology according to the clustering rule.

A virtual Internet Protocol (service view) is provided by a cluster, which is equivalent to a service gateway, all requests are intercepted by the service view and then forwarded, a client inside the cluster can access a behind pod only through the virtual network address, and does not need to concern the specific pod number and pod address in the cluster, even if the pod address changes, the pod address can be shielded by the virtual network address.

ebpf (extended berkeley packet filter) is aimed at providing a method for filtering packets and avoiding useless packet copying from kernel space to user space, an ebpf program is attached to a specified kernel code path, and when the code path is executed, the corresponding ebpf program is executed; the ebpf program attaches to the network socket, performs traffic filtering, traffic classification, and performs actions of the network classifier.

In one embodiment, a client inside the cluster accesses a virtual network address on the main cluster, and selects a real node pod based on an ebpf program loaded by the second component and a cluster rule according to the virtual network address; the virtual network address comprises an internal cluster address segment and a multi-cluster address segment, wherein the internal cluster address segment corresponds to a cluster internal service vip and a port, the multi-cluster address segment corresponds to a federal service vip and a port, the zero-delay load balance from a cluster internal pod to service access is achieved through the cluster internal service vip and the port, and the zero-delay load balance from the multi-cluster pod to service access is achieved through the federal service vip and the port.

In one embodiment, when the cluster rule fails to be configured, the data is forwarded through an external load balancer; and if the configuration of the cluster rule fails, the second component does not load the ebpf program and can complete data forwarding by passing through a stateless load balancer through a default route, so that the access of a client to the service is not influenced, and the effect of disaster recovery backup is achieved.

In the embodiment, the service endpoint information of the main cluster and the sub-cluster is collected through the main cluster, the main cluster and the sub-cluster generate the internal load balancing rule of the cluster and the load balancing rule of multiple clusters according to the service endpoint information, the data is forwarded based on the expanded Berkeley data packet filter technology, the seamless switching of the client access service to the pod-to-pod access can be realized, but the method is different from the method that the conversion rule is added on a kernel netfilter framework by the existing iptables and ipv technologies, but the ebpf technology is used for conversion on a data layer, and the seamless switching of the client access service to the pod-to-pod access can be efficiently realized; the service of multiple clusters is supported, so that the service function is not limited in the cluster, and when a service party deploys a load balancing service, the service party is not limited in one cluster, and cluster resources can be utilized more efficiently.

Fig. 2 is a flowchart illustrating a method for collecting service endpoint information of multiple sub-clusters in an embodiment of the present disclosure, and as shown in fig. 2, the method for collecting service endpoint information of multiple sub-clusters in an embodiment of the present disclosure includes the following steps:

s202, establishing headless service in the main cluster and the sub-cluster; and acquiring the service endpoint information of the main cluster and the sub-cluster based on the headless service.

A headless service is a logical abstraction and access entry of a cluster to a group of pod providing the same service at the back end, and is a special service, which specifies service. None, which is not allocated with a virtual network address in actual operation, may return all pod addresses and DNS through a DNS (domain name system) that resolves a service; the common service can only return the virtual network address of the service by analyzing the DNS of the service, and the service endpoint information of a plurality of sub-clusters is acquired by headless service, so that the IP address resource in the cluster is not consumed, and the resource is saved.

In one embodiment, a DNS alias is created to refer to a service name, preventing the service name from changing.

In one embodiment, the main cluster monitors a plurality of sub-clusters and headless services of the main cluster, and service endpoint information of the headless services can be efficiently obtained through filtering by the label selector.

When a cluster is in the process of creating service, a point is searched according to a tag selector of the service, accordingly, endpoint with the same name as the service is created, and when the IP address of the point changes, the content of the endpoint changes; after receiving the request, the service finds the address to be forwarded through endpoint.

In one embodiment, for headless services defining a tag selector, an endpoint controller creates service endpoint information endpoints records in an API (Application Programming Interface), modifies a DNS configuration and returns all the point addresses through which the backend points of a service can be directly reached; for example, an headless service that creates a mysql in the cluster generates an endpoint named mysql, which is the IP address of the point associated with the service and the contractor port provided for this service.

S204, distributing a virtual network address for the main cluster, wherein the virtual network address comprises: an internal cluster address field, a multi-cluster address field.

In one embodiment, a client in the cluster can access a behind pod only through a virtual network address, and the zero-delay load balancing function from the pod internal pod to service access and the zero-delay load balancing function from the multi-cluster pod to service access are achieved based on the virtual network address and the cluster internal load balancing rule distributed to the main cluster and the multi-cluster load balancing rule.

S206, the main cluster issues the service information in the user-defined resource type to a plurality of sub-clusters, and the rear-end addresses of the sub-clusters are collected.

In one embodiment, the main cluster applies the custom resource CR in the custom resource type CRD to obtain service information in the CR, sends the service information to the plurality of sub-clusters, collects a back-end address backhaul of the sub-clusters, and can select a real back-end pod through the back-end address.

In one embodiment, service endpoint information may be added to the service information.

In the above embodiment, the service access control plane in the multiple clusters does not depend on whether the federate cluster or the external cluster mechanism provides support, and supports a service multiple cluster zero-delay load balancing method, so that the limitation that the service cannot be supported across the clusters is solved, the dependence of the service user on a single cluster is released, and a multiple cluster zero-delay load balancing function is provided.

Fig. 3 shows a flowchart of a method for balancing loads of multiple clusters in an embodiment of the present disclosure, and as shown in fig. 3, the method for balancing loads of multiple clusters in the embodiment of the present disclosure includes the following steps:

s302, a control plane selects a main cluster from a plurality of clusters to create a global service resource object to collect service endpoint information of all the clusters, and each cluster acquires the global service information to configure the cluster data plane conversion rule.

Fig. 4 is a schematic diagram illustrating a control plane for multi-cluster load balancing in an embodiment of the present disclosure, and as shown in fig. 4, the control plane includes: a main cluster 401, a first sub-cluster 402, a second sub-cluster 403, a first sub-cluster node 404, a second sub-cluster node 405;

defining service CRD to be deployed to a main cluster 401, selecting one main cluster 401 from a plurality of sub-clusters according to actual needs, deploying a first cluster component for each cluster, and allocating a global virtual network address to the main cluster 401 through the first cluster component, wherein the virtual network address comprises: an internal cluster address field, a multi-cluster address field.

The main cluster 401 applies the CR created by the service CRD, takes out the service information in the CR and issues the service information to the associated cluster for collecting the multi-cluster back-end address backhaul; creating a header service in a cluster where a first cluster component is located, namely a main cluster 401, a first sub-cluster 402 and a second sub-cluster 403, for collecting endpoint information of the cluster where the first cluster component is located; the master cluster 401 monitors the threads services of all clusters, filters endpoint information of the threads services through the label, and statically adds the endpoint information to the service information.

Each cluster node, i.e. the first sub-cluster node 404 and the second sub-cluster node 405, deploys a second cluster component, monitors the service of the sub-cluster in which it is located, configures the load balancing rule in the cluster, monitors the service of the main cluster 401, and configures the load balancing rule of multiple clusters.

S304, the data plane, each cluster is configured according to the cluster rule generated by the control plane, and ebpf technology is applied to perform message conversion on the bidirectional flow of the data packet.

Fig. 5 is a schematic diagram illustrating a data plane of multi-cluster load balancing in an embodiment of the present disclosure, where, as shown in fig. 5, the data plane includes: a first cluster 501, a second cluster 502, a first service endpoint 503, a second service endpoint 504, an external load balancer 505;

the intra-cluster client accesses the intra-cluster service vip and the port of the first cluster 501, and selects the real pod and the first service endpoint 503 through the ebpf program loaded by the second cluster 502 component, so that zero-delay load balancing from the intra-cluster pod to the service access is achieved.

The federated service vip is a control plane for multi-cluster communication, a client inside the cluster accesses the federated service vip and a port of the second cluster 502, and selects a real pod and the second service endpoint 504 through an ebpf program loaded by the second cluster 502 component, so that a zero-latency load balancing function from multi-cluster pod to service access is achieved.

A client outside the cluster accesses the federated service vip and the port of the second cluster 502, and if the ebpf program is not loaded on the second cluster 502 component, data forwarding can be completed through the external load balancer 505 by default routing, so that the access of the client to the service is not affected, and the effect of disaster recovery backup is achieved.

The intra-cluster clients, the off-cluster clients may be various electronic devices including, but not limited to, smart phones, tablets, laptop portable computers, desktop computers, and the like; the clients of the applications installed in different clients are the same, or clients of the same type of application based on different operating systems. The specific form of the application client may also be different based on different terminal platforms, for example, the application client may be a mobile phone client, a PC client, or the like.

The nodeport is a port accessible by an external system, and on the basis of the nodeport, an external load balancer 505 is created by means of the micro service provider cloud-provider, and the request is forwarded to the nodeport.

The main data structure used by ebpf program is ebpf map, which is a general data structure used to transfer data in kernel or kernel and user space, and its name "map" also means that key is needed for storing and retrieving data.

Maps are created and managed using ebpf () system calls. When a map is successfully created, the file descriptor associated with the map is returned, the map is destroyed while the corresponding file descriptor is closed, and each map defines four values: the type, the maximum number of elements, the byte size of the numerical value, and the byte size of the key, ebpf provides different map types, different maps provide different characteristics, and the actual pod can be selected through the ebpf map.

The following are the common types to be listed:

ebpf _ map _ type _ device _ map: for storing and looking up network device references;

ebpf _ map _ type _ lpm _ trie: a lost-prefix match trie, good for matching IP addresses to a range, the longest prefix match number, is suitable for matching IP addresses to a certain range;

ebpf _ map _ type _ device _ map: for storing and logging up network device references for storing and searching network device references.

S306, in the cluster, the load balancing is realized based on the load balancing rule in the cluster.

In the embodiment, unlike the existing iptables and ipv technologies, the switching rule is added on the kernel netfilter framework, and ebpf technology is used for switching in the data layer, so that seamless switching of the client access service to pod-to-pod access can be efficiently realized

Based on the same inventive concept, the embodiment of the present disclosure further provides a multi-cluster load balancing apparatus, such as the following embodiments. Because the principle of solving the problem of the embodiment of the apparatus is similar to that of the embodiment of the method, reference may be made to the implementation of the embodiment of the apparatus, and repeated descriptions are omitted.

Fig. 6 is a schematic diagram of a multi-cluster load balancing apparatus in an embodiment of the present disclosure, and as shown in fig. 6, the multi-cluster load balancing apparatus 6 includes: a master cluster selection module 601, a service endpoint collection module 602, a cluster rule configuration module 603 and a data forwarding module 604;

a master cluster selecting module 601, selecting one from the plurality of sub-clusters and determining the selected one as a master cluster;

a service endpoint collecting module 602, configured to collect service endpoint information of the master cluster and the slave clusters through the master cluster;

the cluster rule configuration module 603 configures cluster rules for the cluster by the main cluster and the sub-cluster according to the service endpoint information, where the cluster rules include: a cluster internal load balancing rule and a multi-cluster load balancing rule;

the data forwarding module 604 forwards data based on the extended berkeley packet filter technique according to the clustering rules.

In the above embodiment, the service endpoint information of the main cluster and the sub-cluster is collected by the main cluster, the main cluster and the sub-cluster generate a cluster internal load balancing rule and a multi-cluster load balancing rule according to the service endpoint information, and the data is forwarded based on the expanded berkeley packet filter technology, so that the seamless switching of the client access service to the pod access can be realized, but the switching rule is added on a kernel netfilter frame by the existing iptables and ipv technologies, but the ebpf technology is used for switching the data layer, so that the seamless switching of the client access service to the pod access can be efficiently realized; the service of multiple clusters is supported, so that the service function is not limited in the cluster, and when a service party deploys a load balancing service, the service party is not limited in one cluster, and cluster resources can be utilized more efficiently.

Based on the same inventive concept, the embodiment of the present disclosure further provides a multi-cluster load balancing system, as in the following embodiments. Because the principle of the system embodiment for solving the problem is similar to that of the method embodiment, the implementation of the system embodiment may refer to the implementation of the method embodiment, and repeated details are not described again.

Fig. 7 is a schematic diagram of a multi-cluster load balancing system in an embodiment of the present disclosure, which is introduced by taking a pod with two different services accessing two clusters as an example.

The service is a service abstraction of a group of containers, is equivalent to a load balancer of the group of containers, and is responsible for distributing requests to corresponding service containers; kube-proxy is a simple network proxy and load balancer.

The two different services, namely the first service71 and the second service72, can realize seamless switching of the client access service to the pod-to-pod access based on multi-cluster load balancing;

that is, the first service71 may access the first cluster node two 712, the first cluster node four 714 of the first cluster 73 and the second cluster node two 722, the second cluster node four 724 of the second cluster 74 based on the cluster rule;

the second service72 can access the first cluster node one 711, the first cluster node three 713 of the first cluster 73 and the second cluster node one 721, the second cluster node three 723 of the second cluster 74 based on the cluster rule, and the problem that the service function is limited in the cluster in the related art is solved.

In the related art, the cluster kubernets riube-proxy supports three modes: userpace, iptables and ipv;

the userspace mode supported by kube-proxy is obsolete;

iptables mode supported by kube-proxy: the iptables is dynamically managed by the kube-proxy, the kube-proxy is not responsible for forwarding any more, the trend of a data packet is completely determined by iptables rules, and the iptables rules are continuously increased along with the increase of service, so that a kernel is very busy; for example, there are 34 iptabels rules for 2 services and 8 service containers, and as svc and pod increase in the cluster, the rules in iptables open up to expand rapidly, resulting in performance degradation, and in some extreme cases, even rule loss occurs, and the failure is difficult to reproduce and troubleshoot.

Compared with the iptables, the performance of the ipvs mode supported by the kube-proxy is greatly improved, but the ipvs cannot provide functions of packet filtering, address camouflage, SNAT and the like, so that the ipvs mode is matched with the iptables in some scenes, and the influence on the performance is self-evident.

In the embodiment, data is forwarded based on the extended berkeley packet filter technology, seamless switching of client access service to pod-to-pod access can be achieved, but the method is different from the existing iptables and ipvs technology that a conversion rule is added on a kernel netfilter framework, and ebpf technology is used for conversion on a data layer, so that seamless switching of client access service to pod-to-pod access can be achieved efficiently; the service of multiple clusters is supported, so that the service function is not limited in the cluster, and when a service party deploys a load balancing service, the service party is not limited in one cluster, and cluster resources can be utilized more efficiently.

As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or program product. Accordingly, various aspects of the present disclosure may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.

An electronic device 800 according to this embodiment of the disclosure is described below with reference to fig. 8. The electronic device 800 shown in fig. 8 is only an example and should not bring any limitations to the functionality and scope of use of the embodiments of the present disclosure.

As shown in fig. 8, electronic device 800 is in the form of a general purpose computing device. The components of the electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one memory unit 820, and a bus 830 that couples the various system components including the memory unit 820 and the processing unit 810.

Wherein the storage unit stores program code that is executable by the processing unit 810 to cause the processing unit 810 to perform steps according to various exemplary embodiments of the present disclosure as described in the "exemplary methods" section above in this specification.

For example, the processing unit 810 may perform the following steps of the above method embodiments: selecting one from the plurality of sub-clusters to determine as a main cluster; collecting service endpoint information of the main cluster and the sub-clusters through the main cluster; the main cluster and the sub-cluster configure cluster rules for the clusters according to the service endpoint information, and the cluster rules comprise: the method comprises the following steps of (1) load balancing rules inside clusters and load balancing rules of multiple clusters; according to the clustering rules, data is forwarded based on the extended berkeley packet filter technique.

For example, the processing unit 810 may perform the following steps of the above method embodiments: creating headless services within the master cluster and the sub-clusters; acquiring service endpoint information of a main cluster and a sub-cluster based on headless service; assigning a virtual network address to the master cluster, the virtual network address comprising: an internal cluster address segment, a multi-cluster address segment; and the main cluster sends the service information in the user-defined resource type to the plurality of sub-clusters and collects the back-end addresses of the sub-clusters.

The storage unit 820 may include readable media in the form of volatile memory units such as a random access memory unit (RAM) 8201 and/or a cache memory unit 8202, and may further include a read only memory unit (ROM) 8203.

The storage unit 820 may also include a program/utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.

Bus 830 may be any of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 800 may also communicate with one or more external devices 840 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 800, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 800 to communicate with one or more other computing devices. Such communication may occur over input/output (I/O) interfaces 850. Also, the electronic device 800 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 860. As shown, the network adapter 860 communicates with the other modules of the electronic device 800 via the bus 830. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium, which may be a readable signal medium or a readable storage medium. On which a program product capable of implementing the above-described method of the present disclosure is stored. In some possible embodiments, various aspects of the disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the disclosure as described in the "exemplary methods" section above of this specification, when the program product is run on the terminal device.

For example, the program product in the embodiments of the present disclosure, when executed by a processor, implements a method comprising: selecting one from the plurality of sub-clusters to determine as a main cluster; collecting service endpoint information of the main cluster and the sub-clusters through the main cluster; the main cluster and the sub-cluster configure cluster rules for the clusters according to the service endpoint information, and the cluster rules comprise: a cluster internal load balancing rule and a multi-cluster load balancing rule; according to the clustering rules, data is forwarded based on the extended berkeley packet filter technique.

For example, the program product in the embodiments of the present disclosure, when executed by a processor, implements a method comprising: selecting a main cluster from a plurality of clusters to create a global service resource object to collect service endpoint information of all the clusters, and acquiring the global service information by each cluster to configure the cluster data plane conversion rule; each cluster is configured according to the cluster rule generated by the control plane, and ebpf technology is applied to perform message conversion on the bidirectional flow of the data packet; and in the cluster, realizing load balancing based on a cluster internal load balancing rule.

More specific examples of the computer-readable storage medium in the present disclosure may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

In the present disclosure, a computer readable storage medium may include a propagated data signal with readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Alternatively, program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

In particular implementations, program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + +, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In situations involving remote computing devices, the remote computing devices may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to external computing devices (e.g., through the internet using an internet service provider).

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

Moreover, although the steps of the methods of the present disclosure are depicted in the drawings in a particular order, this does not require or imply that the steps must be performed in this particular order, or that all of the depicted steps must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions, etc.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. A multi-cluster load balancing method is applied to nodes and comprises the following steps:

selecting one from the plurality of sub-clusters to determine as a main cluster;

collecting service endpoint information of the main cluster and the sub-cluster through the main cluster;

forwarding data based on an extended Berkeley packet filter technique according to the clustering rules.

2. The method of multi-cluster load balancing according to claim 1, further comprising: setting a custom resource type to be deployed to the main cluster, and determining the main cluster as a global service resource object.

3. The method of claim 1, further comprising: allocating a virtual network address to the master cluster, the virtual network address comprising: an internal cluster address field, a multi-cluster address field.

4. The method of multi-cluster load balancing according to claim 2, further comprising: the main cluster issues the service information in the user-defined resource type to a plurality of sub-clusters, and collects the back-end addresses of the sub-clusters, wherein the service information comprises: the service endpoint information.

5. The method of claim 4, wherein the collecting, by the master cluster, the service endpoint information of the master cluster and the sub-clusters comprises:

creating headless services within the master cluster and the sub-clusters;

and acquiring the service endpoint information of the main cluster and the sub-cluster based on headless service.

6. The method of multi-cluster load balancing according to claim 5, further comprising:

the main cluster monitors headless services of the sub-clusters and the main cluster;

filtering the service endpoint information for the headless service by a tag selector.

7. The method of multi-cluster load balancing according to claim 1, further comprising: and when the cluster rule is failed to be configured, forwarding data through an external load balancer.

8. The method of claim 1, wherein selecting one of the plurality of sub-clusters to determine as the primary cluster comprises:

acquiring performance data of the sub-cluster;

and selecting the main cluster according to the performance data.

9. A multi-cluster load balancing apparatus, comprising:

the service endpoint collecting module is used for collecting the service endpoint information of the main cluster and the sub-clusters through the main cluster;

10. An electronic device, comprising:

a processor; and

a memory for storing executable instructions of the processor;

wherein the processor is configured to perform the multi-cluster load balancing method of any one of claims 1 to 8 via execution of the executable instructions.

11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the multi-cluster load balancing method according to any one of claims 1 to 8.