KR20220164840A - Load balancer manage system, method, program in a cloud native environment and the load balancer created by this method - Google Patents

Abstract

The present invention relates to a load balancer management system capable of flexibly coping with variable traffics. More specifically, in the present invention, a CR management unit generates a custom resource (CR) based on a load balancer node generation request. The custom resource includes at least one parameter information defining a load balancer. The CR management unit stores the generated custom resource in a database. An operator performs at least one of generating, removing, and changing at least one load balancer node (LB node) to correspond to the stored at least one custom resource.

Description

System, method, program for managing load balancer in cloud native environment, and load balancer created by this method

The present invention relates to a system for managing a load balancer in a cloud-native environment, a method, and a load balancer created by the method, and more specifically, control for creating and removing load balancers flexibly according to network conditions in a cloud-native environment. A method and a load balancer created by the method.

Cloud data center environments contain large collections of interconnected servers that provide computing and/or storage capacity to run a variety of applications. For example, a data center may include facilities that host applications and services for subscribers, ie, customers of the data center. And data centers can host all infrastructure equipment, such as networking and storage systems, redundant power supplies and environmental controls.

Data centers are interconnected through a high-speed switch fabric provided by one or more layers of physical network switches and routers.

Due to the nature of data centers, numerous virtual machines (VMs) exist, and these virtual machines can be easily created or removed. Likewise, a load balancer that load balances traffic to virtual machines whose number and location are constantly changing must also be created and removed flexibly according to network conditions.

Load balancers are typically implemented as physical hardware devices that are separately connected to the data center. When relying on such a physical hardware device, it is difficult to provide flexible processing according to a rapidly changing amount of traffic or a change in the number of virtual machines. If the physical equipment is placed by predicting the maximum traffic that can flow into the data center, not only a lot of equipment purchase costs are required, but it can be an uneconomical design in terms of cost and performance in low traffic situations.

In addition, data center networks must be designed to handle exponentially increasing traffic and rapid changes in traffic patterns. This is because a sudden increase in a particular traffic flow can concentrate the load on a particular function, causing malfunction and sequentially affecting other related functions.

Therefore, research on a load balancer management system capable of coping with rapidly changing traffic patterns without difficulty and being economical at the same time is required.

An object of the present invention is to provide a load balancer management system and control method optimized in a cloud native environment.

Another problem to be solved by the present invention is to provide a load balancer capable of supporting multi-tenants.

Another problem to be solved by the present invention is to provide a load balancer management system that dynamically changes according to rapidly changing traffic patterns.

Another problem to be solved by the present invention is to provide a container-type load balancer that can be easily controlled at a software level on a Linux operating system.

Another problem to be solved by the present invention is to provide a load balancer in the form of a container that corresponds to the performance of a hardware load balancer.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

According to one aspect of the present invention in order to solve the above or other problems, CR management unit based on the load balancer creation request, creating a user-specified resource (CR, Custom Resource); The user-specified resource includes at least one parameter information defining a load balancer, and storing the created user-specified resource in a database; and performing at least one of creating, removing, and changing at least one load balancer node (LB Node) so that the operator corresponds to the stored at least one user-specified resource. Control of the load balancer management system. provides a way

A step of allocating a VIP (Virtual IP) to the load balancer to which the VIP allocator is added may be further included.

The generated load balancer may be implemented in a device driver (DD) stack of Linux network stacks.

The load balancer may be implemented as an Extended Berkeley Packet Filter (eBPF)/eXpress Data Path (XDP).

According to another aspect of the present invention to solve the above or other problems, CR management unit for creating a custom resource (CR, Custom Resource) based on the load balancer creation request; The user-specified resource includes at least one parameter information defining a load balancer, and a database for storing the created user-specified resource; and an operator performing at least one of creating, removing, and changing at least one load balancer node (LB Node) to correspond to the stored at least one user-specified resource.

A VIP allocation unit for allocating a VIP (Virtual IP) to the load balancer to be added may be further included.

The generated load balancer may be implemented in a device driver (DD) stack of Linux network stacks.

The load balancer may be implemented as an Extended Berkeley Packet Filter (eBPF)/eXpress Data Path (XDP).

Effects of the system and method for managing a load balancer in a cloud-native environment according to the present invention are described as follows.

According to at least one of the embodiments of the present invention, there is an advantage in that optimized load balancer management is possible in a cloud native environment.

In addition, according to at least one of the embodiments of the present invention, there is an advantage in that a load balancer capable of supporting multi-tenants can be provided.

In addition, according to at least one of the embodiments of the present invention, there is an advantage in that a relatively economical load balancer can be provided through a container-based load balancer compared to an existing hardware-type load balancer.

Additionally, according to at least one of the embodiments of the present invention, there is an advantage in providing a load balancer management system that dynamically changes according to rapidly changing traffic patterns.

A further scope of the applicability of the present invention will become apparent from the detailed description that follows. However, since various changes and modifications within the spirit and scope of the present invention can be clearly understood by those skilled in the art, it should be understood that the detailed description and specific examples such as preferred embodiments of the present invention are given as examples only.

1 is a diagram showing the structure of a load balancer management system 100 according to an embodiment of the present invention.
Figure 2 shows a conceptual diagram in which the load balancer management system 100 according to an embodiment of the present invention is connected to a switch fabric.
Figure 3 illustrates the concept of a control sequence for creating an additional LB node 203-3 according to one embodiment of the present invention.
4 shows a control flow chart for creating an additional LB node 203-3 according to one embodiment of the present invention.
5 illustrates a control sequence in which the LB node 203 processes inbound traffic according to an embodiment of the present invention.
6 illustrates an example in which a MAC address and a VID included in a packet header are changed according to an embodiment of the present invention.
7 is a diagram illustrating a Linux network stack for explaining a location where a load balancer node is executed according to an embodiment of the present invention.
8 shows an example of a hash table 801 according to one embodiment of the present invention.
9 shows the relationship between the LBN core 905 of the Linux kernel and the LBN controller 910 of the user space 707 according to an embodiment of the present invention.
10 shows the experimental environment of this experiment.
11 to 13 show the throughput of each scenario when traffic consisting only of frames of a specific size is transmitted at full speed of a 25G NIC card.
14 shows throughput results in units of BPS (L2 and L1 levels) for IMIX traffic. 15 shows Troughput results in units of PPS for IMIX traffic.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar elements are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

The present invention relates to a virtualized computing infrastructure in a cloud-native environment, and more particularly, to a virtualized computing infrastructure deployed in a network in a virtual execution element (eg, a virtual machine (VM)). It is about configuring load balancers for virtual machines (or containers) as virtual execution elements.

Containerization refers to a virtualization method based on operating system level virtualization. Containers are characterized as lightweight, portable executables for applications that are isolated from each other or from the host.

Because containers are not tightly coupled to the host hardware computing environment, applications can be linked to container images and run as a single, lightweight package on any host or virtual host that supports the underlying container architecture. Because of these characteristics, containers can solve various problems that can arise when running software in different computing environments.

In addition, as a container is an environment in which an operating system is virtualized, it is relatively lightweight compared to a virtual machine, so it is possible to support more container instances than existing virtual machines under the same conditions, and to create and remove them quickly. Often short-lived containers can be created and moved more efficiently than virtual machines. And containers can be managed as groups of logically related elements. In some orchestration platforms, for example Kubernetes, these elements are called "pods", and groups of these pods are called "clusters".

Hereinafter, the terms pod and cluster are used in the present invention, but these are only terms to refer to elements and their groups, and the present invention will not be limited to the Kubernetes platform.

An embodiment of the present invention proposes an installable cloud architecture in which clusters, their roles and relationships are defined. It is also a containerized, high-performance load balancer that can be easily managed through orchestration tools such as Kubernetes and can distribute traffic using Extended Berkeley Packet Filter (eBPF) / eXpress Data Path (XDP) within the Linux kernel. We propose a cloud architecture that can provide

In a Kubernetes-based cloud, applications in the form of containers or virtual machines can be easily created and removed to handle incoming traffic. Accordingly, data centers require load balancers that distribute traffic to multiple containers or virtual machines with virtual IPs (VIPs). In particular, the load balancer needs to be able to forward incoming traffic to multiple containers/virtual machines whose number and location change frequently.

In addition, the load balancer must be flexible and scalable according to the network environment and must be able to deliver traffic stably under any circumstances.

To meet these requirements, the present invention proposes deploying or managing a load balancer in the form of a container that can be deployed and managed through an orchestration tool (eg Kubernetes). In this way, using a container-based load balancer will have the advantage of being able to properly deploy and manage cloud resources according to user requests.

In addition, the load balancer according to an embodiment of the present invention proposes to operate in a layer 2 layer (L2) direct server return (DSR) mode in which a destination MAC (Media Access Control Address) address and VLAN ID are modulated. When operating in this L2DSR mode, the load of the load balancer can be reduced while increasing the response speed by directly responding to the client without going through the load balancer. In addition, in order to support multi-tenant, the VLAN ID can be modulated to perform a load balancing service for a specific tenant. Hereinafter, in the present invention, an identifier corresponding to a specific tenant is used as a VLAN ID, but this is only a term for distinguishing tenants, and the present invention will not be limited to VLANs.

In the present invention, a policy for deploying a containerized load balancer (API for creation and deletion, status monitoring, etc.) and setting packet matching rules in the load balancer is proposed. In addition, we propose a load balancer that can expect sufficient performance to operate in the Linux operating system.

In the present invention, cloud components (pods) required for IaaS (Infrastructure-as-a-Service) are defined, and then the components are classified into clusters according to their characteristics (or functions). This cluster will be described with reference to FIGS. 1 and 2 together.

1 is a diagram showing the structure of a load balancer management system 100 according to an embodiment of the present invention. Figure 2 shows a conceptual diagram in which the load balancer management system 100 according to an embodiment of the present invention is connected to a switch fabric.

Hereinafter, it will be described with reference to FIGS. 1 and 2 together.

The load balancer management system 100 according to an embodiment of the present invention may be configured to include a service cluster 110 , a load balancer (LB) cluster 120 and a LB control cluster 130 . However, additional clusters can be configured in addition to the clusters mentioned above. The components shown in FIG. 1 are not essential for implementing the load balancer management system 100, so the load balancer management system 100 described in this specification has more, or fewer components than the components listed above. can have elements.

An embodiment of the present invention proposes a method for generating a cloud-based service that is stable and scalable step by step in a bare metal server based on a cluster defined (configured) as shown in FIG. 1 . In one embodiment of the present invention, creation (distribution) of load balancers and load balancing rules are automated based on a custom resource definition (CRD). For example, a CRD can be a custom object provided by the Kubernetes platform API. In addition, the load balancer management system 100 according to an embodiment of the present invention implements a load balancer using an extended Berkeley Packet Filter (eBPF) / eXpress Data Path (XDP) to ensure sufficient load balancing performance for the cloud. can

A pod according to an embodiment of the present invention may mean the smallest distributable computing unit that can be created and managed by the load balancer management system 100 . And a pod can be composed of a group of one or more containers, which can share storage and network, and the specification of how to run the containers. A Pod's content can be collocated and scheduled together.

In one embodiment of the present invention, virtual execution elements may be distributed in a virtualization environment using a cluster-based framework. A cluster can be installed on one or more nodes. Each node where the cluster is installed is divided into master node (or master) and worker node (or worker) according to its role. The master node controls the entire cluster.

Pods installed on the master node generally have an API server that provides cluster functions in API form, a database that stores cluster data, a scheduler that determines which nodes pods are assigned to, and manages cluster resources. and an allocating controller, and a Domain Name System (DNS) server. However, pods installed on the master node will not be limited to the listed functions.

Pods installed on worker nodes generally include an agent that executes commands delivered from the API server of the master node, a proxy that manages the network between pods, and a runtime that actually runs the pods. However, pods installed on worker nodes will not be limited to the listed functions.

The service cluster 110 is a configuration that performs actual IaaS services. The service master pod 112 of the service cluster 110 belongs to the master node, and the service agent pod 111 and service API pod 113 belong to the worker node. Although only two service groups are shown in the drawing, it is apparent that a larger number of service groups may be provided.

The service API pod 113 running on the worker node is a pod that runs OpenStack API servers such as "Nova", "Neutron", "Keystone", "Cinder", "Glance", "Horizon", and "Octavia". . Using the aforementioned OpenStack API is just one example, and it will be apparent that other APIs performing the same function can be substituted. For example, "Neutron" is an API for networking control, and it is obvious that other networking control APIs can be applied instead of "Neutron".

The first and second service groups 200-1 and 200-2 shown in FIG. 2 include virtual machines 202-1 to 202-8 created by users.

Among the worker nodes constituting the service cluster 110, 'DHCP-agent', 'ovs (OpenVSwitch) Service Agent pods 111 such as -agent' and 'nova-compute' are installed. The installed service agent pod 111(s) supports the virtual machines 202-1 to 202-4 installed in the servers 201-1 to 201-4 to use resources such as computing and network. That is, the service agent pod 111(s) supports each virtual machine 202-1 to 202-8 to be controlled by the service API.

The LB cluster 120 is a configuration that performs load balancing. The LB master pod 122 belongs to the master node, and the LB node pod 121 belongs to the worker node.

The LB node pod 121 serves as at least one LB node (203-1, 203-2) (LBN, Load Balancer Node) that performs load balancing. The LB nodes 203-1 and 203-2 according to an embodiment of the present invention may be provided in a form in which only one pod is executed on one physical machine (server). This is to use the server's resources (cpu, memory) as a dedicated load balancing server that uses only load balancing resources. When referring to one LB node below, it is expressed as an LB node 203.

As shown in FIG. 2, the LB node pod 121 includes two LB nodes 203-1 and 203-2, but is not limited thereto. In particular, as will be described later, the number of LB nodes 203-1 and 203-2 may be increased or decreased in synchronization with CR (Custom resource) stored in the database of the LB master pod 122.

In the LB master pod 122, basic pod(s) for operating the LB cluster 120 are installed.

The LB control cluster 130 is a component that controls the LB cluster 12 . The LB control master pod 132 belongs to the master node, and the LB control node pod 131 belongs to the worker node.

In the LB control master pod 132, basic pod(s) for operating the LB control cluster 130 are installed.

The LB control node pod 131 may include an API server and an operator that controls the LB nodes 203-1 and 203-2. The API server and operator will be described in detail later with reference to FIG. 3 .

In one embodiment of the present invention, in order to implement a load balancer supporting multi-tenants, it is proposed to logically divide networks of different tenants through VLAN IDs (VIDs). According to this embodiment, even if VMs of different tenants have the same IP, traffic of different networks can be distinguished by VID, so multi-tenant support will be possible. As described above, in the present invention, an identifier corresponding to a specific tenant is used as a VLAN ID, but this is only a term for distinguishing tenants, and the present invention will not be limited to VLANs. For example, in the case of VXLAN, a case of using VNI (Virtual Network Identifier, or VXLAN Segment ID) instead of VID will also be included in the present invention.

Referring to FIG. 2, the first to fourth servers 201-1 to 201-4, which are computing nodes on which the first to eighth VMs 202-1 to 202-8 are executed, are two-layer leaf- It is connected to the top of rack (TOR) leaf switches 210-1 and 210-2 of the spine fabric 230. Although it is shown that only two servers are connected to each leaf switch, it is obvious that the number of servers may vary without being limited thereto.

The fabric configuration shown in FIG. 2 is described as an example of Virtual Extensible LAN (VXLAN)-Ethernet VPN (EVPN), but it will be apparent that various fabric configurations may be included in the present invention.

Also, each of the servers 201-1 to 201-4 may configure at least one virtual machine 202-1 to 202-8. Although the number of virtual machines 201-1 to 202-8 configured in each server 201-1 to 201-4 is shown as two in FIG. 2, it will not be limited thereto.

In each server (201-1 to 201-4, physical machine) constituting the load balancer management system 100 according to an embodiment of the present invention, one port (eth0) is the first TOR of the service network 220. and the second leaf switches 210-1 and 210-2.

Traffic of users received from the network 220 is a two-layer leaf-spine fabric composed of leaf switches 210-1 and 210-2, border leaf switches 212 and spine switches 211-1 and 211-2 ( 230) to move through. In Figure 2, the external router is omitted.

According to an embodiment of the present invention, when packets generated by the first to eighth VMs 202-1 to 202-8, which are computing nodes, go out of the node, the VLAN header along with the VID of the corresponding network is added to the original packet. can be added to In addition, since the leaf-spine fabric 230 is set to VXLAN (Virtual Extensible LAN)-EVPN (Ethernet VPN) or EVPN-VXLAN, a VXLAN header may be added to the original packet instead of a VLAN header. Each of the leaf switches 210-1 and 210-2 has a VXLAN Terminal End Point (VTEP).

Virtual Extensible LAN (VXLAN) is an encapsulation protocol that uses tunneling to provide data center connectivity that extends Layer 2 connectivity over the underlying Layer 3 network. VTEP is an entity that encapsulates and decapsulates packets, and is the start/end point of a VXLAN tunnel.

In order for the LB nodes 203-1 and 203-2 to perform load balancing, the following preliminary work is required. Each server connected to the border leaf switch 212 is labeled as being used for the purpose of the LB nodes 203-1 and 203-2. Each labeled server is joined to the LB cluster 120, and LB nodes 203-1 and 203-2 are installed in each server in the form of a pod.

Each of the LB nodes 203-1 and 203-2 thus installed is assigned to one of the LB node groups. 3 LB nodes are grouped into a group and handle the same VIP. For reference, when an LB node creation request comes in, 3 LB nodes are created by default. Although three LB nodes according to an embodiment of the present invention constitute one group, it is obvious that the number may be changed according to network conditions without necessarily being limited thereto.

External routers can evenly distribute traffic across the three LB nodes via the Equal-cost multi-path routing (ECMP) protocol. Using the ECMP protocol is only one example, and the present invention will not be limited thereto. Since each LB node and external router are configured to establish BGP peers, inbound traffic can be routed from the external router to each LB node.

In the detailed description of the present invention, for convenience, it is expressed as one LB node (203-1, 203-2), but this may mean a group of LB nodes.

3 and 4 below, operations of the LB nodes 203-1 and 203-2 according to an embodiment of the present invention will be described.

Figure 3 illustrates the concept of a control sequence for creating an additional LB node 203-3 according to one embodiment of the present invention. 4 shows a control flow chart for creating an additional LB node 203-3 according to one embodiment of the present invention.

The service API pod 113 operates the LB creation unit 311, which is an API for creating and controlling LB nodes. For example, the LB generator 311 may provide an API for creating and controlling an LB node based on 'Octavia API' among APIs provided by OpenStack. Creation of the LB node starts when the LB creation unit 311 receives an LB creation request (S301) from the user.

Upon receiving the S301 request, the LB generator 311 forwards a port creation request (S302) to be used by the additional LB node 203-3 to the 'network manager 312' of OpenStack. For example, the network management unit 312 may provide an API related to networking control based on 'Neutron API' among APIs provided by OpenStack.

The network management unit 312 creates a port in response to the port creation request (S302) (S303). The network manager 312 returns the VIP to be used in the additional LB node 203-3 to the LB generator 311 along with the port information generated in response (S304).

When the driver 313 included in the LB generator 311 receives the port information, forwards the LB creation request to the API-server 301 of the LB control node pod 131 along with the generated port information including the VIP (S305).

The CR management unit 302 included in the API server 301 is a component that creates, modifies, and deletes a CR (Custom Resource) based on a predefined CRD with respect to an LB node. Upon receiving the request of S305, the CR management unit 302 generates (S306) a third CR 350-3 based on the port information and VIP information included in the request of S305, and stores it in the database 303 (S306). S307).

In the database 303, the first and second CRs 350-1 and 350-2 corresponding to the existing first and second LB nodes 203-1 and 203-2, respectively, may be pre-stored. In addition, the third CR 350-3 generated by the CR management unit 302 may be added and stored. The database 303 according to an embodiment of the present invention may be etcd, which is a key-value store of the Kubernetes platform.

The operator 304 monitors the CR stored in the database 303 (S308).

The operator 304 monitors the CR stored in the database 303 and performs at least one of creation, deletion, and change of the corresponding LB node according to the creation, modification, and deletion of the CR. The operation of the operator 304 to continuously create, remove, and change the LB nodes 203-1 to 203-3 so as to correspond to the CRs stored in the database 303 is called "Reconciling".

As a result of monitoring in S308, when the third CR 350-3 stored in the database 303 is checked, the operator 304 creates the third LB node 203-3 (S309 or requests creation) .

Hereinafter, a control sequence for processing inbound traffic will be described with reference to FIG. 5 .

5 illustrates a control sequence in which the LB node 203 processes inbound traffic according to an embodiment of the present invention. The control sequence of FIG. 5 will be described with reference to FIG. 2 together. An example related to the flowchart of FIG. 5 is a scenario in which inbound traffic is directed to the first LB node 203-1 and forwarded to the fifth VM 202-5.

Inbound traffic directed to a VIP corresponding to a specific LB node 203 reaches the corresponding LB node 203 via an external router (not shown) and a border leaf switch 212 . In the scenario, the first LB node 203-1 receives a packet of inbound traffic toward its own VIP (S501).

The first LB node 203-1 performs hashing on the header information included in the received packet (S502). Based on the hashing result, the first LB node 203-1 obtains a MAC address and VID to be changed (S503). A detailed method of obtaining the MAC address and VID will be described later with reference to FIGS. 8 and 9 .

Subsequently, the first LB node 203-1 replaces the MAC address and VID included in the packet header with the MAC address and VID obtained in step S503 (S504).

6 illustrates an example in which a MAC address and a VID included in a packet header are changed according to an embodiment of the present invention. 6(a) shows header information 601-1 before change, and FIG. 6(b) shows header information 601-2 after change.

The header information 601-1 before change includes a VID 602-1 before change, a source MAC 603-1 before change, and a destination MAC 604-1 before change. In the above step S503, the first LB node 203-1 acquires the MAC address and VID to be changed by hashing the header information 601-1 before the change (S503).

First, the first LB node 203-1 changes the source MAC 603-1 before change to the destination MAC 604-1 before change. This is because the destination before the change becomes the new source after the change. Then, the first LB node 203-1 replaces the destination MAC 604-1 before the change with the MAC address obtained in step S503, and converts the destination MAC 604-2 after the change. That is, the destination MAC 604-2 after change will be the MAC of the fifth VM 202-5. Similarly, the first LB node 203-1 replaces the VID 602-1 before the change with the VID obtained in step S503 and converts it to the VID 602-2 after the change.

Returning to FIG. 5 again, the first LB node 203-1 returns a packet in which the MAC address and VID are replaced to the border leaf switch 212 (S504).

The border leaf switch 212 encapsulates the packet returned from the first LB node 203-1 (S505). As an example, the border leaf switch 212 can be encapsulated with VXLAN. And the border leaf switch 212 is a second leaf switch that is a TOR switch to which the destination MAC 604-2 after change is connected based on the destination MAC 604-2 after change included in the header of the returned packet. It is transferred to (210-2) (S506).

The second leaf switch 210-2 releases the encapsulation of the VXLAN head of the received packet, and forwards the packet to the fifth VM 202-5 corresponding to the destination MAC 604-2 after the change (S507 )do.

After confirming whether the loopback IP is the first LB node 203-1, the fifth VM 202-5 accepts the forwarded packet if the first LB node 203-1 is correct (S508). If not, the fifth VM 202-5 may reject the forwarded packet.

After processing the accepted packet, the fifth VM 202-5 may respond to the sender of the packet by exchanging the source MAC and the destination MAC (S509). That is, after exchanging the source MAC and the destination MAC, the fifth VM 202-5 returns the packet to the second leaf switch 210-2 (S510).

The second leaf switch 210-2 performs encapsulation (for example, VXLAN encapsulation) on the returned packet (S511) and transfers it to the border leaf switch 212 via the second spine switch 211-2 ( S512). The border leaf switch 212 cancels encapsulation (S513) and transmits the packet to the network 220 through an external router (S514).

According to the foregoing, the path of the packet according to an embodiment of the present invention is exported to the network 220 without passing through the LB node. This method is called Direct Server Return (DSR). According to the DSR method, there is an advantage that the load on the LB node is reduced because the packet does not pass through the LB node when leaving the data center.

Furthermore, the load balancer management system 100 according to an embodiment of the present invention proposes a method for achieving speed that does not lag behind the hardware type while the load balancer is implemented in a software form.

In one embodiment of the present invention, the first important matter to be considered is the location of the load balancer implemented in Linux, and the second is the implementation method. The location of the load balancer will be described with reference to FIG. 7 .

7 is a diagram illustrating a Linux network stack for explaining a location where a load balancer node is executed according to an embodiment of the present invention.

Looking at the network side of Linux from the lowest level to the highest level, in general, NIC (701, Network Interface Controller), device driver (702, Device Driver), traffic controller (703, Traffic Controller), netfilter (704, Netfilter), TCP Stack 705, socket layer 706 and userspace 707. The closer you get to the userspace 707 (the higher you go), the more functions (or codes) can be performed, and the more state information exists. On the other hand, the closer you get to the NIC 701 (the lower you go), the fewer functions (or codes) you can use, but faster performance can be expected. That is, when handling packets in Linux, performance improves as the data path through which packets are processed is performed in a lower layer.

In particular, when a packet is transmitted from the device driver 702 to the network stack, there is a problem in that a memory copy phenomenon that adversely affects performance occurs and that packets are unavoidably allocated to the sk_buff structure.

Therefore, the load balancer management system 100 according to an embodiment of the present invention proposes to perform load balancing in a relatively low network stack of Linux. Specifically, an embodiment of the present invention proposes to improve the performance of a forwarding plane that processes packets in a kernel by utilizing an eXpress Data Path (XDP) and an eBPF (Extended Berkeley Packet Filter) virtual machine.

The load balancer management system 100 according to an embodiment of the present invention utilizes eBPF/XDP to process packets in the device driver 702 layer before the packets are transferred to the networking layer of the kernel. BPF allows programs inserted into the kernel in the user space 707 to be executed in the form of a lightweight virtual machine. Those programs are hooked up to specific hooks in the kernel. eBPF refers to an extended version of classic BPF that additionally supports extended registers and instruction sets, and maps (key/value store with no size limit). XDP is a type of eBPF hook and refers to a set of commands that operate within the device driver 207.

8 shows an example of a hash table 801 according to one embodiment of the present invention. 9 shows the relationship between the LBN core 905 of the Linux kernel and the LBN controller 910 of the user space 707 according to an embodiment of the present invention.

The control unit 902 of the LBN controller 910 inserts a program into the LBN core 905 and updates the second hash table storage 903 used by the LBN core 905.

The first hash table storage 901 included in the LBN controller 910 means a storage for loading rules stored in the second hash table storage 903 of the LB node 203 . The update method of the first and second hash table storages 901 and 903 will be described later.

The LBN core 905 controls the overall operation of the LB node 203.

The second hash table storage 903 stores rules for the LB node 203 to perform load balancing. For example, when a packet of inbound traffic is received, the second hash table storage 903 stores a rule about which virtual machine to forward the received packet to. These rules may be stored in the form of a hash table 801 in the example of FIG. 8 .

The second hash table storage 903 according to an embodiment of the present invention may be implemented as a BPF map, which is a key/value storage in the kernel. The second hash table storage 903 can be updated only through the LBN controller 910 .

When adding or changing rules related to load balancing, first, the LBN controller 910 converts the rules stored in the second hash table storage 903 (in the form of a hash table as shown in the example of FIG. 8) to the first hash table storage ( 901) to load. In addition, the LBN controller 910 updates the rules loaded in the first hash table storage 901 to reflect the changes. Then, the LBN controller 910 may add or change a rule in the second hash table storage 903 by overwriting the updated rule. To this end, the LBN controller 910 according to an embodiment of the present invention may keep (Sync) the same rules stored in the first and second hash table storages 901 and 903 .

Hereinafter, a method of applying the rules stored in the second hash table storage 903 will be described.

As shown in step S501 of FIG. 5, when the packet is received by the LB node 203, the LBN core 905 of the received LB node 203 executes two hash functions to obtain target information (destination MAC and VID) do.

Referring to FIG. 8 (a) and (b), the hash table according to an embodiment of the present invention may include first and second hash tables 801 and 802. The first hash table 801 is used to find a location where the second hash table 802 is stored using a first hash function to be described later. The second hash table 802 is used to find the last virtual machine (VM) to which the packet is to be forwarded using the second hash function.

Referring to the second hash table 802 shown in FIG. 8 , a table for each VIP may exist separately. Hash table A (802A) for the first VIP (VIP 1), hash table B (802B) for the second VIP (VIP 2), and hash table C (802C) for the third VIP (VIP 3). can In the illustrated example, the hash tables A to C (802A to 802C) are divided into rows on one hash table 801, but this is only one example and stored in the form of a column or a separate table It could be.

Classification of VIPs on the second hash table 802 may be classification of services. For example, a first service may be assigned to a first VIP, and a second service may be assigned to a second VIP. That is, when multiple services are provided in one data center, individual VIPs can be assigned to each service. In addition, hash tables A to C (802A to 802C) corresponding to each assigned VIP may be provided.

Each item of the second hash table 802 includes destination MAC information of a target virtual machine (VM) to which traffic is to be delivered. For example, in the illustrated hash table A (802A), 'MAC A' is the MAC address for virtual machine A, 'MAC B' is the MAC address for virtual machine B, and 'MAC C' is the MAC address for virtual machine C. '. Similarly, 'MAC D' and 'MAC E' are included in hash table B (802B), and 'MAC F' and 'MAC G' are included in hash table C (802C).

First, the LBN core 905 of the LB node 203 receiving the packet applies the first hash function to the first hash table 801 to determine the memory address (one of 0, 1024, and 2048) of the corresponding VIP, , one of the hash tables A to C (802A to 802C) is selected based on the identified memory address.

Parameters of the first hash function include at least one of a destination IP (VIP), a destination port, a protocol, a VID, and a segment type. In one example, the segment type is set to VLAN, but is not necessarily limited thereto, and the segment type may be defined so that other protocols may be used instead of VLAN. Equation 1 is an example of the first hash function.

After one of the hash tables A to C (802A to 802C) is selected, the LBN core 905 applies a second hash function. Parameters of the second hash function include at least one of a 5-tuple (source IP, source port, destination IP, destination port, protocol) and VID of a packet header. Through the second hash function, the MAC address and VID of the last virtual machine (VM) to which the packet will be delivered can be obtained (S503 of FIG. 5 described above). In the example, each VIP hash table has the same VID, but virtual machines existing in the same service group may have different VIDs. Equation 2 is an example of the second hash function.

Meanwhile, in order for the load balancer management system 100 to perform efficient load balancing, a sorting method for the items of the hash table 801 as in the above-described example of FIG. 8 is important. This is because virtual machines are easily joined and deleted from the load-balancing target pool, and items stored in each hash table 801 are frequently replaced (deleted or added). Even if only information about one virtual machine is added or deleted from the hash table 801, the position of information about another virtual machine that already exists may be changed for reasons such as maintaining a ratio of items in the hash table 801.

In this way, when the hashing value of the hash function is changed and the target virtual machine for a specific VIP is changed, it is called “hashing disruption”. This "hashing disruption" can lead to unintentional termination of unrelated connections. For connections that are terminated unintentionally, an additional process is required to eventually re-establish the connection with another target. Therefore, an embodiment of the present invention proposes a hash table mechanism that supports minimal “hashing disruption”.

To this end, one embodiment of the present invention proposes to use the Maglev hashing mechanism, which is an improved version of the consistent hashing algorithm. This is because Maglev hashing allows the positions of existing entries to be changed as little as possible when new entries are added to the hash table 801 or when existing entries are deleted.

Following Maglev hashing, two processes, one called "Permutation" and one called "Population", place the entries of each virtual machine in a hash table. At this time, "Permutation" is a process of generating a two-dimensional array by obtaining a non-overlapping sequence for each item. And "Population" refers to the process of arranging seats fairly in order by item.

The number of virtual machine entries supported by the hash table 801 according to Maglev hashing in an embodiment of the present invention may be defined as MAX_VIPS * (MAX_VMS * FR). MAX_VIPS is the number of VIPs supported by one LBN. The default value is 4096 and can be changed according to the network condition. MAX_VMS refers to the number of virtual machines that can be processed per VIP. FR is a parameter that is multiplied by MAX_VMS, and as this value increases, “hashing disruption” decreases and can be used to assign a weight to each VIP.

- Experimental proof of load balancer performance according to an embodiment of the present invention

The performance of an LBN implemented in software can determine how close it is to the packet rate in the case of a direct wired connection. Depending on the degree of proximity, it will be possible to judge whether the LB can be used in the actual network. If it is similar to the case where it is directly connected by wire (hereinafter referred to as “Loopback” in the experimental results shown in FIGS. 11 to 15), it can be regarded as satisfactory performance as an LBN.

To this end, in this experiment, the performance of LBN was measured through the RFC2544 standard, which is widely used to measure the performance of physical devices such as firewalls and routers. This standard specifies the performance test conditions so that the performance of the company's products can be accurately compared. In addition to RFC2544, tests similar to real traffic patterns called IMIX were also conducted to meet the characteristics of data centers.

IMIX refers to a traffic pattern in which frames included in traffic are mixed at a ratio determined by frame size. Therefore, IMIX traffic can be seen as traffic that mimics traffic patterns that can occur in actual cloud networks.

In the following, the property values set in the experiment are explained in detail, and then the performance of the RFC2544 test and the IMIX test is explained in order.

10 shows the experimental environment of this experiment.

A. Experiment environment setup

As defined in RFC2544, the experimental environment was configured such that one switch (1002) is connected to two servers (1001-1 and 1001-2). The tester server 1001-1 serves as a tester, and the DUT server 1001-2 serves as a device under test (DUT). For each test scenario (scenario A and scenario B), the tester server 1001-1 generates a packet and sends it to the switch 1002, and the DUT server 1001-2 receives the packet and obtains a performance indicator such as "throughput". It consists of

In this experiment, "TRex", an open source traffic generator, was installed to measure the performance of the tester server (1001-1). "TRex" is a software instrument that realizes low cost and high performance by utilizing Intel's DPDK, and is characterized by generating stateful and stateless traffic. "TRex" allows you to write scenarios in Python scripts to perform tests that comply with RFC standards. Not only that, but it can perform tests similar to commercial physical tester equipment.

Among the cores of the tester server 1001-1, all CPU cores except for the minimum core for executing the tester server 1001-1 are configured for “TRex” packet transmission. An LBN is installed in the DUT server 1001-2 to change the header of the packet received from the switch 1002 into the MAC address and VID of the tester server 1001-1 and forward it to the switch 1002.

Both the tester server (1001-1) and the DUT server (1001-2) used a 2.10GHz CPU (total of 16 cores, 11MB L3 cache) and four 32GB RAMs. And Ubuntu 20.04 with kernel 5.4 was installed on each server (1001-1, 1001-2). The NIC card used for each server 1001-1 and 1001-2 is a Mellanox dual port 25G. One port of the tester server 1001-1 was configured to perform RX only and the other port to perform TX only. On the other hand, the DUT server 1001-2 has only one port, and the corresponding port performs both RX and TX. I turned off the kernel's rxvlan and txvlan offload features on both servers (1001-1 and 10001-2) to see packets with VLAN headers at the kernel level. In the tester server (1001-1), MLX5 poll mode driver, which is a driver for using Mellanox's NIC card as DPDK, was installed.

Iptables, part of the Linux Netfilter project, provides packet filtering and control (NAT, etc.) functions. Among these, DNAT is simply a function that can be used for load balancing. That is, traffic can be load balanced using multiple DNAT rules that change the destination address of incoming packets. For this reason, Iptables DNAT was selected as a comparison target to verify the performance of LBN. In addition, a loopback test was performed by directly connecting the TX port and RX port of the test server, and this performance was set as the upper limit of the load balancing performance.

LBN performs load balancing with multiple virtual machines (VMs) per VIP. Therefore, in this experiment, we set the number of VIPs and the number of virtual machines as parameters that can be modified.

In addition, two scenarios in which LBN is used are set. The first is a scenario load balancing 1 VIP and 255 virtual machines (scenario A in FIG. 10), and the other is a scenario load balancing 128 VIPs and 4000 virtual machines (scenario B in FIG. 10) to be. In the case of testing for multiple VIPs, traffic was randomly selected from the range of VIPs to be sent. The range of VIPs sent by the tester server (1001-1) is equal to the range of the VIP list of rules added to the LBN's hash table. LBNs belonging to the same group have the same rules. Since this test is for L2DSR, the destination IP of the traffic does not change, but the VID and destination MAC address are changed to the VID and MAC address of one of the relevant actual destination virtual machines. To create Iptables DNAT that does the same thing as LBN, we applied the same rules for 128 VIPs to Iptables DNAT. The switch 1002 connecting the tester server 1001-1 and the DUT server 1001-2 simply performs a bridge role of flooding incoming packets to a port toward the other connected server. In the case of a typical L2DSR configuration, each port on the switch connected to the LBN server is set to trunk mode to allow traffic connected to various VLANs, but in the experimental environment, the ports are set to access mode to allow only certain VLANs to transmit and receive. It became.

B. Performance testing according to RFC2544

In this section, we present the settings of experiments performed according to the RFC2544 standard and show the throughput (units of BPS and PPS).

A total of 7 types of frame sizes (64, 128, 256, 512, 1024, 1280, 1510 (byte)) were tested in this experiment. Specifically, for each frame size, four types of tests were performed:

1) loopback

2) DNAT from Iptables

3) Scenario A for LBN (#VIP = 1, #Real-VM = 255)

4) Scenario B for LBN (#VIP = 128, #Real-VM = 4,000)

In one test trial, UDP traffic of each frame size was transmitted for 60 seconds at the maximum speed of a 25G NIC card, and the average throughput of three trials was used as the final result. The interval between each trial was set at 30 s so as not to interfere with each other between trials.

11 to 13 show the throughput of each scenario when traffic consisting only of frames of a specific size is transmitted at full speed of a 25G NIC card.

In particular, FIGS. 11 and 12 show the throughput of the BPS unit at the L2 and L1 levels, respectively. 13 shows the throughput of the PPS unit at the L2 level. From the results for all frame sizes, the performance of Loopback can be accepted as the best performance that LB can reach.

11 and 12, it can be seen that the smaller the frame size, the higher the frame processing overhead and the lower the BPS. The result of FIG. 11 is the result after removing all of the headers of layer 1 and 2 of the frame, and the result of FIG. 12 is the result of removing only the header of layer 1 of the frame, so FIG. 12 will have better performance.

For a given number, we can interpret the test results in three ways.

1) How close the performance of LBN is to loopback

2) What is the performance difference between Iptables DNAT and LBN?

3) What is the difference in performance when the number of rules applied to LBN is different?

In the case of BPS for L1 and L2 (FIGS. 11 and 12), when the frame size is 64, the difference between loopback and LBN is less than about 5 to 6 Gbps. That is, the LBN is about 24% lower than the actual physical maximum performance. When the frame size is 128, the difference is only about 3%, and it can be seen that the performance is almost the same for larger frame sizes.

Excluding the unavoidable performance degradation that occurs to handle the smallest frame size of 64, the performance of LBN proves to be remarkably high. Even in the worst performing case (frame size of 64), LBN provides 16x better performance than Iptables DNAT. When the difference is smallest (when the frame is 1,510), the difference is about double.

Comparing the two scenarios with different number of rules for LBN, in the worst-performing case (with a frame size of 64), the performance decreases by about 4.2% as the number of rules increases. However, it can be seen that the number of rules does not significantly affect the performance, as there is little difference for different frame sizes.

FIG. 13 shows the results of FIG. 11 in units of PPS. Contrary to BPS, it can be seen that the smaller the frame size, the higher the PPS because more packets are transmitted. When the frame size is 64, the difference between loopback and LBN is less than about 7 to 8 Mpps. Percentage values for the above three aspects will be the same as those mentioned in relation to FIGS. 11 and 12 .

C. Performance tests according to IMIX

14 shows throughput results in units of BPS (L2 and L1 levels) for IMIX traffic. 15 shows Troughput results in units of PPS for IMIX traffic.

IMIX's size ratio has changed over time. However, there are commonly used ratios for each frame size. The ratios of each frame size used in the experiment are 60%, 20% and 20% for frames with frame sizes of 64, 512 and 1510, respectively. Since the test was performed by mixing traffic of various frame sizes, it can be seen that it is similar to the actual network environment. Referring to Fig. 14, the difference in BPS measured at the L2 level between loopback and LBN is less than about 0.7 Gbps. This means that the LBN is only about 2% below the theoretical maximum performance. And LBN provides 27 times better performance than Iptables DNAT. When comparing two scenarios with different numbers of rules for LBN, it can be seen that there is almost no difference. Similar to the RFC255 test, we can see that the number of rules does not significantly affect the performance of LBN.

Above, the embodiment of the control method of the load balancer management system according to the present invention has been described, but this is described as at least one embodiment, whereby the technical idea of the present invention and its configuration and operation are not limited, The scope of the technical idea of the present invention is not limited / limited by the drawings or the description referring to the drawings. In addition, the concepts and embodiments of the present invention presented in the present invention can be used by those skilled in the art as a basis for modifying or designing other structures to achieve the same purpose of the present invention. , Modified or changed equivalent structure by a person skilled in the art to which the present invention belongs is bound by the technical scope of the present invention described in the claims, and does not depart from the spirit or scope of the invention described in the claims. Various changes, substitutions and modifications are possible within the limits.

Claims (21)

In the control method of a system that manages a load balancer (LB, Load Balancer) in a cloud-native environment,
Based on the load balancer node creation request, CR management unit creating a custom resource (CR, Custom Resource);
The user-specified resource includes at least one parameter information defining a load balancer,
CR management unit storing the created user-specified resource in a database; and
Including the step of the operator performing at least one of creating, removing, and changing at least one load balancer node (LB Node) to correspond to the stored at least one user-specified resource,
How to control the load balancer management system.
According to claim 1,
Further comprising allocating a VIP (Virtual IP) to the load balancer to which the VIP allocation unit is to be added.
How to control the load balancer management system.
According to claim 1,
The generated load balancer is implemented in the device driver (DD) stack of the Linux network stack.
How to control the load balancer management system.
According to claim 3,
The load balancer is implemented as an Extended Berkeley Packet Filter (eBPF) / eXpress Data Path (XDP),
How to control the load balancer management system.
In a load balancer management system that manages a load balancer (LB, Load Balancer) in a cloud-native environment,
CR management unit for creating a custom resource (CR, Custom Resource) based on the load balancer node creation request;
The user-specified resource includes at least one parameter information defining a load balancer,
a database for storing the created user-specified resource; and
Including an operator that performs at least one of creating, removing, and changing at least one load balancer node (LB Node) to correspond to the stored at least one user-specified resource,
Load balancer management system.
According to claim 5,
Further comprising a VIP allocation unit for allocating a VIP (Virtual IP) to the load balancer to be added,
Load balancer management system.
According to claim 5,
The generated load balancer is implemented in the device driver (DD) stack of the Linux network stack.
Load balancer management system.
According to claim 7,
The load balancer is implemented as an Extended Berkeley Packet Filter (eBPF) / eXpress Data Path (XDP),
Load balancer management system.
In the control method of the load balancer node,
LBN core receiving a packet from the border leaf switch;
applying, by the LBN core, first and second hash functions to the received packet;
obtaining at least one of a destination address and a VID from a hash table based on a result of applying the first and second hash functions by the LBN core;
replacing, by the LBN core, at least one of the destination address and VID of the received packet with the obtained destination MAC address and VID;
Transmitting the replaced packet by the LBN core to the border leaf switch,
How to control load balancer nodes.
According to claim 9,
The first and second hash functions are applied to the header of the received packet,
How to control load balancer nodes.
The method of claim 9, wherein the parameters of the first hash function,
Including at least one of a destination IP (VIP), destination port, protocol, VID, and segment type for the received packet,
How to control load balancer nodes.
The method of claim 9, wherein the parameter of the second hash function,
At least one of a source IP, a source port, a destination IP, a destination port, a protocol, and a VID of a packet header for the received packet,
How to control load balancer nodes.
The method of claim 9, wherein the acquiring step,
Acquiring, by the LBN core, a Virtual IP (VIP) based on a first hashing result of the first hash function,
How to control load balancer nodes.
The method of claim 13, wherein the acquiring step,
The LBN core further comprising obtaining a destination MAC address based on the second hashing result of the second hash function and the first hashing result.
How to control load balancer nodes.
For load balancer nodes,
hash table storage to store hash tables; and
Including an LBN core for receiving packets from a border leaf switch,
The LBN core,
Applying first and second hash functions to the received packet;
Obtaining at least one of a destination address and a VID on the stored hash table based on a result of applying the first and second hash functions;
Replace at least one of the destination address and VID of the received packet with the obtained destination MAC address and VID;
The LBN core transmits the replaced packet to the border leaf switch,
load balancer node.
According to claim 15,
The first and second hash functions are applied to the header of the received packet,
load balancer node.
16. The method of claim 15, wherein the parameters of the first hash function,
Including at least one of a destination IP (VIP), destination port, protocol, VID, and segment type for the received packet,
load balancer node.
16. The method of claim 15, wherein the parameters of the second hash function,
At least one of a source IP, a source port, a destination IP, a destination port, a protocol, and a VID of a packet header for the received packet,
load balancer node.
According to claim 15,
The LBN core obtains a VIP (Virtual IP) based on the first hashing result of the first hash function.
load balancer node.
According to claim 19,
The LBN core obtains a destination MAC address based on the second hashing result and the first hashing result of the second hash function.
load balancer node.
A computer program stored in a medium to be combined with hardware to execute the method of any one of claims 1 to 4.

Images