WO2023165137A1

WO2023165137A1 - Cross-cluster network communication system and method

Info

Publication number: WO2023165137A1
Application number: PCT/CN2022/124573
Authority: WO
Inventors: 王琨; 赵建星; 樊建刚; 牛丽; 田文杰
Original assignee: 京东科技信息技术有限公司
Priority date: 2022-03-02
Filing date: 2022-10-11
Publication date: 2023-09-07
Also published as: CN114640556A

Abstract

The present disclosure provides a cross-cluster network communication system and method. The network communication system comprises: a first cluster used for determining a target address of an original data packet as a second cluster, and encapsulating the original data packet to generate an encapsulated data packet; a first gateway component provided corresponding to the first cluster and used for receiving the encapsulated data packet sent by the first cluster; a second gateway component provided corresponding to the second cluster and used for receiving the encapsulated data packet sent by the first gateway component and sending the encapsulated data packet to the second cluster; and the second cluster used for receiving the encapsulated data packet sent by the second gateway component and de-encapsulating the encapsulated data packet to obtain the original data packet. Communication among different clusters is realized.

Description

A cross-cluster network communication system and method

Cross References to Related Applications

This disclosure claims priority to Chinese Patent Application No. 2022102001198, entitled "A Cross-Cluster Network Communication System and Method," filed with the China Intellectual Property Office on March 2, 2022, which is incorporated by reference The entire content of the Chinese patent application disclosure is incorporated herein.

technical field

The present disclosure relates to the technical field of cloud platforms, and in particular to a cross-cluster network communication system and method.

Background technique

Cloud Native provides powerful container orchestration capabilities, opens up network interfaces, and supports customized and flexible container networks. In the cloud-native network solution, each container group has an independent IP address, and the container group runs on a flat network, and the containers can be directly connected to each other. A container group network interface protocol defines the interface specification of the container network. The container network can be configured through a custom plug-in. However, the existing container group network interface protocol can only realize the communication between container groups in a single cluster. Consider the intercommunication of multi-cluster container groups.

Contents of the invention

The present disclosure provides a cross-cluster network communication system and method, which realize communication between different clusters.

According to a first aspect of the present disclosure, a cross-cluster network communication system is provided, and the network communication system includes:

The first cluster is configured to determine the target address of the original data packet as the second cluster, and encapsulate the original data packet to generate an encapsulated data packet;

The first gateway component is set corresponding to the first cluster, and is used to receive the encapsulated data packet sent by the first cluster;

The second gateway component is set corresponding to the second cluster, and is used to receive the encapsulated data packet sent by the first gateway component, and send the encapsulated data packet to the second cluster;

The second cluster is configured to receive the encapsulated data packet sent by the second gateway component, and decapsulate the encapsulated data packet to obtain the original data packet.

According to the cross-cluster network communication system provided by the present disclosure, the network communication system further includes:

The first target node set in the first cluster is used to encapsulate the original data packet, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway component through the sending port;

The second target node set in the second cluster is used to receive the encapsulated data packet sent by the second gateway component through the receiving port, and decapsulate the encapsulated data packet to obtain the original data packet;

Wherein, the source address of the original data packet is the first destination node, and the destination address of the original data packet is the second destination node.

According to the cross-cluster network communication system provided by the present disclosure, wherein:

A first target node is set in the first cluster, a second target node is set in the second cluster, and the first target node includes a first target container group;

The source address of the original data packet is the first target container group, and the target address of the original data packet is the second target node;

The first target container group is used to encapsulate the original data packet, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway component through the sending port of the first target node;

The second target node is configured to receive the encapsulated data packet sent by the second gateway component through the receiving port, and decapsulate the encapsulated data packet to obtain the original data packet.

A first target node is set in the first cluster, a second target node is set in the second cluster, and the second target node includes a second target container group;

The source address of the original data packet is the first target node, and the target address of the original data packet is the second target container group;

The first target node is used to encapsulate the original data packet, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway component through the sending port;

The second target container group is used to receive the original data packet sent by the second target node according to the target address, wherein the original data packet sent by the second target node is the encapsulation data sent by the second gateway component received by the second target node through the receiving port packet, which is obtained by decapsulating the encapsulated data packet.

A first target node is set in the first cluster, a second target node is set in the second cluster, the first target node includes a first target container group, and the second target node includes a second target container group;

The source address of the original data packet is the first target container group, and the target address of the original data packet is the second target container group;

The first gateway route is set between the first cluster and the first gateway component, and is used to send the encapsulated data packet to the first gateway component corresponding to the first cluster;

The second gateway route is set between the first gateway component and the second gateway component, and is used to send the encapsulated data packet in the first gateway component to the second gateway component corresponding to the second cluster; and

The third gateway route is set between the second gateway component and the second cluster, and is used to send the encapsulated data packet in the second gateway component to the second cluster.

According to a second aspect of the present disclosure, there is provided a cross-cluster network communication method for the network communication system described in any one of the above, the method comprising:

Determining the destination address of the original data packet as the second cluster by the first cluster;

Encapsulate the original data packet through the first cluster to generate an encapsulated data packet;

sending the encapsulated data packet to the first gateway component corresponding to the first cluster through the first cluster;

Send the encapsulated data packet to the second gateway component corresponding to the second cluster through the first gateway component, so that the second gateway component sends the encapsulated data packet to the second cluster, and decapsulates the encapsulated data packet through the second cluster , to get the original packet.

According to the cross-cluster network communication method provided by the present disclosure, the first cluster includes the first target node, and the second cluster includes the second target node; the source address of the original data packet is the first target node, and the target address of the original data packet is the second target node. Two target nodes;

The method also includes:

encapsulating the original data packet by the first target node, generating the encapsulated data packet, and sending the encapsulated data packet to the first gateway component via the sending port of the first target node;

The encapsulated data packet sent by the second gateway component is received through the receiving port of the second target node, and the encapsulated data packet is decapsulated to obtain the original data packet.

According to the cross-cluster network communication method provided by the present disclosure, the first cluster includes the first target node, the first target node includes the first target container group, and the second cluster includes the second target node; the source address of the original data packet is the first The target container group, the target address of the original data packet is the second target node;

The method also includes:

Encapsulating the original data packet through the first target container group, generating the encapsulated data packet, and sending the encapsulated data packet to the first gateway component via the sending port of the first target node;

The encapsulated data packet sent by the second gateway component is received through the receiving port of the second target node, and the encapsulated data packet is decapsulated to obtain an original data packet.

According to the cross-cluster network communication method provided in the present disclosure, the first cluster includes a first target node, the second cluster includes a second target node, and the second target node includes a second target container group; The source address of the original data packet is the first target node, and the target address of the original data packet is the second target container group;

The method also includes:

encapsulating the original data packet by the first target node, generating the encapsulated data packet, and sending the encapsulated data packet to the first gateway component through a sending port;

The original data packet sent by the second target node according to the target address is received through the second target container group, wherein the original data packet sent by the second target node is that the second target node receives the The encapsulated data packet sent by the second gateway component is obtained by decapsulating the encapsulated data packet.

According to the cross-cluster network communication method provided in the present disclosure, the first cluster includes a first target node, the first target node includes a first target container group, the second cluster includes a second target node, and The second target node includes a second target container group; the source address of the original data packet is the first target container group, and the target address of the original data packet is the second target container group;

The method also includes:

Encapsulate the original data packet through the first target container group, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway component through the sending port of the first target node;

According to a third aspect of the present disclosure, there is provided an electronic device, including a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the program, any of the above-mentioned The cross-cluster network communication method described in the item.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the cross-cluster network communication method as described in any one of the above is implemented. .

According to a fifth method of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the cross-cluster network communication method as described in any one of the above is implemented.

The present disclosure provides a cross-cluster network communication system and method. The target address of the original data packet is determined as the second cluster through the first cluster, and the first gateway component corresponding to the first cluster cannot directly identify the original data packet. Therefore, Encapsulating the original data packet enables the first gateway component to receive the encapsulated original data; through the first gateway component, the encapsulated data packet is sent again to the second gateway component corresponding to the second cluster, and finally the second gateway component will encapsulate The data packet is sent to the second cluster, and the second cluster decapsulates the encapsulated data packet to obtain the original data packet. With the help of the first gateway component corresponding to the first cluster and the second gateway component corresponding to the second cluster, the communication between different clusters is realized. communication between.

Description of drawings

In order to more clearly illustrate the technical solutions in the present disclosure or the prior art, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are the present invention For some disclosed embodiments, those skilled in the art can also obtain other drawings based on these drawings without any creative work.

FIG. 1 is one of the schematic structural diagrams of a cross-cluster network communication system according to an embodiment of the present disclosure;

FIG. 2 is the second schematic structural diagram of a cross-cluster network communication system according to an embodiment of the present disclosure;

Fig. 3 is a third structural schematic diagram of a cross-cluster network communication system according to an embodiment of the present disclosure;

FIG. 4 is a fourth schematic structural diagram of a cross-cluster network communication system according to an embodiment of the present disclosure;

FIG. 5 is a fifth schematic structural diagram of a cross-cluster network communication system according to an embodiment of the present disclosure;

6 is a schematic diagram of the principles of the Cni program and the Agent program according to an embodiment of the disclosure;

7 is a block diagram of Pod communication between Cluster1 and Cluster2 according to an embodiment of the present disclosure;

FIG. 8 is a schematic flowchart of a cross-cluster network communication method according to an embodiment of the present disclosure;

Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments It is a part of the embodiments of the present disclosure, but not all of them. Based on the embodiments in the embodiments of the present disclosure, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the embodiments of the present disclosure.

Cloud native is a software development technique that takes full advantage of cloud computing, deploying applications as microservices using a software technology stack. In existing technologies, cloud-native applications are built as a set of microservices running in Docker containers, orchestrated in Kubernetes, and managed and deployed using DevOps and GitOps workflows. The advantage of using Docker containers is the ability to package all the software and environment configuration required for execution into a single executable package. Containers run in a virtualized environment, isolating the contained application from its environment.

Kubernetes, abbreviated as K8s, is a container cluster management system initiated and maintained by the Google team. The underlying layer is based on container technologies such as Docker and Rkt, providing powerful application management and resource management and scheduling capabilities. K8s has complete cluster management capabilities, including multi-level security protection and access mechanisms, multi-tenant application support capabilities, transparent service registration and service discovery mechanisms, built-in intelligent load balancers, powerful fault discovery and self-healing functions, Service rolling upgrade and online expansion capabilities, scalable resource automatic scheduling mechanism, and multi-granularity resource allocation quota management capabilities. At the same time, K8s provides comprehensive management tools that cover all aspects including development, test deployment, and operation and maintenance monitoring. Therefore, K8s is a brand-new distributed architecture solution based on container technology, and it is a one-stop, complete distributed system development and support platform.

In the existing technology, there are two types of communication between Pods in a single cluster, one is the Underlay network, and all Pods are directly connected by a three-layer network or a two-layer network, such as Calico's Bgp network; the other is Overlay In the network, communication between Pods requires an encapsulation/decapsulation process, such as Calico's IPIP network and Flannel's Vxlan network.

For the Underlay network, all Nodes and Pods are on the Layer 3 or Layer 2 network plane, and cross-cluster communication can be realized directly through the routing configuration of the switch; for the Overlay network, PodIP is a private network address, and the switches between Nodes will not have Pods For the routing of the network segment, the communication between Pods needs to encapsulate the corresponding Overlay header, find the route to the destination host according to the outer header, and then send it to the destination Pod after decapsulation.

However, this communication method is suitable for a single cluster. For different clusters, Pods have their own private network addresses. A solution is needed to open up cross-cluster private networks. Therefore, embodiments of the present disclosure provide a cross-cluster For the network communication system between Pods, the Overlay network is selected. Specifically, the Overlay network is to build a virtual network on top of the existing physical network. The upper-layer application is only related to the virtual network. Under the condition of no large-scale modification, the bearing of the application on the network can be realized, and it can be separated from other network services, and the basic network technology based on IP is the main one.

Embodiments of the present disclosure provide a cross-cluster network communication system. The network communication system includes multiple clusters, and each cluster is correspondingly provided with a gateway component.

Cluster (Cluster) is a collection of computing, storage and network resources, K8s uses these resources to run various container-based applications. Each component point of the cluster is called a node (Node), and the combination of nodes forms a cluster. The responsibility of Node is to run the container application. Node is responsible for monitoring and reporting the status of the container, and managing the life cycle of the container according to the requirements of the Cluster. Node runs on the Linux operating system, which can be a physical machine or a virtual machine.

The gateway device refers to Gateway, abbreviated as (GW). Gateway is the gateway service framework under the SpringCloud technology stack. In the SpringCloud-based microservice environment, external requests will reach the Gateway, and the Gateway forwards, filters, authenticates, and Fusing and other pre-operations.

The Gateway running in the Kubernetes environment can get the Kubernetes Service list if the Spring-Cloud-Kubernetes framework is used. Therefore, the Gateway can assume the role of a gateway and forward external requests to the Services in Kubernetes.

Referring to FIG. 1 , it is one of the schematic structural diagrams of a cross-cluster network communication system provided by an embodiment of the present disclosure, including:

The first cluster 11 is configured to determine the target address of the original data packet as the second cluster, and encapsulate the original data packet to generate an encapsulated data packet.

Specifically, the first cluster is represented by Cluster1, the second cluster is represented by Cluster2, and the original data packet refers to the data packet in Cluster1.

Encapsulation refers to hiding the attributes and implementation details of the original data packet, only exposing the interface to the outside world, and controlling the access level of reading and modifying attributes in the program. In the embodiments of the present disclosure, encapsulation refers to the outside of the original data packet Add a packet header of the Overlay network. Correspondingly, the message header can be understood as some information segments. The message is the unit sent by the network. During the transmission process, it will be continuously encapsulated into groups, packets, and frames for transmission. The way of encapsulation is to add some information segments. The added information segments It is the header.

Correspondingly, IP tunneling technology is used to encapsulate the original data packet. IP tunneling technology is a process in which a router encapsulates one network layer protocol into another protocol for transmission across the network to another router.

IP tunneling technology is a data packet encapsulation technology, which encapsulates the original IP packet (whose header contains the original sender and final destination) in the data payload of another data packet (called encapsulated IP packet) for transmission .

The original data packet is encapsulated through the IP tunneling technology, and the encapsulated data packet is called an encapsulated data packet.

The first gateway component 12 is set corresponding to the first cluster, and is used for receiving the encapsulated data packet sent by the first cluster.

Specifically, the first gateway component refers to the Gateway1 correspondingly set in the first cluster, and sends the encapsulated data packet to the Gateway1 set in the first cluster.

Correspondingly, one or more gateway components can be set in the first cluster, and when the scale of the cluster is large, it supports horizontal expansion to add gateway components to achieve load balancing.

The second gateway component 13 is set corresponding to the second cluster, and is used for receiving the encapsulated data packet sent by the first gateway component, and sending the encapsulated data packet to the second cluster.

Specifically, the second gateway component refers to Gateway2 set in the second cluster, and similarly, one or more gateway components may also be set in the second cluster.

Correspondingly, Gateway1 of the first cluster sends the encapsulated data packet to Gateway2 of the second cluster, and then Gateway2 of the second cluster sends the encapsulated data packet to the second cluster.

The second cluster 14 is configured to receive the encapsulated data packet sent by the second gateway component, and decapsulate the encapsulated data packet to obtain the original data packet.

Specifically, decapsulation is the inverse process of encapsulation, and the header of the added Overlay network is removed to obtain the original data.

Correspondingly, the encapsulated data packet is decapsulated by Node2 in Cluster2 to obtain the original data packet, that is, the data packet in Cluster1 is obtained.

The present disclosure provides a cross-cluster network communication system. The target address of the original data packet is determined as the second cluster through the first cluster. The first gateway component corresponding to the first cluster cannot directly identify the original data packet. Therefore, the original Encapsulation of the data packet enables the first gateway component to receive the encapsulated original data; the encapsulated data packet is sent to the second gateway component corresponding to the second cluster through the first gateway component again, and finally the second gateway component will encapsulate the data packet sent to the second cluster, the second cluster decapsulates the encapsulated data packet to obtain the original data packet, and realizes the communication between different clusters by means of the first gateway component corresponding to the first cluster and the second gateway component corresponding to the second cluster communication.

Referring to FIG. 2 , the second schematic structural diagram of a cross-cluster network communication system provided by an embodiment of the present disclosure includes:

The first target node 21 set in the first cluster is used to encapsulate the original data packet, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway component 22 through the sending port;

The second target node 24 arranged in the second cluster is used to receive the encapsulated data packet sent by the second gateway component 23 through the receiving port, and decapsulate the encapsulated data packet to obtain the original data packet;

Specifically, the cluster includes multiple nodes, and the nodes are represented by Node, the first target node is represented by Node1, and the second target node is represented by Node2.

Correspondingly, Node1 encapsulates the original data packet, generates an encapsulated data packet, and sends the encapsulated data packet to Gateway1 through the sending port, and Node2 receives the encapsulated data packet sent by Gateway2 through the receiving port, and decapsulates the encapsulated data packet to obtain the original data pack.

The cross-cluster network communication system provided by the embodiments of the present disclosure implements Overlay network communication between nodes in different clusters.

Referring to FIG. 3 , the third schematic structural diagram of a cross-cluster network communication system provided by an embodiment of the present disclosure includes:

The first target container group 31 is used to encapsulate the original data packet, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway component 32 via the sending port of the first target node;

The second target node 34 is configured to receive the encapsulated data packet sent by the second gateway component 33 through the receiving port, and decapsulate the encapsulated data packet to obtain the original data packet.

Specifically, a container group refers to a Pod, which is a basic management unit in K8s, rather than a container (Container). A Pod is a layer of encapsulation of K8s on a container. Container composition.

Pod can support multiple containers to share network addresses and file systems in a Pod, and services can be combined and completed in simple and efficient ways such as inter-process communication and file sharing. When a Pod contains multiple containers, these containers always run on the same On the working node (Node), a Pod will never span multiple working nodes.

Correspondingly, the first target container group refers to Pod1 in Node1. Pod1 encapsulates the original data to generate an encapsulated data packet, and Pod1 sends the encapsulated data packet to Gateway1 corresponding to Cluster1.

Referring to FIG. 4 , the fourth schematic structural diagram of a cross-cluster network communication system provided by an embodiment of the present disclosure includes:

The first target node 41 is configured to encapsulate the original data packet, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway component 42 through the sending port;

The second target container group 44 is used to receive the original data packet sent by the second target node according to the target address, wherein the original data packet sent by the second target node is received by the second target node through the receiving port and sent by the second gateway component 43 The encapsulated data packet is obtained by decapsulating the encapsulated data packet.

Specifically, the second target container group refers to Pod2 in Node2. Send the encapsulated data packet in Gateway1 to Gateway2 corresponding to the second cluster, and Gateway2 sends the encapsulated data packet to Pod2 of Node2 in the second cluster. Pod2 decapsulates the encapsulated data packet to obtain the original data packet.

The cross-cluster network communication system provided by the embodiments of the present disclosure realizes Overlay network communication between container groups and nodes in different clusters.

Referring to FIG. 5 , the fifth structural diagram of a cross-cluster network communication system provided by an embodiment of the present disclosure includes:

The first target container group 51 is used to encapsulate the original data packet, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway component 52 via the sending port of the first target node;

The second target container group 54 is used to receive the original data packet sent by the second target node according to the target address, wherein the original data packet sent by the second target node is received by the second target node through the receiving port and sent by the second gateway component 53 The encapsulated data packet is obtained by decapsulating the encapsulated data packet.

Specifically, a first target node Node1 is set in the first cluster Cluster1, Pod1 is set in Node1, a second target node Node2 is set in the second cluster Cluster2, and Pod2 is set in Node2.

The cross-cluster network communication system provided by the embodiments of the present disclosure implements Overlay network communication between container groups in different clusters.

Based on any of the above embodiments, the system also includes:

The second gateway route is set between the first gateway component and the second gateway component, and is used to send the encapsulated data packet in the first gateway component to the second gateway component corresponding to the second cluster;

It can be understood that routing refers to the process in which a router receives a data packet from one interface, directs it according to the destination address of the data packet, and forwards it to another interface. In this step, the encapsulated data packet is sent from the sending node to the first gateway component.

Specifically, the first gateway route may be understood as route 1 pointing to Gateway1, and route 1 is generated by the Route-Controller. A route pointing to Gateway1 is determined by the sending node in the first cluster, and the encapsulated data packet is sent to the first gateway component, ie, Gateway1, based on the route of Gateway1.

The second gateway route is route 2 pointing to Gateway2, and route 2 is generated by Route-Controller. Send the encapsulated data packet in Gateway1 to Gateway2 corresponding to the second cluster based on route 2.

The third route is route 3 pointing to the second cluster (which can be understood as pointing to the receiving node Node2 in the second cluster), and the route 3 is generated by the Node-Controller. Gateway2 sends the encapsulated data packet to Node2 in the second cluster based on route 3. It can be understood that the fourth route pointing to the second target container group can be determined through Node2, and the encapsulated data packet is sent to the second target container group of Node2 based on the fourth route. The fourth route is route 4 pointing to Node2, and route 4 It is generated by Node-Controller. Send the encapsulated data packet to Pod2 of Node2 based on route 4.

It can be understood that before setting the first gateway route, the second gateway route and the third gateway route, it is necessary to configure route-related information. The specific process is the following steps 1-3:

Step 1. Create a CRD (Custom Resource Definition) to represent the routing information of the Overlay network.

Wherein, in the CRD structure that is created, use TunnelType to represent the type of Overlay tunnel, what use in the embodiment of the present disclosure is IP tunneling technology, Route is concrete route information, and Destination is the purpose IP address of route (purpose IP address is the IP address of the destination cluster, which is the second cluster in the embodiment of the disclosure), Remote is the next hop list of the route (the next hop includes the first gateway component and the second gateway component in the embodiment of the disclosure), if It is multiple destination IPs, forming an equal-cost route (ECMP, Equal Cost Multi-path).

Equal-cost routing (ECMP) can be understood as in a network environment where there are multiple different links to the same destination address, if traditional routing technology is used, the data packet sent to the destination address can only use one of the links, and the other Links are in a backup state or invalid state, and it takes a certain amount of time to switch between each other in a dynamic routing environment, and the equivalent multipath routing protocol can use multiple links at the same time in this network environment, which not only increases the transmission bandwidth, but also can Back up the data transmission of the failed link without delay or packet loss. The biggest feature of ECMP is that it realizes multi-path load balancing under the condition of equal value.

Step 2, use the Cni program to configure the container network, and use the Agent program to configure the container Overlay network route.

Specifically, the Cni program is implemented using the K8s standard Cni interface, which is used to assign an IP (IPAM function) to the Pod, and is used to configure the route pointing to the Pod.

The role of the Agent program is to interact with the Apiserver to obtain Pod, Node, and RouteCRD information, and configure container Overlay network routing.

Apiserver is the core of the cluster and is responsible for the communication between the functional modules of the cluster. Each functional module in the cluster stores information into etcd through Apiserver. ) to achieve information exchange between modules.

Step 3, the Agent program is initialized through the Config configuration file, starts the corresponding controller (Controller), List/Watch Apiserver and sends the Overlay route.

Among them, Controller includes: Pod-Controller, Node-Controller, Route-Controller.

Specifically, the Agent program is initialized through the Config configuration file. The configuration structure is: Kubeconfig is the address of the authentication file communicating with the Apiserver, TlIPRoute/TlNodeRoute/TlRoute respectively indicate the switch to enable the Pod-Controller, Node-Controller, and Route-Controller, and TlRouteLabel It is the label selector of RouteCRD monitored by Route-controller.

Specifically, refer to FIG. 6 , which is a schematic diagram of the principles of the Cni program and the Agent program provided by the embodiment of the present disclosure.

1. Pod-Controller

The function is to List/Watch Pod information, issue the Overlay route pointing to the Pod, and use it for Pod cross-Node communication, for example, ip route add{PodIP}via{NodeIP}dev tunl0 onlink, the destination IP is the IP of the Pod, and the next hop is The NodeIP where the Pod is located, and the outbound interface is tunl0.

2. Node-Controller

The role is to List/Watch Node information. In table 16, the Overlay route pointing to Node is issued for the intercommunication between Pod and Node. For example, ip route add{NodeIP}via{NodeIP}dev tunl0 onlink table 16, destination IP and next The jumps are the IP of the Node, and the outgoing interface is the tunl0 port. Among them, table 16 stores Overlay routing information about Node.

3. Route-Controller

The function is to List/Watch RouteCRD information, and issue the Overlay route pointing to the destination network segment (the network segment of other clusters), which is mainly used for inter-cluster routing, such as ip route add{Destination}onlink nexthop via{GW1}dev tunl0 weight 2 onlink nexthop via{GW2}dev tunl0 weight 2 onlink, the destination is a network segment or IP, the next hop is the IP of the gateway (multiple next hops form ECMP), and the outbound interface is tunl0.

Further, taking the communication between Pods in different clusters as an example, the implementation of the present disclosure will be further explained. Referring to FIG. 7 , which is a block diagram of Pod communication between Cluster1 and Cluster2 provided by the embodiment of the present disclosure, The specific process of Pod communication between Cluster1 and Cluster2 is steps 11-14.

In the first cluster, the first cluster is represented by Cluster1, the node is represented by Node1, and the container group is represented by Pod1; in the second cluster, the second cluster is represented by Cluster2, the node is represented by Node2, and the container group is represented by Pod2; GW1 represents The first cluster corresponds to the configured gateway, and GW2 indicates the second cluster corresponds to the configured gateway. By default, relevant routing information has been configured during communication, which are directly represented by route 1, route 2, route 3 and route 4.

Step 11, obtain the original data packet of Pod1, determine the route 1 according to the IP of Pod2, encapsulate the original data packet, generate an encapsulated data packet, and send it from the network card of Node1.

Specifically, the original data packet comes out of Pod1, and the route is searched according to the destination IP (Pod2 IP). First search in table 16, if there is no suitable route, then check the table main, and confirm that there is route 1 of the outbound interface tun10 (the route pointing to GW1), The original data is encapsulated by IP tunneling technology to generate an encapsulated data packet, (in the encapsulated data packet, the original IP is the IP of Node1, and the destination IP is the IP of GW1) and then sent from the network card (eth) of Node1.

Among them, table 16 stores Overlay routing information about Node, and table main stores related routing information about Pod and gateway components.

Step 12: After receiving the encapsulated data packet, GW1 decapsulates it, searches for the route according to the IP of the next hop to determine route 2, encapsulates the data packet again, and sends it out from the network card of GW1.

Specifically, after GW1 receives the encapsulated data packet, it decapsulates the IPIP header, searches for the route, finds that there is route 2 with an outbound interface tun10, and the next hop is the gateway component GW2 of Cluster2, and then encapsulates the IPIP header again (the original IP in the encapsulated data packet is The IP of GW1 of Cluster1, the destination IP is the IP of GW2 of Cluster2), and then sent from the network card (eth) of GW1.

Step 13: After receiving the encapsulated data packet, GW2 of Cluster2 decapsulates it, determines route 3 according to Pod2 IP, encapsulates the original data packet again, and sends it out from the network card of GW2.

Specifically, after receiving the encapsulated data packet, GW2 of Cluster2 decapsulates the IPIP header, searches for the route according to the destination IP (Pod2 IP), determines that there is a route 3 (pointing to the Pod2 IP) of the outbound interface tun10, and encapsulates the IPIP header again (in The original IP in the encapsulated data packet is the IP of GW2, and the destination IP is the IP of GW2 in Cluster2), and then sent from the network card (eth) of GW2.

Step 14, after receiving the encapsulated data packet, Node2 decapsulates it to obtain the original data packet, determines that there is a route 4 pointing to Pod2 in Node2, and forwards the original data packet directly to Pod2.

Specifically, after Node2 receives the encapsulated data packet, it decapsulates the IPIP header, looks up the route according to the inner destination IP (Pod2), looks up the table main table, determines that there is a route 4 pointing to veth, and forwards it directly to the veth of Pod2.

The cross-cluster network communication system provided by the embodiments of the present disclosure realizes communication between container groups in different clusters.

The following describes the cross-cluster network communication method provided by the embodiments of the present disclosure, which is used in the network communication system described in any one of the above, specifically including:

Specifically referring to FIG. 8 , which is a schematic flowchart of a cross-cluster network communication method provided by an embodiment of the present disclosure, the method includes:

810. Determine the destination address of the original data packet as the second cluster by using the first cluster.

820. Encapsulate the original data packet through the first cluster to generate an encapsulated data packet.

830. Send the encapsulated data packet to the first gateway component corresponding to the first cluster through the first cluster.

840. Send the encapsulated data packet to the second gateway component corresponding to the second cluster through the first gateway component, so that the second gateway component sends the encapsulated data packet to the second cluster, and the encapsulated data packet is processed by the second cluster Decapsulate to get the original data packet.

The present disclosure provides a cross-cluster network communication method. The target address of the original data packet is determined as the second cluster through the first cluster. The first gateway component corresponding to the first cluster cannot directly identify the original data packet. Therefore, the original Encapsulation of the data packet enables the first gateway component to receive the encapsulated original data; the encapsulated data packet is sent to the second gateway component corresponding to the second cluster through the first gateway component again, and finally the second gateway component will encapsulate the data packet sent to the second cluster, the second cluster decapsulates the encapsulated data packet to obtain the original data packet, and realizes the communication between different clusters by means of the first gateway component corresponding to the first cluster and the second gateway component corresponding to the second cluster communication.

Based on any of the above embodiments, the first cluster includes a first target node, and the second cluster includes a second target node; the source address of the original data packet is the first target node, and the target address of the original data packet is the second target node;

The method also includes:

Based on any of the above embodiments, the first cluster includes a first target node, the first target node includes a first target container group, and the second cluster includes a second target node; the source address of the original data packet is the first target container group, The destination address of the original data packet is the second destination node;

The method also includes:

Based on any of the above embodiments, the first cluster includes the first target node, the second cluster includes the second target node, and the second target node includes the second target container group; the source address of the original data packet is the first target node, and the original The destination address of the data packet is the second destination container group;

The method also includes:

encapsulating the original data packet through the first target node, generating the encapsulated data packet, and sending the encapsulated data packet to the first gateway component through the sending port;

The original data packet sent by the second target node according to the target address is received through the second target container group, wherein the original data packet sent by the second target node is an encapsulated data packet sent by the second gateway component received by the second target node through the receiving port, Obtained by decapsulating the encapsulated data packet.

Based on any of the above embodiments, the first cluster includes a first target node, the first target node includes a first target container group, the second cluster includes a second target node, and the second target node includes a second target container group; the original data The source address of the packet is the first target container group, and the target address of the original data packet is the second target container group;

The method also includes:

FIG. 9 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG. 9, the electronic device may include: a processor (processor) 910, a communication interface (Communications Interface) 920, a memory (memory) 930 and a communication bus 940, Wherein, the processor 910 , the communication interface 920 , and the memory 930 communicate with each other through the communication bus 940 . The processor 910 can call the logic instructions in the memory 930 to execute a cross-cluster network communication system, the network communication system includes: a first cluster, used to determine the target address of the original data packet as the second cluster, and The data packet is encapsulated to generate an encapsulated data packet; the first gateway component is set corresponding to the first cluster, and is used to receive the encapsulated data packet sent by the first cluster; the second gateway component is set corresponding to the second cluster, and is used to receive the second cluster An encapsulated data packet sent by a gateway component, and the encapsulated data packet is sent to a second cluster; the second cluster is used to receive the encapsulated data packet sent by the second gateway component, and decapsulate the encapsulated data packet to obtain original data Bag.

In addition, the above-mentioned logic instructions in the memory 930 may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium. Several instructions are included to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .

On the other hand, the present disclosure also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by the computer, the computer can execute A cross-cluster network communication system provided above, the network communication system includes: a first cluster, used to determine the target address of the original data packet as the second cluster, and encapsulate the original data packet to generate an encapsulated data packet; The first gateway component is set corresponding to the first cluster, and is used to receive the encapsulated data packet sent by the first cluster; the second gateway component is set corresponding to the second cluster, and is used to receive the encapsulated data packet sent by the first gateway component, and Send the encapsulated data packet to the second cluster; the second cluster is used to receive the encapsulated data packet sent by the second gateway component, and decapsulate the encapsulated data packet to obtain the original data packet.

In yet another aspect, the present disclosure also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is implemented to implement the above-mentioned cross-cluster network communication system provided by each, The network communication system includes: a first cluster, which is used to determine the target address of the original data packet as the second cluster, and encapsulates the original data packet to generate an encapsulated data packet; the first gateway component is set corresponding to the first cluster, and uses For receiving the encapsulated data packet sent by the first cluster; the second gateway component is set corresponding to the second cluster, and is used to receive the encapsulated data packet sent by the first gateway component and send the encapsulated data packet to the second cluster; the second cluster , for receiving the encapsulated data packet sent by the second gateway component, and decapsulating the encapsulated data packet to obtain the original data packet.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place , or can also be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without any creative efforts.

Through the above description of the implementations, those skilled in the art can clearly understand that each implementation can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic discs, optical discs, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

A cross-cluster network communication system, including:

The first cluster is configured to encapsulate the original data packet to generate an encapsulated data packet;

A first gateway component, set corresponding to the first cluster, for receiving the encapsulated data packet sent by the first cluster;

A second gateway component, set corresponding to the second cluster, for receiving the encapsulated data packet sent by the first gateway component, and sending the encapsulated data packet to the second cluster; and

The second cluster is configured to receive the encapsulated data packet sent by the second gateway component, and decapsulate the encapsulated data packet to obtain an original data packet.
The cross-cluster network communication system according to claim 1, further comprising:

The first target node set in the first cluster is used to encapsulate the original data packet, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway component through a sending port; and

The second target node set in the second cluster is used to receive the encapsulated data packet sent by the second gateway component through a receiving port, and decapsulate the encapsulated data packet to obtain an original data packet;

Wherein, the source address of the original data packet is the first destination node, and the destination address of the original data packet is the second destination node.
The cross-cluster network communication system according to claim 1, wherein:

A first target node is set in the first cluster, a second target node is set in the second cluster, and the first target node includes a first target container group;

The source address of the original data packet is the first target container group, and the target address of the original data packet is the second target node;

The first target container group is configured to encapsulate the original data packet, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway via the sending port of the first target node components;

The second target node is configured to receive the encapsulated data packet sent by the second gateway component through a receiving port, and decapsulate the encapsulated data packet to obtain an original data packet.
The cross-cluster network communication system according to claim 1, wherein:

A first target node is set in the first cluster, a second target node is set in the second cluster, and the second target node includes a second target container group;

The source address of the original data packet is the first target node, and the target address of the original data packet is the second target container group;

The first target node is configured to encapsulate the original data packet, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway component through a sending port;

The second target container group is configured to receive the original data packet sent by the second target node according to the target address, wherein the original data packet sent by the second target node is received by the second target node through a receiving port The encapsulated data packet sent by the second gateway component is obtained by decapsulating the encapsulated data packet.
The cross-cluster network communication system according to claim 1, wherein:

A first target node is set in the first cluster, a second target node is set in the second cluster, the first target node includes a first target container group, and the second target node includes a second target container group;

The source address of the original data packet is the first target container group, and the target address of the original data packet is the second target container group;

The first target container group is configured to encapsulate the original data packet, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway via the sending port of the first target node components;

The second target container group is configured to receive the original data packet sent by the second target node according to the target address, wherein the original data packet sent by the second target node is received by the second target node through a receiving port The encapsulated data packet sent by the second gateway component is obtained by decapsulating the encapsulated data packet.
The cross-cluster network communication system according to claim 1, further comprising:

The first gateway route is set between the first cluster and the first gateway component, and is used to send the encapsulated data packet to the first gateway component corresponding to the first cluster;

The second gateway route is set between the first gateway component and the second gateway component, and is used to send the encapsulated data packet in the first gateway component to the second gateway component corresponding to the second cluster middle;

The third gateway route is set between the second gateway component and the second cluster, and is used to send the encapsulated data packet in the second gateway component to the second cluster.
A cross-cluster network communication method, comprising:

Encapsulating the original data packet through the first cluster to generate an encapsulated data packet;

sending the encapsulated data packet to a first gateway component corresponding to the first cluster through the first cluster;

sending the encapsulated data packet to a second gateway component corresponding to a second cluster through the first gateway component, so that the second gateway component sends the encapsulated data packet to the second cluster; and

The encapsulated data packet is decapsulated by the second cluster to obtain the original data packet.
The cross-cluster network communication method according to claim 7, wherein the first cluster includes a first target node, and the second cluster includes a second target node; the source address of the original data packet is the first target node node, the destination address of the original data packet is the second destination node;

The method also includes:

Encapsulating the original data packet by the first target node, generating the encapsulated data packet, and sending the encapsulated data packet to the first gateway component via the sending port of the first target node;

The encapsulated data packet sent by the second gateway component is received through the receiving port of the second target node, and the encapsulated data packet is decapsulated to obtain an original data packet.
The cross-cluster network communication method according to claim 7, wherein the first cluster includes a first target node, the first target node includes a first target container group, and the second cluster includes a second target node; the source address of the original data packet is the first target container group, and the target address of the original data packet is the second target node;

The method also includes:

Encapsulate the original data packet through the first target container group, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway component through the sending port of the first target node;

The encapsulated data packet sent by the second gateway component is received through the receiving port of the second target node, and the encapsulated data packet is decapsulated to obtain an original data packet.
The cross-cluster network communication method according to claim 7, wherein the first cluster includes a first target node, the second cluster includes a second target node, and the second target node includes a second target container group; the source address of the original data packet is the first target node, and the target address of the original data packet is the second target container group;

The method also includes:

encapsulating the original data packet by the first target node, generating the encapsulated data packet, and sending the encapsulated data packet to the first gateway component through a sending port;

The original data packet sent by the second target node according to the target address is received through the second target container group, wherein the original data packet sent by the second target node is that the second target node receives the The encapsulated data packet sent by the second gateway component is obtained by decapsulating the encapsulated data packet.
The cross-cluster network communication method according to claim 7, wherein the first cluster includes a first target node, the first target node includes a first target container group, and the second cluster includes a second target node, the second target node includes a second target container group; the source address of the original data packet is the first target container group, and the target address of the original data packet is the second target container group;

The method also includes:

Encapsulate the original data packet through the first target container group, generate the encapsulated data packet, and send the encapsulated data packet to the first gateway component through the sending port of the first target node;

The original data packet sent by the second target node according to the target address is received through the second target container group, wherein the original data packet sent by the second target node is that the second target node receives the The encapsulated data packet sent by the second gateway component is obtained by decapsulating the encapsulated data packet.
An electronic device comprising:

processor; and

A memory, the memory stores a computer program that can run on the processor, wherein, when the computer program is executed by the processor, the electronic device executes the cross-connected program described in any one of claims 7 to 11 The network communication method of the cluster.
A non-transitory computer-readable storage medium, on which a computer program is stored, wherein, when the computer program is executed by a processor, the cross-cluster network communication method according to any one of claims 7 to 11 is executed.
A computer program product, comprising a computer program, wherein, when the computer program is executed by a processor, the cross-cluster network communication method according to any one of claims 7 to 11 is executed.