CN115086312A - Method and system for realizing kubernets service cross-cluster communication - Google Patents
Method and system for realizing kubernets service cross-cluster communication Download PDFInfo
- Publication number
- CN115086312A CN115086312A CN202210504839.3A CN202210504839A CN115086312A CN 115086312 A CN115086312 A CN 115086312A CN 202210504839 A CN202210504839 A CN 202210504839A CN 115086312 A CN115086312 A CN 115086312A
- Authority
- CN
- China
- Prior art keywords
- cluster
- tunnel
- tunnel gateway
- service
- deploying
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- 238000004891 communication Methods 0.000 title claims abstract description 28
- 230000007246 mechanism Effects 0.000 claims abstract description 24
- RJKFOVLPORLFTN-LEKSSAKUSA-N Progesterone Chemical compound C1CC2=CC(=O)CC[C@]2(C)[C@@H]2[C@@H]1[C@@H]1CC[C@H](C(=O)C)[C@@]1(C)CC2 RJKFOVLPORLFTN-LEKSSAKUSA-N 0.000 claims abstract description 6
- 230000004044 response Effects 0.000 claims description 11
- 230000008676 import Effects 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 230000000977 initiatory effect Effects 0.000 claims 3
- 230000008447 perception Effects 0.000 abstract 1
- 102100033121 Transcription factor 21 Human genes 0.000 description 6
- 101150109289 tcf21 gene Proteins 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 101710119687 Transcription factor 21 Proteins 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000005641 tunneling Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/46—Interconnection of networks
- H04L12/4633—Interconnection of networks using encapsulation techniques, e.g. tunneling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/66—Arrangements for connecting between networks having differing types of switching systems, e.g. gateways
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0668—Management of faults, events, alarms or notifications using network fault recovery by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/10—Network architectures or network communication protocols for network security for controlling access to devices or network resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Cardiology (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a method and a system for realizing kubernet service cross-cluster communication, which relate to the technical field of cross-cluster communication, and the method comprises the following steps: deployment step: distributing different CIDR blocks for different clusters, selecting a cluster deployment cluster medium, deploying a network tunnel gateway and a navigation agent for each cluster, and finally deploying a routing agent for each node of each cluster; the working steps are as follows: after the corresponding deployment is completed, the network tunnel gateway, the navigation agent and the routing agent start to do preparation work: the method comprises the steps of creating necessary CRDs, mutually discovering and establishing network tunnels through cluster intermediaries by network tunnel gateways, exposing the Services of a cluster where the navigation agent is located through the cluster intermediaries and obtaining the Services of other clusters, and obtaining an endpoint CR set by the tunnel gateways through an Informer mechanism by a routing agent. The invention can realize service non-perception cross-cluster communication.
Description
Technical Field
The invention relates to the technical field of cross-cluster communication, in particular to a method and a system for realizing cross-cluster communication of kubernets service.
Background
Kubernets is a brand-new container technology-based distributed architecture solution, and is a container cluster management system of Google open source, and Kubernets is k8s for short. Kubernets currently has become a de facto standard for container orchestration. With the large-scale use of kubernets, the problems of limited bearing capacity, limited disaster tolerance and the like of kubernets single clusters are highlighted, single clusters become unsatisfiable, multi-cluster arrangement becomes a natural solution, and cross-cluster communication of services is a problem to be solved for realizing multi-cluster arrangement.
The invention patent with the publication number of CN113032126A discloses a cross-cluster communication system and a method for a high-concurrency cloud workflow scheduling engine, wherein a controller module of the system is containerized and deployed to a Master cluster of Kubernetes in a Service and Deployment mode; the generator module is containerized and deployed in each cluster in a Shell script mode, and each cluster deploys a unique generator container in a Service and Deployment mode; the controller module and the generator module provide access to the outside in a Service NodePort mode and cross cluster communication in a gPC mode; the distributed database is independently deployed in each cluster and used for storing resource request records of the scheduler module; the cache database is deployed by adopting a distributed architecture and is used for storing global workflow information records.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a method and a system for realizing the kubernets service cross-cluster communication.
According to the method and the system for realizing the kubernets service cross-cluster communication, the scheme is as follows:
in a first aspect, a method for implementing kubernets service cross-cluster communication is provided, the method including:
deployment step: distributing different CIDRs for different clusters, selecting a cluster deployment cluster medium, deploying a network tunnel gateway and a navigation agent for each cluster, and finally deploying a routing agent for each node of each cluster;
the working steps are as follows: after the corresponding deployment is completed, the network tunnel gateway, the navigation agent and the routing agent start to do preparation work: the method comprises the steps of creating necessary CRDs, mutually discovering and establishing network tunnels through cluster intermediaries by network tunnel gateways, exposing the Services of a cluster where a navigation agent is located through the cluster intermediaries and obtaining the Services of other clusters, and obtaining an end point CR set by the tunnel gateways through an Informer mechanism by a routing agent.
Preferably, the selecting one cluster deployment cluster intermediary in the deploying step specifically includes:
step S1.1: creating a cluster CRD comprising:
cluster _ cidr array: CIDRs distributed by different clusters;
name: a cluster name;
service _ cidr array: CIDRs of service and pod distributed by each cluster;
step S1.2: creating a tunnel gateway endpoint (CRD) comprising:
name: a cluster name;
private _ ip: ip in the cluster;
public _ ip: ip for other cluster tunnel gateways to access;
step S1.3: configuring the ServiceAccount and RBAC rules for other clusters to safely access.
Preferably, the deploying the network tunnel gateway for each cluster in the deploying step specifically includes:
step S2.1: each cluster selects two nodes as a master node/a slave node respectively;
step S2.2: the ServiceAccount and access address of the access cluster intermediary K8s API are configured.
Preferably, the deploying the navigation agent for each cluster in the deploying step includes:
step S3.1: creating a cluster ServiceExport CRD, comprising:
name and namespace: name and name space for service, contained in default metadata;
status: deriving a status for the service, including success and failure;
step S3.2: creating a cluster ServiceImport CRD, comprising:
name and namespace: service name and namespace, contained in default metadata;
ports array: a list of exposed ports;
ips array: an ip list;
step S3.3: deploying a navigation agent component on a node, and configuring the CIDR information and the cluster name mapping relation of each cluster through a configuration file;
step S3.4: deploying a navigation proxy CoreDNS plug-in, configuring the CoreDNS to forward a DNS request for a clusterset.
Preferably, the starting of preparation work after the deployment of the tunnel gateway in the working step includes:
step S4.1: the slave node sends a heartbeat data packet to the master node through network connection at regular time, and when finding that the master node loses response, the slave node considers that the master node fails and switches the slave node to the master node to continue working;
step S4.2: the tunnel gateway establishes cluster information CR and tunnel gateway information CR of a cluster in a cluster medium;
step S4.3: the tunnel gateway queries and acquires cluster CR and tunnel gateway CR of other nodes in real time through an actor mechanism, and caches the cluster CR and the tunnel gateway CR in the ETCD of the current cluster;
step S4.4: when the tunnel gateway acquires a tunnel gateway endpoint CR creation event of other clusters, acquiring a public _ ip field value of the tunnel gateway, trying to establish network connection, and retrying connection later if the public _ ip field value fails;
step S4.5: when the cluster tunnel gateway receives the connection created by other cluster tunnel gateways, the cluster tunnel for creating the connection is inquired from the cache of the cluster, if the cluster tunnel is inquired, the connection creation is completed, and if the cluster tunnel is not inquired, the connection is refused.
Preferably, the step of starting preparation work by the navigation agent comprises:
step S5.1: creating a Service export CR for each Service needing to be exposed to other clusters in the cluster intermediary through the K8s API call, and enabling other clusters to know which services of the current cluster can be accessed by other clusters;
step S5.2: calling a Service export created in the cluster broker through a K8s API to create a corresponding Service import CR for each created Service export, and enabling other clusters to know the related information including the ip and the port of the Service exposed by the current cluster;
step S5.3: and continuously acquiring the ServiceImport and the ServiceExport through an Informer mechanism, and caching the ServiceImport in the ETCD through the cluster K8s API.
Preferably, the starting of the preparation work by the routing agent in the working step includes:
step S6.1: the routing agent acquires an endpoint CR set by the tunnel gateway through an Informer mechanism, and further acquires the IP address of the tunnel gateway of the current cluster;
step S6.2: the routing agent sends the flow of the cross-cluster to the tunnel gateway through the CNI plug-in, the tunnel gateway sends the flow to the corresponding cluster tunnel gateway, and on the contrary, the flow from the tunnel gateway can be sent to different nodes according to the destination IP of the flow data packet.
In a second aspect, there is provided a system for enabling kubernets service cross-cluster communication, the system comprising:
a deployment module: distributing different CIDRs for different clusters, selecting a cluster deployment cluster medium, deploying a network tunnel gateway and a navigation agent for each cluster, and finally deploying a routing agent for each node of each cluster;
the working module comprises: after the corresponding deployment is completed, the network tunnel gateway, the navigation agent and the routing agent start to do preparation work: the method comprises the steps of creating necessary CRDs, mutually discovering and establishing network tunnels through cluster intermediaries by network tunnel gateways, exposing the Services of a cluster where the navigation agent is located through the cluster intermediaries and obtaining the Services of other clusters, and obtaining an endpoint CR set by the tunnel gateways through an Informer mechanism by a routing agent.
Preferably, the selecting one cluster deployment cluster intermediary in the deployment module specifically includes:
module M1.1: creating a cluster CRD comprising:
cluster _ cidr (array): CIDRs distributed by different clusters;
name: a cluster name;
service _ cidr (array): CIDRs of service and pod distributed by each cluster;
module M1.2: creating a tunnel gateway endpoint (CRD) comprising:
name: a cluster name;
private _ ip: ip in the cluster;
public _ ip: ip for other cluster tunnel gateways to access;
module M1.3: configuring the ServiceAccount and RBAC rules for other clusters to safely access.
Preferably, the deploying the network tunnel gateway for each cluster in the deploying step specifically includes:
module M2.1: each cluster selects two nodes as a master node/a slave node respectively;
module M2.2: the ServiceAccount and access address of the access cluster intermediary K8s API are configured.
Compared with the prior art, the invention has the following beneficial effects:
1. the present invention makes possible the use of the kubernets existing infrastructure, for example: various meta-information is stored and efficiently inquired by using the existing CRD mechanism of k8s, and traffic is routed between a network tunnel gateway and a computing node in a cluster by using a CNI plug-in, so that the deployment and use difficulty is reduced;
2. the invention utilizes official design 'KEP-645' to realize a cross-cluster Service discovery mechanism, thereby increasing the universality and the understandability;
3. the cluster intermediary has high failure tolerance, and the current communication among the cross-cluster services cannot be influenced by failure;
4. the service in the present invention enables unaware cross-cluster communication.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is an overall schematic view of the present invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
The embodiment of the invention provides a method for realizing cross-cluster communication of kubernets service, which can enable the service to access other cluster services as the service of the cluster, wherein the main components involved in the method comprise the following parts:
1. the cluster intermediary: by using the existing CRD mechanism (Custom Resource Definition ) of k8s, a schema Definition (schema) of a Custom Resource (CR) object is defined, the Custom Resource object can be generated according to the CRD and specific data through a Kubernets API, various cluster meta-information is stored in a centralized manner, and all clusters can conveniently and efficiently query and acquire information change in real time through an own actor mechanism of a k8s client. The following are some examples of stored data: cluster-related static information: a Pod (container group consisting of related containers) and Service (a proxy mechanism exposed as a web Service for a group of Pod applications), etc.; ip addresses of network tunnel gateways used between clusters to communicate with each other, etc.
2. A tunnel gateway: the inter-cluster network communication is realized by establishing a safe network tunnel between the tunnel gateways among different clusters, and the tunnel gateway is deployed in each cluster. Network tunnels have many mature implementations, so a communication gateway only provides an interface of a plug-in, and the specific implementation can depend on the existing tools: various implementations of IPSec (network security protocol), WireGuard (VPN program), VXLAN (virtual local area network extension, a tunneling technique, implementing a virtual two-layer switch in a three-layer network), etc. Because the high availability of the communication gateway is very important, the tunnel gateway supports the high availability of a master-slave mode, the master is responsible for communication under normal conditions, and when the master fails, the slave is switched to the master to continue working.
3. The navigation agent: cross-cluster service discovery is achieved by implementing the official design "KEP-1645". The navigation agent is deployed in each cluster, and is responsible for storing Service information (ServiceExport CRD) which is exposed to other clusters by the cluster into the cluster broker for other cluster query, and simultaneously, the navigation agent can query the Service (serviceimport CRD) which is exposed by other clusters in the cluster broker; the navigation agent creates a DNS service by utilizing a CoreDNS plug-in mechanism, the DNS service has a domain name of' clusterset.
4. The routing agent: the routing agent utilizes the plug-in mechanism of the prior CNI (Container Network Interface, Network resource of user management Container) to pass the traffic requested to be routed to other cluster endpoints through the navigation agent of the cluster, and similarly, the traffic incoming from other clusters is routed to the destination endpoint from the navigation agent.
Referring to fig. 1, a method for implementing a kubernets service cross-cluster communication provided by the present invention mainly includes the following steps:
deployment step: and distributing different CIDRs for different clusters, selecting a cluster deployment cluster medium, deploying a network tunnel gateway and a navigation agent for each cluster, and finally deploying a routing agent for each node of each cluster.
The working steps are as follows: after the corresponding deployment is completed, the network tunnel gateway, the navigation agent and the routing agent start to do preparation work: the method comprises the steps of creating necessary CRDs, mutually discovering and establishing network tunnels through cluster intermediaries by network tunnel gateways, exposing the Services of a cluster where the navigation agent is located through the cluster intermediaries and obtaining the Services of other clusters, and obtaining an endpoint CR set by the tunnel gateways through an Informer mechanism by a routing agent.
In the step of specifically deploying, selecting one cluster deployment cluster intermediary specifically comprises:
step S1.1: creating a cluster CRD comprising:
cluster _ cidr (array): CIDRs (class Inter-Domain Routing, misclassification Inter-Domain Routing, which can be used for ip representation and ip segment representation) distributed by different clusters;
name: a cluster name;
service _ cidr (array): CIDRs of service and pod distributed by each cluster;
step S1.2: creating a tunnel gateway endpoint (CRD) comprising:
name: a cluster name;
private _ ip: ip in the cluster;
public _ ip: ip for other cluster tunnel gateways to access;
step S1.3: the ServiceAccount (account information for authentication) and RBAC (role based access control) rules are configured for secure access by other clusters.
In the deploying step, the deploying the network tunnel gateway for each cluster specifically includes:
step S2.1: and each cluster selects two nodes as a master node/a slave node respectively, so that high availability of the network tunnel gateway is ensured.
Step S2.2: the ServiceAccount and access address of the kubernets API (kubernets services network interface) accessing cluster broker are configured.
In the deploying step, deploying the navigation agent for each cluster includes:
step S3.1: creating a cluster ServiceExport CRD (explained) comprising:
name and namespace: name and name space for service, contained in the default metadata field;
status: deriving a status for the service, including success and failure;
step S3.2: creating a cluster ServiceImport CRD, comprising:
name and namespace: service name and namespace, contained in default metadata;
ports array: a list of exposed ports;
ips array: an ip list;
step S3.3: deploying a navigation agent component on a node, and configuring the CIDR information and the cluster name mapping relation of each cluster through a configuration file;
step S3.4: a navigation agent CoreDNS (a component of kubernets, implementing DNS functionality) plug-in is deployed, which is configured to forward DNS requests for "clusterset.
Next, in the working step, starting preparation work after the tunnel gateway is deployed includes:
step S4.1: the slave node sends a heartbeat data packet to the master node through network connection at regular time, and when finding that the master node loses response, the slave node considers that the master node fails, switches the slave node into the master node and continues working.
Step S4.2: the tunnel gateway uses the cluster CRD and the information of the cluster where the tunnel gateway belongs in step S1.1 to create a cluster CR (CR) (custom resource) through a cluster intermediary kubernets API, customizes resources, performs addition and deletion modification on the resources through the kubernets API, and can learn change in real time, and uses the tunnel gateway endpoint CRD and the information of the tunnel gateway of the cluster where the tunnel gateway belongs in step S1.2 to create the tunnel gateway CR through the cluster intermediary kubernets API.
Step S4.3: the tunnel gateway queries and acquires the cluster CR and the tunnel gateway CR of other nodes in real time through an actor mechanism, and caches the cluster CR and the tunnel gateway CR to an ETCD (Kubernets database component) of the current cluster.
Step S4.4: when the tunnel gateway acquires the creation event of the tunnel gateway end point CR of other clusters, the public _ ip field value of the tunnel gateway is acquired, network connection is tried to be established, and if the network connection fails, the connection is retried at a later time.
Step S4.5: when the cluster tunnel gateway receives the connection created by other cluster tunnel gateways, the cluster tunnel for creating the connection is inquired from the cache of the cluster, if the cluster tunnel is inquired, the connection creation is completed, and if the cluster tunnel is not inquired, the connection is refused.
In the working step, the navigation agent starts to do preparation work and comprises the following steps:
step S5.1: the ServiceExport CR is created in the cluster broker for each Service that needs to be exposed to other clusters through the K8s API call, letting the other clusters know which services the current cluster has for other clusters to access.
Step S5.2: and calling the Service export CR for each created Service import CR in the cluster intermediary through the K8s API, and enabling other clusters to know information such as ip and port of the Service exposed by the current cluster.
Step S5.3: the serviceImport CR and the serviceExport CR are continuously acquired through an Informer mechanism, and the serviceImport CR is cached in the ETCD through the cluster K8s API.
In the working step, the route agent starts to make preparation work and comprises the following steps:
step S6.1: and the routing agent acquires the end point CR set by the tunnel gateway through an Informer mechanism, and further acquires the IP address of the tunnel gateway of the current cluster.
Step S6.2: the routing agent sends the flow path of the cross-cluster to the tunnel gateway through the CNI plug-in, the tunnel gateway sends the flow to the corresponding cluster tunnel gateway, and on the contrary, the flow coming out of the tunnel gateway can be sent to different Pod or Service according to the destination IP of the flow data packet.
So far, the preparation work is completed, and the workflow is illustrated by a scene description of cross-cluster communication: pod 1's pod1 resolves to an ip connection through service2 domain name under cluster 2 namespace default and accesses service2.
When pod1 pod1 passes: when the IP connection with domain name resolution of "service 2.default. service. clusterset. local" accesses service2 of cluster 2, the request packet is forwarded to the tunnel gateway node of cluster 1, the tunnel gateway of cluster 1 forwards the traffic to the tunnel gateway of destination cluster 2, the tunnel gateway of cluster 2 forwards the traffic to service2, and after the request processing is completed, the packet of the response returns the original route.
The method specifically comprises the following steps:
(1): pod 1's pod1 query via DNS
Ip of "service 2.default. service. clusterset. local" domain name, DNS query request is sent to CoreDNS.
(2): CoreDNS resolves the "service 2.default. service. cluster. local" domain name to get the ip of service2.
The step (2) comprises the following steps:
(2.1): a navigation agent plug-in of the cluster 1 acquires Service corresponding to a domain name;
(2.2): inquiring information of the serviceImport corresponding to the Service through the serviceImport CR cached in the step S5.3, if the information is not successfully exposed, returning NXDomain error, if the information is successfully exposed, inquiring ip information corresponding to the Service through the serviceExport CR cached in the step S5.3, if the inquiry is successful, returning ip, and if the inquiry is failed, returning NXDomain error;
(3): according to the rule configured by the routing agent in step S6.2, the destination ip is not in the cluster, and the packet is forwarded to the tunnel gateway node of the cluster 1.
(4): and when receiving the data packet, the cluster 1 tunnel gateway forwards the flow to the tunnel gateway of the cluster 2 matched with the current destination ip.
The step (4) comprises the following steps:
(4.1): inquiring the cluster name corresponding to the ip as a cluster 2 through the CIDRs information configured in the step S3.3 and distributed by the cluster;
(4.2): querying the tunnel gateway CR cached in the cluster 1 in the step S4.3 in the cluster medium through an Informer mechanism, and finding (4.1) the tunnel gateway CR corresponding to the queried cluster 2;
(4.3): the cluster 1 and the cluster 2 are connected through the steps S4.4 and S4.5, so that the current data packet is sent to the tunnel gateway of the cluster 2;
(5): when the cluster 2 tunnel receives the data packet sent by the cluster 1, checking whether the target IP of the current data packet belongs to the CIDR block of the cluster 2, if not, rejecting the target IP; and if so, forwarding the traffic to the service2 corresponding to the destination IP according to the created routing rule in the step S6.2.
(6): and (4) sending a response data packet by the service2 after the Pod of the service2 proxy processes the request, and forwarding the response data packet to the cluster 2 tunnel gateway node according to the principle of the step (3).
(7): and (4) when the cluster 2 tunnel gateway receives the data packet, according to the principle of the step (4), the response data packet is forwarded to the tunnel gateway of the cluster 1 matched with the destination ip of the response data packet.
(8): when the cluster 1 tunnel gateway receives the request response packet, the response packet is forwarded to the pod1 matching the destination ip of the current response packet according to the principle described in step (5).
The embodiment of the invention provides a method and a system for realizing kubernets service cross-cluster communication, which can realize multi-cluster arrangement and non-sensing cross-cluster communication; the cross-cluster Service discovery mechanism is realized by using official design 'KEP-645', so that the universality and the understandability are increased; the cluster intermediary has high tolerance for failure, and the failure does not affect the current communication between the cross-cluster services.
Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, units provided by the present invention as pure computer readable program code, the system and its various devices, modules, units provided by the present invention can be fully implemented by logically programming method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and various devices, modules and units thereof provided by the present invention can be regarded as a hardware component, and the devices, modules and units included therein for implementing various functions can also be regarded as structures within the hardware component; means, modules, units for performing the various functions may also be regarded as structures within both software modules and hardware components for performing the method.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.
Claims (10)
1. A method for realizing cross-cluster communication of kubernets service, which is characterized by comprising the following steps:
deployment step: distributing different CIDR blocks for different clusters, selecting a cluster deployment cluster medium, deploying a network tunnel gateway and a navigation agent for each cluster, and finally deploying a routing agent for each node of each cluster;
the working steps are as follows: after the corresponding deployment is completed, the network tunnel gateway, the navigation agent and the routing agent start to do preparation work: the method comprises the steps of creating necessary CRDs, mutually discovering and establishing network tunnels through cluster intermediaries by network tunnel gateways, exposing the Services of a cluster where the navigation agent is located through the cluster intermediaries and obtaining the Services of other clusters, and obtaining an endpoint CR set by the tunnel gateways through an Informer mechanism by a routing agent.
2. The method of claim 1, wherein the step of deploying selects a cluster deployment cluster intermediary that comprises:
step S1.1: creating a cluster CRD comprising:
cluster _ cidr array: CIDRs distributed by different clusters;
name: a cluster name;
service _ cidr array: CIDRs of service and pod distributed by each cluster;
step S1.2: creating a tunnel gateway endpoint (CRD) comprising:
name: a cluster name;
private _ ip: ip in the cluster;
public _ ip: ip for other cluster tunnel gateways to access;
step S1.3: configuring the ServiceAccount and RBAC rules for other clusters to safely access.
3. The method of claim 1, wherein the deploying a network tunnel gateway for each cluster in the deploying step specifically comprises:
step S2.1: each cluster selects two nodes as a master node/a slave node respectively;
step S2.2: the ServiceAccount and access address of the access cluster intermediary K8s API are configured.
4. The method of claim 1, wherein the deploying a navigation agent for each cluster in the deploying step comprises:
step S3.1: creating a cluster ServiceExport CRD, comprising:
name and namespace: name and name space for service, contained in default metadata;
status: deriving a status for the service, including success and failure;
step S3.2: creating a cluster ServiceImport CRD, comprising:
name and namespace: service name and namespace, contained in default metadata;
ports array: a list of exposed ports;
ips array: an ip list;
step S3.3: deploying a navigation agent component on a node, and configuring the CIDR information and the cluster name mapping relation of each cluster through a configuration file;
step S3.4: deploying a navigation proxy CoreDNS plug-in, configuring the CoreDNS to forward a DNS request for a clusterset.
5. The method of claim 1, wherein the step of initiating preparation after deployment of a tunnel gateway comprises:
step S4.1: the slave node sends a heartbeat data packet to the master node through network connection at regular time, and when finding that the master node loses response, the slave node considers that the master node fails, switches the slave node into the master node and continues working;
step S4.2: the tunnel gateway establishes cluster information CR and tunnel gateway information CR of a cluster in a cluster medium;
step S4.3: the tunnel gateway queries and acquires cluster CR and tunnel gateway CR of other nodes in real time through an actor mechanism, and caches the cluster CR and the tunnel gateway CR in the ETCD of the current cluster;
step S4.4: when the tunnel gateway acquires a tunnel gateway endpoint CR creation event of other clusters, acquiring a public _ ip field value of the tunnel gateway, trying to establish network connection, and retrying connection later if the public _ ip field value fails;
step S4.5: when the cluster tunnel gateway receives the connection created by other cluster tunnel gateways, the cluster tunnel for creating the connection is inquired from the cache of the cluster, if the cluster tunnel is inquired, the connection creation is completed, and if the cluster tunnel is not inquired, the connection is refused.
6. The method of claim 1, wherein the step of initiating preparation by the navigation agent comprises:
step S5.1: creating a Service export CR for each Service needing to be exposed to other clusters in the cluster intermediary through the K8s API call, and enabling other clusters to know which services of the current cluster can be accessed by other clusters;
step S5.2: calling a Service export created in the cluster broker through a K8s API to create a corresponding Service import CR for each created Service export, and enabling other clusters to know the related information including the ip and the port of the Service exposed by the current cluster;
step S5.3: and continuously acquiring the ServiceImport and the ServiceExport through an Informer mechanism, and caching the ServiceImport in the ETCD through the cluster K8s API.
7. The method of claim 1, wherein the step of initiating preparation by the routing agent comprises:
step S6.1: the routing agent acquires an endpoint CR set by the tunnel gateway through an Informer mechanism, and further acquires the IP address of the tunnel gateway of the current cluster;
step S6.2: the routing agent sends the flow of the cross-cluster to the tunnel gateway through the CNI plug-in, the tunnel gateway sends the flow to the corresponding cluster tunnel gateway, and on the contrary, the flow from the tunnel gateway can be sent to different nodes according to the destination IP of the flow data packet.
8. A system for enabling kubernets service cross-cluster communication, comprising:
a deployment module: distributing different CIDRs for different clusters, selecting a cluster deployment cluster medium, deploying a network tunnel gateway and a navigation agent for each cluster, and finally deploying a routing agent for each node of each cluster;
the working module comprises: after the corresponding deployment is completed, the network tunnel gateway, the navigation agent and the routing agent start to do preparation work: the method comprises the steps of creating necessary CRDs, mutually discovering and establishing network tunnels through cluster intermediaries by network tunnel gateways, exposing the Services of a cluster where the navigation agent is located through the cluster intermediaries and obtaining the Services of other clusters, and obtaining an endpoint CR set by the tunnel gateways through an Informer mechanism by a routing agent.
9. The system of claim 6, wherein the selecting one of the deployment modules to deploy a cluster intermediary comprises:
module M1.1: creating a cluster CRD comprising:
cluster _ cidr (array): CIDRs distributed by different clusters;
name: a cluster name;
service _ cidr (array): CIDRs of service and pod distributed by each cluster;
module M1.2: creating a tunnel gateway endpoint CRD, comprising:
name: a cluster name;
private _ ip: ip in the cluster;
public _ ip: ip for other cluster tunnel gateways to access;
module M1.3: configuring the ServiceAccount and RBAC rules for other clusters to safely access.
10. The system of claim 6, wherein the deploying a network tunnel gateway for each cluster in the deploying step specifically comprises:
module M2.1: each cluster selects two nodes as a master node/a slave node respectively;
module M2.2: the ServiceAccount and access address of the access cluster intermediary K8s API are configured.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210504839.3A CN115086312A (en) | 2022-05-10 | 2022-05-10 | Method and system for realizing kubernets service cross-cluster communication |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210504839.3A CN115086312A (en) | 2022-05-10 | 2022-05-10 | Method and system for realizing kubernets service cross-cluster communication |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115086312A true CN115086312A (en) | 2022-09-20 |
Family
ID=83247462
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210504839.3A Pending CN115086312A (en) | 2022-05-10 | 2022-05-10 | Method and system for realizing kubernets service cross-cluster communication |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115086312A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115242877A (en) * | 2022-09-21 | 2022-10-25 | 之江实验室 | Spark collaborative calculation and operation method and device for multiple K8s clusters |
| CN116896499A (en) * | 2023-06-12 | 2023-10-17 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | kubernetes Pod network error checking system and method |
| US11954525B1 (en) | 2022-09-21 | 2024-04-09 | Zhejiang Lab | Method and apparatus of executing collaborative job for spark faced to multiple K8s clusters |
| CN117978406A (en) * | 2024-02-20 | 2024-05-03 | 国网江苏省电力有限公司信息通信分公司 | Heterogeneous multi-container cluster scheduling method, system, equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111885123A (en) * | 2020-07-06 | 2020-11-03 | 苏州浪潮智能科技有限公司 | Construction method and device of cross-K8 s target service access channel |
| CN112751913A (en) * | 2020-12-22 | 2021-05-04 | 联奕科技股份有限公司 | Network communication method and system across Kubernetes cluster |
| US20210311762A1 (en) * | 2020-04-02 | 2021-10-07 | Vmware, Inc. | Guest cluster deployed as virtual extension of management cluster in a virtualized computing system |
| CN113572689A (en) * | 2021-09-24 | 2021-10-29 | 深圳市信润富联数字科技有限公司 | Microservice gateway management method, system, device, readable storage medium and product |
| CN114153566A (en) * | 2021-12-20 | 2022-03-08 | 浪潮电子信息产业股份有限公司 | Cross-processor architecture multi-container inter-cluster service discovery method, device and equipment |
| CN114443059A (en) * | 2020-10-30 | 2022-05-06 | 中国联合网络通信集团有限公司 | Kubernets cluster deployment method, device and equipment |
-
2022
- 2022-05-10 CN CN202210504839.3A patent/CN115086312A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210311762A1 (en) * | 2020-04-02 | 2021-10-07 | Vmware, Inc. | Guest cluster deployed as virtual extension of management cluster in a virtualized computing system |
| CN111885123A (en) * | 2020-07-06 | 2020-11-03 | 苏州浪潮智能科技有限公司 | Construction method and device of cross-K8 s target service access channel |
| CN114443059A (en) * | 2020-10-30 | 2022-05-06 | 中国联合网络通信集团有限公司 | Kubernets cluster deployment method, device and equipment |
| CN112751913A (en) * | 2020-12-22 | 2021-05-04 | 联奕科技股份有限公司 | Network communication method and system across Kubernetes cluster |
| CN113572689A (en) * | 2021-09-24 | 2021-10-29 | 深圳市信润富联数字科技有限公司 | Microservice gateway management method, system, device, readable storage medium and product |
| CN114153566A (en) * | 2021-12-20 | 2022-03-08 | 浪潮电子信息产业股份有限公司 | Cross-processor architecture multi-container inter-cluster service discovery method, device and equipment |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115242877A (en) * | 2022-09-21 | 2022-10-25 | 之江实验室 | Spark collaborative calculation and operation method and device for multiple K8s clusters |
| CN115242877B (en) * | 2022-09-21 | 2023-01-24 | 之江实验室 | Spark collaborative computing and operating method and device for multiple K8s clusters |
| US11954525B1 (en) | 2022-09-21 | 2024-04-09 | Zhejiang Lab | Method and apparatus of executing collaborative job for spark faced to multiple K8s clusters |
| CN116896499A (en) * | 2023-06-12 | 2023-10-17 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | kubernetes Pod network error checking system and method |
| CN116896499B (en) * | 2023-06-12 | 2024-03-19 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | kubernetes Pod network error checking system and method |
| CN117978406A (en) * | 2024-02-20 | 2024-05-03 | 国网江苏省电力有限公司信息通信分公司 | Heterogeneous multi-container cluster scheduling method, system, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109032755B (en) | Container service hosting system and method for providing container service | |
| CN115086312A (en) | Method and system for realizing kubernets service cross-cluster communication | |
| CN107947961B (en) | SDN-based Kubernetes network management system and method | |
| CN112840322B (en) | Single-node and multi-node data storage library system in network routing environment | |
| CN113572831B (en) | Communication method, computer equipment and medium between Kubernetes clusters | |
| TW200835265A (en) | Address resolution request mirroring | |
| US8572201B2 (en) | System and method for providing a directory service network | |
| CN101263696A (en) | Routing data packets from a multihomed host | |
| US8478898B2 (en) | System and method for routing directory service operations in a directory service network | |
| CN112491984B (en) | Container editing engine cluster management system based on virtual network bridge | |
| CN112165502B (en) | Service discovery system, method and second server | |
| CN103618801A (en) | Method, device and system for sharing P2P (Peer-to-Peer) resources | |
| WO2020057445A1 (en) | Communication system, method, and device | |
| JP2005295124A (en) | Path table synchronizing method, network device, and path table synchronizing program | |
| CN115190103A (en) | Service grid-based service domain name resolution method, device and equipment | |
| CN107077429B (en) | Method for reading data, equipment and system | |
| CN102316154B (en) | Optimize the access to the resource based on federation infrastructure | |
| CN114201362A (en) | Prometheus-based enterprise-level high-availability monitoring system and implementation method | |
| CN111770159A (en) | Service discovery system, method, first server and storage medium | |
| US9922031B2 (en) | System and method for efficient directory performance using non-persistent storage | |
| CN113904973B (en) | Route updating method, medium, device and computing equipment | |
| KR100375796B1 (en) | Direct interpersonal distributed network system for communication by using jointly-owned back-up storage device on the web | |
| TW200534633A (en) | Method of automatically transferring router functionality | |
| CN112073449B (en) | Kubernetes-based environment switching processing method and equipment | |
| CN117453380B (en) | Cluster container group scheduling method, system and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |