CN111552543A - Container management and control method and processing node - Google Patents
Container management and control method and processing node Download PDFInfo
- Publication number
- CN111552543A CN111552543A CN202010440720.5A CN202010440720A CN111552543A CN 111552543 A CN111552543 A CN 111552543A CN 202010440720 A CN202010440720 A CN 202010440720A CN 111552543 A CN111552543 A CN 111552543A
- Authority
- CN
- China
- Prior art keywords
- container
- information
- node
- state
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
Abstract
The application relates to a container management and control method and a processing node, wherein after a management node removes container information of a first container based on an obtained deletion request of the first container and stops receiving and/or forwarding a connection request facing the first container, a service node removes routing information of the first container and feeds back first removal state information of the first container routing information to the management node, the management node sets the state of the first container to be a second state under the condition that the management node determines that the service node meets the removal condition of the first container routing information based on at least part of the obtained first removal state information, and the service node triggers and executes deletion operation of the first container based on the second state of the first container. Therefore, the execution sequence of each component processing flow in the container deleting process is strictly controlled, the correct processing logic of the container deleting can be ensured, and data loss or service interruption in the container deleting process is avoided.
Description
Technical Field
The present application belongs to the field of container cluster management, and in particular, relates to a container management and control method and a processing node.
Background
The container cluster management system is an open-source platform, comprises a management node and at least one service node, and can realize the functions of automatic deployment, automatic capacity expansion and reduction, maintenance and the like of a container cluster based on each node. K8s (kubernets) is a more common container cluster management system, created by Google, an open source version of the Google large-scale container management technology Borg.
Currently, when a container is deleted in a container cluster management system such as K8s, an API server in a management node first sets a container state to a Terminating state, and each component in the cluster node monitors the state and synchronously executes a respective processing flow until the container is deleted.
However, in the process of deleting the container, the processing flows of each component are executed concurrently, the execution sequence of each flow cannot be strictly controlled, the service may be affected due to an unreasonable execution sequence, and accordingly, the container data may be lost or the service may be interrupted.
Disclosure of Invention
In view of this, the present application provides a container management and control method and a processing node, which strictly control the execution sequence of the processing flow of each component in the container deletion process, ensure the correct processing logic for container deletion, and avoid data loss or service interruption in the container deletion process.
Therefore, the application discloses the following technical scheme:
a method of vessel management, the method comprising:
obtaining a delete request for the first container;
removing the container information of the first container and stopping receiving and/or forwarding the connection request facing the first container; the management node stores the corresponding relation information of at least one container of the service and the bearing service;
obtaining first removal state information of first container routing information of at least one service node; the first removal state information is generated when the serving node removes routing information for the first container after the management node has removed the container information and stopped the receiving and/or forwarding operations;
setting the state of the first container to a second state in the case that the service node is determined to meet the removal condition of the first container routing information based on at least part of the obtained first removal state information;
wherein the respective service node performs a container deletion operation on the first container based on the second state.
Preferably, after obtaining the delete request for the first container, the method further includes: setting a state of the first container to a first state;
the removing the container information of the first container and stopping receiving and/or forwarding the connection request facing the first container includes:
monitoring the state of the first container;
based on the monitored first state, removing the container information of the first container and stopping receiving and/or forwarding a connection request towards the first container.
Preferably, before obtaining the first removal state information of the first container routing information of the at least one service node, further comprises:
sending a first notification message to each service node;
the first notification message includes: the management node has removed the container information of the first container and has stopped receiving and/or forwarding the second removal state information of the connection request.
Preferably, the obtaining the first removal state information of the at least one service node includes:
receiving a second notification message of at least one service node;
the second notification message includes: the service node has removed the first removal state information of the first container routing information.
Preferably, wherein determining whether the serving node satisfies the removal condition for the first container routing information based on the obtained at least part of the first removal state information comprises:
determining the number of service nodes with the first container routing information removed according to the number of the received second notification messages, and recording the number of the service nodes in a preset data structure field;
determining whether the number of serving nodes reaches a number threshold;
if the number threshold is reached, the service node meets the removal condition of the first container routing information.
A container management and control method is applied to a service node, and comprises the following steps:
obtaining second removal state information of the management node; the second removal state information represents that the management node has removed the container information of the first container and has stopped receiving and/or forwarding the connection request facing the first container;
removing routing information of the first container;
sending first removal state information of the first container routing information to the management node, so that the management node sets the state of the first container to be a second state when determining that the service node meets the removal condition of the first container routing information based on the first removal state information of at least one service node;
executing a predetermined task process before the first container is stopped, based on the second state;
and executing container deletion operation on the first container under the condition that a termination condition is met.
Preferably, the obtaining the second removal state information of the management node includes:
acquiring a first notification message sent by a management node;
the first notification message includes: the management node has removed the container information of the first container and has stopped receiving and/or forwarding the second removal state information of the connection request facing the first container;
after obtaining a deletion request for a first container, the management node sets the state of the first container to a first state, removes container information of the first container based on the first state, and stops receiving and/or forwarding a connection request facing the first container.
Preferably, the sending the first removal state information of the first container routing information to the management node includes:
after removing the routing information, sending a second notification message to the management node;
wherein the second notification message comprises: the service node has removed the first removal state information of the first container routing information.
Preferably, the executing the predetermined task processing before the first container is stopped includes:
and storing the cache region data of the first container.
Preferably, the performing a container deletion operation on the first container when the termination condition is satisfied includes:
obtaining network state information of the first container and/or storage condition information of the cache region data;
when the network state information meets the state condition, controlling the first container to stop working;
and deleting the first container when the time length of the first container stopping working reaches a time length threshold value and/or the cache area data is saved.
Preferably, after the deleting operation of the first container is completed, the method further includes:
sending a third notification message to the management node, where the third notification message includes deletion state information that the service node has deleted the first container, so that the management node sets the state of the first container to a third state.
A processing node, comprising:
a memory for storing at least one set of computer instructions;
and the processor is used for implementing the container management and control method applied to the management node or implementing the container management and control method applied to the service node by executing the instruction set stored in the memory.
According to the above scheme, according to the container management and control method and the processing node provided by the application, after the management node removes the container information of the first container based on the obtained deletion request for the first container and stops receiving and/or forwarding the connection request facing the first container, the service node removes the routing information of the first container and feeds back the first removal state information of the first container routing information to the management node, the management node sets the state of the first container to the second state when determining that the service node meets the removal condition for the first container routing information based on at least part of the obtained first removal state information, and the service node triggers and executes the deletion operation for the first container based on the second state of the first container. Therefore, the execution sequence of each component processing flow in the container deleting process is strictly controlled, the correct processing logic of the container deleting can be ensured, and data loss or service interruption in the container deleting process is avoided.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a schematic diagram of a process for deleting a container in a container cluster management system architecture in the prior art;
fig. 2 is a schematic flowchart of a container management method applied to a management node according to an embodiment of the present application;
fig. 3 is another schematic flow chart of a container management method applied to a management node according to an embodiment of the present application;
FIG. 4 is a container deletion intermediate state transition diagram provided by an embodiment of the present application;
fig. 5 is a schematic flowchart of a container management method applied to a management node according to an embodiment of the present application;
fig. 6 is a schematic flowchart of a container management method applied to a management node according to an embodiment of the present application;
FIG. 7 is an exemplary diagram of a newly added (extended) count field provided by an embodiment of the present application;
fig. 8 is a schematic processing diagram of deleting a container in an improved container cluster management system architecture according to an embodiment of the present application;
fig. 9 is a schematic flowchart of a container management and control method applied to a service node according to an embodiment of the present application;
fig. 10 is a schematic flowchart of another container management method applied to a service node according to an embodiment of the present application;
fig. 11 is a schematic structural diagram of a processing node according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
First, with reference to fig. 1, a structure composition of a container cluster management system and an implementation process of deleting a container in the conventional art are briefly described.
As shown in fig. 1, the container cluster management system includes a management node and a plurality of service nodes connected to the management node. The management node comprises an API (interface) server, an Etcd database and an endpoint controller, wherein the API server is responsible for monitoring container management requests, such as requests for deleting a container and creating a container, and triggering a corresponding container management process, the Etcd database synchronously stores the states of the containers based on the container management of the API server, and records access addresses of all the points corresponding to the service by adopting an endpoint object, and the endpoint controller is one of the components of the cluster controller and is responsible for monitoring the service and the change of the corresponding point and maintaining all the endpoint objects in the service list based on the monitored service and the change of the corresponding point. The service node comprises a kubbelet component, a kube-proxy and a Prestop module, wherein the kubbelet component is used for processing tasks issued by the management node to the node, the kubbe-proxy adopts an iptables mode to configure load balancing, load distribution is carried out based on a routing forwarding rule provided by an iptables table, the Prestop module is used for providing a Prestop hook to be responsible for task processing before container stop, in addition, a pod used for loading the container is operated on the service node, and the container is used for deploying and operating services.
Taking K8s as an example, when the K8s cluster deletes a container, the API server first sets the state of the container requested to be deleted to the Terminating state. Each component in the cluster node (including the management node and the service node) monitors the state and starts to synchronously execute respective processing flows, for example, kubel calls a prestop hook to request a container to store cache data, an endpoint controller removes the information of the container (the point where the container is located) from a service list, and stops the request forwarding to the container; the kubel sends a SIGTERM signal to the container to inform the container to clear resources; and sending a SIGKILL signal by the kubelet to forcibly delete the container, notifying an API server after deletion, and setting the state of the container to be terminal by the API server.
However, in the process of deleting the container, the flows are executed concurrently, the sequence of each flow cannot be strictly controlled, and the service may be affected because the execution sequence is not reasonable. For example, a prestop hook may not keep the cached data finished yet, kubelet may start to forcibly delete the container, or an endpoint controller may not stop forwarding a new service connection request, kubelet may start to send a SIGTERM signal to notify the container to clear the resources and close the network connection, etc., which may result in data loss and service interruption.
Therefore, the application discloses a container management and control method and a processing node, which are used for at least solving the technical problems existing in the container deleting process in the traditional technology, ensuring the correct processing logic of the container deleting process and avoiding data loss or service interruption in the container deleting process.
In an alternative embodiment, referring to fig. 2, a method for container management is provided, and the method of this embodiment may be applied to, but is not limited to, a management node of a container cluster management system in a container cluster management scenario.
As shown in fig. 2, the above-mentioned container management and control method applied to the management node may include the following processing procedures:
When the container resource provided by the container cluster is used to deploy a use object (such as a service provider) for implementing a corresponding service (service), and when the service-based business scale and other factors have container capacity reduction requirements, a deletion request for a specified container (i.e., the first container) can be submitted to a management node of the container cluster management system.
The management node may accordingly listen for the request and obtain the request. Specifically, the API server in the management node may monitor and obtain the delete request for the first container.
The management node stores the corresponding relation information of the service and at least one container for bearing the service.
In implementation, the correspondence information may be implemented in the form of an endpoint object, where endpoint is a resource object in container management and is used to record access addresses of all points corresponding to one service. When a service is created in the cluster, an endpoint object corresponding to the service is generated and used for recording address information such as ip addresses and ports of all the points corresponding to the service.
Therefore, the correspondence information between the service stored by the management node and the at least one container carrying the service may be specifically implemented as: address information of all the pods corresponding to the service is served, or association relationship between the pods and the containers can be recorded in combination. Such information may be recorded and maintained in the Etcd database of the management node in the form of a service list or the like.
The pod is the smallest unit of the container cluster management system for implementing container management, the container is contained in a pod, and one pod usually has one container or multiple logically associated containers, and the removal/deletion or creation of the container is implemented by removing/deleting or creating the pod. A complete service is deployed on only one container or multiple logically-associated containers running on one pod, and one service may be deployed and run on containers of multiple pods based on a business scale requirement to implement multiple runs of the same service, or, if the business scale is small, one service may be deployed on only one container of one pod (only one run correspondingly).
In the embodiment of the present application, the deletion of the container is essentially a deletion of the pod, and conversely, the deletion of the container is realized by deleting the pod, which is the minimum management unit in the container management, so that, for the deletion operation of the first container, if the first container is the only container on the pod where the first container is located, the deletion of the first container itself is meant, and if the first container is one of a plurality of associated containers on the pod where the first container is located, the deletion of the plurality of associated containers on the pod where the first container is located is essentially simultaneous deletion.
Therefore, for the deletion request of the first container, the information carried by the deletion request may be the information of the specific container to be deleted, such as a container identifier, and may also carry the information of the pod where the container to be deleted is located, such as a pod identifier.
In response to the obtained deletion request for the first container, the management node triggers a deletion process for the requested first container.
In the process of deleting the container, the management node first removes (i.e., deletes) the container information of the first container from the correspondence information between the service stored in the management node and the at least one container carrying the service. Specifically, the information about the pod where the first container is located may be deleted from the service list, such as deleting the ip address and the port of the pod where the first container is located, and stopping receiving and/or forwarding the connection request for the first container.
Here, the connection request refers to a service connection request submitted to a service when a user (e.g., a user of an application) needs to use the service deployed in the container. When the container information of the first container is removed, the receiving and/or forwarding of the connection request facing the first container is stopped, so that a new service connection for the first container to be deleted is not constructed in the process of deleting the first container, and the service interruption of the new service connection caused by the subsequent deletion of the first container is avoided.
The first removal state information is generated when the service node removes the routing information of the first container after the management node has removed the container information of the first container and stops the receiving and/or forwarding operations. That is, the service node removes the routing information of the first container at the service node itself after the management node has removed the container information of the first container and stops receiving and/or forwarding the connection request directed to the first container.
If the service node directly removes the routing information of the first container from itself based on monitoring of the API server (that is, monitoring that the first container needs to be deleted) before the management node removes the container information of the first container and stops receiving and/or forwarding the connection request for the first container (e.g., in a processing manner of the conventional technology), a service connection request for the first container forwarded by the management node may be caused, a routing failure may be triggered due to a missing routing information, and for a front-end user, a desired service may not be successfully acquired. The method and the system solve the problem through strict control of the sequence of execution of the actions of the management node and the service node.
The container cluster management system comprises a plurality of service nodes, wherein each service node maintains a routing forwarding rule for providing a service connection request so as to realize load balancing of different pods, and the routing forwarding rule can be specifically realized in a forwarding table form, such as an iptables table. After monitoring that the management node has removed the container information of the first container and stops receiving and/or forwarding the connection request for the first container, the service node may specifically delete forwarding table information related to the first container from the iptables.
Optionally, each service node in the container cluster management system performs the above-mentioned route information deleting operation for the first container.
In addition, after deleting the routing information of the first container, the service node may feed back, to the management node, first removal state information for representing a deletion state (e.g., a state indicating whether the deletion is successful or not) of the routing information of the first container.
The management node can know whether each service node successfully removes the routing information of the first container or not based on the obtained first removal state information, and further determine whether each service node meets the removal condition of the routing information of the first container or not.
The removal conditions described above may be set as: all service nodes (all service nodes which normally operate) in the container cluster management system successfully delete the reason information of the first container; alternatively, it may be set such that: the reason information that the service node (normally operating service node) reaching the set number threshold in the container cluster management system successfully deletes the first container is not limited.
If the removal condition is not met, the management node continues to wait and receive the first removal state information of the service node in the system until the state of the first container is set to be the second state.
The second state may specifically be a Terminating state, i.e. a Terminating state.
The Terminating state is a state adopted when the container is deleted in the conventional technology, and according to the description of the container deleting process in the conventional technology, it can be known that the API server directly sets the state of the container requested to be deleted as Terminating when receiving a deletion request for the container.
The present application continues to use the Terminating status, but differs from the conventional technology in that the state of the first container is set to the state after the management node removes the container information of the first container, stops receiving and/or forwarding the connection request for the first container, and the service node removes the routing information of the first container, that is, different from the setting timing of the conventional technology, and subsequently, the service node running the first container triggers a container deleting operation for the first container to complete deletion of the first container based on the monitored Terminating status, it is easy to understand that, since the series of processes of the above step 201 and step 204 are already performed before the Terminating status, the container deleting process (no longer including the step 204) is performed in response to the Terminating status of the first container, which is different from the process performed by the conventional technology in response to the Terminating status, thus, the response strategy for this Terminating state is also different from the conventional technique.
In the present application, the state of the first container is set to the second state (Terminating state) at the above timing, which essentially corresponds to dividing the container deleting process into two stages from a macroscopic perspective: a pre-delete phase (pre-processing phase) and a delete phase.
The processing procedure of step 201 and step 204 can be understood as a pre-deletion phase, and based on the processing procedure of this phase, it may be implemented to block, for the first container to be deleted, the route forwarding of the new service connection request (the service connection request facing the first container generated in the time period from the reception of the first container deletion request to the completion of the deletion of the first container) of the front-end user to the first container, and accordingly, it may avoid the interruption of the service corresponding to this part of the service request due to the deletion of the first container.
Specifically, in the pre-deletion phase, if the user issues a service connection request, and the service to which the service connection request is directed operates in other containers (pod) in addition to the first container, that is, multiple copies of the service are operated, the management node and the service node respectively remove the container information (pod information) and the routing information of the first container, so that the connection request of the user is forwarded to other containers providing the service (excluding the possibility of forwarding the route to the first container) based on the updated container information (pod information) and routing information of the requested service in the management node and the service node, and thus, subsequent deletion of the first container does not cause interruption of the service corresponding to the new connection.
Of course, for services that run with a single share based on only one container (one pod), deletion of the container will necessarily result in revocation of the service, and the user front-end request will not be responded to.
The process of the delete phase will be described in detail later.
The embodiment strictly controls the execution sequence of the processing flows of the management node and each component of the processing node in the container deleting process, can ensure the correct processing logic of the container deleting process, and avoids data loss or service interruption in the container deleting process.
In an alternative embodiment, referring to fig. 3, after obtaining a delete request for a first container (step 201), the container management and control method applied to a management node may further include:
step 201', the state of the first container is set to a first state.
The first state may specifically be pre _ terminate.
After receiving a delete request for a first container, an API server in a management node first sets the first container to a first state, i.e., a pre _ terminate state.
Alternatively, this pre _ terminate state of the first container may be recorded and saved at the same time, e.g. into the Etcd database of the management node.
Referring to fig. 4 in combination, the first state (pre _ terminate state) is a state where the container deletion processing is newly added (extended) in the embodiment of the present application, and in the conventional technology, two states are involved in the container deletion processing: the terminal state is that the API server sets the container to the state when receiving a deletion request for the container, and the terminal state is that the API server sets the container to the state when receiving a notification of the service node that the deletion of the container is completed.
On this basis, as shown in fig. 3, the step 202 of removing the container information of the first container and stopping receiving and/or forwarding the connection request for the first container may specifically be implemented as the following step 2021 and 2022:
In an implementation, the Endpoint controller in the management node may monitor the container status, and after monitoring the above-mentioned first status of the first container, such as the pre _ terminate status, remove (i.e., delete) the container information of the first container from the correspondence information between the service stored in the management node itself and at least one container carrying the service.
Specifically, the information about the pod where the first container is located may be deleted from the service list, such as deleting the ip address and the port of the pod where the first container is located, and stopping receiving and/or forwarding the connection request for the first container.
In the embodiment of the application, when a container deletion request is received, the container to be deleted is set to be a newly added first state (pre _ terminate) first, only the related processing of the pre-deletion stage is executed based on the pre _ terminate state, and the first container is set to be the Terminating state after the related processing of the pre-deletion stage is finished, so that the execution sequence of each flow step in the container deletion process is effectively controlled.
In an optional embodiment, referring to fig. 5, before obtaining the first removal state information of the first container routing information of the at least one service node (step 203), the method for container management applied to the management node may further include:
step 203', sends a first notification message to each serving node.
The first notification message includes: the management node has removed the container information of the first container and has stopped receiving and/or forwarding the second removal state information of the connection request.
That is, after completing removing the container information of the first container and stopping receiving and/or forwarding the connection request facing the first container, the management node may send a notification message, i.e., the above-mentioned first notification message, to each service node in the container cluster management system to notify each service node that it (management node) has completed removing the container information of the first container and stop receiving and/or forwarding the event of the connection request facing the first container, so that each service starts to perform the operation of removing the routing information of the first container.
The first notification message may be implemented in the form of data information or an instruction signal.
For the form of the data information, the second removal state information carried in the first notification message may be any agreed information that can be used to indicate that the management node has removed the container information of the first container and has stopped receiving and/or forwarding the connection request, such as a word, or a group of letters, special characters or numbers, or a combination of any several of the letters, the numbers and the special characters.
In the implementation form of the instruction signal, the management node may send an agreed instruction signal to the service node, where the agreed instruction signal can be used to indicate that the management node has removed the container information of the first container and has stopped receiving and/or forwarding the connection request, and is detected and received by a listening pin of a corresponding listening device (e.g., a listening device of the kube-proxy) of the service node.
The function of sending the first notification message to the respective service nodes may be performed by an API server or an endpoint controller in the management node.
By sending the first notification message carrying the second removal state information to each service node in the cluster, each service node can trigger execution of operation processing on the routing information of the first container based on the received message, so that strict control of the execution sequence of each step is realized.
In an alternative embodiment, referring to fig. 6, in the above container management and control method applied to a management node, the step 203 of obtaining first removal state information of at least one service node may include:
the second notification message includes: the service node has removed the first removal state information of the first container routing information.
That is, the service node may specifically feed back the first removal state information to the management node in the form of a notification message, and preferably, the first removal state may indicate that the service node has removed the first container routing information.
The second notification message may also be implemented in the form of data information or an instruction signal.
For the form of the data information, the first removal status information carried in the second notification message may be any agreed information that can be used to indicate that the service node has removed the first container routing information, such as a word, or a group of letters, special characters, or numbers, or a combination of any several of the letters, the numbers, and the special characters, and the like, which is not limited herein.
In the implementation form of the command signal, the service node may send an agreed command signal to the management node, which can be used to indicate that the service node has removed the first container routing information, and the agreed command signal is detected and received by a listening pin of a corresponding listening device (e.g., a listening device of an API server in the management node) of the management node.
And the management node receives a second notification message which is fed back by a certain service node and carries the first removal state information, namely the service node is considered to finish deleting the first container routing information.
Correspondingly, determining whether the service node satisfies the removal condition for the first container routing information based on the obtained at least part of the first removal state information may be implemented by the following processing procedures:
determining the number of service nodes with the first container routing information removed according to the number of the received second notification messages, and recording the number of the service nodes in a preset data structure field;
determining whether the number of serving nodes reaches a number threshold;
if the number threshold is reached, the service node meets the removal condition of the first container routing information.
The number threshold may be set to the number of all service nodes in the cluster management node that are normally operating, or set to a value smaller than but close to the number of all service nodes that are normally operating.
In the implementation, a count field may be added to a corresponding data structure of a functional program (computer program) of the API server for implementing a processing function thereof, and the field is initialized to 0 for a deletion processing procedure of each container, and in the processing procedure of deleting a container, when the API server receives a second notification message (indicating that the service node has completed deleting the first container routing information) fed back by one service node, a value of the field is increased by 1, so that the number of service nodes in the cluster that have completed deleting the first container routing information may be known based on reading a value of the field.
Specifically, in actual implementation, it is usually the responsibility of the kube-proxy in the service node to delete the forwarding table information related to the routing rule of the pod corresponding to the first container from the iptables, so as to, as shown in fig. 7, in this embodiment, add a pod data structure field in the functional program of the API server in advance: and the Kube-proxy return _ count is used for recording the number of Kube-proxy executed tasks, wherein the API server in the management node increases the value of the field of the Kube-proxy return _ count by 1 every time the API server receives a second notification message of one service node.
Referring to fig. 8 and fig. 1 in combination, in the present embodiment, a structural improvement is performed on a management node of a container cluster management system, a controller, such as a proxy controller in fig. 8, is additionally arranged in the management node of the system, and the controller is used to monitor a change in a value of a return _ count field and determine whether a value of the monitored field reaches a set threshold, so as to know whether each service node in the system meets a removal condition, and when the value is determined to be yes, an API server is notified, so that the API server sets the state of a first container to a second state, such as the Terminating state, and when the value is determined to be no, the API server continues to monitor the value of the field.
In implementation, the functions of value monitoring, condition determination and the like of the newly added field may be added to the API server or the endpoint controller as an extended function of the API server or the endpoint controller. In addition, the implementation manner based on the added field may not be adopted, and other manners may also be adopted, for example, when the API server does not receive the second notification message of the service node beyond the set time length, the number of service nodes in the cluster from which the removal of the first container routing information is completed is obtained based on a means for counting the number of currently received second notification messages.
In match with the container management and control method applied to the management node, the present application also provides a container management and control method applied to the service node, and referring to fig. 9, in an alternative embodiment, the container management method applied to the service node may include:
and step 901, obtaining second removal state information of the management node.
Each service node can obtain second removal state information of the management node by monitoring the change of a service list which is maintained by the end point controller in the management node (for example, monitoring whether the container or the point information in the service list is deleted) and the forwarding action of the end point controller to the service connection request, wherein the second removal state information represents that the management node has removed the container information of the first container and has stopped receiving and/or forwarding the connection request facing the first container.
In implementation, after the management node has removed the container information of the first container and has stopped receiving and/or forwarding the connection request for the first container, the management node may send a first notification message to each service node; the first notification message includes: second removal state information characterizing that the management node has removed the container information of the first container and has stopped receiving and/or forwarding connection requests towards the first container.
Therefore, each service node can obtain the second removal state information by monitoring the first notification message.
The first notification message may be implemented in the form of data information or an instruction signal, and for more detailed implementation, reference may be made to the above description related to the first notification message, which is not described herein again.
In an implementation, the first notification message in the form of the data information or the instruction signal may be specifically listened to by the kube-proxy in the service node.
And 902, removing the routing information of the first container.
After the first notification message is monitored by the kube-proxy of the service node, the routing information of the first container is removed (i.e., deleted) from the routing forwarding rule table maintained by the service node for providing the service connection request, for example, the relevant forwarding table information of the pod where the first container is located is removed from the iptables table.
After removing the routing information of the first container, the kube-proxy feeds back first removal state information of the routing information of the first container to the API server of the management node so as to inform the API server of the management node whether the first container has been removed by the kube-proxy or not.
Alternatively, the kube-proxy may send the first removal state information indicating that the first container routing information has been successfully removed to the API server only in case that the first container routing information is successfully removed, and not send in case of an incomplete removal or a removal failure.
The kube-proxy may specifically send a second notification message to an API server of the management node, where the second notification message includes: the service node has removed the first removal state information of the first container routing information.
The second notification message may also be implemented in the form of data information or an instruction signal, and for more detailed implementation, reference may be made to the above description related to the second notification message, which is not described again.
The API server of the management node receives a second notification message carrying the first removal state information fed back by the kube-proxy of a certain service node, considers that the service node has completed/successfully removed the routing information of the first container, and may determine whether the number of the obtained second notification messages (the number of the second notification messages, specifically, may be obtained by monitoring the value of the return _ count) satisfies a number threshold, to determine whether the service node in the container cluster management system satisfies the removal condition, and if so, set the state of the first container to the second state, such as the Terminating state described above.
And 904, executing the predetermined task processing before the first container stops based on the second state.
Specifically, the Kubelet in the service node is responsible for monitoring the state setting information of the API server in the management node on the first container, and when the state of the first container is monitored to be set to the second state (the monitoring state), the Prestop module is triggered to start executing the task processing before the container stops based on the Prestop hook.
The processing of the task before the container stop is performed by using the Prestop hook may include, but is not limited to, saving the buffer data of the first container to provide a data basis for at least a source-tracing or analysis of the running information of the first container.
In the process that the first container is in the terminating state, although the container information and the routing information of the first container are deleted at the management node and the service node respectively based on the processing in the pre-deletion stage (it is ensured that no new service connection request is forwarded to the first container in the process of deleting the first container), there may still be a service connection that has been established with the first container before the container information and the routing information of the first container are deleted or before a deletion request for the first container is received (that is, a user terminal with a service requirement has established a service connection with the first container before the management node obtains the deletion request for the first container or before the container information of the first container is deleted).
For the situation, in order to ensure that the service of the existing connection is not interrupted, the normal operation of the first container needs to be maintained, the operation status of the existing connection of the first container is monitored, and when the monitored operation status information represents that the data processing of the existing connection is completed (i.e., the service is stopped, which correspondingly indicates that the front-end user no longer uses the service running in the first container), the first container is deleted.
In addition, it is easily understood that deletion of the first container may be premised on that the storage of the cache area data of the first container is completed.
Thus, the above-described termination condition may be set as: the time length of the first container stopping working reaches the time length threshold value and/or the first container cache region data is saved. Among them, the termination condition may be preferably set as: the time length of the first container stopping working reaches the time length threshold value and the cache area data of the first container is stored.
Please refer to fig. 8 and fig. 1 in combination, in order to implement the above processing, this embodiment further improves the service node, wherein a container deletion monitoring agent is added in the PreStop module of the service node, the container deletion monitoring agent is used to monitor the network connection state of the existing connection of the first container to be deleted, if a connection termination packet on the existing connection is monitored or the throughput of the existing connection is monitored to be lower than a set throughput threshold, it is considered that the data processing of the existing connection is completed, at this time, the operation of the first container may be stopped, and accordingly, the operation of the pod where the first container is located may be stopped.
The Kubelet can request the first container to stop working by sending a SIGTERM command to the first container, and the first container responds to the command after receiving the command, cleans resources, closes network connection and stops working.
The monitored network connection state information may specifically be, but is not limited to, TCP (transmission control protocol) network connection state information, and the connection termination packet may correspondingly be a FIN, an ACK packet, and the like.
When the time length of the first container stopping operation reaches the time length threshold value, the first container can be deleted, and the deletion of the first container obviously cannot cause service interruption of the existing connection.
In addition, the storage condition information of the first container cache region data can be acquired, and when the acquired storage condition information represents that the cache region data of the first container is stored, deleting the first container does not correspondingly cause the loss of the first container cache region data.
Wherein, based on the set termination condition: the time length of the first container stopping working reaches the time length threshold value and/or the first container cache region data is saved. The first container may be deleted if the length of time that the first container has been out of service reaches a length threshold and/or if the first container buffer data has been saved.
In order to avoid service interruption of the service running on the first container, ensure that the complete cache region data of the first container is stored, and avoid data loss, the first container may be deleted under the condition that the time length for which the first container stops working reaches the time length threshold and the cache region data of the first container is already stored. Specifically, the first container may be deleted based on a SIGKILL command issued by kubelet.
The operation of deleting the first container routing information, which is performed by the service node based on the second removal state information of the management node, belongs to the processing of the pre-deletion phase described above, and specifically, the processing may be triggered by the service node based on monitoring of container information indicating that the first container is removed and second removal state information indicating that the forwarding of the connection request for the first container is stopped of the management node after the API server of the management node sets the state of the first container to the first state (pre _ terminate state) based on the received deletion operation for the first container and removes the container information of the first container based on the first state and stops forwarding the connection request for the first container.
After monitoring the second state (Terminating state) of the first container, the service node performs a series of processing such as processing of a predetermined task before the first container is stopped, and a first container deletion operation when a termination condition is satisfied, and belongs to the processing in the deletion stage.
In this embodiment, the service node removes the routing information of the first container after the management node has removed the container information of the first container and stops receiving and/or forwarding the connection request for the first container, and executes the predetermined task processing before the first container is stopped and executes the container deletion operation on the first container when it is monitored that the management node sets the state of the first container to the second state, so that the execution sequence of the processing flows of the components in the container deletion process is strictly controlled, the correct processing logic of the container deletion process can be ensured, and data loss or service interruption in the container deletion process is avoided.
In an alternative embodiment, as shown in fig. 10, the above container management and control method applied to the service node may further include the following processing after the first container is deleted:
The third notification message includes deletion status information that the serving node has deleted the first container to cause the managing node to set the status of the first container to a third status.
Wherein the third state may specifically be a terminated state.
The third notification message may be implemented in the form of data information or an instruction signal.
For the form of the data message, the deletion status information carried in the third notification message may be any agreed information that can be used to indicate that the service node has deleted the first container, such as a word, or a group of letters, special characters, or numbers, or a combination of any several of the letters, the numbers, and the special characters.
In the implementation form of the command signal, the service node may send an agreed command signal to the management node, which can be used to indicate that the service node has deleted the first container, and the agreed command signal is detected and received by a corresponding snoop pin of a corresponding snoop device (e.g., a snoop device of the API server) of the management node.
Based on the obtained deletion state information of the deleted first container, the API server of the management node correspondingly sets the first container to a third state (a terminated state) to achieve synchronization with the actual state of the first container (the deleted actual state); further optionally, the third state (terminated state) of the first container may also be recorded to an Etcd database of the management node.
Illustratively, as shown in fig. 8, according to the container management and control method disclosed in the present application, the management node and the service node in the container cluster management system may delete a container through the following process flows:
1) the API server receives a container deletion request, and firstly, the container to be deleted is set to be in a pre _ terminate state;
2) the Endpoint controller monitors a pre _ terminate state, removes information of a container to be deleted from a service list, and stops forwarding a connection request facing the container;
3) the method comprises the following steps that a kube-proxy on each service node monitors an event of endpoint list change and stopping forwarding of a connection request facing a container, a forwarding table related to the container is deleted from an iptables, and an API server is notified after deletion;
4) when receiving a notification message of a service node, the API server increases the return _ count of the kube-proxy field by 1;
5) a Proxy controller in a management node monitors the change of the number of the kube-Proxy for completing the task (namely, a return _ count value), judges whether all the kube-Proxy have completed the task of deleting the container routing forwarding table (of course, whether the kube-Proxy for completing the task of deleting the container routing forwarding table reaches a set threshold value can also be judged), and if all the tasks are completed, informs an API server to set the container state as terminating;
6) the Kubelet component monitors the terminating state, triggers the Prestop module to start executing the task processing before the container stops (such as storing the data in the container cache region), loads the container deletion monitoring agent, and monitors the TCP network connection state of the existing connection of the to-be-deleted container, the throughput of each existing connection, FIN, ACK message and the like;
7) when the Kubelet component detects a connection termination message (FIN, ACK message, etc.) or detects that the throughput is lower than a certain threshold, it is considered that the existing connection data is processed, and the pod operation can be stopped. The Kubelet sends a SIGTERM command to the container to request the container to stop working;
8) the kubelet component waits for a period of time and ensures that the cache has been saved, and then sends a SIGKILL command to perform a container delete operation.
The embodiment of the present application also discloses a processing node, and referring to fig. 11, the processing node includes a memory 1101 and a processor 1102.
A memory 1101 for storing at least one set of computer instructions.
The memory 1101 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
The set of instructions in the memory 1101 may be embodied in the form of a computer program.
The processor 1102 is configured to implement the container management method applied to the management node according to any one of the embodiments above, or implement the container management method applied to the service node according to any one of the embodiments above, by executing the instruction set stored in the memory.
The processor 1102 may be a Central Processing Unit (CPU), an application-specific integrated circuit (ASIC), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, etc.
Besides, the processing node may further include a communication interface, a communication bus, and other components. The memory, the processor and the communication interface communicate with each other via a communication bus.
The communication interface is used for processing communication between the node and other devices. The communication bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like, and may be divided into an address bus, a data bus, a control bus, and the like.
For a specific processing procedure of the processor in the processing node by executing the instruction set, reference may be made to the related description of the above embodiments, and details are not described again.
It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other.
For convenience of description, the above system or apparatus is described as being divided into various modules or units by function, respectively. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.
From the above description of the embodiments, it is clear to those skilled in the art that the present application can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present application may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments of the present application.
Finally, it is further noted that, herein, relational terms such as first, second, third, fourth, and the like may be used solely to distinguish one instance or operation from another instance or operation without necessarily requiring or implying any actual such relationship or order between such instances or operations. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.
Claims (12)
1. A container management and control method is applied to a management node, and comprises the following steps:
obtaining a delete request for the first container;
removing the container information of the first container and stopping receiving and/or forwarding the connection request facing the first container; the management node stores the corresponding relation information of at least one container of the service and the bearing service;
obtaining first removal state information of first container routing information of at least one service node; the first removal state information is generated when the serving node removes routing information for the first container after the management node has removed the container information and stopped the receiving and/or forwarding operations;
setting the state of the first container to a second state in the case that the service node is determined to meet the removal condition of the first container routing information based on at least part of the obtained first removal state information;
wherein the respective service node performs a container deletion operation on the first container based on the second state.
2. The method of claim 1, after obtaining the delete request for the first container, further comprising: setting a state of the first container to a first state;
the removing the container information of the first container and stopping receiving and/or forwarding the connection request facing the first container includes:
monitoring the state of the first container;
based on the monitored first state, removing the container information of the first container and stopping receiving and/or forwarding a connection request towards the first container.
3. The method of claim 1, prior to obtaining the first removal state information for the first container routing information for the at least one serving node, further comprising:
sending a first notification message to each service node;
the first notification message includes: the management node has removed the container information of the first container and has stopped receiving and/or forwarding the second removal state information of the connection request.
4. The method of claim 1, the obtaining first removal state information for at least one serving node, comprising:
receiving a second notification message of at least one service node;
the second notification message includes: the service node has removed the first removal state information of the first container routing information.
5. The method of claim 4, wherein determining whether the serving node satisfies a removal condition for the first container routing information based on the obtained at least part of the first removal state information comprises:
determining the number of service nodes with the first container routing information removed according to the number of the received second notification messages, and recording the number of the service nodes in a preset data structure field;
determining whether the number of serving nodes reaches a number threshold;
if the number threshold is reached, the service node meets the removal condition of the first container routing information.
6. A container management and control method is applied to a service node, and comprises the following steps:
obtaining second removal state information of the management node, wherein the second removal state information represents that the management node has removed the container information of the first container and has stopped receiving and/or forwarding the connection request facing the first container;
removing routing information of the first container;
sending first removal state information of the first container routing information to the management node, so that the management node sets the state of the first container to be a second state when determining that the service node meets the removal condition of the first container routing information based on the first removal state information of at least one service node;
executing a predetermined task process before the first container is stopped, based on the second state;
and executing container deletion operation on the first container under the condition that a termination condition is met.
7. The method of claim 6, the obtaining second removal state information for a management node, comprising:
acquiring a first notification message sent by a management node;
the first notification message includes: the management node has removed the container information of the first container and has stopped receiving and/or forwarding the second removal state information of the connection request facing the first container;
after obtaining a deletion request for a first container, the management node sets the state of the first container to a first state, removes container information of the first container based on the first state, and stops receiving and/or forwarding a connection request facing the first container.
8. The method of claim 6, the sending first removal state information for first container routing information to the management node, comprising:
after removing the routing information, sending a second notification message to the management node;
wherein the second notification message comprises: the service node has removed the first removal state information of the first container routing information.
9. The method of claim 6, the performing a predetermined task processing prior to the first container stopping, comprising:
and storing the cache region data of the first container.
10. The method of claim 9, the performing a container deletion operation on the first container upon satisfaction of a termination condition, comprising:
obtaining network state information of the first container and/or storage condition information of the cache region data;
when the network state information meets the state condition, controlling the first container to stop working;
and deleting the first container when the time length of the first container stopping working reaches a time length threshold value and/or the cache area data is saved.
11. The method of claim 6, after completing the delete operation for the first container, further comprising:
sending a third notification message to the management node, where the third notification message includes deletion state information that the service node has deleted the first container, so that the management node sets the state of the first container to a third state.
12. A processing node, comprising:
a memory for storing at least one set of computer instructions;
a processor for implementing the method of any one of claims 1 to 5, or for implementing the method of any one of claims 6 to 11, by executing a set of instructions stored on the memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010440720.5A CN111552543A (en) | 2020-05-22 | 2020-05-22 | Container management and control method and processing node |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010440720.5A CN111552543A (en) | 2020-05-22 | 2020-05-22 | Container management and control method and processing node |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111552543A true CN111552543A (en) | 2020-08-18 |
Family
ID=72004923
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010440720.5A Pending CN111552543A (en) | 2020-05-22 | 2020-05-22 | Container management and control method and processing node |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111552543A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112084004A (en) * | 2020-09-02 | 2020-12-15 | 中国电力科学研究院有限公司 | Container detection and maintenance method and system for container application |
| CN113626054A (en) * | 2021-08-16 | 2021-11-09 | 聚好看科技股份有限公司 | Business service updating method and device |
| CN113645069A (en) * | 2021-08-06 | 2021-11-12 | 联想(北京)有限公司 | Cluster data processing method and cluster |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102333029A (en) * | 2011-06-23 | 2012-01-25 | 北京新媒传信科技有限公司 | Routing method in server cluster system |
| CN105245373A (en) * | 2015-10-12 | 2016-01-13 | 天津市普迅电力信息技术有限公司 | Construction and operation method of container cloud platform system |
| CN108196940A (en) * | 2017-12-29 | 2018-06-22 | 华为技术有限公司 | Delete the method and relevant device of container |
-
2020
- 2020-05-22 CN CN202010440720.5A patent/CN111552543A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102333029A (en) * | 2011-06-23 | 2012-01-25 | 北京新媒传信科技有限公司 | Routing method in server cluster system |
| CN105245373A (en) * | 2015-10-12 | 2016-01-13 | 天津市普迅电力信息技术有限公司 | Construction and operation method of container cloud platform system |
| CN108196940A (en) * | 2017-12-29 | 2018-06-22 | 华为技术有限公司 | Delete the method and relevant device of container |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112084004A (en) * | 2020-09-02 | 2020-12-15 | 中国电力科学研究院有限公司 | Container detection and maintenance method and system for container application |
| CN113645069A (en) * | 2021-08-06 | 2021-11-12 | 联想(北京)有限公司 | Cluster data processing method and cluster |
| CN113626054A (en) * | 2021-08-16 | 2021-11-09 | 聚好看科技股份有限公司 | Business service updating method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111552543A (en) | Container management and control method and processing node | |
| EP3800926B1 (en) | Alarm method and device | |
| CN112868206A (en) | Method, system and computer readable medium for providing service broker functionality in a telecommunications network core using a service based architecture | |
| EP2442541A1 (en) | Computer telephony integration device and resource management method and incoming call control method | |
| CN105681216A (en) | System and method for network service access control and load balance under NFV (network functions virtualization) architecture | |
| WO2019024679A1 (en) | Method for upgrading network function and upgrade management entity | |
| CN112202918A (en) | Load scheduling method, device, equipment and storage medium for long connection communication | |
| WO2021057605A1 (en) | Device management method, device, system and device, and storage medium | |
| CN112564990B (en) | Management method for switching audio management server | |
| WO2020252724A1 (en) | Log processing method and device, and computer-readable storage medium | |
| CN111130979A (en) | Method and equipment for connecting branch node with central node in SDWAN (software development wide area network) scene | |
| CN113037599A (en) | Message processing method, device and storage medium | |
| WO2023116701A1 (en) | Data information pushing method and apparatus based on internet of things platform, and device and medium | |
| CN111817953A (en) | Method and device for electing master equipment based on Virtual Router Redundancy Protocol (VRRP) | |
| CN111314241A (en) | Task scheduling method and scheduling system | |
| CN110661836B (en) | Message routing method, device and system, and storage medium | |
| CN109688011B (en) | Agent selection method and device based on OpenStack | |
| US9967163B2 (en) | Message system for avoiding processing-performance decline | |
| CN115640169A (en) | Method, system, device and storage medium for ensuring that a master cluster stops providing services | |
| CN107483637B (en) | NFS-based client link management method and device | |
| CN112100556B (en) | Method and system for optimizing message pushing mode | |
| CN111629054B (en) | Message processing method, device and system, electronic equipment and readable storage medium | |
| JP2015114952A (en) | Network system, monitoring control unit, and software verification method | |
| CN114598593A (en) | Message processing method, system, computing device and computer storage medium | |
| CN113515458A (en) | Method and system for reducing resource consumption of test environment based on Envoy plug-in |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |