CN110569109A - container updating method, control node and edge node - Google Patents
container updating method, control node and edge node Download PDFInfo
- Publication number
- CN110569109A CN110569109A CN201910858081.1A CN201910858081A CN110569109A CN 110569109 A CN110569109 A CN 110569109A CN 201910858081 A CN201910858081 A CN 201910858081A CN 110569109 A CN110569109 A CN 110569109A
- Authority
- CN
- China
- Prior art keywords
- container
- resource
- updated
- node
- edge
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/65—Updates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
Abstract
the embodiment of the application provides a container updating method, a control node and an edge node, and relates to the technical field of container management. By carrying out resource locking on the target container group to which the container to be updated belongs, the edge node deletes the container to be updated from the target container group instead of directly deleting the target container group when receiving the update instruction. Because the resources used by the container to be updated are borne in the target container group and the resources of the target container group are locked, the resources of the container to be updated borne by the target container group can be prevented from being released, so that the resources of the container to be updated can be utilized to create a corresponding new version container, and even under the condition of insufficient resources, the container can be updated.
Description
Technical Field
the present application relates to the field of container management technologies, and in particular, to a container updating method, a control node, and an edge node.
background
The container cluster management system provides a series of complete functions such as deployment and operation, resource scheduling, service discovery and dynamic expansion for containerized application, and improves the convenience of large-scale container cluster management. The existing container cluster management system mainly provides the following two ways to update the container: firstly, creating a new version of container, and then deleting an old version of container; second, the old version of the container is deleted before the new version of the container is created.
However, when the resources of the container cluster are insufficient, if the first method is adopted, the resources for creating a new version of the container cannot be applied, and the updating of the container cannot be realized; if the second method is adopted, the resources released by deleting the old version of the container may be preempted by other applications, so that the new version of the container cannot be created, and further the updating of the container fails.
Therefore, the existing container cluster management system cannot realize the updating of the container under the condition of insufficient cluster resources.
disclosure of Invention
In view of the above, an object of the present invention is to provide a container update method, a control node and an edge node, so as to solve the problem in the prior art that the container update cannot be implemented under the condition of insufficient cluster resources.
In order to achieve the above purpose, the embodiments of the present application employ the following technical solutions:
In a first aspect, an embodiment of the present application provides a container update method, which is applied to a control node in a container cluster management system, and the method includes:
Sending a locking notice to an edge node, wherein the locking notice comprises a container identifier of a container to be updated, and enabling the edge node to add a resource locking label to a target container group to which the container to be updated belongs;
Sending an update instruction for the container to be updated to the edge node, so that the edge node deletes the container to be updated from the target container group and creates a new version container corresponding to the container to be updated according to the update instruction when the target container group to which the container to be updated belongs is checked to have the resource locking tag.
In an alternative embodiment, the method further comprises:
Judging whether the edge node is in a resource tension state or not according to the resource information reported by the edge node;
Wherein the lock notification is issued by the control node when the edge node is in a resource-strained state.
In an optional embodiment, the resource information includes: the amount of remaining resources of the edge node;
The judging whether the edge node is in a resource tension state according to the resource information reported by the edge node comprises the following steps:
Judging whether the residual resource quantity meets the preset resource specification of the container to be updated or not according to the residual resource quantity reported by the edge node;
And if not, determining that the edge node is in a resource tension state.
in an optional embodiment, the resource information includes: region information of the edge node;
the judging whether the edge node is in a resource tension state according to the resource information reported by the edge node comprises the following steps:
determining a target area to which the edge node belongs according to the area information reported by the edge node, and acquiring the residual resource amount of each edge node belonging to the target area;
judging whether edge nodes with residual resource quantity meeting the preset resource specification of the container to be updated exist in the target area or not;
And if not, determining that the edge node is in a resource tension state.
In a second aspect, an embodiment of the present application provides a container updating method, which is applied to an edge node in a container cluster management system, and the method includes:
Receiving a locking notification sent by a control node, wherein the locking notification comprises a container identifier of a container to be updated, and adding a resource locking label to a target container group to which the container to be updated belongs according to the locking notification;
Receiving an update instruction aiming at the container to be updated and sent by the control node, and judging whether the target container group to which the container to be updated belongs has the resource locking label or not according to the update instruction;
And if the target container group has the resource locking label, deleting the container to be updated from the target container group, and creating a new version container corresponding to the container to be updated.
in an alternative embodiment, the lock notification is sent by the control node when the edge node where the container to be updated is located is in a resource-tight state; the method further comprises the following steps:
and if the target container group does not have the resource locking label, deleting the target container group, creating a new target container group, and creating a new version container corresponding to the container to be updated in the new target container group.
in an alternative embodiment, the method further comprises:
After the new version container is created, the resource lock tag of the target container group is deleted.
in an optional embodiment, the creating a new version container corresponding to the container to be updated includes:
and creating the new version container according to the parameter information of the container to be updated and the container mirror image information of the new version.
In a third aspect, an embodiment of the present application provides a container update method, which is applied to a container cluster management system, where the system includes a control node and an edge node that are communicatively connected to each other; the control node comprises a main node process and an edge resource controller component, and the edge node comprises a proxy node process and an edge resource reservation component;
the edge resource controller component sends a locking notification to the agent node process through the master node process, wherein the locking notification comprises a container identifier of a container to be updated;
The agent node process receives the locking notice and adds a resource locking label for a target container group to which the container to be updated belongs according to the container identifier in the locking notice;
The edge resource controller component sends an update instruction aiming at the container to be updated to the agent node process through the main node process;
and the agent node process receives the updating instruction, judges whether the target container group to which the container to be updated belongs has a resource locking label, if so, calls the edge resource reservation component, deletes the container to be updated from the target container group, and creates a new version container corresponding to the container to be updated.
in an alternative embodiment, the method further comprises:
The edge resource controller component obtains resource information reported by the edge node through the master node process, and judges whether the edge node is in a resource tension state or not according to the resource information;
wherein the lock notification is issued by the edge resource controller component through the master node process when the edge node is in a resource-strained state.
in an alternative embodiment, the method further comprises:
after the edge resource reservation component creates the new version container, sending a creation success notification to the agent node process;
and the agent node process receives the creation success notification and deletes the resource locking label of the target container group.
in a fourth aspect, an embodiment of the present application provides a control node, including a processor and a machine-readable storage medium, where the machine-readable storage medium has stored thereon machine-executable instructions, which when invoked, cause the processor to implement the method of any one of the foregoing embodiments.
in a fifth aspect, an embodiment of the present application provides an edge node, including a processor and a machine-readable storage medium, where the machine-readable storage medium has stored thereon machine-executable instructions, and when the machine-executable instructions are called, the processor is caused to implement the method newly described in any one of the foregoing embodiments.
In a sixth aspect, embodiments of the present application provide a machine-readable storage medium storing machine-executable instructions, which when executed, implement the method of any one of the foregoing embodiments.
In a seventh aspect, an embodiment of the present application provides a machine-readable storage medium, which stores machine-executable instructions, and when the machine-executable instructions are executed, the method described in any one of the foregoing embodiments is implemented.
compared with the prior art, the container updating method, the control node and the edge node provided by the embodiment of the application have the following beneficial effects:
According to the container updating method, the control node and the edge node, the target container group to which the container to be updated belongs is subjected to resource locking, so that the edge node deletes the container to be updated from the target container group instead of directly deleting the target container group when receiving the updating instruction. Because the resources used by the container to be updated are borne in the target container group and the resources of the target container group are locked, the resources of the container to be updated borne by the target container group can be prevented from being released, so that the resources of the container to be updated can be utilized to create a corresponding new version container, and even under the condition of insufficient resources, the container can be updated.
in order to make the aforementioned objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
in order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 illustrates an architecture diagram of a container cluster management system according to an embodiment of the present application;
FIG. 2 is a schematic flow chart diagram illustrating a container update method according to a first embodiment of the present disclosure;
FIG. 3 is a flow chart illustrating a container update method according to a second embodiment of the present application;
FIG. 4 is a schematic diagram illustrating a distribution of edge nodes according to a second embodiment of the present application;
FIG. 5 is a flow chart illustrating a container update method according to a third embodiment of the present application;
FIG. 6 is a schematic flow chart diagram illustrating a container update method according to a fourth embodiment of the present disclosure;
Fig. 7 is a block diagram illustrating a control node according to a fifth embodiment of the present application;
Fig. 8 is a block diagram illustrating an edge node according to a sixth embodiment of the present application.
Icon: 10-a container cluster management system; 100-a control node; 110 — a master node process; 120-an edge resource controller component; 200. 300, 400-edge nodes; 201-group of containers; 210-proxy node process; 220-edge resource reservation component; 101. 202-a processor; 102. 203-machine-readable storage media; 103. 204-system bus.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present application without making any creative effort, shall fall within the protection scope of the present application.
it is to be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element. Furthermore, the features in the embodiments of the present application may be combined with each other without conflict.
Referring to fig. 1, fig. 1 is a schematic diagram illustrating an architecture of a container cluster management system 10 according to an embodiment of the present disclosure. The container cluster management system 10 includes a control node 100 and at least one edge node, such as the edge nodes 200 and 300 shown in fig. 1, communicatively coupled to each other. The control node 100 and the edge node may be any devices having network communication functions and data processing functions, for example, servers, hosts, etc., which are not limited in this embodiment. In some examples, the control node 100 may be a data center server and the edge nodes may be servers that are closer to the user.
At least one container group is deployed on each edge node, at least one container can run in each container group, and each container corresponds to one service application and is used for providing services required by the service application for a user. The container group is packaged with storage resources, network resources, configuration information, container restart strategies and the like required by container operation.
the control node 100 is used to manage the edge nodes and the container groups and containers on the edge nodes. In detail, the master node process 110 may be deployed on the control node 100, and the proxy node process may be deployed on the edge node, for example, the proxy node process 210 is deployed on the edge node 200, and the communication between the control node 100 and the edge node 200 may be implemented through the interaction between the master node process 110 and the proxy node process 210. In a specific example, when a certain container in the container cluster needs to be updated, the master node process 110 of the control node 100 may determine an edge node where the container to be updated is located, and schedule the proxy node process of the edge node to perform an update operation on the container to be updated.
illustratively, the container cluster management system 10 in the embodiment of the present application may be, for example, a kubernets system, which is an open source version of Google internal scheduling system Borg and is a Platform-as-a-Service (Platform-as-a-Service) Platform with wide application. Correspondingly, the master node process may be a master-server process running on the control node, the proxy node process may be a kubel process running on the edge node, and the container group may be a Pod, for example. It is understood that the container cluster management system 10 may be other similar systems, and the embodiment is not limited thereto.
First embodiment
Referring to fig. 2, fig. 2 is a flowchart illustrating a container updating method according to this embodiment, which can be applied to the control node 100 in the container cluster management system 10 shown in fig. 1, and the following describes each step of the method in detail.
step S210, sending a locking notification to an edge node, where the locking notification includes a container identifier of a container to be updated, so that the edge node adds a resource locking tag to a target container group to which the container to be updated belongs.
Step S220, sending an update instruction for the container to be updated to the edge node, so that the edge node deletes the container to be updated from the target container group according to the update instruction when it is checked that the target container group to which the container to be updated belongs has the resource locking tag, and creates a new version container corresponding to the container to be updated.
In step S210, when the container needs to be updated, the control node 100 may notify the edge node where the container to be updated belongs to perform resource locking on the target container group to which the container to be updated belongs.
there are various ways of determining the container to be updated. For example, the designation may be made by the user. For another example, the edge node may report the operation state information of the container on the edge node to the control node 100 periodically, and the control node 100 may monitor the container according to the operation state information of the container reported by each edge node, and when it is monitored that the mirror image information or other attribute information of any container changes, may determine the container as a container to be updated. For example, it may be determined that the container a is a container to be updated when it is monitored that the mirror information of the container a in the container group 201 shown in fig. 1 is changed.
After determining the container a to be updated, the control node 100 may determine, according to the operation state information of the container reported by each edge node, an edge node where the container a to be updated is located, for example, the edge node 200 shown in fig. 1, and then send a lock notification to the edge node 200, where the lock notification includes a container identifier "a".
The edge node 200 receives the locking notification, and finds the container group to which the container indicated by the container identifier "a" belongs, that is, the container group 201, according to the container identifier "a" carried in the locking notification. The container group 201 may be regarded as the target container group described in this embodiment.
In one embodiment, the resource lock tag may be included in the lock notification, in which case the edge node 200 may add the resource lock tag in the lock notification to the container group 201 when determining that the container group 201 is the target container group. In another embodiment, the resource lock label may be recorded in the edge node 200, and when the edge node 200 determines that the container group 201 is the target container group, the edge node adds the resource lock label stored by the node to the container group 201.
the adding of the resource locking tag to the container group 201 refers to setting the resource locking tag as the attribute information of the container group 201. The resource locking tag may be any identifier that can be recognized by the edge node, for example, the identifier may be a character "keep," which is not limited in this embodiment.
After determining the container a to be updated, the control node 100 further sends an update instruction to the edge node 200 where the container a to be updated is located through step S220, where the update instruction may include an identifier of the container a to be updated, and the update instruction is used to instruct the edge node 200 to perform an update operation on the container a to be updated.
In the related art, when an edge node receives an update instruction of a control node for a certain container, the edge node deletes a container group to which the container belongs, reconstructs the container group, and creates a new version container in the reconstructed container group. Research shows that the container group carries the resources required by the container operation, and once the container group is deleted, the carried resources are also released. Under the condition of insufficient resources, the released resources of the container group after deletion are preempted by other services, so that the container group running a new version container cannot be rebuilt, and further the updating of the container cannot be realized.
Based on the above analysis, in this embodiment, a resource locking tag is skillfully added to the container group 201 to which the container a to be updated belongs in step S210, so as to implement resource locking on the container group 201 to which the container a to be updated belongs, so that when the edge node 200 receives an update instruction for the container a to be updated, it is detected whether the container group 201 to which the container a to be updated belongs has the resource locking tag, if so, the container group 201 is retained, the container a to be updated is deleted from the container group 201, and a new version container a' corresponding to the container a to be updated is created.
Therefore, the resources of the container A to be updated can not be released in the updating process, the resources are prevented from being preempted by other services, and the new version container A' can be created by using the resources after the container A to be updated is deleted, so that the container updating is realized.
Second embodiment
referring to fig. 3, fig. 3 is a flowchart illustrating a container updating method applied to a control node 100 according to the present embodiment, and a specific implementation flow of the method is similar to that of the first embodiment. The difference is that the present embodiment defines the timing at which the control node sends the lock notification.
in this case, the container updating method provided by the present embodiment includes the following steps.
Step S310, according to the resource information reported by the edge node, judging whether the edge node is in a resource tension state.
Step S320, if the edge node is in a resource-deficient state, sending the locking notification to the edge node, where the locking notification includes a container identifier of the container to be updated, so that the edge node adds a resource locking tag to a target container group to which the container to be updated belongs.
Step S330, sending an update instruction for the container to be updated to the edge node, so that the edge node deletes the container to be updated from the target container group and creates a new version container corresponding to the container to be updated according to the update instruction when it is checked that the target container group to which the container to be updated belongs has the resource locking tag.
With regard to the specific implementation flows of step S320 and step S330, reference may be made to the relevant description in the first embodiment.
based on the analysis of the first embodiment, the case where a container on an edge node cannot be updated usually occurs in the case where the resources of the edge node are insufficient. Therefore, the control node 100 may notify the edge node to lock the resource of the target container group to which the container to be updated belongs when the resource of the edge node to which the container to be updated belongs is insufficient.
Therefore, under the condition of sufficient resources, the old version container can be destroyed by deleting the container group to which the container to be updated belongs, and then the new version container is created based on the newly-built container group, so that the environment of the obtained new version container is brand-new and clean, and errors are not easy to occur.
In this embodiment, an edge node is a node closer to a user, and a corresponding container is deployed on the edge node, so that a service can be quickly provided for the user.
It has been found through research that in practical applications, when updating containers on edge nodes, it is generally desirable that the updated containers can provide services to users at substantially the same speed. Therefore, when updating the container, the updated new version container needs to run on the edge node where the old version container is located, or run on the same edge node as the edge node where the old version container is located.
Taking the container a as an example, when monitoring that the container a needs to be updated, the control node 100 preferentially deploys the new version container a ' on the edge node 200 where the container a is located, and if the resources of the edge node 200 are insufficient, deploys the new version container a ' on the edge node 300 belonging to the same area as the edge node 200, so that the service speed of the new version container a ' is substantially consistent with that of the old version container a.
based on the above analysis, there are various ways to determine whether the edge node where the container to be updated is located is in a resource-deficient state according to the resource information reported by the edge node.
In this embodiment, the resource information is reported periodically by the edge node, and may include a remaining resource amount and area information of the edge node, where the remaining resource amount may include, for example, a remaining Central Processing Unit (CPU) resource, a memory resource, and the like. The area information may include a geographic area to which the edge node belongs and an Autonomous System (AS), where the geographic area may be, for example, a city area such AS shenzhen and guangzhou, and the autonomous system may be divided according to different network service providers, for example, may be divided into a unicom autonomous system, a telecommunication autonomous system, and the like.
The container cluster management system 10 includes edge nodes distributed in a plurality of areas, and area information of each edge node is used to define an area to which the edge node belongs. For example, three edge nodes 200, 300 and 400 are shown in fig. 4 in communication with the control node 100, each operating with multiple containers, where the locale of the edge node 200 is "guangzhou" and the autonomous system is "telecom", the locale of the edge node 300 is "guangzhou" and the autonomous system is "telecom", and the locale of the edge node 400 is "shenzhen" and the autonomous system is "Unicom".
In the first way, the control node may send the lock notification when the edge node resource where the container to be updated is located is insufficient, and then step S310 may be implemented by the following procedure:
Judging whether the residual resource quantity meets the preset resource specification of the container to be updated or not according to the residual resource quantity reported by the edge node;
if yes, determining that the edge node is not in a resource tension state;
And if not, determining that the edge node is in a resource tension state.
the preset resource specification refers to the amount of resources required for operating the container to be updated.
in the second way, the control node may send the lock notification when the resources of each edge node are insufficient in the same area to which the edge node where the container to be updated belongs. In this case, step S310 may be implemented by a flow as follows:
Determining a target area to which the edge node belongs according to the area information reported by the edge node, and acquiring the residual resource amount of each edge node belonging to the target area;
Judging whether edge nodes with residual resource quantity meeting the preset resource specification of the container to be updated exist in the target area or not;
And if so, determining that the edge node is not in a resource tension state.
And if not, determining that the edge node is in a resource tension state.
The preset resource specification refers to the amount of resources required for operating the container to be updated.
Referring to fig. 4, the flow of the second method is described in detail below by taking the container a to be updated as an example:
The control node 100 may determine that the area information of the edge node 200 where the container a to be updated is located includes "guangzhou" and "telecommunications", and may determine that the edge node belongs to a telecommunications autonomous system area in the guangzhou area according to the area information, and then may determine that the edge nodes whose area information includes "guangzhou" and "telecommunications" have 200 and 300 according to the resource information reported by each edge node, and then obtain respective remaining resource amounts of the edge nodes 200 and 300 from the resource information reported by each edge node, and respectively determine whether the respective remaining resource amounts of the edge nodes 200 and 300 reach a preset resource specification of the container a to be updated (for example, a resource amount required for operating the container a to be updated).
if the remaining resource amount of any one of the edge nodes 200 and 300 reaches the resource amount required for running the container a to be updated, it can be determined that the edge node 200 is not in a resource tension state. If the remaining resource amounts of the edge nodes 200 and 300 do not reach the resource amount required for running the container a to be updated, it is determined that the edge node 200 is in a resource tense state.
third embodiment
Referring to fig. 5, fig. 5 is a container updating method provided by the present embodiment, which can be applied to an edge node, such as the edge node 200, in the container cluster management system 10 shown in fig. 1. The individual steps of the method are explained in detail below.
step S510, receiving a locking notification sent by a control node, where the locking notification includes a container identifier of a container to be updated, and adding a resource locking tag to a target container group to which the container to be updated belongs according to the locking notification.
step S520, receiving an update instruction for the container to be updated sent by the control node, and determining whether the target container group to which the container to be updated belongs has the resource locking tag according to the update instruction. If yes, go to step S530; if not, step S540 may be performed.
Step S530, delete the container to be updated from the target container group, and create a new version container corresponding to the container to be updated.
And step S540, deleting the target container group, creating a new target container group, and creating a new version container corresponding to the container to be updated in the new target container group.
In step S510, the edge node 200 may receive the lock notification through the proxy node process 210, recognize an identifier of the container to be updated carried in the lock notification, for example, recognize an identifier "a", and further determine, by searching, a container group 201 to which the container a to be updated belongs, and then add a resource lock tag to the container group 201, that is: the resource lock tag is set as the attribute information of the container group 201. The resource locking identifier may be obtained from the locking notification, or may be pre-stored on the edge node 200, which is not limited in this embodiment.
Regarding step S520, there are various ways for the control node to send the lock notification.
In an implementation manner of this embodiment, the lock notification may be sent by the control node to the edge node where the container to be updated is located after the container to be updated is determined. In this case, when the edge node 200 receives an update instruction for the container to be updated, the target container group to which the container to be updated belongs generally has a resource locking tag, and the edge node 200 may perform container update according to the flows shown in step S520 and step S530.
In another implementation manner in this embodiment, the lock notification may be sent by the control node when the edge node where the container to be updated is located is in a resource-tight state. The specific manner of determining whether the edge node is in the resource tension state is similar to that in the second embodiment, and reference may be made to the related description in the second embodiment.
in this case, if the edge node 200 is in a resource-tight state, the edge node 200 will have a resource lock tag, and the update operation may be performed on the container to be updated according to the flow shown in step S520 and step S530. If the edge node 200 is not in the resource-tight state, the edge node 200 will not have the resource locking label, and the container to be updated may be updated according to the flow of step S520 and step S540 at any time.
It should be noted that, in the present embodiment, the manner and the flow of the control node 100 sending the lock notification and the update instruction to the edge node are similar to those in the first embodiment and the second embodiment, and specific reference may be made to the detailed description of the above embodiments.
Through the design, the container to be updated can be updated under the condition of keeping the container group to which the container to be updated belongs. Because the resources of the container to be updated are borne in the container group to which the container to be updated belongs, the resources of the container to be updated cannot be released under the condition that the container group is not deleted, so that the problem that the container cannot be updated due to the fact that the resources of the container to be updated are preempted by other services after being released under the condition that the resources are insufficient is solved.
Further, in this embodiment, after creating the new version container, the edge node 200 may delete the resource lock tag of the target container group, so as to avoid locking the target container group for a long time. Therefore, under the condition of sufficient resources, the container to be updated can be destroyed by deleting the target container group, and the new version container is created in the newly-built target container group, so that the new version container runs in a brand-new and clean environment, and the running is not easy to make mistakes.
fourth embodiment
In the container cluster management system 10, the container update method provided by the foregoing embodiment may have different implementations. One implementation provided by the present embodiment is explained below.
Referring to fig. 1 again, in the present embodiment, an Edge-resource-controller (Edge-resource-controller) component 120 may be disposed in the control node 100, and may be capable of interacting with the main node process 110 and invoking an interface of the main node process 110; an Edge-resource-reserving (Edge-resource-ordering) component 220 may be provided at the Edge node 200, which is capable of interacting with the proxy node process 210 and invoking an interface of the proxy node process 210.
Referring to fig. 6, fig. 6 is a schematic diagram illustrating an operation flow of the container updating method on the container cluster management system 10, which may specifically include the following steps.
In step S610, the edge resource controller component 120 sends a lock notification to the proxy node process 210 through the master node process 110, where the lock notification includes the container identifier of the container to be updated.
In this embodiment, the proxy node process of each edge node reports the resource information of the node and the running state information of the container on the node to the master node process 110 periodically. The edge resource controller component 120 may obtain the operation state information of the container of each edge node from the master node process 110, and when it is monitored that container mirror image information or attribute information of any container is changed according to the obtained operation state information, may determine the container as a container to be updated.
When it is determined that the container is to be updated, the edge resource controller component 120 may call the interface of the master node process 110 to send the lock notification to the proxy node process 210, as per step S610.
step S620, the agent node process 210 receives the locking notification, and adds a resource locking tag to the target container group to which the container to be updated belongs according to the container identifier in the locking notification.
in step S630, the edge resource controller component 120 sends an update instruction for the container to be updated to the proxy node process through the master node process.
Step S640, the agent node process 210 receives the update instruction, determines whether the target container group to which the container to be updated belongs has a resource locking tag, and if so, invokes the edge resource reservation component 220, deletes the container to be updated from the target container group, and creates a new version container corresponding to the container to be updated.
The agent node process 210 may transmit the parameter information of the container to be updated and the container image information of the new version as call parameters to the edge resource reservation component 220, and then call the edge resource reservation component 220. In the calling process, the edge resource reservation component 220 may send a SIGTEAM signal to the container to be updated according to the container identifier of the container to be updated in the update instruction to control the container to be updated to exit, delete the container to be updated from the target container group after the container to be updated exits, and create the container according to the received call parameter, where the created container is the new version container.
Optionally, the parameter information may include, for example, identification information of a container to be updated, configuration file information, environment variable information, and the like, which is not limited in this embodiment. The proxy node process 210 may invoke the edge resource reservation component 220 in Inter-process communication (IPC) fashion.
Through the newly added edge resource controller component 120 and edge resource reservation component 220, and the process shown in fig. 6, the edge resource controller component 120 invokes an interface of the master node process 110, controls the proxy node process 210 to lock resources of the container group to which the container to be updated belongs, and before the proxy node process 210 tries to delete the container group to which the container to be updated belongs according to the update instruction, determines whether resources of the container group to which the container to be updated belongs are locked, if so, the container group is retained, and the container to be updated is updated under the condition that the container group is retained. Therefore, the resources of the container to be updated are not released in the updating process, so that the resources are prevented from being preempted by other services, and the new version container can be created by using the resources after the container to be updated is deleted, so that the container is updated.
further, referring to the foregoing description of the second embodiment, a case that a container on an edge node cannot be updated generally occurs when a resource of the edge node cannot be updated, and therefore, the container updating method provided in this embodiment may further include the following steps:
The edge resource controller component 120 obtains the resource information reported by the edge node 200 from the master node process 110, and determines whether the edge node 200 is in a resource tension state according to the resource information.
The edge resource controller component 120 may fetch the resource information from the host node process 110 in a hypertext transfer protocol (HTTP) snooping (Watch) manner. The manner and the flow of the edge resource controller 120 determining whether the edge node 200 is in the resource tension state based on the resource information are similar to those of the solution provided in the second embodiment, and are not described herein again.
The above-described lock notification may be sent by the edge resource controller component 120 when the edge node 200 is in a resource-stressed state. Therefore, under the condition of sufficient resources, the old version container can still be destroyed by deleting the container group to which the container to be updated belongs, so that the environment of the subsequently created new version container is clean and brand new, and the operation is not easy to make mistakes.
The edge resource reservation component 220 may return a create success notification to the proxy node process 210 after creating the new version container so that the proxy node process 210 may delete the resource lock tag of the target container group. Therefore, the resource locking of the target container group can be avoided for a long time, and the environment of the updated container can be ensured to be clean and brand new under the condition of sufficient resources.
for the description of each step of this embodiment, reference may be made to the detailed description of the relevant steps in the foregoing three embodiments.
Fifth embodiment
referring to fig. 7, fig. 7 is a block diagram of a control node 100 according to the present embodiment. The control node 100 comprises a processor 101 and a machine-readable storage medium 102, the processor 101 and the machine-readable storage medium 102 being communicatively connected via a system bus 103. The machine-readable storage medium 102 stores machine-executable instructions that when invoked may cause the processor 101 to implement the steps of the container update methods of the first and second embodiments of the present application.
it should be understood that the structure shown in fig. 7 is a schematic diagram of the control node 100, and that the control node 100 may also include more or less components than shown in fig. 7, or have a completely different configuration than shown in fig. 7. For example, the control node 100 may also include a communication component for communicating with edge nodes or other control nodes in the container cluster management system 10.
it should be noted that the control node 100 may perform the steps of the container updating method according to the first embodiment of the present application, and has the beneficial effects of the container updating methods according to the first embodiment and the second embodiment.
Sixth embodiment
referring to fig. 8, an edge node 200 is shown as an example, which is a block diagram of an edge node. The edge node 200 includes a processor 202 and a machine-readable storage medium 203, the processor 202 and the machine-readable storage medium 203 may be communicatively coupled via a system bus 204. The machine-readable storage medium 203 stores machine-executable instructions that when invoked may cause the processor 202 to implement the steps of the container update method of the third embodiment of the present application.
It should be understood that the architecture shown in fig. 8 is merely a schematic diagram of edge node 200, and that edge node 200 may include more or fewer components than shown in fig. 8, or have a completely different configuration than shown in fig. 8. For example, the edge node 200 may also include a communication component for communicating with a control node or other edge nodes in the container cluster management system 10.
The architecture of other edge nodes in the container cluster management system 10 is similar to the edge node 200, and will not be described herein.
It should be noted that the edge node 200 may perform the steps of the container updating method according to the third embodiment of the present application, and has the beneficial effects of the container updating method according to the third embodiment.
to sum up, according to the container updating method, the control node, and the edge node provided in the embodiment of the present application, by performing resource locking on the target container group where the container to be updated is located, when receiving the update instruction, the edge node deletes the container to be updated from the target container group, instead of directly deleting the target container group. Because the resources used by the container to be updated are borne in the target container group and the resources of the target container group are locked, the resources of the container to be updated borne by the target container group can be prevented from being released, so that the resources of the container to be updated can be utilized to create a corresponding new version container, and even under the condition of insufficient resources, the container can be updated.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The embodiments described above are merely illustrative, and the flowcharts and block diagrams in the figures, for example, illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
the foregoing description has been given for illustrative purposes only and is not intended to be limiting, and numerous modifications and variations will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (15)
1. A container updating method is applied to a control node in a container cluster management system, and comprises the following steps:
Sending a locking notice to an edge node, wherein the locking notice comprises a container identifier of a container to be updated, and enabling the edge node to add a resource locking label to a target container group to which the container to be updated belongs;
Sending an update instruction for the container to be updated to the edge node, so that the edge node deletes the container to be updated from the target container group and creates a new version container corresponding to the container to be updated according to the update instruction when the target container group to which the container to be updated belongs is checked to have the resource locking tag.
2. The method of claim 1, further comprising:
Judging whether the edge node is in a resource tension state or not according to the resource information reported by the edge node;
wherein the lock notification is issued by the control node when the edge node is in a resource-strained state.
3. the method of claim 2, wherein the resource information comprises: the amount of remaining resources of the edge node;
The judging whether the edge node is in a resource tension state according to the resource information reported by the edge node comprises the following steps:
Judging whether the residual resource quantity meets the preset resource specification of the container to be updated or not according to the residual resource quantity reported by the edge node;
And if not, determining that the edge node is in a resource tension state.
4. The method of claim 2, wherein the resource information comprises: region information of the edge node;
The judging whether the edge node is in a resource tension state according to the resource information reported by the edge node comprises the following steps:
determining a target area to which the edge node belongs according to the area information reported by the edge node, and acquiring the residual resource amount of each edge node belonging to the target area;
Judging whether edge nodes with residual resource quantity meeting the preset resource specification of the container to be updated exist in the target area or not;
And if not, determining that the edge node is in a resource tension state.
5. A container updating method applied to an edge node in a container cluster management system, the method comprising:
Receiving a locking notification sent by a control node, wherein the locking notification comprises a container identifier of a container to be updated, and adding a resource locking label to a target container group to which the container to be updated belongs according to the locking notification;
Receiving an update instruction aiming at the container to be updated and sent by the control node, and judging whether the target container group to which the container to be updated belongs has the resource locking label or not according to the update instruction;
and if the target container group has the resource locking label, deleting the container to be updated from the target container group, and creating a new version container corresponding to the container to be updated.
6. The method according to claim 5, wherein the lock notification is issued by the control node when the edge node where the container to be updated is in a resource-tight state; the method further comprises the following steps:
And if the target container group does not have the resource locking label, deleting the target container group, creating a new target container group, and creating a new version container corresponding to the container to be updated in the new target container group.
7. The method of claim 5 or 6, further comprising:
After the new version container is created, the resource lock tag of the target container group is deleted.
8. The method according to claim 5 or 6, wherein the creating of the new version container corresponding to the container to be updated comprises:
And creating the new version container according to the parameter information of the container to be updated and the container mirror image information of the new version.
9. A container updating method is applied to a container cluster management system, the system comprises a control node and an edge node which are mutually connected in a communication mode, the control node comprises a main node process and an edge resource controller component, and the edge node comprises a proxy node process and an edge resource reservation component; the method comprises the following steps:
The edge resource controller component sends a locking notification to the agent node process through the master node process, wherein the locking notification comprises a container identifier of a container to be updated;
The agent node process receives the locking notice and adds a resource locking label to a target container group to which the container to be updated belongs according to the container identifier in the locking notice;
the edge resource controller component sends an update instruction aiming at the container to be updated to the agent node process through the main node process;
and the agent node process receives the updating instruction, judges whether the target container group to which the container to be updated belongs has a resource locking label, if so, calls the edge resource reservation component, deletes the container to be updated from the target container group, and creates a new version container corresponding to the container to be updated.
10. the method of claim 9, further comprising:
the edge resource controller component acquires resource information reported by the edge node from the master node process, and judges whether the edge node is in a resource tension state or not according to the resource information;
Wherein the lock notification is issued by the edge resource controller component through the master node process when the edge node is in a resource-strained state.
11. The method according to claim 9 or 10, characterized in that the method further comprises:
After the edge resource reservation component creates the new version container, sending a creation success notification to the agent node process;
And the agent node process receives the creation success notification and deletes the resource locking label of the target container group.
12. A control node comprising a processor and a machine-readable storage medium having stored thereon machine-executable instructions that, when invoked, cause the processor to implement the method of any one of claims 1 to 4.
13. An edge node comprising a processor and a machine-readable storage medium having stored thereon machine-executable instructions that, when invoked, cause the processor to implement the method as claimed in any one of claims 5 to 8.
14. A machine-readable storage medium having stored thereon machine-executable instructions which, when executed, implement the method of any one of claims 1-4.
15. A machine-readable storage medium having stored thereon machine-executable instructions which, when executed, implement the method of any one of claims 5-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910858081.1A CN110569109B (en) | 2019-09-11 | 2019-09-11 | Container updating method, control node and edge node |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910858081.1A CN110569109B (en) | 2019-09-11 | 2019-09-11 | Container updating method, control node and edge node |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110569109A true CN110569109A (en) | 2019-12-13 |
| CN110569109B CN110569109B (en) | 2022-05-31 |
Family
ID=68779123
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910858081.1A Active CN110569109B (en) | 2019-09-11 | 2019-09-11 | Container updating method, control node and edge node |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110569109B (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111290767A (en) * | 2020-01-20 | 2020-06-16 | 中国科学院计算技术研究所 | Container group updating method and system with service quick recovery function |
| CN111324360A (en) * | 2020-01-09 | 2020-06-23 | 华中科技大学 | Container mirror image construction method facing edge calculation |
| CN111552494A (en) * | 2020-04-24 | 2020-08-18 | 星环信息科技（上海）有限公司 | Method, device, system and medium for managing container group |
| CN111917588A (en) * | 2020-08-10 | 2020-11-10 | 南方电网数字电网研究院有限公司 | Edge device management method, device, edge gateway device and storage medium |
| CN112181597A (en) * | 2020-10-12 | 2021-01-05 | 成都精灵云科技有限公司 | Dynamic updating method for use limit of container resource |
| WO2021238301A1 (en) * | 2020-05-27 | 2021-12-02 | 华为技术有限公司 | Application update method, apparatus and device and medium |
| CN113986468A (en) * | 2021-11-04 | 2022-01-28 | 马上消费金融股份有限公司 | Container injection method and device |
| CN114866790A (en) * | 2022-03-25 | 2022-08-05 | 上海哔哩哔哩科技有限公司 | Live streaming scheduling method and device |
| CN115145690A (en) * | 2022-06-30 | 2022-10-04 | 小米汽车科技有限公司 | Cluster resource expansion method and device and container cluster management system |
| US11693649B2 (en) | 2021-04-06 | 2023-07-04 | International Business Machines Corporation | Autonomous Kubernetes operator creation and management |
| CN116431188A (en) * | 2023-04-24 | 2023-07-14 | 江苏博云科技股份有限公司 | Edge application SOTA upgrading method under cloud edge cooperative strategy |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106874125A (en) * | 2017-01-13 | 2017-06-20 | 北京元心科技有限公司 | Method and device for sharing system resources among multi-container systems |
| CN107515776A (en) * | 2017-07-18 | 2017-12-26 | 深信服科技股份有限公司 | The uninterrupted upgrade method of business, node to be upgraded and readable storage medium storing program for executing |
| US20180089001A1 (en) * | 2016-09-29 | 2018-03-29 | Intel Corporation | Cloud container resource binding and tasking using keys |
| WO2018164782A1 (en) * | 2017-03-10 | 2018-09-13 | Pure Storage, Inc. | Synchronously replicating datasets and other managed objects to cloud-based storage systems |
| CN109301936A (en) * | 2018-10-11 | 2019-02-01 | 云南电网有限责任公司红河供电局 | A kind of intelligent substation O&M information monitoring system based on container scheduling architecture |
-
2019
- 2019-09-11 CN CN201910858081.1A patent/CN110569109B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180089001A1 (en) * | 2016-09-29 | 2018-03-29 | Intel Corporation | Cloud container resource binding and tasking using keys |
| CN106874125A (en) * | 2017-01-13 | 2017-06-20 | 北京元心科技有限公司 | Method and device for sharing system resources among multi-container systems |
| WO2018164782A1 (en) * | 2017-03-10 | 2018-09-13 | Pure Storage, Inc. | Synchronously replicating datasets and other managed objects to cloud-based storage systems |
| CN107515776A (en) * | 2017-07-18 | 2017-12-26 | 深信服科技股份有限公司 | The uninterrupted upgrade method of business, node to be upgraded and readable storage medium storing program for executing |
| CN109301936A (en) * | 2018-10-11 | 2019-02-01 | 云南电网有限责任公司红河供电局 | A kind of intelligent substation O&M information monitoring system based on container scheduling architecture |
Non-Patent Citations (2)
| Title |
|---|
| 杜威科: "基于Kubernetes的大数据流式计算Spark平台设计与实现", 《中国优秀博硕士学位论文全文数据库（硕士）信息科技辑（月刊）》 * |
| 杜威科: "基于Kubernetes的大数据流式计算Spark平台设计与实现", 《中国优秀博硕士学位论文全文数据库（硕士）信息科技辑（月刊）》, no. 02, 15 February 2018 (2018-02-15), pages 139 - 280 * |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111324360B (en) * | 2020-01-09 | 2022-04-01 | 华中科技大学 | Container mirror image construction method and system for edge calculation |
| CN111324360A (en) * | 2020-01-09 | 2020-06-23 | 华中科技大学 | Container mirror image construction method facing edge calculation |
| CN111290767A (en) * | 2020-01-20 | 2020-06-16 | 中国科学院计算技术研究所 | Container group updating method and system with service quick recovery function |
| CN111290767B (en) * | 2020-01-20 | 2021-06-08 | 中国科学院计算技术研究所 | Container group updating method and system with service quick recovery function |
| CN111552494A (en) * | 2020-04-24 | 2020-08-18 | 星环信息科技（上海）有限公司 | Method, device, system and medium for managing container group |
| CN111552494B (en) * | 2020-04-24 | 2021-05-11 | 星环信息科技(上海)股份有限公司 | Method, device, system and medium for managing container group |
| WO2021238301A1 (en) * | 2020-05-27 | 2021-12-02 | 华为技术有限公司 | Application update method, apparatus and device and medium |
| CN111917588A (en) * | 2020-08-10 | 2020-11-10 | 南方电网数字电网研究院有限公司 | Edge device management method, device, edge gateway device and storage medium |
| CN111917588B (en) * | 2020-08-10 | 2023-06-06 | 南方电网数字电网科技(广东)有限公司 | Edge device management method, device, edge gateway device and storage medium |
| CN112181597A (en) * | 2020-10-12 | 2021-01-05 | 成都精灵云科技有限公司 | Dynamic updating method for use limit of container resource |
| CN112181597B (en) * | 2020-10-12 | 2024-01-19 | 成都精灵云科技有限公司 | Dynamic updating method for container resource use limit |
| US11693649B2 (en) | 2021-04-06 | 2023-07-04 | International Business Machines Corporation | Autonomous Kubernetes operator creation and management |
| CN113986468A (en) * | 2021-11-04 | 2022-01-28 | 马上消费金融股份有限公司 | Container injection method and device |
| CN114866790A (en) * | 2022-03-25 | 2022-08-05 | 上海哔哩哔哩科技有限公司 | Live streaming scheduling method and device |
| CN114866790B (en) * | 2022-03-25 | 2024-02-27 | 上海哔哩哔哩科技有限公司 | Live stream scheduling method and device |
| CN115145690A (en) * | 2022-06-30 | 2022-10-04 | 小米汽车科技有限公司 | Cluster resource expansion method and device and container cluster management system |
| CN115145690B (en) * | 2022-06-30 | 2023-08-04 | 小米汽车科技有限公司 | Cluster resource expansion method and device and container cluster management system |
| CN116431188A (en) * | 2023-04-24 | 2023-07-14 | 江苏博云科技股份有限公司 | Edge application SOTA upgrading method under cloud edge cooperative strategy |
| CN116431188B (en) * | 2023-04-24 | 2023-09-29 | 江苏博云科技股份有限公司 | Edge application SOTA upgrading method under cloud edge cooperative strategy |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110569109B (en) | 2022-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110569109B (en) | Container updating method, control node and edge node | |
| US11296923B2 (en) | Network fault originator identification for virtual network infrastructure | |
| CN106790092B (en) | Remote procedure call server control system and method | |
| US10218780B2 (en) | Cloud based peer assisted updates in a device management environment | |
| CN110557433A (en) | Resource management method, platform, cloud gateway, system and storage medium | |
| CN111163173A (en) | Cluster configuration method and device, server and readable storage medium | |
| CN116566984B (en) | Routing information creation method and device of k8s container cluster and electronic equipment | |
| CN110557270B (en) | Network slice management method and device | |
| CN111277628B (en) | Server configuration method and device and service server | |
| CN108062244B (en) | Reptile task canceling method and device | |
| CN109344202B (en) | Data synchronization method and management node | |
| CN112243243A (en) | Network slicing implementation method, entity and system | |
| CN115225645A (en) | Service updating method, device, system and storage medium | |
| CN115640066A (en) | Security detection method, device, equipment and storage medium | |
| GB2607871A (en) | Improvements in and relating to multi-access edge computing (MEC) | |
| CN111385422B (en) | Reserved telephone traffic management method and device | |
| CN113783712A (en) | Default gateway management method, gateway manager, server and storage medium | |
| CN113094074A (en) | Service cluster updating method and device | |
| KR20150112127A (en) | Local resource sharing method of machine to machine component and apparatus thereof | |
| CN104734923B (en) | The execution control method and system of application task, message server | |
| CN114721746B (en) | Trino catalog hot loading method and system | |
| CN111404980B (en) | Data storage method and object storage system | |
| CN110024445B (en) | Mobile device roaming method and apparatus based on network profile of user | |
| CN116975477A (en) | Application service calling method, device, equipment and storage medium | |
| CN117632531A (en) | Method, device and storage medium for processing messages in cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |