CN112559142B - Container control method, device, edge computing system, medium and equipment - Google Patents
Container control method, device, edge computing system, medium and equipment Download PDFInfo
- Publication number
- CN112559142B CN112559142B CN201910919303.6A CN201910919303A CN112559142B CN 112559142 B CN112559142 B CN 112559142B CN 201910919303 A CN201910919303 A CN 201910919303A CN 112559142 B CN112559142 B CN 112559142B
- Authority
- CN
- China
- Prior art keywords
- container
- stop
- controlling
- containers
- physical device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000011156 evaluation Methods 0.000 claims abstract description 77
- 238000004364 calculation method Methods 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 20
- 238000004422 calculation algorithm Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000011217 control strategy Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45575—Starting, stopping, suspending or resuming virtual machine instances
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Warehouses Or Storage Devices (AREA)
Abstract
The invention discloses a control method, a device, an edge computing system, a medium and equipment for a container, wherein the control method is used for controlling a plurality of containers on physical equipment and comprises the following steps: acquiring resource use information and container parameters of each container in a plurality of containers; calculating a start-stop evaluation value of each container according to the resource use information of the container and the container parameters; and controlling the container according to a preset start-stop strategy according to the start-stop evaluation value. The decision right of the container start-stop judgment is executed by the physical equipment of the edge node, so that the overall availability of the edge computing service is improved, the reporting process of the edge node to the center node is reduced, the timeliness of container start-stop control is improved, and the problem that the container start-stop cannot be executed due to the failure of the center node is avoided.
Description
Technical Field
The present invention relates to the field of internet technologies, and in particular, to a method and apparatus for controlling a container, an edge computing system, a medium, and a device.
Background
Edge computing refers to providing near-end services on the side near the object or data source, using an open platform with integrated network, computing, storage, and application core capabilities. Because the application program of the edge calculation is initiated at the edge side, the edge calculation can realize that most of data operation and data control sink to the local equipment without depending on a cloud, which clearly greatly improves the processing efficiency of data, lightens the load of the cloud and a central database, and generates faster network service response.
In edge computing systems, utilizing container service technology at an edge node to provide edge computing services to users can generally be employed, providing great flexibility in building container technology operating system instances on demand. For ease of understanding, a container service may be understood as a group of containers, which may contain multiple containers. The different container services are sensitive to resource usage to different extents, but the computational logic performed by multiple containers belonging to the same container service is sensitive to resource usage to approximately the same extent. Multiple containers belonging to the same container service may be respectively provided on different physical devices, and multiple container services may be created on the same physical device. For example, a physical device a and a physical device B, where a container service a and a container service B may be created on the physical device a, and a container service B may be created on the physical device B; the container service a comprises a container a1, a container a2 and a container a3, all containers in the container service a are created on the physical device A, the container service B comprises a container B1, a container B2, a container B3, a container B4 and a container B5, the container B1, the container B2 and the container B3 in the container service B are created on the physical device A, and the container B4 and the container B5 in the container service B are created on the physical device B.
In the computing process, the container corresponds to an instance, and the instance can be understood as a process, and as the implementation of resource isolation between containers is imperfect, the interaction between containers created on the same physical device is large. When a container created on the same physical device is abnormal, the use amount of resources is increased, and the container created on the physical device is possibly greatly influenced, so that the influenced container cannot normally execute calculation.
In the prior art, a central node controls the starting and stopping of containers created on an edge physical device, the central node continuously receives the conditions of resource occupation and the like of each container reported from the edge node, the central node judges which container needs to be started and which container needs to be stopped according to the received conditions, and sends a judging result to the edge node, and the edge node executes the judgment made by the central node. In addition, the central node generally does not consider the importance of the container service and the overall situation of the container service in the process of judging the start and stop of the container, so that the following problems exist in the control of the start and stop of the container in the prior art:
1. The start and stop of the container cannot be controlled according to the importance of the container service, and a control strategy cannot be executed according to the overall use condition of the container service, so that the important container service is easily influenced;
2. when different containers on one physical device are mutually affected, the edge node needs to report the central node to make a decision, and the timeliness of the start and stop control is affected; and the occupied bandwidth resource is larger because frequent reporting is needed.
Disclosure of Invention
In order to solve the technical problems, the invention provides a control method and device of a container, an edge computing system, a medium and equipment.
The invention provides a control method of a container, which is used for controlling a plurality of containers on physical equipment and comprises the following steps:
acquiring resource usage information and container parameters of each container in the plurality of containers;
calculating a start-stop evaluation value of each container according to the resource use information of the container and the container parameters;
and controlling the container according to the start-stop evaluation value and a preset start-stop strategy. .
The method also has the following characteristics: the calculating the start-stop evaluation value of each container according to the resource use information of the container and the container parameters comprises the following steps:
And giving a preset weight value to the resource use information and the container parameters of the containers, and carrying out weighted summation on the resource use information and the container parameters of each container according to the preset weight value to obtain the start-stop evaluation value of each container.
The method also has the following characteristics: the method for controlling the container according to the start-stop evaluation value and the preset start-stop strategy comprises the following steps:
when the operation state of the hardware of the physical equipment is in a first operation state, controlling the container with the start-stop evaluation value meeting a first preset condition to stop; and/or the number of the groups of groups,
and when the operating state of the hardware of the physical equipment is in the second operating state, controlling the start-stop evaluation value to meet the container start of the second preset condition.
The method also has the following characteristics: when the operation state of the hardware of the physical device is in the first operation state, controlling the container with the start-stop evaluation value meeting the first preset condition to stop includes:
when the running state of the CPU of the physical device meets any one of the following conditions, controlling the container with the start-stop evaluation value meeting the first preset condition to stop:
the first condition is that the utilization rate of all CPUs on the physical equipment is greater than or equal to a first stop limit value;
The average load value of any CPU is larger than or equal to a second stop limit value;
and in the third condition, the number of the running CPUs on the physical equipment is greater than or equal to a third stop limit value.
The method also has the following characteristics: when the operation state of the hardware of the physical device is in the second operation state, controlling the start-stop evaluation value to meet the container start of the second preset condition includes:
when the running state of the CPU of the physical device meets all of the following conditions, controlling the start-stop evaluation value to meet the container start of a second preset condition:
the first condition is that the utilization rate of all CPUs on the physical equipment is smaller than or equal to a first starting limit value;
the load average value of any CPU is smaller than or equal to a second starting limit value;
and under the third condition, the number of the running CPUs on the physical equipment is smaller than or equal to a third starting limit value.
The method also has the following characteristics: the container parameters comprise importance values and stopping rates, wherein the stopping rate of a container is the ratio of the stopping rate of a container service to which the container belongs to the average stopping rate of all container services, and the stopping rate of the container service is the ratio of the number of containers which are stopped in the same container service to the number of all containers in the container service;
The resource usage information includes an occupancy rate of the container to a CPU of the physical device, an occupancy rate to a memory of the physical device, and/or an occupancy rate to an I/O of the physical device.
The method also has the following characteristics: the obtaining of the resource usage information and the container parameters for each of the plurality of containers includes:
obtaining container parameters for each of the plurality of containers from a central node;
the occupancy rate of each container of the plurality of containers to the hardware of the physical device is obtained from the physical device.
The invention also provides an apparatus for controlling a container, the apparatus being provided on a physical device at an edge node, comprising:
the acquisition module is used for acquiring the resource use information and the container parameters of each container in the plurality of containers;
the calculation module is used for calculating the start-stop evaluation value of each container according to the resource use information of the container and the container parameters;
and the control module is used for controlling the container according to the start-stop evaluation value and a preset start-stop strategy.
The device also has the following characteristics: the computing module includes:
the assignment unit is used for assigning a preset weight value to the resource use information and the container parameters of the container;
And the computing unit is used for obtaining the start-stop evaluation value of each container after carrying out weighted summation on the resource use information of each container and the container parameters according to the preset weight value.
The device also has the following characteristics: the control module includes:
the first control unit is used for controlling the container with the start-stop evaluation value meeting a first preset condition to stop when the running state of the hardware of the physical equipment is in a first running state;
and/or the number of the groups of groups,
and the second control unit is used for controlling the start and stop evaluation value to meet the container start of a second preset condition when the running state of the hardware of the physical equipment is in a second running state.
The device also has the following characteristics: the first control unit is used for controlling the container with the start-stop evaluation value meeting the first preset condition to stop when the running state of the CPU of the physical device meets any one of the following conditions:
the first condition is that the utilization rate of all CPUs on the physical equipment is greater than or equal to a first stop limit value;
the average load value of any CPU is larger than or equal to a second stop limit value;
and in the third condition, the number of the running CPUs on the physical equipment is greater than or equal to a third stop limit value.
The device also has the following characteristics: the second control unit is configured to control the start-up and stop evaluation value to meet a container start-up of a second preset condition when the running state of the CPU of the physical device meets all of the following conditions:
the first condition is that the utilization rate of all CPUs on the physical equipment is smaller than or equal to a first starting limit value;
the load average value of any CPU is smaller than or equal to a second starting limit value;
and under the third condition, the number of the running CPUs on the physical equipment is smaller than or equal to a third starting limit value.
The device also has the following characteristics: the acquisition module comprises:
a container parameter acquiring unit configured to acquire a container parameter of each of the plurality of containers from a center node;
and the resource use information acquisition unit is used for acquiring the occupancy rate of each container in the plurality of containers to the hardware of the physical device from the physical device.
The invention also provides an edge computing system, which comprises a central node and an edge node, wherein a plurality of physical devices are arranged at the edge node, each of the plurality of physical devices is provided with the device for controlling the container, and the container on the physical device is controlled by using the container control method.
The present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method of controlling a container as described above.
The present invention also provides an edge computing device comprising: a transceiver, a memory, a processor;
the transceiver is used for receiving and transmitting messages;
the memory is used for storing instructions and data;
the processor is configured to read the instructions and data stored in the memory to perform the container control method as described above.
According to the method and the device, the starting and stopping of the containers are controlled according to the resource use information and the container parameters of each container in the plurality of containers and the preset starting and stopping strategy, so that important tasks can be effectively prevented from being stopped or started.
The decision right of the container start-stop judgment is executed by the physical equipment of the edge node, so that the overall availability of the edge computing service is improved, the reporting process of the edge node to the center node is reduced, the timeliness of container start-stop control is improved, and the problem that the container start-stop cannot be executed due to the failure of the center node is avoided.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention. In the drawings:
FIG. 1 is an overall flow diagram of container control in an embodiment;
FIG. 2 is a flow chart of container stop control in an embodiment;
FIG. 3 is a flow chart of container start-up control in an embodiment;
fig. 4 is a structural view of an apparatus for controlling a container in the embodiment.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention. It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be arbitrarily combined with each other.
The application provides a control method of a container, which is used for controlling a plurality of containers on physical equipment of an edge node, is mainly used for controlling the start and stop of the containers, and transfers the control decision right of the start and stop of the containers from a center node to the edge node, so that the data transmission between the edge node and the center node is effectively reduced, the occupation of bandwidth is reduced, and the bandwidth resource is saved.
As shown in fig. 1, a control method of a container, the control method is used for controlling a plurality of containers on a physical device, and the control method includes:
s10, acquiring resource use information and container parameters of each container in a plurality of containers;
s20, calculating a start-stop evaluation value of each container according to the resource use information of the container and the container parameters and a preset algorithm;
s30, controlling the container according to a preset start-stop strategy according to the start-stop evaluation value.
In the control method, the resource use information and the container parameters of each container are selected as parameters to control the start and stop of the container, so that the containers which execute important services can be distinguished according to the importance of the container service of the container, and the containers which execute important services are prevented from being stopped or started, so that the control process of the start and stop of the containers is more intelligent and more reliable.
In S10, the container parameters of each container include an importance value and a stopping rate, where the stopping rate of a container indicates a probability that all instances (instances are in one-to-one correspondence with containers) of a busy container service on an edge node using the container service are stopped, i.e., a container service that avoids a busy service from being stopped for all services. The stopping rate of a container is a ratio of the stopping rate of the container service to which the container belongs to the average stopping rate of the entire container service. The stopping rate of the container service to which the container belongs is a ratio of the number of containers which are stopped in the same container service to the total number of containers in the container service, for example, 100 containers are created in one container service, and 30 containers in the 100 containers are stopped, so that the stopping rate of the container service is 30%. The average stop rate of the total container service is a ratio of the total number of containers stopped in the total number of containers created by all the container services in the total network to the total number of containers created by all the container services in the total network. For example, there are 10 container services in the whole network, and 100 containers are created in each container service, and there are 1000 containers in the whole network, wherein the number of containers to be stopped is 200, and the average stop rate of the whole container service is 20%. The stopping rate of the container was then 30% to 20%, i.e. 1.5. The stopping rate is selected as a main calculation evaluation standard, and a weight value is given to the stopping rate in the calculation process, so that the situation that containers in the same container service are possibly stopped completely is effectively reduced.
Since it is clear which container is more important in the process of creating the container, in order to ensure the accuracy of the container importance value, the container importance value is specified by the system when the container is created, or may be set by the system administrator in an input manner. In the present application, the container importance value of each container is an importance value of a container service to which the container belongs, and the importance value is mainly used for determining the priority of stopping, and the higher the importance value, the lower the probability that the container is stopped when the stopping decision is made. Since the importance value of the container is used as an important evaluation parameter, the importance of the container service can be distinguished, and important business can be prevented from being stopped or started.
The resource usage information includes occupancy of the container to the CPU of the physical device, occupancy of the memory of the physical device, and/or occupancy of the I/O of the physical device. The occupancy rate of the CPU includes at least one of the overall utilization rate of the CPU, the utilization rate of each core of the CPU, and the utilization rates of the CPU load of 1min, 5min, and 15min, and it is preferable to consider all the contents related to the utilization rate of the CPU at the same time in order to evaluate the occupancy rate of the CPU as a whole. The occupancy rate of the memory of the physical device includes available memory, used memory, idle memory, and the like. The I/O occupancy rate of the physical device comprises the number of read/write per second, the size of read/write bytes, the actual I/O occupancy time and the like of each disk. When the resource usage information is actually selected, one of the occupancy rate of the CPU and the occupancy rate of the memory of the physical device and the occupancy rate of the I/O of the physical device can be selected, and two or all of the occupancy rates can be selected for more effectively evaluating the resource usage information. And setting a preset weight value for each parameter, and carrying out weighted summation calculation according to the preset weight value or carrying out average calculation, and taking the three parameters as resource use information after integral evaluation. It should be noted that, the above-mentioned resource usage information may be preferably used to respectively count and calculate the average value, the median value, the maximum value and the minimum value of each parameter included in the above-mentioned resource usage information in a collection period, such as 3 minutes, or 30 seconds, so as to control the start-stop of the container. In the specific calculation process, one of the average value, the median value, the maximum value and the minimum value of each parameter can be selected for calculation, and the specific selection method is optional.
Further, the acquiring the resource usage information and the container parameters of each of the plurality of containers in S10 includes:
obtaining container parameters for each of a plurality of containers from a central node;
the occupancy of hardware of the physical device by each of the plurality of containers is obtained from the physical device.
In the method, the central node mainly bears the creation and deletion of the container, coordinates with the edge node, issues instructions for the creation of the container and the state maintenance of the start and stop of the container, but judges that the start and stop of the container are made by physical equipment positioned at the edge node in the running process of the container. That is, the central node is only responsible for issuing commands to the edge nodes, informing the edge nodes of the container stop or start, but specifically which container should stop and which container should start is decided by the edge nodes, that is, the edge nodes are mainly responsible for the actual container creation, start, stop and other actions, and meanwhile, the edge nodes report the container change to the central node. The container parameters for the containers are obtained from the central node, which in addition to performing the above-mentioned tasks, needs to record and store the importance value of each container to transmit the container importance value to the edge node when needed by the edge node. Meanwhile, since the edge nodes periodically send the number of times that the containers are stopped to the central node, the central node needs to count the number of times that different containers in the same container service on multiple edge nodes are stopped. Because the calculation judgment of the decision of starting or stopping the container is made at the edge node, the edge node does not need to report all the parameters used in the calculation process to the center node any more and only needs to report the final decision, so the number of times and the data volume of reporting to the center node by the edge node are reduced, and the problem of usability of starting and stopping the container caused by the failure of the center node is avoided. In addition, the central node is only responsible for static configuration, container start-stop state and record of container start-stop, and is not responsible for summarizing and storing of historical values of each container on each edge node, so that the start-stop control decision right of the container is given to the edge nodes, and the availability of the whole edge computing service is improved.
Further, in S20, calculating, according to the resource usage information of the containers and the container parameters, the start-stop evaluation value of each container according to the preset algorithm includes:
and giving a preset weight value to the resource use information and the container parameters of the containers, and carrying out weighted summation on the resource use information and the container parameters of each container according to the preset weight value to obtain the start-stop evaluation value of each container.
Specifically, since the resource usage information of the container includes an occupancy rate of the container to the CPU of the physical device, an occupancy rate to the memory of the physical device, and/or an occupancy rate to the I/O of the physical device, the container parameters include an importance value and a stop rate. Therefore, when calculating the start-stop evaluation value of each container according to a preset algorithm, it is necessary to assign weight values to the CPU occupancy rate of the container to the physical device, the memory occupancy rate to the physical device, the I/O occupancy rate to the physical device, the importance value, and the stop rate, respectively. The weight value is determined according to the actual situation, in a specific embodiment, the weight of the importance value of a container is 0.5, the weight of the stopping rate is 0.08, the occupancy rate of the container to the CPU of the physical device is 0.14, the occupancy rate of the container to the memory of the physical device is 0.14, and the occupancy rate of the container to the I/O of the physical device is 0.14. When the weighted summation calculation is performed, the importance value is multiplied by 0.5, the stopping rate is multiplied by 0.08, the occupancy rate of the CPU of the physical equipment by the container is multiplied by 0.14, the occupancy rate of the memory of the physical equipment by the container is multiplied by 0.14, the occupancy rate of the I/O of the physical equipment by the container is multiplied by 0.14, and the obtained data is summed to obtain the start-stop evaluation value of the container. It will be understood, of course, that in actual calculation, the importance value, the stop rate, the CPU occupancy rate of the physical device by the container, the memory occupancy rate of the physical device by the container, and the I/O occupancy rate of the physical device by the container need to be weighted and summed for each container to obtain the start/stop evaluation value of each container. In addition, in the specific calculation process, when the number of parameters used for weighted summation calculation is reduced, the weight value is correspondingly adjusted in an adaptive manner. After the start-stop evaluation value of each container is obtained, a start-stop evaluation value table with numerical values arranged from large to small is actually obtained for use in the subsequent method. It will be appreciated that the start-stop rating value table may be a table that does not actually exist and may be a piece of code stored in a physical hard disk.
Further, as shown in fig. 2, in S30, the method for controlling the container according to the preset start-stop strategy according to the start-stop evaluation value includes:
and S31, when the operation state of the hardware of the physical equipment is in the first operation state, controlling the container with the start-stop evaluation value meeting the first preset condition to stop.
In executing S31, specifically, when the operation state of the CPU of the physical device satisfies any one of the following conditions, the container whose start-stop evaluation value satisfies the first preset condition is controlled to stop:
the first condition, the utilization rate of all CPUs on the physical device is greater than or equal to a first stop limit value;
the average load value of any CPU is larger than or equal to a second stop limit value;
the number of CPUs running on the physical device under the condition three is greater than or equal to the third stop limit value.
That is, when the hardware operation state of the physical device satisfies any one of the first condition, the second condition and the third condition, the hardware operation state of the physical device is considered to be in the first operation state, and at this time, the container whose start-stop evaluation value satisfies the first preset condition is controlled to stop operation. The first preset condition refers to the containers arranged in the top n in the obtained start-stop evaluation value table. In a specific embodiment, the utilization rate of all CPUs on the physical device is greater than or equal to the first stop limit value, at this time, the operation state of the CPUs of the physical device is in the first operation state, and the first n containers arranged in the start-stop evaluation value table satisfy the first preset condition, so that the edge node may obtain a decision, that is, control the first n containers in the start-stop evaluation value table to stop operation, and send the decision to the central node, and the central node controls the containers meeting the first preset condition on the edge node to stop operation according to the decision.
In another specific embodiment, the number of CPUs running on the physical device is greater than or equal to the third stop limit value, and the load average of one CPU is greater than or equal to the second stop limit value, at this time, the running state of the CPU of the physical device is in the first running state, and the first n containers arranged in the start-stop evaluation value table satisfy the first preset condition, so the edge node may obtain a decision, that is, control the first n containers in the start-stop evaluation value table to stop running, and send the decision to the central node, and the central node controls the containers meeting the first preset condition on the edge node to stop running according to the decision. It will be appreciated, of course, that the first stop limit value in the first condition, the second stop limit value in the second condition, and the third stop limit value in the third condition may be set according to the specific situation. For example, the first stop limit may be 90%, the second stop limit may be 100, and the third stop limit may be 100% of the number of CPUs running on the physical device.
Further, in S30, as shown in fig. 3, the method for controlling the container according to the preset start-stop strategy according to the start-stop evaluation value includes:
And S32, when the operating state of the hardware of the physical equipment is in the second operating state, controlling the container with the start-stop evaluation value meeting the second preset condition to start.
In the specific execution of the control of the start-stop of the container, S31 and S32 may be used separately as control methods, respectively, and it is preferable to control the start-stop of the container using S31 and S32 simultaneously.
When the operation state of the hardware of the physical device is in the second operation state, controlling the start-stop evaluation value to meet the container start of the second preset condition includes:
when executing S32, when the operation state of the CPU of the physical device satisfies all of the following conditions, controlling the container start whose start-stop evaluation value satisfies the second preset condition:
the utilization rate of all CPUs on the physical equipment is smaller than or equal to a first starting limit value;
the load average value of any CPU is smaller than or equal to a second starting limit value;
the number of CPUs running on the physical device under the third condition is less than or equal to the third start-up limit value.
Namely, when the hardware running state of the physical device meets the first condition, the second condition and the third condition at the same time, the hardware running state of the physical device is considered to be in the second running state, and at the moment, the container with the start-stop evaluation value meeting the second preset condition is controlled to start running. Wherein the second preset condition refers to the last m containers in the obtained start-stop evaluation value table. In a specific embodiment, the utilization rate of all CPUs is less than or equal to the first start limit value, at the same time, the load average value of any one CPU is less than or equal to the second start limit value, and the number of CPUs running on the physical device is less than or equal to the third start limit value, at this time, the CPUs on the physical device simultaneously meet the first condition, the second condition and the third condition, which means that the CPUs on the physical device run in the second running state, and the containers m after being arranged in the start-stop evaluation value table meet the second preset condition, so that the edge node can obtain a decision, that is, control the start-up operation of the containers m after being arranged in the start-stop evaluation value table, and send the decision to the central node, and the central node controls the start-up operation of the containers meeting the second preset condition on the edge node according to the decision. It will be appreciated that the first start limit value in the first condition, the second start limit value in the second condition, and the third start limit value in the third condition may be set according to the specific situation. For example, the first start-up limit may be 60%, the second start-up limit may be 10, and the third start-up limit may be 50% of the number of CPUs running on the physical device.
Here, for different models, the above three conditions for determining whether the operation state of the CPU of the physical device is in the first operation state and the above three conditions for determining whether the operation state of the CPU of the physical device is in the second operation state may be changed accordingly. That is, the evaluation index when judging the operation state of the hardware of the physical device varies for different models, and may be increased or decreased according to actual situations. For example, the judgment data of the above conditions of the CPU-intensive physical device, the large memory-based physical device, and the I/O-intensive physical device may change. For example, the first stop limit value of the CPU-intensive physical device is 95%, while the first stop limit value of the I/O-intensive physical device is 80%, and the change of the remaining parameters can be adjusted according to specific practical situations, which will not be described herein. In addition, only a control method using a CPU of a physical device as an evaluation main body is disclosed in the present application, and in the actual use process, the occupancy rate of a container to a memory of the physical device or the occupancy rate of an I/O of the physical device may be used as a control method of a rating main body, and the control concept of the whole control method using the CPU as the rating main body is substantially the same as the control method using the CPU as the rating main body, and is not described herein again.
In the process of controlling the start and stop of the containers, after one container is stopped, the container still needs to be judged after entering the next resource use information acquisition period and judgment period, and if the container still occupies large hardware resources of the physical equipment, the container is easy to stop. However, in actual use, after the container is stopped, the occupation of the hardware resources of the physical device is also reduced correspondingly, so that the container is started in a high probability in the container starting judgment process. Then, if a container is running, it may occupy too much resources, and is not a container for handling important tasks, it may happen that it is stopped during a stop judgment, started again during an immediately subsequent start, and then stopped again. In this case, there is a case where one container is stopped and started a plurality of times, resulting in waste of resources. To avoid this, in the actual control process, if a container is started and stopped more than a predetermined number of times, such as 10 times or 20 times, in the start-up judgment process, the container is marked in the next start-up judgment process and is not started. The container may also be removed from the container start-stop evaluation value table to ensure that the container is not restarted. The central node monitors the starting and stopping of each container and the actual running state of the container in real time, and when a container is repeatedly started and stopped for a plurality of times, the central node processes the node in the mode.
As shown in fig. 4, the present invention further provides an apparatus for controlling a container, where the apparatus is disposed on a physical device located at an edge node, and is used as an execution body to execute the above-mentioned control method for starting and stopping the container.
The device for controlling the container comprises:
the acquisition module is used for acquiring the resource use information and the container parameters of each container in the plurality of containers;
the computing module is used for computing the start-stop evaluation value of each container according to the resource use information of the container and the container parameters and a preset algorithm;
and the control module is used for controlling the container according to a preset start-stop strategy according to the start-stop evaluation value.
Wherein, because the resource usage information of the container and the acquisition method of the parameter of the container are different, the acquisition module comprises:
a container parameter acquiring unit configured to acquire a container parameter of each of the plurality of containers from the center node;
and the resource use information acquisition unit is used for acquiring the occupancy rate of each container in the plurality of containers to the hardware of the physical device from the physical device.
Further, in the specific calculation, the calculation module is disposed on the physical device located at the edge node, so as to reduce the calculation amount of the central server to a certain extent, and reduce the load of the central server, where the calculation module includes:
The assignment unit is used for assigning a preset weight value to the resource use information of the container and the container parameter;
and the computing unit is used for obtaining the start-stop evaluation value of each container after carrying out weighted summation on the resource use information of each container and the container parameters according to the preset weight value.
Further, the control module includes:
and the first control unit is used for controlling the container with the start-stop evaluation value meeting the first preset condition to stop when the running state of the hardware of the physical equipment is in the first running state.
Specifically, the first control unit is configured to control, when the running state of the CPU of the physical device meets any one of the following conditions, stopping the container whose start-stop evaluation value meets the first preset condition:
the first condition, the utilization rate of all CPUs on the physical device is greater than or equal to a first stop limit value;
the average load value of any CPU is larger than or equal to a second stop limit value;
the number of CPUs running on the physical device under the condition three is greater than or equal to the third stop limit value.
Still further, the control module further includes:
and the second control unit is used for controlling the container with the start-stop evaluation value meeting the second preset condition to start when the running state of the hardware of the physical equipment is in the second running state.
Specifically, the second control unit is configured to control the start-up and stop of the container whose evaluation value satisfies the second preset condition when the running state of the CPU of the physical device satisfies all of the following conditions:
the utilization rate of all CPUs on the physical equipment is smaller than or equal to a first starting limit value;
the load average value of any CPU is smaller than or equal to a second starting limit value;
the number of CPUs running on the physical device under the third condition is less than or equal to the third start-up limit value.
In the actual use process, the same physical device can be provided with only the first control unit or only the second control unit, and preferably, the same physical device is provided with the first control unit and the second control unit at the same time.
The application also provides an edge computing system, which comprises a center node and an edge node, wherein a plurality of physical devices are arranged at the edge node, each of the plurality of physical devices is provided with the device for controlling the container, and the container on the physical device is controlled by using the container control method.
The application also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor implements the above-mentioned container control method.
The application also provides an edge computing device comprising: a transceiver, a memory, a processor. The transceiver is used for receiving and transmitting messages, the memory is used for storing instructions and data, and the processor is used for reading the instructions and the data stored in the memory so as to execute the control method of the container.
The above description may be implemented alone or in various combinations and these modifications are within the scope of the present invention.
Those of ordinary skill in the art will appreciate that all or a portion of the steps of the methods described above may be implemented by a program that instructs associated hardware, and the program may be stored on a computer readable storage medium such as a read-only memory, a magnetic or optical disk, etc. Alternatively, all or part of the steps of the above embodiments may be implemented using one or more integrated circuits, and accordingly, each module/unit in the above embodiments may be implemented in hardware or may be implemented in a software functional module. The present invention is not limited to any specific form of combination of hardware and software.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that an article or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such article or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of additional identical elements in an article or apparatus that comprises the element.
The above embodiments are only for illustrating the technical scheme of the present invention, not for limiting the same, and the present invention is described in detail with reference to the preferred embodiments. It will be understood by those skilled in the art that various modifications and equivalent substitutions may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention, and the present invention is intended to be covered by the scope of the appended claims.
Claims (16)
1. A control method of a container, the control method being for controlling a plurality of containers on a physical device, the control method comprising:
acquiring resource use information and container parameters of each container in the plurality of containers, wherein the container parameters comprise importance values and stopping rates, and the stopping rates of the containers are the ratio of the stopping rate of the container service of the container to the average stopping rate of all container services, wherein the stopping rate of the container service is the ratio of the number of containers which are stopped in the same container service and the number of all containers in the container service;
calculating a start-stop evaluation value of each container according to the resource use information of the container and the container parameters;
and controlling the container according to the start-stop evaluation value and a preset start-stop strategy.
2. The method of controlling a container according to claim 1, wherein calculating a start-stop evaluation value of each container based on the resource usage information of the container and the container parameter comprises:
and giving a preset weight value to the resource use information and the container parameters of the containers, and carrying out weighted summation on the resource use information and the container parameters of each container according to the preset weight value to obtain the start-stop evaluation value of each container.
3. The method for controlling a container according to claim 1 or 2, wherein the method for controlling the container according to a preset start-stop strategy according to the start-stop evaluation value comprises:
when the operation state of the hardware of the physical equipment is in a first operation state, controlling the container with the start-stop evaluation value meeting a first preset condition to stop; and/or when the operation state of the hardware of the physical equipment is in the second operation state, controlling the start-stop evaluation value to meet the container start of the second preset condition.
4. The method for controlling a container according to claim 3, wherein controlling the stopping of the container, in which the start-stop evaluation value satisfies a first preset condition, when the operation state of the hardware of the physical device is in the first operation state, comprises:
When the running state of the CPU of the physical device meets any one of the following conditions, controlling the container with the start-stop evaluation value meeting the first preset condition to stop:
the first condition is that the utilization rate of all CPUs on the physical equipment is greater than or equal to a first stop limit value;
the average load value of any CPU is larger than or equal to a second stop limit value;
and in the third condition, the number of the running CPUs on the physical equipment is greater than or equal to a third stop limit value.
5. The method for controlling a container according to claim 3, wherein controlling the start-stop evaluation value to satisfy a second preset condition when the operation state of the hardware of the physical device is in the second operation state comprises:
when the running state of the CPU of the physical device meets all of the following conditions, controlling the start-stop evaluation value to meet the container start of a second preset condition:
the first condition is that the utilization rate of all CPUs on the physical equipment is smaller than or equal to a first starting limit value;
the load average value of any CPU is smaller than or equal to a second starting limit value;
and under the third condition, the number of the running CPUs on the physical equipment is smaller than or equal to a third starting limit value.
6. A method for controlling a container according to claim 1 or 2,
the resource usage information includes an occupancy rate of the container to a CPU of the physical device, an occupancy rate to a memory of the physical device, and/or an occupancy rate to an I/O of the physical device.
7. The method of controlling a container according to claim 1, wherein acquiring the resource usage information and the container parameters of each of the plurality of containers comprises:
obtaining container parameters for each of the plurality of containers from a central node;
the occupancy rate of each container of the plurality of containers to the hardware of the physical device is obtained from the physical device.
8. An apparatus for controlling a container, the apparatus being disposed on a physical device at an edge node, comprising:
the system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is used for acquiring resource use information and container parameters of each container in a plurality of containers, the container parameters comprise an importance value and a stopping rate, the stopping rate of the container is a ratio of the stopping rate of a container service to which the container belongs to an average stopping rate of all container services, and the stopping rate of the container service is a ratio of the number of containers which are created in the same container service and stopped to the number of all containers in the container service;
The calculation module is used for calculating the start-stop evaluation value of each container according to the resource use information of the container and the container parameters;
and the control module is used for controlling the container according to the start-stop evaluation value and a preset start-stop strategy.
9. The apparatus for controlling a container of claim 8, wherein the computing module comprises:
the assignment unit is used for assigning a preset weight value to the resource use information and the container parameters of the container;
and the computing unit is used for obtaining the start-stop evaluation value of each container after carrying out weighted summation on the resource use information of each container and the container parameters according to the preset weight value.
10. The apparatus for controlling a container according to claim 8 or 9, wherein the control module comprises:
the first control unit is used for controlling the container with the start-stop evaluation value meeting a first preset condition to stop when the running state of the hardware of the physical equipment is in a first running state; and/or a second control unit, configured to control the start-stop evaluation value to meet a container start-up of a second preset condition when the operation state of the hardware of the physical device is in a second operation state.
11. The apparatus for controlling a container according to claim 10, wherein the first control unit is configured to control the stopping of the container, for which the start-stop evaluation value satisfies a first preset condition, when an operation state of the CPU of the physical device satisfies any one of the following conditions:
the first condition is that the utilization rate of all CPUs on the physical equipment is greater than or equal to a first stop limit value;
the average load value of any CPU is larger than or equal to a second stop limit value;
and in the third condition, the number of the running CPUs on the physical equipment is greater than or equal to a third stop limit value.
12. The apparatus for controlling a container according to claim 10, wherein the second control unit is configured to control the start-up and stop of the container whose evaluation value satisfies a second preset condition when the operation state of the CPU of the physical device satisfies all of the following conditions:
the first condition is that the utilization rate of all CPUs on the physical equipment is smaller than or equal to a first starting limit value;
the load average value of any CPU is smaller than or equal to a second starting limit value;
and under the third condition, the number of the running CPUs on the physical equipment is smaller than or equal to a third starting limit value.
13. The apparatus for controlling a container according to claim 8, wherein the acquisition module comprises:
a container parameter acquiring unit configured to acquire a container parameter of each of the plurality of containers from a center node;
and the resource use information acquisition unit is used for acquiring the occupancy rate of each container in the plurality of containers to the hardware of the physical device from the physical device.
14. An edge computing system comprising a central node and an edge node, a plurality of physical devices being provided at the edge node, wherein each of the plurality of physical devices is provided with the apparatus for controlling a container according to any one of claims 8 to 13, and wherein the container on the physical device is controlled using the method for controlling a container according to any one of claims 1 to 7.
15. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method of controlling a container according to any one of claims 1 to 7.
16. An edge computing device, the edge computing device comprising: a transceiver, a memory, a processor;
The transceiver is used for receiving and transmitting messages;
the memory is used for storing instructions and data;
the processor is configured to read instructions and data stored in the memory to perform the method of controlling the container according to claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910919303.6A CN112559142B (en) | 2019-09-26 | 2019-09-26 | Container control method, device, edge computing system, medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910919303.6A CN112559142B (en) | 2019-09-26 | 2019-09-26 | Container control method, device, edge computing system, medium and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112559142A CN112559142A (en) | 2021-03-26 |
| CN112559142B true CN112559142B (en) | 2023-12-19 |
Family
ID=75030196
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910919303.6A Active CN112559142B (en) | 2019-09-26 | 2019-09-26 | Container control method, device, edge computing system, medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112559142B (en) |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105045656A (en) * | 2015-06-30 | 2015-11-11 | 深圳清华大学研究院 | Virtual container based big data storage and management method |
| CN106547628A (en) * | 2016-11-29 | 2017-03-29 | 北京元心科技有限公司 | Multi-system resource release method and device |
| CN106557353A (en) * | 2016-11-04 | 2017-04-05 | 天津轻工职业技术学院 | A kind of container carries the server performance index Evaluation Method of service application |
| CN106992887A (en) * | 2017-04-05 | 2017-07-28 | 国家电网公司 | The implementation method of application example elastic telescopic based on container, apparatus and system |
| CN107533469A (en) * | 2015-05-29 | 2018-01-02 | 英特尔公司 | The container of graphics processing unit resource is accessed |
| CN107908461A (en) * | 2017-12-01 | 2018-04-13 | 广东电网有限责任公司佛山供电局 | A kind of resource dynamic retractility device and implementation method based on Docker |
| CN108093046A (en) * | 2017-12-18 | 2018-05-29 | 江苏润和软件股份有限公司 | A kind of cloud application container resource regulating method based on feedforward control |
| US10033756B1 (en) * | 2017-10-26 | 2018-07-24 | Hytrust, Inc. | Methods and systems for holistically attesting the trust of heterogeneous compute resources |
| CN108399100A (en) * | 2017-02-06 | 2018-08-14 | 中国移动通信集团公司 | A kind of method of adjustment and device of container resource |
| CN108897627A (en) * | 2018-07-23 | 2018-11-27 | 南京叠嘉信息科技有限公司 | For the Docker dynamic dispatching algorithm of exemplary container |
| CN108958882A (en) * | 2018-06-06 | 2018-12-07 | 麒麟合盛网络技术股份有限公司 | A kind of container method of adjustment, device and system |
| CN109144727A (en) * | 2018-08-21 | 2019-01-04 | 郑州云海信息技术有限公司 | The management method and device of resource in cloud data system |
| CN109697153A (en) * | 2018-12-28 | 2019-04-30 | 浙江省公众信息产业有限公司 | Monitoring method, monitoring system and computer readable storage medium |
| CN109714400A (en) * | 2018-12-12 | 2019-05-03 | 华南理工大学 | A kind of energy optimization resource scheduling system and its method towards container cluster |
| CN110134495A (en) * | 2019-05-21 | 2019-08-16 | 山东大学 | A kind of container is across the online moving method of host, storage medium and terminal device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10558533B2 (en) * | 2017-12-07 | 2020-02-11 | Red Hat, Inc. | Reducing service disruptions in a micro-service environment |
-
2019
- 2019-09-26 CN CN201910919303.6A patent/CN112559142B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107533469A (en) * | 2015-05-29 | 2018-01-02 | 英特尔公司 | The container of graphics processing unit resource is accessed |
| CN105045656A (en) * | 2015-06-30 | 2015-11-11 | 深圳清华大学研究院 | Virtual container based big data storage and management method |
| CN106557353A (en) * | 2016-11-04 | 2017-04-05 | 天津轻工职业技术学院 | A kind of container carries the server performance index Evaluation Method of service application |
| CN106547628A (en) * | 2016-11-29 | 2017-03-29 | 北京元心科技有限公司 | Multi-system resource release method and device |
| CN108399100A (en) * | 2017-02-06 | 2018-08-14 | 中国移动通信集团公司 | A kind of method of adjustment and device of container resource |
| CN106992887A (en) * | 2017-04-05 | 2017-07-28 | 国家电网公司 | The implementation method of application example elastic telescopic based on container, apparatus and system |
| US10033756B1 (en) * | 2017-10-26 | 2018-07-24 | Hytrust, Inc. | Methods and systems for holistically attesting the trust of heterogeneous compute resources |
| CN107908461A (en) * | 2017-12-01 | 2018-04-13 | 广东电网有限责任公司佛山供电局 | A kind of resource dynamic retractility device and implementation method based on Docker |
| CN108093046A (en) * | 2017-12-18 | 2018-05-29 | 江苏润和软件股份有限公司 | A kind of cloud application container resource regulating method based on feedforward control |
| CN108958882A (en) * | 2018-06-06 | 2018-12-07 | 麒麟合盛网络技术股份有限公司 | A kind of container method of adjustment, device and system |
| CN108897627A (en) * | 2018-07-23 | 2018-11-27 | 南京叠嘉信息科技有限公司 | For the Docker dynamic dispatching algorithm of exemplary container |
| CN109144727A (en) * | 2018-08-21 | 2019-01-04 | 郑州云海信息技术有限公司 | The management method and device of resource in cloud data system |
| CN109714400A (en) * | 2018-12-12 | 2019-05-03 | 华南理工大学 | A kind of energy optimization resource scheduling system and its method towards container cluster |
| CN109697153A (en) * | 2018-12-28 | 2019-04-30 | 浙江省公众信息产业有限公司 | Monitoring method, monitoring system and computer readable storage medium |
| CN110134495A (en) * | 2019-05-21 | 2019-08-16 | 山东大学 | A kind of container is across the online moving method of host, storage medium and terminal device |
Non-Patent Citations (3)
| Title |
|---|
| Container-Based Complex Programming Skills Training Platform;Wei Wang等;《2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS)》;全文 * |
| 容器技术在移动核心网的应用;何宇锋;林奕琳;张琳峰;;移动通信(第03期);全文 * |
| 电网应用架构容器管理规范;廖辉;胡永华;马文;;信息系统工程(第08期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112559142A (en) | 2021-03-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9645856B2 (en) | Resource health based scheduling of workload tasks | |
| US11252220B2 (en) | Distributed code execution involving a serverless computing infrastructure | |
| CN111614570B (en) | Flow control system and method for service grid | |
| US7627618B2 (en) | System for managing data collection processes | |
| CN110221915B (en) | Node scheduling method and device | |
| US20140165061A1 (en) | Statistical packing of resource requirements in data centers | |
| CN112269641B (en) | Scheduling method, scheduling device, electronic equipment and storage medium | |
| US11838384B2 (en) | Intelligent scheduling apparatus and method | |
| CN113032102B (en) | Resource rescheduling method, device, equipment and medium | |
| US20210357269A1 (en) | Quality of service scheduling with workload profiles | |
| JP2008152618A (en) | Job assignment program, method and device | |
| JP7079998B1 (en) | Cluster capacity expansion method and equipment | |
| CN103248622B (en) | A kind of Online Video QoS guarantee method of automatic telescopic and system | |
| CN113342477A (en) | Container group deployment method, device, equipment and storage medium | |
| CN112214288B (en) | Pod scheduling method, device, equipment and medium based on Kubernetes cluster | |
| CN112559142B (en) | Container control method, device, edge computing system, medium and equipment | |
| CN115562841A (en) | Cloud video service self-adaptive resource scheduling system and method | |
| CN114936089A (en) | Resource scheduling method, system, device and storage medium | |
| CN112000477A (en) | Method, device, equipment and medium for load balancing in pod | |
| CN116719632B (en) | Task scheduling method, device, equipment and medium | |
| CN111459651A (en) | Load balancing method, device, storage medium and scheduling system | |
| CN111327663A (en) | Bastion machine distribution method and equipment | |
| CN112817728B (en) | Task scheduling method, network device and storage medium | |
| US20230354101A1 (en) | Resource allocation device, resource allocation method, and control circuit | |
| CN118118494A (en) | Method, device, computer equipment and storage medium for dynamic control of resources |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |