CN113791863B - Virtual container-based power Internet of things proxy resource scheduling method and related equipment - Google Patents
Virtual container-based power Internet of things proxy resource scheduling method and related equipment Download PDFInfo
- Publication number
- CN113791863B CN113791863B CN202110914882.2A CN202110914882A CN113791863B CN 113791863 B CN113791863 B CN 113791863B CN 202110914882 A CN202110914882 A CN 202110914882A CN 113791863 B CN113791863 B CN 113791863B
- Authority
- CN
- China
- Prior art keywords
- physical node
- resource
- load
- physical
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 230000004044 response Effects 0.000 claims abstract description 26
- 239000013598 vector Substances 0.000 claims description 45
- 230000008859 change Effects 0.000 claims description 11
- 238000012937 correction Methods 0.000 claims description 7
- 238000012216 screening Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 230000006872 improvement Effects 0.000 abstract description 5
- 238000005516 engineering process Methods 0.000 description 14
- 238000004422 calculation algorithm Methods 0.000 description 12
- 238000004891 communication Methods 0.000 description 11
- 238000012217 deletion Methods 0.000 description 5
- 230000037430 deletion Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000005457 optimization Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
Abstract
The disclosure provides a virtual container-based power internet of things proxy resource scheduling method and related equipment, wherein the method comprises the following steps: load information of each physical node is obtained; scheduling resources of the physical nodes according to the load information; carrying out load resource prediction on each running service, and calculating the expected value of the number of containers required by the service; acquiring a true value of the number of containers, judging the magnitude relation between the true value and the expected value, responding to the fact that the true value is smaller than the expected value, selecting a first optimal physical node from the physical nodes after resource scheduling, and deploying a new container; and/or in response to determining that the true value is greater than the expected value, selecting a second optimal physical node from the physical nodes after resource scheduling, and deleting the old container. The method and the device realize the load balancing of the container cluster and the improvement of the resource utilization rate, and enhance the performance of the device in the electric power Internet of things.
Description
Technical Field
The disclosure relates to the technical field of computer virtualization, in particular to a virtual container-based power internet of things proxy resource scheduling method and related equipment.
Background
The wide application of the electric power internet of things generates huge amounts of data, and the virtualization technology provides a solution for the storage management of the data, so that the method has become a widely accepted server resource sharing mode, wherein the container technology becomes a main deployment model of the virtualization technology due to the lightweight property of the container technology, the traditional container cluster management strategy has the advantages of simple structure, insufficient stability, difficulty in ensuring efficient utilization of resources, low resource utilization rate and poor load balancing degree of the container clusters.
Disclosure of Invention
Accordingly, an object of the present disclosure is to provide a virtual container-based power internet of things proxy resource scheduling method and related devices.
In a first aspect of the present disclosure, a method for scheduling power internet of things proxy resources based on a virtual container is provided, including: load information of each physical node is obtained; scheduling resources of the physical nodes according to the load information; carrying out load resource prediction on each running service, and calculating the expected value of the number of containers required by the service; acquiring a true value of the number of containers, judging the magnitude relation between the true value and the expected value, responding to the fact that the true value is smaller than the expected value, selecting a first optimal physical node from the physical nodes after resource scheduling, and deploying a new container on the first optimal physical node; and/or in response to determining that the true value is greater than the expected value, selecting a second optimal physical node from the physical nodes after resource scheduling, and deleting old containers on the second optimal physical node.
Further, the load information includes: CPU utilization of the physical nodeMemory utilizationNetwork bandwidth utilization->IO bandwidth utilization->Load resource utilization->Wherein the load resource utilization is +.>For said CPU utilization->Memory utilization->Network bandwidth utilizationAnd IO bandwidth utilization->Is the maximum value of (a).
Further, the resource scheduling for the physical node according to the load information specifically includes: screening the load resource utilization rateLess than 40% of the physical nodes are divided into a first cluster and a second cluster with the same number; wherein the total amount of load resources used by the first cluster is lower than the total amount of load resources used by the second cluster; and dispatching the containers of the physical nodes in the first cluster to the physical nodes of the second cluster.
Further, the load resource prediction for each running service specifically includes:
a. initializing a prediction update coefficient vector w, a load resource prediction vector y and a load resource vector o at the time t;
b. predicting a predicted value of the resource required by the service at the moment t, and calculating a predicted error;
c. adjusting the prediction update coefficient vector w according to the prediction error, and calculating a mean square error corresponding to the adjusted prediction update coefficient vector w;
d. Repeating the steps b and c until the mean square error is smaller than a given threshold value or the repetition number is larger than the maximum correction number;
e. and updating the load resource prediction vector y and the load resource vector o, and predicting load resources required by service at the time t+1.
Further, the obtaining the real value of the container number, and judging the magnitude relation between the real value and the expected value specifically includes: acquiring a true value of the number of containers, and calculating a difference value flag between the true value of the number of containers and the expected value;
the response to determining that the true value is smaller than the expected value, selecting a first optimal physical node from the physical nodes after resource scheduling, and deploying a new container on the first optimal physical node, specifically including: in response to determining that the difference flag <0, obtaining a first set and a second set of the physical nodes for deploying containers from the resource-scheduled physical nodes; calculating the weight of each physical node according to the physical node load condition of the first set; selecting the physical node with the largest weight as a first optimal physical node, deploying a new container on the first optimal physical node, and repeating the |flag| times;
The response to determining that the true value is greater than the expected value, selecting a second optimal physical node from the physical nodes after resource scheduling, and deleting old containers on the second optimal physical node, specifically including: in response to determining that the difference flag >0, obtaining a third set of physical nodes of the container in which the service has been deployed from the resource-scheduled physical nodes; calculating the weight of each physical node according to the physical node load condition of the third set; selecting the physical node with the largest weight as a second optimal physical node, deleting old containers on the second optimal physical node, and repeating the |flag| times.
Further, the first set includes a container for deploying a new container and post-deployment load resource utilizationThe second set comprising +.>Is a physical node of (a); and when no physical node exists in the first set, the physical nodes of the second set are clustered into the first set.
Further, the weight represents the resource utilization rate of the physical node after deploying or deleting the containerIncluding the resource utilization within and between the physical nodes >Is a change in the degree of equilibrium of (a).
In a second aspect of the present disclosure, a virtual container-based power internet of things proxy resource scheduling device is provided, including:
the information acquisition module is configured to acquire load information of each physical node;
the node scheduling module is configured to schedule resources of the physical nodes according to the load information;
a resource prediction module configured to perform load resource prediction for each running service, and calculate a container number expected value required by the service;
the container scheduling module is configured to acquire the real value of the number of containers, judge the magnitude relation between the real value and the expected value, respond to the fact that the real value is smaller than the expected value, select a first optimal physical node from the physical nodes after resource scheduling, and deploy a new container on the first optimal physical node; and/or in response to determining that the true value is greater than the expected value, selecting a second optimal physical node from the physical nodes after resource scheduling, and deleting old containers on the second optimal physical node.
Further, the container scheduling module is specifically configured to obtain a real value of the number of containers, and calculate a difference value flag between the real value of the number of containers and the expected value; in response to determining that the difference flag <0, obtaining a first set and a second set of the physical nodes for deploying containers from the resource-scheduled physical nodes; calculating the weight of each physical node according to the physical node load condition of the first set; selecting the physical node with the largest weight as a first optimal physical node, deploying a new container on the first optimal physical node, and repeating the |flag| times; in response to determining that the difference flag >0, obtaining a third set of physical nodes of the container in which the service has been deployed from the resource-scheduled physical nodes; calculating the weight of each physical node according to the physical node load condition of the third set; selecting the physical node with the largest weight as a second optimal physical node, deleting old containers on the second optimal physical node, and repeating the |flag| times.
In a third aspect of the present disclosure, there is provided an electronic device comprising a memory, a processor and a computer program stored on the memory and executed by the processor, the processor implementing the method of any one of the above when executing the program.
From the above, the present disclosure provides a virtual container-based power internet of things proxy resource scheduling method and related devices, which realizes load balancing of container clusters and improvement of resource utilization rate, and enhances performance of devices in the power internet of things; according to the load resource use condition of each physical node, resource scheduling is carried out, the condition that the nodes are used too much is avoided, and the node resource utilization rate is improved; the optimal physical node deployment or deletion container is selected, so that the resource utilization rate and the balance degree of each physical node are further improved; by predicting the load resource in advance, the dynamic scheduling of the container is better performed, and the performance of each physical node under the condition of traffic emergency can be improved.
Drawings
In order to more clearly illustrate the technical solutions of the present disclosure or related art, the drawings required for the embodiments or related art description will be briefly described below, and it is apparent that the drawings in the following description are only embodiments of the present disclosure, and other drawings may be obtained according to these drawings without inventive effort to those of ordinary skill in the art.
Fig. 1 is a flow chart of a virtual container-based power internet of things proxy resource scheduling method according to an embodiment of the disclosure;
FIG. 2 is a flow chart of a load resource prediction algorithm according to an embodiment of the disclosure;
fig. 3 is a schematic structural diagram of a virtual container-based power internet of things proxy resource scheduling device according to an embodiment of the disclosure;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.
Detailed Description
For the purposes of promoting an understanding of the principles and advantages of the disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the same.
It should be noted that unless otherwise defined, technical or scientific terms used in the embodiments of the present disclosure should be given the ordinary meaning as understood by one of ordinary skill in the art to which the present disclosure pertains. The terms "first," "second," and the like, as used in embodiments of the present disclosure, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that elements or items preceding the word are included in the element or item listed after the word and equivalents thereof, but does not exclude other elements or items. The terms "connected" or "connected," and the like, are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", etc. are used merely to indicate relative positional relationships, which may also be changed when the absolute position of the object to be described is changed.
As described in the background art, the power internet of things is a network in which intelligent sensing devices are widely deployed in each link of power production, power transmission, power consumption and power management, so as to realize safe and reliable transmission, collaborative processing, unified service and application integration of information. The electric power internet of things improves the utilization efficiency of electric power facilities, and provides important technical support for links such as power grid generation, power transmission, power transformation, power distribution and power utilization. Currently, the development of the internet of things industry has risen to the national strategy.
The internet of things creates a huge amount of data, and how to better store and manage such data to improve device performance is a challenge. Cloud computing provides a solution to store and manage the data it generates and accumulates.
The cloud computing technology is a computing mode provided for a user through a network, so that the user can conveniently and rapidly acquire computing resources according to own requirements. The cloud computing technology regards the whole computer resource as a whole, divides all requests into smaller fragments according to a certain rule, distributes the smaller fragments to different servers for simultaneous processing, greatly reduces the calculated amount of a single server and improves the calculation efficiency. In this process, the user only needs to send requests and receive service results, and does not have to pay attention to the intermediate process.
The virtualization technology is one of core technologies of cloud computing, and the computing resources are uniformly managed by logically isolating physical resources, so that the resource utilization rate is improved. Common virtualization technologies are virtual machine technology and container technology. The virtual machine technology is used for simulating the whole operating system and all hardware, and occupies a large amount of resources, so that the starting speed is low. Compared to virtual machine technology, container technology is much lighter. In the electric power internet of things scenario, the device is generally limited by resources, and a lightweight application program is also required to be adopted, so that unnecessary overhead is reduced.
Docker is an implementation engine of a container that can efficiently deploy, execute, and manage the container. Because the granularity of the containers of the Docker is smaller, more containers can be deployed on the same physical node, and the management and the scheduling of the containers are more complex. The traditional container cluster management strategy has the advantages of simple structure, insufficient stability, difficulty in ensuring efficient resource utilization, low resource utilization rate and poor load balance of the container clusters.
The applicant finds out in the process of realizing the present disclosure that if the optimal node deployment or deletion container is selected according to the use condition of the load resource, the resource utilization rate and the balance degree of each physical node can be improved, in order to avoid the situation that the original node is used too much, the container of the node can be scheduled according to the load information, the resource utilization rate is further improved, the change trend of the resource required by each service can be predicted in advance, and the container scheduling can be better facilitated.
The technical scheme of the present disclosure will be described in detail by specific examples.
Referring to fig. 1, a flow chart of a virtual container-based power internet of things proxy resource scheduling method according to an embodiment of the disclosure includes the following steps:
s1, acquiring load information of each physical node, and providing a basis for the subsequent steps.
The entire container cluster environment is available (N s ,S s ,C s ) Representation, where N s Representing a collection of physical nodes, one of which is forA representation; s is S s Representing a set of services, one of which is for +.>A representation; c (C) s Representing a collection of containers, one of which is for +.>A representation; with |N s I represents N s The number of physical nodes in (1) by->Representation->The number of containers, i, j, k, represents the sequence number.
Each service may contain multiple containers, each of which can only run on one physical node, usingRepresenting service->The number of containers is contained, each physical node contains a certain CPU, memory, network bandwidth and IO bandwidth resource, and the use is made ofCPU utilization rate, memory utilization rate, network bandwidth utilization rate, IO bandwidth utilization rate and load resource utilization rate of the physical node i are represented, and the load resource utilization rate is +.>CPU utilization is taken->Memory utilization- >Network bandwidth utilization->And IO bandwidth utilization->Is the maximum value of (a).
S2, carrying out resource scheduling on the physical nodes according to the load information, and avoiding starting more nodes.
The method specifically comprises the following steps: screening the load resource utilization rateLess than 40% of the physical nodes are divided into a first cluster and a second cluster with the same number; wherein the total amount of load resources used by the first cluster is lower than the total amount of load resources used by the second cluster; and dispatching the containers of the physical nodes in the first cluster to the physical nodes of the second cluster, so that the node resource utilization rate is improved.
And S3, carrying out load resource prediction on each running service, calculating the expected value of the number of containers required by the service, and carrying out dynamic scheduling of the containers better through advanced prediction.
Referring to fig. 2, a flowchart of a load resource prediction algorithm according to an embodiment of the disclosure, where load resource prediction is performed for each running service, includes:
a. initializing a prediction update coefficient vector w, a load resource prediction vector y and a load resource vector o at the time t;
b. predicting a predicted value of the resource required by the service at the moment t, and calculating a predicted error;
c. Adjusting the prediction update coefficient vector w according to the prediction error, and calculating a mean square error corresponding to the adjusted prediction update coefficient vector w;
d. repeating the steps b and c until the mean square error is smaller than a given threshold value or the repetition number is larger than the maximum correction number;
e. and updating the load resource prediction vector y and the load resource vector o, and predicting load resources required by service at the time t+1.
By predicting the load resource in advance, the dynamic scheduling of the container is better performed, and the performance of each physical node under the condition of traffic emergency can be improved.
The code for the load resource prediction algorithm may be designed as follows:
input: predictive update coefficient vector w (including predictive update coefficient at time t-q- > time t-1), load resource predictive vector y (including predictive value at time t-q- > time t-1), load resource vector o (including true value at time t-q- > time t-1), and resource yt required for service at time t
And (3) outputting: predicted value at time t+1
In the code of the load resource prediction algorithm, w represents a prediction update coefficient vector, a group of coefficients is maintained, and the load resource at the next moment is predicted; y represents a load resource prediction vector, and maintains predicted values of load resources at the last q moments; o represents a load resource vector, and the true values of the load resources at the last q moments are stored; q represents the vector length, which is a constant, and the load resource at the next moment is predicted through the real data of the load resources at the past q moments; p represents the adjustment rate, is a constant and is generally smaller than 1, the magnitude of the value determines the magnitude of the iterative step of the prediction update coefficient vector, the convergence speed is influenced, if p is too small, the convergence is too slow, and if the adjustment rate p is too large, the algorithm is easy to be not converged; dif represents a prediction error, which is the difference between the true value and the predicted value; the MSE represents the mean square error, and represents the mean square error between the predicted value and the true value of the load resource, and is used for considering whether the predicted vector w of the load resource is reasonable or not; the expectErr represents a given error threshold, if the mean square error MSE > expectErr, it means that the load resource prediction vector w is unreasonable, and the deviation needs to be corrected continuously; and (3) maxTryCount represents the maximum correction times, and if the maxTryCount times deviation is corrected for the load resource prediction vector w, and w is still unreasonable, stopping correcting the deviation, and preventing excessive deviation correction.
Lines 1-4 of the code of the load resource prediction algorithm can be understood as checking whether there is enough data accumulation, if not, updating the prediction update coefficient vector w, the load resource prediction vector y and the load resource vector o; lines 5-12 represent entering a loop, wherein the whole loop is used for adjusting a prediction update coefficient vector w so as to better predict the value at the time t+1, predicting a prediction value prediction and a prediction error dif of resources required by service at the current time t, adjusting the prediction update coefficient vector w by using the prediction error, and finally calculating the mean square error MSE of errors of the prediction value and a true value, and jumping out of the loop if the prediction error is smaller than a given threshold value expectErr or the number of loops is larger than the maximum correction number maxTryCount; lines 13-19 update vectors y, o and predict the resources needed for service at time t+1.
S4, acquiring a true value of the number of containers, judging the size relation between the true value and the expected value, responding to the fact that the true value is smaller than the expected value, selecting a first optimal physical node from the physical nodes after resource scheduling, and deploying a new container on the first optimal physical node; and/or in response to determining that the true value is greater than the expected value, selecting a second optimal physical node from the physical nodes after resource scheduling, and deleting an old container on the second optimal physical node, so that load balancing of a container cluster and improvement of resource utilization rate are realized, and performance of equipment in the electric power Internet of things is enhanced.
Further, the obtaining the real value of the container number, and judging the magnitude relation between the real value and the expected value specifically includes: acquiring a true value of the number of containers, and calculating a difference value flag between the true value of the number of containers and the expected value;
the response to determining that the true value is smaller than the expected value, selecting a first optimal physical node from the physical nodes after resource scheduling, and deploying a new container on the first optimal physical node, specifically including: in response to determining that the difference flag <0, obtaining a first set and a second set of the physical nodes for deploying containers from the resource-scheduled physical nodes; calculating the weight of each physical node according to the physical node load condition of the first set; selecting the physical node with the largest weight as a first optimal physical node, deploying a new container on the first optimal physical node, and repeating the |flag| times;
the response to determining that the true value is greater than the expected value, selecting a second optimal physical node from the physical nodes after resource scheduling, and deleting old containers on the second optimal physical node, specifically including: in response to determining that the difference flag >0, obtaining a third set of physical nodes of the container in which the service has been deployed from the resource-scheduled physical nodes; calculating the weight of each physical node according to the physical node load condition of the third set; selecting the physical node with the largest weight as a second optimal physical node, deleting old containers on the second optimal physical node, and repeating the |flag| times.
According to the method, the container scheduling is carried out according to the difference value between the actual value and the expected value of the number of the containers, namely according to the change trend of the required resources, the containers are deleted when the number of the containers is large, and new containers are established when the number of the containers is small, so that the balance degree of the container clusters is further improved, and the resource utilization rate is improved; it should be noted that if the difference value flag=0, the expected value representing the container is the same as the actual value, and the scheduling of the container is not required.
The container may be deployed or deleted, and may be operated based on manual allocation, based on resource usage and performance conditions when running different loads, based on a dynamic weighted scheduling algorithm, or based on historical conflicts, and the like, which is not specifically limited.
Further, the first set includes a container for deploying a new container and post-deployment load resource utilizationThe second set comprising +.>Is a physical node of (a); when no physical node exists in the first set, the physical nodes of the second set are collected into the first set, so that a container is deployed for the node with relatively low resource utilization rate, the resource utilization rate can be improved, and the operation efficiency is also improved.
The weight in this step represents the resource utilization of the physical node after deploying or deleting the containerIncluding the resource utilization within and between the physical nodes>The balance degree change of the (2) can more comprehensively represent the balance degree of each physical node, and is favorable for optimizing and balancing resources.
In general, the virtual container-based power internet of things proxy resource scheduling method realizes the load balance of the container cluster and the improvement of the resource utilization rate, and enhances the performance of the equipment in the power internet of things; according to the load resource use condition of each physical node, resource scheduling is carried out, the condition that the nodes are used too much is avoided, and the node resource utilization rate is improved; the optimal physical node deployment or deletion container is selected, so that the resource utilization rate and the balance degree of each physical node are further improved; by predicting the load resource in advance, the dynamic scheduling of the container is better performed, and the performance of each physical node under the condition of traffic emergency can be improved.
In some embodiments, the code of the virtual container-based power internet of things proxy resource scheduling algorithm may be designed as follows:
input: without any means for
And (3) outputting: without any means for
In codes of a virtual container-based power internet of things proxy resource scheduling algorithm, num represents a container number expected value required by the service;a true value representing the number of containers; filter represents a filter; workable1 represents the first set of physical nodes; workable2 represents the second set of physical nodes; workable3 represents a third set of the physical nodes; />Representing resource utilization of the physical node after deployment of the container>Is a change in the degree of equilibrium of (a); />Representing the resource utilization of said physical node after deletion of the container +.>Is a change in the degree of equilibrium of (a).
The 1 st-9 th lines of codes of the power internet of things proxy resource scheduling algorithm based on the virtual containers can be understood as checking nodes with the load resource utilization rate lower than 40%, and if the number is more than or equal to 2, scheduling containers of half nodes with fewer node resources in the nodes to half nodes with more resources so as not to start excessive nodes; lines 10-11 can be understood as calling a load resource prediction algorithm for each running service, and obtaining and recording a predicted value of the load resource of the service k; lines 12-13 represent calculating a flag as the difference between the number of containers contained in the service and the number of containers required by the service, i.e., the difference between the actual value and the expected value; lines 14-24 represent flag <0, indicating insufficient containers, requiring the addition of new containers; lines 25-33 represent flag >0, indicating that the container is redundant, requiring the redundant container to be deleted.
Further, the method comprises the steps of,and->Is calculated as follows:
wherein alpha and beta are weight coefficients for adjusting the proportion of the optimization target;representing the change of the balance degree of various resource utilization rates inside the physical nodes after the container is deployed; />Representing the change of the balance degree of various resource utilization rates in the physical node after deleting the container; />Negative between physical nodes after table deployment of containersThe change of the balance degree of the utilization rate of the carrier resources; />Representing a change in the degree of balance of load resource utilization between physical nodes after deletion of the container.
Further, the method comprises the steps of,and->Is calculated as follows:
can be usedRepresenting the standard deviation between the internal resource utilization of the physical node i, then, in the equation,representing standard deviation between internal resource utilization rates before a physical node i deploys or deletes a container;representing internal resource utilization after physical node i deploys a containerStandard deviation between rates; />Representing the standard deviation between the internal resource utilization after the physical node i deletes the container.
Can be usedRepresenting the standard deviation between the load resource utilization of all physical nodes, then in the formula, +.>Representing standard deviation between load resource utilization of all physical nodes before the physical node i deploys or deletes the container; / >Representing standard deviation between load resource utilization rates of all physical nodes after the physical node i deploys the container; />Representing the standard deviation between the load resource utilization of all physical nodes after the physical node i deletes the container.
Further, the method comprises the steps of,and->Is calculated as follows:
/>
in the method, in the process of the invention,representing an average value of the utilization rate of the internal resources of the physical node; />Representing an average of resource utilization between physical nodes.
The optimization target of the algorithm is the optimization degree of the internal balance of the physical nodes and the load balance among the physical nodes, and the final total optimization target is as follows:
that is to sayOr->The largest physical node of the values, wherein +.>The physical node with the largest numerical value is the first optimal physical node, and a new container is deployed on the node; />The largest physical node of the numerical value is the firstTwo optimal physical nodes, deleting old containers on the nodes; thereby achieving the purpose of dynamic scheduling of the container and improving the utilization rate and the balance degree of load resources.
It should be noted that the methods of one or more embodiments of the present disclosure may be performed by a single device, such as a computer or server. The method of the embodiment can also be applied to a distributed scene, and is completed by mutually matching a plurality of devices. In the case of such a distributed scenario, one of the devices may perform only one or more steps of the methods of one or more embodiments of the present disclosure, the devices interacting with each other to accomplish the methods.
The foregoing has described certain embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
Based on the same inventive concept, corresponding to the method of any embodiment, the disclosure further provides a virtual container-based power internet of things proxy resource scheduling device, referring to fig. 3, including:
the information acquisition module 301 is configured to acquire load information of each physical node;
a node scheduling module 302 configured to schedule resources for the physical node according to the load information;
a resource prediction module 303 configured to perform load resource prediction for each running service, and calculate a desired value of the number of containers required by the service;
the container scheduling module 304 is configured to obtain a real value of the number of containers, determine a magnitude relation between the real value and the expected value, select a first optimal physical node from the physical nodes after resource scheduling in response to determining that the real value is smaller than the expected value, and deploy a new container on the first optimal physical node; and/or in response to determining that the true value is greater than the expected value, selecting a second optimal physical node from the physical nodes after resource scheduling, and deleting old containers on the second optimal physical node.
For convenience of description, the above system is described as being functionally divided into various modules, respectively. Of course, the functions of the various modules may be implemented in the same one or more pieces of software and/or hardware when implementing one or more embodiments of the present disclosure.
The container scheduling module is specifically configured to obtain a real value of the number of containers, and calculate a difference value flag between the real value of the number of containers and the expected value; in response to determining that the difference flag <0, obtaining a first set and a second set of the physical nodes for deploying containers from the resource-scheduled physical nodes; calculating the weight of each physical node according to the physical node load condition of the first set; selecting the physical node with the largest weight as a first optimal physical node, deploying a new container on the first optimal physical node, and repeating the |flag| times; in response to determining that the difference flag >0, obtaining a third set of physical nodes of the container in which the service has been deployed from the resource-scheduled physical nodes; calculating the weight of each physical node according to the physical node load condition of the third set; selecting the physical node with the largest weight as a second optimal physical node, deleting old containers on the second optimal physical node, and repeating the |flag| times.
The system of the above embodiment is used for implementing the corresponding method in the foregoing embodiment, and has the beneficial effects of the corresponding virtual container-based power internet of things proxy resource scheduling method embodiment, which is not described herein again.
Based on the same inventive concept, the present disclosure also provides an electronic device corresponding to the method of any embodiment, which includes a memory, a processor, and a computer program stored on the memory and capable of running on the processor, where the processor implements the virtual container-based power internet of things proxy resource scheduling method according to any embodiment when executing the program.
Fig. 4 shows a more specific hardware architecture of an electronic device according to this embodiment, where the device may include: a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040, and a bus 1050. Wherein processor 1010, memory 1020, input/output interface 1030, and communication interface 1040 implement communication connections therebetween within the device via a bus 1050.
The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit ), microprocessor, application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc. for executing relevant programs to implement the technical solutions provided in the embodiments of the present disclosure.
The Memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory ), static storage device, dynamic storage device, or the like. Memory 1020 may store an operating system and other application programs, and when the embodiments of the present specification are implemented in software or firmware, the associated program code is stored in memory 1020 and executed by processor 1010.
The input/output interface 1030 is used to connect with an input/output module for inputting and outputting information. The input/output module may be configured as a component in a device (not shown) or may be external to the device to provide corresponding functionality. Wherein the input devices may include a keyboard, mouse, touch screen, microphone, various types of sensors, etc., and the output devices may include a display, speaker, vibrator, indicator lights, etc.
Communication interface 1040 is used to connect communication modules (not shown) to enable communication interactions of the present device with other devices. The communication module may implement communication through a wired manner (such as USB, network cable, etc.), or may implement communication through a wireless manner (such as mobile network, WIFI, bluetooth, etc.).
Bus 1050 includes a path for transferring information between components of the device (e.g., processor 1010, memory 1020, input/output interface 1030, and communication interface 1040).
It should be noted that although the above-described device only shows processor 1010, memory 1020, input/output interface 1030, communication interface 1040, and bus 1050, in an implementation, the device may include other components necessary to achieve proper operation. Furthermore, it will be understood by those skilled in the art that the above-described apparatus may include only the components necessary to implement the embodiments of the present description, and not all the components shown in the drawings.
The electronic device of the foregoing embodiment is configured to implement the corresponding virtual container-based power internet of things proxy resource scheduling method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which is not described herein.
Those of ordinary skill in the art will appreciate that: the discussion of any of the embodiments above is merely exemplary and is not intended to suggest that the scope of the disclosure, including the claims, is limited to these examples; the technical features of the above embodiments or in the different embodiments may also be combined under the idea of the present disclosure, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in details for the sake of brevity.
Additionally, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown within the provided figures, in order to simplify the illustration and discussion, and so as not to obscure the embodiments of the present disclosure. Furthermore, the devices may be shown in block diagram form in order to avoid obscuring the embodiments of the present disclosure, and this also accounts for the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform on which the embodiments of the present disclosure are to be implemented (i.e., such specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the disclosure, it should be apparent to one skilled in the art that embodiments of the disclosure can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative in nature and not as restrictive.
While the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of those embodiments will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
The disclosed embodiments are intended to embrace all such alternatives, modifications and variances which fall within the broad scope of the appended claims. Accordingly, any omissions, modifications, equivalents, improvements, and the like, which are within the spirit and principles of the embodiments of the disclosure, are intended to be included within the scope of the disclosure.
Claims (4)
1. A virtual container-based power Internet of things proxy resource scheduling method comprises the following steps:
load information of each physical node is obtained; the load information includes: CPU utilization of the physical nodeMemory utilization->Network bandwidth utilization->IO bandwidth utilization->Load resource utilization->Said load resource utilization->For said CPU utilization->Memory utilization->Network bandwidth utilizationAnd IO bandwidth utilization->Maximum value of (2);
and carrying out resource scheduling on the physical node according to the load information, wherein the resource scheduling comprises the following steps of: screening the load resource utilization rateLess than 40% of the physical nodes are divided into a first cluster and a second cluster with the same number; wherein the total amount of load resources used by the first cluster is lower than the total amount of load resources used by the second cluster; scheduling containers of physical nodes in the first cluster to physical nodes of the second cluster;
Carrying out load resource prediction on each running service, and calculating the expected value of the number of containers required by the service; the load resource prediction for each running service specifically comprises the following steps: a. initializing a prediction update coefficient vector w, a load resource prediction vector y and a load resource vector o at the time t; b. predicting a predicted value of the resource required by the service at the moment t, and calculating a predicted error; c. adjusting the prediction update coefficient vector w according to the prediction error, and calculating a mean square error corresponding to the adjusted prediction update coefficient vector w; d. repeating the steps b and c until the mean square error is smaller than a given threshold value or the repetition number is larger than the maximum correction number; e. updating the load resource prediction vector y and the load resource vector o, and predicting load resources required by service at the time t+1;
acquiring a true value of the number of containers, judging the magnitude relation between the true value and the expected value, responding to the fact that the true value is smaller than the expected value, selecting a first optimal physical node from the physical nodes after resource scheduling, and deploying a new container on the first optimal physical node; and/or in response to determining that the true value is greater than the expected value, selecting a second optimal physical node from the physical nodes after resource scheduling, and deleting old containers on the second optimal physical node; the step of obtaining the real value of the number of the containers and judging the size relation between the real value and the expected value specifically comprises the following steps: acquiring a true value of the number of containers, and calculating a difference value flag between the true value of the number of containers and the expected value; the response to determining that the true value is smaller than the expected value, selecting a first optimal physical node from the physical nodes after resource scheduling, and deploying a new container on the first optimal physical node, specifically including: in response to determining that the difference flag <0, obtaining a first set and a second set of the physical nodes for deploying containers from the resource-scheduled physical nodes; calculating the weight of each physical node according to the physical node load condition of the first set; selecting the physical node with the largest weight as a first optimal physical node, deploying a new container on the first optimal physical node, and repeating the |flag| times; the response to determining that the true value is greater than the expected value, selecting a second optimal physical node from the physical nodes after resource scheduling, and deleting old containers on the second optimal physical node, specifically including: in response to determining that the difference flag >0, obtaining a third set from the resource-scheduled physical nodes; calculating the weight of each physical node according to the physical node load condition of the third set; selecting the physical node with the largest weight as a second optimal physical node, deleting old containers on the second optimal physical node, repeating the |flag| times;
The first set is a container and part for deploying new containersPost-deployment load resource utilization The second set is the set of physical nodes for deploying new containers and post-deployment load resource utilization +.>Is a collection of physical nodes of (a); when no physical node exists in the first set, the physical nodes of the second set are clustered into the first set; the third set is a set of physical nodes of the container where the service has been deployed.
2. The method of claim 1, wherein the weight represents a resource utilization of the physical node after deploying or deleting a containerIncluding the resource utilization within and between the physical nodesIs a change in the degree of equilibrium of (a).
3. An electric power thing allies oneself with proxy resource scheduling device based on virtual container includes:
the information acquisition module is configured to acquire load information of each physical node; the load information includes: CPU utilization of the physical nodeMemory utilization->Network bandwidthUtilization rate->IO bandwidth utilization->Load resource utilization->Said load resource utilization->For said CPU utilization->Memory utilization->Network bandwidth utilization- >And IO bandwidth utilization->Maximum value of (2);
the node scheduling module is configured to schedule resources of the physical nodes according to the load information, and specifically comprises the following steps: screening the load resource utilization rateLess than 40% of the physical nodes are divided into a first cluster and a second cluster with the same number; wherein the total amount of load resources used by the first cluster is lower than the total amount of load resources used by the second cluster; scheduling containers of physical nodes in the first cluster to physical nodes of the second cluster;
a resource prediction module configured to perform load resource prediction for each running service, and calculate a container number expected value required by the service; the load resource prediction for each running service specifically comprises the following steps: a. initializing a prediction update coefficient vector w, a load resource prediction vector y and a load resource vector o at the time t; b. predicting a predicted value of the resource required by the service at the moment t, and calculating a predicted error; c. adjusting the prediction update coefficient vector w according to the prediction error, and calculating a mean square error corresponding to the adjusted prediction update coefficient vector w; d. repeating the steps b and c until the mean square error is smaller than a given threshold value or the repetition number is larger than the maximum correction number; e. updating the load resource prediction vector y and the load resource vector o, and predicting load resources required by service at the time t+1;
The container scheduling module is configured to acquire the real value of the number of containers, judge the magnitude relation between the real value and the expected value, respond to the fact that the real value is smaller than the expected value, select a first optimal physical node from the physical nodes after resource scheduling, and deploy a new container on the first optimal physical node; and/or in response to determining that the true value is greater than the expected value, selecting a second optimal physical node from the physical nodes after resource scheduling, and deleting old containers on the second optimal physical node; the step of obtaining the real value of the number of the containers and judging the size relation between the real value and the expected value specifically comprises the following steps: acquiring a true value of the number of containers, and calculating a difference value flag between the true value of the number of containers and the expected value; the response to determining that the true value is smaller than the expected value, selecting a first optimal physical node from the physical nodes after resource scheduling, and deploying a new container on the first optimal physical node, specifically including: responsive to determining the difference flag<0, obtaining a first set and a second set of the physical nodes for deploying containers from the physical nodes after the resource scheduling; calculating the weight of each physical node according to the physical node load condition of the first set; selecting the physical node with the largest weight as a first optimal physical node, and selecting the physical node with the largest weight as the first optimal physical node A new container is deployed on-site, repeating the |flag| times; the response to determining that the true value is greater than the expected value, selecting a second optimal physical node from the physical nodes after resource scheduling, and deleting old containers on the second optimal physical node, specifically including: responsive to determining the difference flag>0, obtaining a third set from the physical nodes after resource scheduling; calculating the weight of each physical node according to the physical node load condition of the third set; selecting the physical node with the largest weight as a second optimal physical node, deleting old containers on the second optimal physical node, repeating the |flag| times; the first set is for deploying new containers and post-deployment load resource utilizationThe second set is the set of physical nodes for deploying new containers and post-deployment load resource utilization +.>Is a collection of physical nodes of (a); when no physical node exists in the first set, the physical nodes of the second set are clustered into the first set; the third set is a set of physical nodes of the container where the service has been deployed.
4. An electronic device comprising a memory, a processor and a computer program stored on the memory and executed by the processor, the processor implementing the method of any one of claims 1 to 2 when the program is executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110914882.2A CN113791863B (en) | 2021-08-10 | 2021-08-10 | Virtual container-based power Internet of things proxy resource scheduling method and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110914882.2A CN113791863B (en) | 2021-08-10 | 2021-08-10 | Virtual container-based power Internet of things proxy resource scheduling method and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113791863A CN113791863A (en) | 2021-12-14 |
| CN113791863B true CN113791863B (en) | 2024-01-23 |
Family
ID=78875847
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110914882.2A Active CN113791863B (en) | 2021-08-10 | 2021-08-10 | Virtual container-based power Internet of things proxy resource scheduling method and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113791863B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105794099A (en) * | 2013-12-11 | 2016-07-20 | Abb 技术有限公司 | Pulse width modulation with faster capacitor balancing |
| CN107045455A (en) * | 2017-06-19 | 2017-08-15 | 华中科技大学 | A kind of Docker Swarm cluster resource method for optimizing scheduling based on load estimation |
| CN108829494A (en) * | 2018-06-25 | 2018-11-16 | 杭州谐云科技有限公司 | Container cloud platform intelligence method for optimizing resources based on load estimation |
| EP3410368A1 (en) * | 2013-06-19 | 2018-12-05 | United States Postal Service | System and method for providing real-time tracking of items in a distribution network |
| CN109918198A (en) * | 2019-02-18 | 2019-06-21 | 中国空间技术研究院 | A kind of emulation cloud platform load dispatch system and method based on user characteristics prediction |
| CN111857953A (en) * | 2020-07-17 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Container cluster management method, device, equipment and readable storage medium |
| CN113110914A (en) * | 2021-03-02 | 2021-07-13 | 西安电子科技大学 | Internet of things platform construction method based on micro-service architecture |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180167487A1 (en) * | 2016-12-13 | 2018-06-14 | Red Hat, Inc. | Container deployment scheduling with constant time rejection request filtering |
| US10740362B2 (en) * | 2017-12-22 | 2020-08-11 | International Business Machines Corporation | Container structure |
| US11182196B2 (en) * | 2019-11-13 | 2021-11-23 | Vmware, Inc. | Unified resource management for containers and virtual machines |
-
2021
- 2021-08-10 CN CN202110914882.2A patent/CN113791863B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3410368A1 (en) * | 2013-06-19 | 2018-12-05 | United States Postal Service | System and method for providing real-time tracking of items in a distribution network |
| CN105794099A (en) * | 2013-12-11 | 2016-07-20 | Abb 技术有限公司 | Pulse width modulation with faster capacitor balancing |
| CN107045455A (en) * | 2017-06-19 | 2017-08-15 | 华中科技大学 | A kind of Docker Swarm cluster resource method for optimizing scheduling based on load estimation |
| CN108829494A (en) * | 2018-06-25 | 2018-11-16 | 杭州谐云科技有限公司 | Container cloud platform intelligence method for optimizing resources based on load estimation |
| CN109918198A (en) * | 2019-02-18 | 2019-06-21 | 中国空间技术研究院 | A kind of emulation cloud platform load dispatch system and method based on user characteristics prediction |
| CN111857953A (en) * | 2020-07-17 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Container cluster management method, device, equipment and readable storage medium |
| CN113110914A (en) * | 2021-03-02 | 2021-07-13 | 西安电子科技大学 | Internet of things platform construction method based on micro-service architecture |
Non-Patent Citations (5)
| Title |
|---|
| Dynamic Resource Scheduling Of Container-based Edge IoT Agents;Yutong Ji et.al.;2021 International Wireless Communications and Mobile Computing (IWCMC);全文 * |
| Performance Tuning and Modeling for Big Data Applications in Docker Containers;Kejiang Ye et.al.;2017 International Conference on Networking, Architecture, and Storage (NAS);全文 * |
| 一种适于Docker容器资源控制的分类预测方法;李植;陈莉君;;计算机与数字工程(12);全文 * |
| 基于Docker容器调度优化方法的研究;吴双艳;中国优秀硕士学位论文全文数据库 (信息科技辑);全文 * |
| 基于服务质量的服务功能链编排和动态优化方法;蔡轶斌;中国优秀硕士学位论文全文数据库 (信息科技辑);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113791863A (en) | 2021-12-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109947567B (en) | Multi-agent reinforcement learning scheduling method and system and electronic equipment | |
| CN112363813A (en) | Resource scheduling method and device, electronic equipment and computer readable medium | |
| CN110231976B (en) | Load prediction-based edge computing platform container deployment method and system | |
| JP7083476B1 (en) | Network access device resource allocation method and equipment | |
| CN108667878A (en) | Server load balancing method and device, storage medium, electronic equipment | |
| CN110198244A (en) | Resource allocation method and device towards isomery cloud service | |
| CN110780914A (en) | Service publishing method and device | |
| CN111131486B (en) | Load adjustment method and device of execution node, server and storage medium | |
| CN111371616B (en) | Virtual network function chain deployment method and system for NUMA (non Uniform memory Access) architecture server | |
| CN105491150A (en) | Load balance processing method based on time sequence and system | |
| CN111431996B (en) | Method, apparatus, device and medium for resource configuration | |
| CN110990154A (en) | Big data application optimization method and device and storage medium | |
| Wang et al. | Task scheduling algorithm based on improved Min-Min algorithm in cloud computing environment | |
| CN113553160A (en) | Task scheduling method and system for edge computing node of artificial intelligence Internet of things | |
| CN114205317B (en) | SDN and NFV-based service function chain SFC resource allocation method and electronic equipment | |
| CN114553723A (en) | Operation method, system, medium and equipment of artificial intelligence training platform | |
| CN114691372A (en) | Group intelligent control method of multimedia end edge cloud system | |
| CN113791863B (en) | Virtual container-based power Internet of things proxy resource scheduling method and related equipment | |
| CN112130927A (en) | Reliability-enhanced mobile edge computing task unloading method | |
| CN109783236A (en) | Method and apparatus for output information | |
| CN114003238B (en) | Container deployment method, device, equipment and storage medium based on transcoding card | |
| CN115525394A (en) | Method and device for adjusting number of containers | |
| CN108933834A (en) | A kind of dispatching method and dispatching device | |
| CN110119300A (en) | The load-balancing method and device of dummy unit cluster | |
| CN113918306A (en) | Method, device, equipment, medium and product for scheduling running resources |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |