CN111752678A - Low-power-consumption container placement method for distributed collaborative learning in edge computing - Google Patents
Low-power-consumption container placement method for distributed collaborative learning in edge computing Download PDFInfo
- Publication number
- CN111752678A CN111752678A CN202010546862.XA CN202010546862A CN111752678A CN 111752678 A CN111752678 A CN 111752678A CN 202010546862 A CN202010546862 A CN 202010546862A CN 111752678 A CN111752678 A CN 111752678A
- Authority
- CN
- China
- Prior art keywords
- server
- container
- edge
- network
- test
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 230000005540 biological transmission Effects 0.000 claims abstract description 5
- 238000012360 testing method Methods 0.000 claims description 54
- 230000003068 static effect Effects 0.000 claims description 30
- 238000004458 analytical method Methods 0.000 claims description 23
- 238000004422 calculation algorithm Methods 0.000 claims description 16
- 238000007405 data analysis Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 6
- 238000012216 screening Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 abstract description 5
- 230000008569 process Effects 0.000 description 19
- 238000005516 engineering process Methods 0.000 description 7
- 238000005265 energy consumption Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000007621 cluster analysis Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 235000008694 Humulus lupulus Nutrition 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000001143 conditioned effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000002054 transplantation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Power Sources (AREA)
Abstract
The invention discloses a low-power-consumption container placement method for distributed collaborative learning in edge computing. Firstly, clustering is carried out by using historical resources of service containers, and then a container resource use prediction model is constructed, so that the prediction of the resource demand of a container to be deployed is realized. Secondly, when the edge server is accessed to the edge cloud computing platform for the first time, the energy efficiency performance and other comparability are tested and calibrated, and the real-time load, hardware resources, network communication and other state information of all online edge servers are obtained through the acquisition module. Thirdly, comprehensively considering all placement possibilities of currently available servers from three aspects of energy equivalence, network transmission overhead and server load balancing, respectively calculating the ranking of each server obtained in the three aspects, and finally determining the server with the highest comprehensive ranking as the final placement position of the container.
Description
Technical Field
The invention relates to a method for realizing optimal placement of containers in an edge cloud computing cluster system, in particular to a method for realizing low-power-consumption container placement by a distributed collaborative learning method aiming at a large-scale distributed deployed edge server cluster in an edge computing environment.
Background
In recent years, with the large deployment of internet of things devices and the rapid popularization of mobile devices, the collected data and service requests start to accumulate in a large amount at the edge of the network. Under the background, the centralized processing mode adopted by the traditional cloud computing cannot meet the increasingly marginal computing resource requirement. Therefore, the concept of Edge computing (Edge computing) comes to the end, and its main feature is to implement all or part of data processing on the Edge computing resource request in the Edge server cluster close to the local, so as to optimize the execution flow of the Edge request. The edge computing fully utilizes idle computing resources at edge positions, computing pressure of the cloud data center can be effectively relieved, and meanwhile, real-time performance of service response is effectively guaranteed by a computing processing mode close to user and source data.
Meanwhile, as one of the underlying basestones of the current cloud native technology, the Container technology (Container) gradually replaces the hypervisor technology based on the virtualization of the hardware abstraction layer due to the characteristics of light weight, rapid deployment, easy transplantation and elastic expansion. Container technology is essentially implemented by creating a set of resource-constrained, and isolated processes on a physical machine. In the process, namespaces and cgroups provided by the linux kernel are mainly utilized, the namespaces are used for isolating process access, and the cgroups are used for isolating resources such as a CPU (Central processing Unit), a memory, an I/O (input/output) and a network. Therefore, the containers running on the same host machine can share the kernel, and the system overhead except the running user load is greatly reduced. In a particular production environment, Docker has become currently the most prevalent container engine. Meanwhile, the rapid development of container management platforms such as Kubernets, Swam and facilities also provides help for container arrangement and management under a distributed cluster architecture, and effectively ensures high availability and strong elastic expansion capability of cloud platform computing services. Although the current container technology is generally applied in a cloud computing platform architecture, the container technology is rarely deployed to a cluster network architecture consisting of edge servers. However, a range of excellent features of container technology remain well suited for service deployment and implementation in edge computing environments.
The overall energy efficiency of the distributed cluster is always one of the main problems to be considered for building and managing the cloud service platform. Considering that the traditional cloud computing cluster is realized by a centralized deployment manner, the method is greatly different from the edge server distributed deployment architecture in the edge computing environment. This results in that the cost of executing the service request of the same user on different geo-location servers may have great differences, which is mainly reflected in the aspects of server operation energy consumption, network transmission cost, request response delay, and the like. Therefore, the placement strategy of its computation load execution container on each edge server must be optimized for a specific user computation service request, thereby achieving low-power operation of the entire edge computation cluster.
In the current research, a traditional cloud computing cluster generally adopts a specific container scheduling algorithm to schedule different types of containers to a server with the highest energy efficiency for operation, so as to improve the energy utilization rate of the whole cluster. The Energy Efficiency of the data center server is generally measured by two indexes, namely Energy Efficiency (EE) and Energy Proportionality (EP). Energy efficiency refers to the ratio between server performance and power, where the performance of a server is often defined as the amount of computational load done per unit time. Different server types, or different computational loads, can significantly impact the energy efficiency of the server. The low-power-consumption container scheduling strategy needs to integrate the relationship between the current overall computing load level and cluster computing resources, close idle or low-energy-efficiency servers in time, and reasonably distribute the working load to the rest of the servers. In addition, the container allocation policy of the work server cannot be standardized to 100% of the computational load, but should be conditioned to a load level at the time of the server peak energy efficiency. According to the latest statistics, the peak energy efficiency of the current server is generally at the 60% -80% working load level. On the other hand, the energy isocratic refers to the variation of the power consumption of the server with the utilization rate. The index measures the energy utilization efficiency of the server under the dynamic load level, and the value range of the index is between 0 and 1. Under an ideal condition, the energy equal ratio value is 1, and the energy consumption and the working load of the server meet the equal ratio change rule at the moment. For example, assuming an ideal server peak power consumption of 100W, it should be taken at the server peak load (100%), and when the server load drops to 0, its power consumption should also drop proportionally to 0W. When the energy equivalence of the server is closer to 1, the energy conversion mechanism of the server is closer to an ideal server, so that the server can save more energy under the condition of dynamic change of the load level. In the container optimization scheduling policy, it is also necessary to consider scheduling the workload to a server with high energy ratio as much as possible.
In summary, for a distributed edge server cluster in an edge computing environment, an optimized container placement strategy that comprehensively considers various features such as container load, server energy consumption, network communication overhead, and the like is urgently needed to achieve low-power-consumption deployment of edge services.
Disclosure of Invention
The whole network service architecture considered by the invention is divided into three layers, namely edge end equipment (user request), an edge server (edge cloud computing platform) and a cloud data center from the edge. The optimization of the edge cloud computing platform is mainly focused, namely how to select an edge server for placing a container, so that the running power consumption generated by the container running is minimized.
The technical scheme adopted by the invention for solving the technical problem is as follows:
And 2, the edge server state collection module stores the received new server static information to a server static information base.
And 3, the server periodically reports the use condition of the dynamic resources to the edge server state collection module, and the edge server state collection module continuously updates the information to the dynamic information base of the server.
and 5, the user service container analysis module transmits the resource use requirement of the container to be created into a server dynamic information base as a screening condition, and traverses and obtains an available server list meeting the container deployment condition.
And 4, the server dynamic information base sends the available server list and the dynamic resource use information of the servers in the list to the server state information comprehensive analysis module.
And 5, according to the available server list, the server state information comprehensive analysis module requests the static information of all the servers in the list from the server static information base.
And 6, analyzing and calculating the performance indexes of container placement under all conditions, including energy geometric proportion, network transmission overhead and server load balance, by the server state information comprehensive analysis module according to the received static and dynamic information of the available server and the resource demand information of the service container to be created.
And 7, the server state information comprehensive analysis module eliminates the servers with the energy efficiency values exceeding the target load interval according to the performance values obtained by the available servers under the three indexes, and finally obtains the EP, the NetI/O and the load balancing ranking table respectively in a performance descending mode.
And 8, accumulating ranking values of each server in three dimensions of EP, NetI/O and load balance by a container placement decision module, deciding a server with the minimum accumulated value (with the highest comprehensive ranking), and sending a container deployment command to the server so as to complete the optimal placement of the container.
The above process involves three parts: cluster analysis based on container historical resource use condition, test and calibration of power consumption characteristics of edge server and optimized container placement based on heuristic algorithm
(1) Cluster analysis based on container historical resource usage
Aiming at various computing requests sent by edge users, the edge cloud service platform creates different container templates for service response. Different computational logic in the container template results in the created container having differentiated resource usage at real runtime. The method includes the steps of collecting use information of resources of different container templates in the historical operation process, dividing the use information into three types including calculation intensive type, network I/O intensive type and mixed type through a comprehensive clustering algorithm, and continuously utilizing a single-dimension clustering algorithm to further subdivide the use information into 5 grades of load intensity for each type. Accordingly, a total of 15(3 x 5) container resource load types can be obtained, and resource usage prediction models of different types of container templates are calculated and established. The model can accurately predict the use condition of the container resources to be deployed on the edge cloud computing platform, and provide the requirement information of the client for the low-power-consumption container placement strategy.
(2) Power consumption feature testing and calibration of edge servers
The edge cloud computing platform connects a large number of heterogeneous edge servers into a cluster of a distributed architecture through a public network. In order to obtain a more reasonable container placement strategy, comprehensive power consumption characteristic analysis needs to be performed on different types of edge servers, wherein the comprehensive power consumption characteristic analysis mainly comprises load level and energy equal ratio testing and calibration during peak energy efficiency performance of the servers. The invention provides an edge server state acquisition module to realize the functions, which can communicate with a server and complete power consumption characteristic test and calibration when the server is firstly accessed into an edge cloud computing cluster, and records power consumption characteristic information obtained by the server and other hardware information into an information base. In addition, the module also needs to collect the dynamic resource usage information (such as CPU, memory, disk and network I/O) of all online servers at regular time, and provides the global resource information of the cloud platform for the low-power-consumption container placement algorithm strategy.
(3) Heuristic algorithm based optimized placement of containers
The invention provides a container scheduling algorithm based on a heuristic algorithm based on resource demand prediction information of a client container and the availability of global real-time resources of a cloud platform. The algorithm comprehensively considers all placement possibilities of the currently available servers from three aspects of energy equivalence, network transmission overhead and server load balancing, calculates the ranking of each server in three aspects through a simulation data analysis module, and then selects the server with the highest comprehensive ranking as the final placement position of the container.
The invention has the beneficial effects that: different from the traditional container optimal placement method, the invention is characterized in that a low-power-consumption container placement strategy for optimizing indexes in three aspects of network I/O overhead, server power consumption and load balance is provided by considering two aspects of client container demand prediction and cloud computing platform hardware resource analysis aiming at a server cluster with distributed deployment characteristics in an edge cloud computing environment.
Drawings
FIG. 1 is a diagram of a distributed collaborative learning low-power container placement methodology architecture in edge computing;
FIG. 2 is a diagram of a process for constructing a container resource usage prediction model;
FIG. 3 is a schematic diagram of testing and calibration of power consumption characteristics of an edge server;
fig. 4 is a flow chart of a container low power consumption placement strategy based on a heuristic algorithm.
Detailed Description
The specific steps of the method of the present invention are described below with reference to fig. 1:
fig. 1 shows the overall structure of a low-power-consumption container placement method for distributed collaborative learning in edge computing. The method comprises the steps that 1, 2 and 3 represent communication between an edge server and an edge server state collection module, and comprises the steps of sending a test control command and returning a test result to a tested server during power consumption characteristic test of the server, and sending static information (power consumption characteristics and hardware resource information) and reporting the use condition of dynamic resources periodically by the edge server to the edge server state collection module. And 4, the edge server state collection module stores the received server static information (power consumption characteristics and hardware resource information) in the server static information base. And 5, the edge server state collection module continuously updates the received dynamic resource use condition to the server dynamic information base. And 6, the user serves the container analysis module and transmits the resource use requirement of the container to be created as a screening condition to the server dynamic information base for screening the available servers. And 7, sending the screening result (the available server list) and the dynamic resource utilization information of the server in the screening result to the server state information comprehensive analysis module on behalf of the server dynamic information base. And 8, requesting the static information of the server from the static information base of the server by the comprehensive analysis module of the server state information according to the available server list, and receiving the return of the matching result of the static information base. 9. 10 and 11 represent available server performance value ranking tables obtained by the server state information comprehensive analysis module according to three indexes of EP, Net and load balance respectively. 12, the ranking table of the performance values of the three indexes of EP, Net and load balance is sent to the container placement decision module to be used as a decision basis. And 13, the container placement decision module sends a container deployment command to the specified server according to the decision result.
The key steps of the present invention are described in more detail below.
(1) Establishment of container resource usage prediction model
As one of the key parts of the invention, the container resource usage prediction model can predict the more accurate resource usage demand of the user service container to be placed, and the resource prediction result is the critical information for realizing the low-power-consumption container placement strategy.
Considering that the user service request is satisfied by creating a container, the edge cloud computing platform provides different container mirror templates for different user service requests. The container images contain specific computing logic frameworks, can meet different computing service functions of users, and can also show various different types of resource use characteristics in the actual operation process. Therefore, by clustering the historical resource usage of these containers, containers in which resource usage behaves similarly can be aggregated into individual specific types. By calculating the average value of the same type of container in the use of the CPU and the network I/O and taking the average value as the representative of the resource use amount of the container of the type, the resource use prediction model of the mirror image template of each type of container can be constructed. Therefore, for a container mirror image needing to be placed, the resource utilization predicted value can be conveniently obtained by judging the type of the container to which the container mirror image belongs. In addition, in order to divide the container types with finer granularity, the invention adopts a method combining multidimensional clustering and single-dimensional clustering.
With reference to fig. 2 and algorithm 1, the following describes the steps of establishing a container resource usage prediction model:
And 2, inputting the container historical resource use condition data set into a comprehensive clustering algorithm, clustering container data in two dimensions of a CPU (Central processing Unit) and a network I/O (input/output) and preliminarily distinguishing three types of intensive calculation, intensive network I/O and mixed type according to a clustering result.
And 3, continuously clustering containers under each subtype in a single-dimension mode according to the result after the primary clustering, and respectively clustering the containers into 5 subtypes (serving as 5 grades of different load intensities), wherein 15(3 × 5) categories are counted. Wherein, the compute intensive containers are clustered by CPU utilization; network I/O intensive clustering with network I/O overhead; the mixed type uses the normalized CPU and the accumulated value of the network I/O overhead as parameters for clustering.
And 4, calculating the CPU average value and the network I/O average value of the container in each sub-class, and taking the CPU average value and the network I/O average value as the predicted values of the operation resources of the container under different load intensities.
And 5, inputting the operation resource predicted values of all the containers and the corresponding categories and load intensities thereof into a database to form a mapping table of the container names and the resource predicted values, thereby completing the establishment of a container resource use prediction model.
As shown in Table 1, the resource usage prediction model map for a compute intensive container, defined as 5 different load strengths, denoted from low to high by I to V, respectively. Under each load intensity, a corresponding resource predicted value is represented by a triple group which respectively represents the predicted CPU utilization rate, the memory utilization rate and the network I/O value.
TABLE 1 computationally intensive Container resource usage prediction model mapping table
(2) Edge server power consumption feature testing and calibration
In the problem of low-power-consumption container placement, the power consumption characteristics of the edge server are main factors influencing the energy consumption of container operation, and the method specifically utilizes two indexes of load level, energy and the like during the peak energy efficiency performance of the server to realize the selection of a subsequent container placement server. Therefore, it is required to perform power consumption feature test and calibration on all servers on the edge cloud computing platform, and the principle structure of the power consumption feature test and calibration is shown in fig. 3. The specific functions of the modules are introduced as follows:
1) the test result acquisition module: and receiving a test result of the test case workload on the tested edge server.
2) A data analysis module: and summarizing test results on the test result acquisition cluster block and the power supply test equipment, and calculating specific power consumption characteristic values.
3) A test task release module: and issuing test tasks of different types and load levels to the tested edge server.
4) A workload management module: and receiving the test tasks of the test task issuing module, and scheduling and managing the specific workload of the corresponding instance running on the edge server.
5) The working load is as follows: receiving task arrangement of the workload management module, and specifically executing a computing task.
6) The power supply test equipment comprises: and measuring the real-time power of the edge server, and submitting the statistical result to the data analysis module.
As shown in table 2, the table is a test record and energy efficiency calculation result table for each load level in the power consumption test and calibration process of the edge server. The load level of the edge server at the peak energy efficiency performance is the maximum value among the 11 energy efficiency performances, and the corresponding load level is the requirement. And the energy equivalence EP can be obtained by the following formula:
therein, EE0Representing the energy efficiency of the server at a load level of 0%, EE1Representing the energy efficiency of the server at a load level of 10%, and so on, EE10Server with load level of 100%Energy efficiency.
Table 2 server power consumption test result table
| Load level | Server performance | Real time power | Energy efficiency |
| 100％ | A | a | A/a |
| 90％ | B | b | B/b |
| …… | …… | …… | …… |
| 20％ | C | c | C/ |
| 10％ | D | d | D/d |
| 0％ | E | e | E/e |
(3) Edge server initialization boot process
The specific implementation of the power consumption characteristic test and calibration process of the edge server is arranged to be performed when a new edge server is accessed to the cluster network for the first time, and is a small part of the initialization and boot process of the whole edge server. The initialization boot process of the whole edge server comprises the following specific steps:
And 2, starting an edge server initialization bootstrap program by an edge server state collection module, sending a request to the server, and requesting to return a hardware address (MAC) of the server.
And 3, after receiving the return result (MAC address), the edge server state collection module sends a search command to the server static information base by taking the MAC value as a search condition.
And 4, after the server static information base is searched, returning the result to the edge server state collection module.
And 6, starting a server power consumption characteristic test and calibration program by the edge server state collection module, wherein the node responsible for the edge server state collection module becomes a test system, and the edge server becomes a tested system.
And 7, a test task issuing module in the test system issues a calibration task, and when the tested system works at a load level of 100%, the calibration result is returned to the test system.
And 8, the test system issues a test task meeting the 100% load level of the system to be tested according to the calibration result.
And 9, executing the test task by the tested system, collecting the performance of the server under the current load level by the test result acquisition module, and submitting the result to the data analysis module. Meanwhile, the current server power value measured on the power supply test equipment is also submitted to the data analysis module.
And 11, calculating and analyzing 11 groups of data acquired in the process by a data analysis module, and determining the load level and energy equivalence ratio of the edge server during peak energy efficiency.
And 12, the edge server state collection module stores the load level and the energy geometric proportion value and other static information of the edge server in the peak energy efficiency performance into a server static information base.
And step 13, the edge server state collection module ends the initialization boot process of the edge server.
In addition to performing the initial boot process and collecting the server static state information, the edge server state collection module also takes over the function of collecting all the online server dynamic state information. The static state information is only collected when the server is accessed to the cloud platform for the first time and is stored in a server static information base; the dynamic state information is collected in a timing mode and is stored in a server dynamic information base. The two kinds of information are stored separately, the data volume recorded by a static information base is huge, and the main operation is query and insertion; the dynamic database has less data volume and needs to be updated and deleted frequently. The entries and descriptions held in the static and dynamic state information bases are listed together, as shown in table 3.
Table 3 server state information table
(4) Decision making process of container low-power-consumption placing method
The container low-power-consumption placement decision process is based on a heuristic strategy, after the edge equipment sends a task request to the cloud computing platform, the scheduling system predicts the resource use requirement of the container to be placed, and then specific container placement is carried out according to the resource availability condition of the cloud computing platform. In the placing process, the dispatching system excludes servers which reach peak energy efficiency performance in the edge cloud computing platform according to the load level of each online server, and then three indexes of remaining undetermined server energy efficiency equal ratio, extra network communication overhead from edge end equipment to the servers and load balance of the servers after container deployment are comprehensively considered.
The first index, energy efficiency equal-proportion (EE) of the rest servers, can be directly obtained from the server static information base, and a ranking list of the energy efficiency equal-proportion of the server to be determined can be obtained through a simple sorting algorithm. And other two indexes need to be obtained through calculation of the server state information comprehensive analysis module.
The second indicator, Net cost of the edge device to the server extra network, is calculated as follows:
Net_cost＝[Container_img/(Net_hardwave-Net_now)]×hop_num
wherein, Container _ img is the size of the mirror image capacity of the Container to be placed, Net _ hardwave and Net _ now respectively represent the hardware bandwidth size of the target server and the current network bandwidth usage, and hop _ num represents the number of route hops from the edge device to the target server.
The third index, load balancing (Balance) of the server after container deployment, is calculated as follows:
Balance＝(CPU-average)2+(Mem-average)2+(Net-average)2
the CPU, the Mem and the Net respectively represent the CPU utilization rate, the memory utilization rate and the network I/O utilization rate of the server after the container deployment, and the CPU, the Mem and the Net are obtained through the following calculation formulas:
here, the CPU _ rate, the Mem _ rate, and the Net _ now respectively represent a CPU utilization rate, a memory utilization rate, and a network bandwidth utilization rate of the server before container deployment; container _ CPU, Container _ mem and Container _ net respectively represent predicted CPU, memory and bandwidth usage of the Container to be deployed; finally, CPU _ hardware, Mem _ hardware and Net _ hardware represent the server hardware CPU, memory and bandwidth configurations, respectively. The average is an average value of CPU, Mem, and Net, and is calculated as follows:
average＝1/3×(CPU+Mem+Net)
all the data are acquired by the server state information comprehensive analysis module through the server static and dynamic information base.
The following describes the decision process of the container low power consumption placement method in detail with reference to fig. 4 and algorithm 2:
Claims (3)
1. the method for placing the low-power-consumption container facing distributed collaborative learning in edge computing is characterized by comprising the following steps of:
step 1, an edge server accesses an edge cloud computing platform, completes the test and calibration of power consumption characteristics in an initialization stage, and then returns static information to an edge server state collection module;
step 2, the edge server state collection module stores the received new server static information to a server static information base;
step 3, the server reports the use condition of the dynamic resources to the edge server state collection module periodically, and the edge server state collection module continuously updates the information to a server dynamic information base;
step 4, the user request is sent to a user service container analysis module, the user service container analysis module predicts the resource use requirement of the container to be created through a container resource use prediction model,
step 5, the user service container analysis module transmits the resource use requirement of the container to be created as a screening condition into a server dynamic information base, and traverses and obtains an available server list meeting the container deployment condition;
step 4, the server dynamic information base sends the available server list and the dynamic resource use information of the servers in the list to the server state information comprehensive analysis module;
step 5, according to the available server list, the server state information comprehensive analysis module requests the static information of all servers in the list from the server static information base;
step 6, the server state information comprehensive analysis module analyzes and calculates performance indexes of container placement under all conditions, including energy geometric proportion, network transmission overhead and server load balance, according to the received available server static and dynamic information and the resource demand information of the service container to be created;
step 7, the server state information comprehensive analysis module excludes servers with energy efficiency values exceeding the target load interval according to the performance values obtained by the available servers under the three performance indexes in the step 6, and obtains an EP (EP), a NetI/O (network interconnection/output) and a load balancing ranking table respectively in a performance descending manner;
step 8, a container placement decision module accumulates ranking values of each server in three dimensions of EP, NetI/O and load balancing, decides the server with the minimum accumulated value, namely the server with the highest comprehensive ranking, and then sends a container deployment command to the server so as to complete the optimal placement of the container;
step 9, the edge cloud computing platform system continuously monitors network information, and if a new server access request is received, the steps 1 to 2 are executed; if a user service request is received, executing the steps 4 to 8; otherwise, continuously executing the step 3 in a circulating way.
2. The method for placing the low-power-consumption container facing the distributed collaborative learning in the edge computing according to claim 1, wherein: the power consumption characteristic test and calibration in the step 1 are realized by the following functional modules:
the test result acquisition module: receiving a test result of the test case workload on the edge server to be tested;
a data analysis module: summarizing test results on the test result acquisition cluster block and the power supply test equipment, and calculating specific power consumption characteristic values;
a test task release module: issuing test tasks of different types and load levels to the edge server to be tested;
a workload management module: receiving a test task of the test task issuing module, and scheduling and managing a specific working load of a corresponding instance on the edge server;
the working load is as follows: receiving task arrangement of a workload management module, and specifically executing a computing task;
the power supply test equipment comprises: and measuring the real-time power of the edge server, and submitting the statistical result to the data analysis module.
3. The method for placing the low-power-consumption container facing the distributed collaborative learning in the edge computing according to claim 1, wherein: the container resource usage prediction model in step 4 is built as follows:
4-1, acquiring a data set for recording the use conditions of historical resources of all containers in a network downloading or local testing mode;
4-2, inputting the container historical resource use condition data set into a comprehensive clustering algorithm, clustering container data in two dimensions of a CPU (Central processing Unit) and a network I/O (input/output) by the comprehensive clustering algorithm, and preliminarily distinguishing three types of a calculation intensive type, a network I/O intensive type and a mixed type according to a clustering result;
4-3, continuously carrying out single-dimension clustering on the containers under each subtype according to the result after the primary clustering, and respectively clustering into five subclasses, wherein fifteen categories are counted; wherein, the compute intensive containers are clustered by CPU utilization; clustering the network I/O intensive containers by using the network I/O overhead; the mixed container takes the normalized CPU and the accumulated value of the network I/O overhead as parameters for clustering;
4-4, calculating the CPU average value and the network I/O average value of the container in each sub-class, and taking the CPU average value and the network I/O average value as the predicted values of the operation resources of the container under different load intensities;
and 4-5, inputting the operation resource predicted values of all the containers and the corresponding categories and load intensities thereof into a database to form a mapping table of the container names and the resource predicted values, thereby completing the establishment of a container resource use prediction model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010546862.XA CN111752678A (en) | 2020-06-16 | 2020-06-16 | Low-power-consumption container placement method for distributed collaborative learning in edge computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010546862.XA CN111752678A (en) | 2020-06-16 | 2020-06-16 | Low-power-consumption container placement method for distributed collaborative learning in edge computing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111752678A true CN111752678A (en) | 2020-10-09 |
Family
ID=72676247
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010546862.XA Pending CN111752678A (en) | 2020-06-16 | 2020-06-16 | Low-power-consumption container placement method for distributed collaborative learning in edge computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111752678A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113204429A (en) * | 2021-05-31 | 2021-08-03 | 深圳供电局有限公司 | Resource scheduling method and system of data center, scheduling equipment and medium |
| CN113467892A (en) * | 2021-07-14 | 2021-10-01 | 广州华多网络科技有限公司 | Distributed cluster resource configuration method and corresponding device, equipment and medium |
| CN114691360A (en) * | 2022-03-21 | 2022-07-01 | 北京有竹居网络技术有限公司 | Resource scheduling method and related equipment thereof |
| CN116361120A (en) * | 2023-05-31 | 2023-06-30 | 山东浪潮科学研究院有限公司 | Method, device, equipment and medium for managing and scheduling heterogeneous resources of database |
| CN117032972A (en) * | 2023-08-15 | 2023-11-10 | 中交路桥科技有限公司 | Slope monitoring system based on cloud network side |
| US11882182B2 (en) | 2021-08-04 | 2024-01-23 | Samsung Electronics Co., Ltd. | Method and a network management system for determining storage node for a network element |
| US11953972B2 (en) | 2022-04-06 | 2024-04-09 | International Business Machines Corporation | Selective privileged container augmentation |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110149396A (en) * | 2019-05-20 | 2019-08-20 | 华南理工大学 | A kind of platform of internet of things construction method based on micro services framework |
-
2020
- 2020-06-16 CN CN202010546862.XA patent/CN111752678A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110149396A (en) * | 2019-05-20 | 2019-08-20 | 华南理工大学 | A kind of platform of internet of things construction method based on micro services framework |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113204429A (en) * | 2021-05-31 | 2021-08-03 | 深圳供电局有限公司 | Resource scheduling method and system of data center, scheduling equipment and medium |
| CN113467892A (en) * | 2021-07-14 | 2021-10-01 | 广州华多网络科技有限公司 | Distributed cluster resource configuration method and corresponding device, equipment and medium |
| CN113467892B (en) * | 2021-07-14 | 2024-06-21 | 广州华多网络科技有限公司 | Distributed cluster resource allocation method and corresponding device, equipment and medium thereof |
| US11882182B2 (en) | 2021-08-04 | 2024-01-23 | Samsung Electronics Co., Ltd. | Method and a network management system for determining storage node for a network element |
| CN114691360A (en) * | 2022-03-21 | 2022-07-01 | 北京有竹居网络技术有限公司 | Resource scheduling method and related equipment thereof |
| US11953972B2 (en) | 2022-04-06 | 2024-04-09 | International Business Machines Corporation | Selective privileged container augmentation |
| CN116361120A (en) * | 2023-05-31 | 2023-06-30 | 山东浪潮科学研究院有限公司 | Method, device, equipment and medium for managing and scheduling heterogeneous resources of database |
| CN116361120B (en) * | 2023-05-31 | 2023-08-15 | 山东浪潮科学研究院有限公司 | Method, device, equipment and medium for managing and scheduling heterogeneous resources of database |
| CN117032972A (en) * | 2023-08-15 | 2023-11-10 | 中交路桥科技有限公司 | Slope monitoring system based on cloud network side |
| CN117032972B (en) * | 2023-08-15 | 2024-05-14 | 中交路桥科技有限公司 | Slope monitoring system based on cloud network side |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111752678A (en) | Low-power-consumption container placement method for distributed collaborative learning in edge computing | |
| CN106776005B (en) | Resource management system and method for containerized application | |
| CN104281496B (en) | The distribution method and distributor of virtual resource | |
| CN103338228B (en) | Cloud computing load balancing dispatching algorithms based on double weighting Smallest connection algorithms | |
| CN104038540B (en) | Method and system for automatically selecting application proxy server | |
| CN104050042B (en) | The resource allocation methods and device of ETL operations | |
| CN106534318B (en) | A kind of OpenStack cloud platform resource dynamic scheduling system and method based on flow compatibility | |
| CN109120715A (en) | Dynamic load balancing method under a kind of cloud environment | |
| CN108108245B (en) | Hybrid scheduling method and system for cloud platform wide-node scientific workflow | |
| CN107426332B (en) | A kind of load-balancing method and system of web server cluster | |
| CN113037877B (en) | Optimization method for time-space data and resource scheduling under cloud edge architecture | |
| CN108170530B (en) | Hadoop load balancing task scheduling method based on mixed element heuristic algorithm | |
| CN105718364A (en) | Dynamic assessment method for ability of computation resource in cloud computing platform | |
| CN101944113B (en) | Cloud computing system-based data acquisition system | |
| CN104731528B (en) | A kind of construction method and system of cloud computing block storage service | |
| CN107566535B (en) | Self-adaptive load balancing method based on concurrent access timing sequence rule of Web map service | |
| CN115134371A (en) | Scheduling method, system, equipment and medium containing edge network computing resources | |
| CN109478147A (en) | Adaptive resource management in distributed computing system | |
| CN105426241A (en) | Cloud computing data center based unified resource scheduling energy-saving method | |
| CN107402926A (en) | A kind of querying method and query facility | |
| Gopu et al. | Optimal VM placement in distributed cloud environment using MOEA/D | |
| CN113553160A (en) | Task scheduling method and system for edge computing node of artificial intelligence Internet of things | |
| CN109408230A (en) | Docker container dispositions method and system based on energy optimization | |
| CN102081624B (en) | Data inquiring method and data inquiring device | |
| Zhang et al. | An energy-aware host resource management framework for two-tier virtualized cloud data centers |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |